Komodo 5 release now available!

[Don]

Is Komodo particularly good compared to other programs (such as Houdini?) Right now it's just a hypothesis - at least for me. I will construct some tests to prove or disprove this. I think Larry believes it is true since he has spent an enormous amount of time getting familiar with Komodo for the book he published and he believes Komodo is the best program of all for opening book preparation.

Why not implement ches960 support then? It would surely help to prove or disprove your hypothesis. Btw. looking at CCRL 40/4 FRC list, I might start spreading a hypothesis too ... Also note the 100 elo gap between #2 and #3 (and between #4 - #5). It would be nice if more strong engines supported FRC.

Yes, I really want to add chess960 support - but first MP and then Gaviota databases. Excellent results for chess960 - actually this is slight support for my "hypothesis" that not all programs play non-standard positions as well as others - which is related the opening book question, especially how an engine might play an opening position that is off-beat and different.

Code: Select all

CCRL 404FRC Rating List - All engines, best versions only

Rank           Engine            ELO   +    -   Score  AvOp  Games
 1 Critter 1.6 64-bit         3289  +22  -22  76.7% -212.8   900
 2 Houdini 2.0 64-bit         3280  +18  -18  69.4% -156.8  1200
 3 Stockfish 2.2.2 64-bit     3182  +17  -17  60.2%  -81.7  1300
 4 Rybka 4 64-bit             3170  +14  -14  61.4%  -87.0  1800
 5 Naum 4.2 64-bit            3029  +12  -11  48.8%   +6.8  3100
 6 Shredder 12                3020  +12  -12  45.3%  +32.3  2900

[Laskos]

Is Komodo particularly good compared to other programs (such as Houdini?) Right now it's just a hypothesis - at least for me. I will construct some tests to prove or disprove this. I think Larry believes it is true since he has spent an enormous amount of time getting familiar with Komodo for the book he published and he believes Komodo is the best program of all for opening book preparation.

Why not implement ches960 support then? It would surely help to prove or disprove your hypothesis. Btw. looking at CCRL 40/4 FRC list, I might start spreading a hypothesis too ... Also note the 100 elo gap between #2 and #3 (and between #4 - #5). It would be nice if more strong engines supported FRC.

Code: Select all

CCRL 404FRC Rating List - All engines, best versions only

Rank           Engine            ELO   +    -   Score  AvOp  Games
 1 Critter 1.6 64-bit         3289  +22  -22  76.7% -212.8   900
 2 Houdini 2.0 64-bit         3280  +18  -18  69.4% -156.8  1200
 3 Stockfish 2.2.2 64-bit     3182  +17  -17  60.2%  -81.7  1300
 4 Rybka 4 64-bit             3170  +14  -14  61.4%  -87.0  1800
 5 Naum 4.2 64-bit            3029  +12  -11  48.8%   +6.8  3100
 6 Shredder 12                3020  +12  -12  45.3%  +32.3  2900

That's interesting. Are you testing on FRC when developing Critter? If yes, then Critter on standard chess could be fitted to play better, very close to Houdini. Also, it means that Critter is very good for opening preparation.

Kai

[Uri Blass]

Some people may view the opening as irrelevant since it is possible to obtain comprehensive opening books (which can be considered part of the game playing system) and that is a legitimate point of view - however it's easy to get a program out of the opening with a few non-standard but not ridiculous moves in which case the program will be on it's own and will need to be able to play the opening well.

I think that if you get the program out of book by non standard moves you do not lose the game but you clearly get relatively worse position in most cases so testing with very short opening books does not give a good estimate what the program can practically do in tournaments when both sides use books(if you choose to give komodo a book with non standard moves then maybe komodo is going to play the opening relatively better than the opponent but it also is going to start from slightly worse position).

If you start from non-standard positions it does not have to be inferior positions but even if it is slightly inferior, that is easily compensated for by the fact that your program is superior in the opening and you can force him into using clock time to do what YOU do better. It's stupid to let him cruise into the part of the game he does best without even taking a time hit.

I am also not sure if komodo is better with very short book at long time control.

I'm not sure either. That is a working hypothesis and I don't believe until I can prove with some experiment. However I do think it's very likely and Larry has a huge amount of experience with Komodo at very long thinking time in his opening book analysis projects and strongly believes Komodo is superior to every other program for this purpose. (which address you next point since this is not short time control stuff.)

Being relatively better in evaluating the opening stage does not help much at long time control when the program already search middle game positions even in the early opening stage.

But I'm talking only about long time controls. I don't know how Komodo does at short time controls with the opening. Maybe it's superior there too?

Mathematically however you have fallen prey to a common fallacy which is that at long time controls the program is thinking 10 times deeper. No, it doesn't work that way. It may be thinking 10 times LONGER but it's not searching 10 times deeper. In the first couple of seconds of search Komodo and any other program has completely most of the depth it's going to unless you are playing correspondence chess, and even then it probably is searching at least half the depth in the first 2 or 3 seconds.

1)I assume that the standard positions are standard because they are based on superior moves and it means that non standard positions are inferior positions.

2)If Komodo is better in the opening then testers may prove it by testing combination of programs and show for example that
if you use komodo at moves 1-20 and houdini later in the game the combination of komodo and houdini get significantly better rating than houdini.

The question is if it is possible to show it.

I would like to see combination of engines tested in the rating list to see if we can show if there are cases when a combination of engines is better than every one of them.

[lkaufman]

Is Komodo particularly good compared to other programs (such as Houdini?) Right now it's just a hypothesis - at least for me. I will construct some tests to prove or disprove this. I think Larry believes it is true since he has spent an enormous amount of time getting familiar with Komodo for the book he published and he believes Komodo is the best program of all for opening book preparation.

Why not implement ches960 support then? It would surely help to prove or disprove your hypothesis. Btw. looking at CCRL 40/4 FRC list, I might start spreading a hypothesis too ... Also note the 100 elo gap between #2 and #3 (and between #4 - #5). It would be nice if more strong engines supported FRC.

Code: Select all

CCRL 404FRC Rating List - All engines, best versions only

Rank           Engine            ELO   +    -   Score  AvOp  Games
 1 Critter 1.6 64-bit         3289  +22  -22  76.7% -212.8   900
 2 Houdini 2.0 64-bit         3280  +18  -18  69.4% -156.8  1200
 3 Stockfish 2.2.2 64-bit     3182  +17  -17  60.2%  -81.7  1300
 4 Rybka 4 64-bit             3170  +14  -14  61.4%  -87.0  1800
 5 Naum 4.2 64-bit            3029  +12  -11  48.8%   +6.8  3100
 6 Shredder 12                3020  +12  -12  45.3%  +32.3  2900

We do want to support frc, but I'm sure you'll agree that MP is a higher priority. I think that in general frc should favor the stronger engine (more chances to make mistakes!), so that is probably why the gap between the top two and the next two is larger than in normal chess. It is of course not unlikely that Critter is better than Houdini at frc, but since it is weaker at regular chess I would want a more significant lead than 9 elo to convince me that it is so.
I would also say that being best at FRC does not mean best at the opening at normal chess, although it would be suggestive of that. To some extent, FRC is a different game, although as a past US Open FRC champion I obviously like and support the game.

[lkaufman]

[/code]

Code: Select all

  Time Control was  1 min + 1 sec  ,  like previous

Tom

So the results were almost reversed with Ponder on; overall Komodo won by a single game. The results could all be due to chance assuming they are of equal strength, though it seems a bit unlikely. It seems more likely that Komodo does not benefit as much from Ponder as Houdini; perhaps Robert has devoted some time to improving Ponder ON play in Houdini, while we have not done much in that department for Komodo. Personally, since I use engines for analysis, I don't care about Ponder on play, but some people do and it is used by IPON, so perhaps we'll have to look into it more.

[MikeGL]

i7 / 940 @ 2.93 , Ponder On , 512 mb , W7 , F11 , 1 thread ea , klo_150\






[/code]

Code: Select all

Different result from ponder off , maybe random ?

I DL the Perfect 2012 book , but it is .ctg . When I get the klo openings to work , they are in .pgn format . Do I need to put the .ctg file in a diff location , or change it somehow to a .pgn database ?

Thanks for the help...... not all of us are geniuses

Tom

Maybe the previous claim of Dimitri Yasenko is correct regarding the
process and resourse priority being higher.
http://talkchess.com/forum/viewtopic.ph ... w=&start=0

I'm not familiar with Fritz11 GUI, but is it really possible to allocate one
core for each engine and a third core for the F11 GUI. If this is possible,
then I think I will dump my Arena and buy a Fritz interface.

[geots]

Some people may view the opening as irrelevant since it is possible to obtain comprehensive opening books (which can be considered part of the game playing system) and that is a legitimate point of view - however it's easy to get a program out of the opening with a few non-standard but not ridiculous moves in which case the program will be on it's own and will need to be able to play the opening well.

I think that if you get the program out of book by non standard moves you do not lose the game but you clearly get relatively worse position in most cases so testing with very short opening books does not give a good estimate what the program can practically do in tournaments when both sides use books(if you choose to give komodo a book with non standard moves then maybe komodo is going to play the opening relatively better than the opponent but it also is going to start from slightly worse position).

If you start from non-standard positions it does not have to be inferior positions but even if it is slightly inferior, that is easily compensated for by the fact that your program is superior in the opening and you can force him into using clock time to do what YOU do better. It's stupid to let him cruise into the part of the game he does best without even taking a time hit.

I am also not sure if komodo is better with very short book at long time control.

I'm not sure either. That is a working hypothesis and I don't believe until I can prove with some experiment. However I do think it's very likely and Larry has a huge amount of experience with Komodo at very long thinking time in his opening book analysis projects and strongly believes Komodo is superior to every other program for this purpose. (which address you next point since this is not short time control stuff.)

Being relatively better in evaluating the opening stage does not help much at long time control when the program already search middle game positions even in the early opening stage.

But I'm talking only about long time controls. I don't know how Komodo does at short time controls with the opening. Maybe it's superior there too?

Mathematically however you have fallen prey to a common fallacy which is that at long time controls the program is thinking 10 times deeper. No, it doesn't work that way. It may be thinking 10 times LONGER but it's not searching 10 times deeper. In the first couple of seconds of search Komodo and any other program has completely most of the depth it's going to unless you are playing correspondence chess, and even then it probably is searching at least half the depth in the first 2 or 3 seconds.

A good place to pose this question to you. And no, it's not a trick question with a trap. It is a serious question:

We all know if you have 3 exes: a normal, an SSE and a popcnt- it is best to run them in console mode to determine the fastest at nps- then use it (even tho that is supposed to be popcnt anyway)

Point being you want the fastest. All other things being equal, you want your engine to be considered "fast". Averaging 1700kns is better than 950kns- even tho an extreme example.

Am I wrong when I say the faster your engine runs, the more it would matter and help in a 2 to 5 min. blitz game. Whereas at 40/40 repeating and slower controls, the speed factor takes on less importance. Because at those controls, time is not that much of a factor- and it is less important how quick you get there- but rather what you do when you do arrive.

Let me add a "part B" to the question. In the pgns for a game, it shows the average kns both engines ran at, and the average depth to which each thought. Why often do I see the engine that is running at much slower kns repeatedly thinking 1 to 2.5 ply depths deeper than the faster engine?

Actually these 2 questions, or 1 question with 2 parts- depending on your POV, is one(s) whose answers are very important to me.



Thanks in advance-

george

[lkaufman]

Some people may view the opening as irrelevant since it is possible to obtain comprehensive opening books (which can be considered part of the game playing system) and that is a legitimate point of view - however it's easy to get a program out of the opening with a few non-standard but not ridiculous moves in which case the program will be on it's own and will need to be able to play the opening well.

I think that if you get the program out of book by non standard moves you do not lose the game but you clearly get relatively worse position in most cases so testing with very short opening books does not give a good estimate what the program can practically do in tournaments when both sides use books(if you choose to give komodo a book with non standard moves then maybe komodo is going to play the opening relatively better than the opponent but it also is going to start from slightly worse position).

If you start from non-standard positions it does not have to be inferior positions but even if it is slightly inferior, that is easily compensated for by the fact that your program is superior in the opening and you can force him into using clock time to do what YOU do better. It's stupid to let him cruise into the part of the game he does best without even taking a time hit.

I am also not sure if komodo is better with very short book at long time control.

I'm not sure either. That is a working hypothesis and I don't believe until I can prove with some experiment. However I do think it's very likely and Larry has a huge amount of experience with Komodo at very long thinking time in his opening book analysis projects and strongly believes Komodo is superior to every other program for this purpose. (which address you next point since this is not short time control stuff.)

Being relatively better in evaluating the opening stage does not help much at long time control when the program already search middle game positions even in the early opening stage.

But I'm talking only about long time controls. I don't know how Komodo does at short time controls with the opening. Maybe it's superior there too?

Mathematically however you have fallen prey to a common fallacy which is that at long time controls the program is thinking 10 times deeper. No, it doesn't work that way. It may be thinking 10 times LONGER but it's not searching 10 times deeper. In the first couple of seconds of search Komodo and any other program has completely most of the depth it's going to unless you are playing correspondence chess, and even then it probably is searching at least half the depth in the first 2 or 3 seconds.

A good place to pose this question to you. And no, it's not a trick question with a trap. It is a serious question:

We all know if you have 3 exes: a normal, an SSE and a popcnt- it is best to run them in console mode to determine the fastest at nps- then use it (even tho that is supposed to be popcnt anyway)

Point being you want the fastest. All other things being equal, you want your engine to be considered "fast". Averaging 1700kns is better than 950kns- even tho an extreme example.

Am I wrong when I say the faster your engine runs, the more it would matter and help in a 2 to 5 min. blitz game. Whereas at 40/40 repeating and slower controls, the speed factor takes on less importance. Because at those controls, time is not that much of a factor- and it is less important how quick you get there- but rather what you do when you do arrive.

Let me add a "part B" to the question. In the pgns for a game, it shows the average kns both engines ran at, and the average depth to which each thought. Why often do I see the engine that is running at much slower kns repeatedly thinking 1 to 2.5 ply depths deeper than the faster engine?

Actually these 2 questions, or 1 question with 2 parts- depending on your POV, is one(s) whose answers are very important to me.



Thanks in advance-

george

It is certainly true that in general, the shorter the time limit, the greater the elo difference that would result from a given speedup. It is probably also true that in general, if you trade speed for more chess knowledge, it is more likely to be a good tradeoff with longer time limits, although in many cases it won't be a good tradeoff at any practical time limit.
As for your second question, if you are talking about unrelated engines, the depth reached depends on other factors than just NPS. Stockfish will reach higher depths than other programs because it prunes more, even if it does less NPS than some other programs.

Does this answer your questions?

[geots]

Some people may view the opening as irrelevant since it is possible to obtain comprehensive opening books (which can be considered part of the game playing system) and that is a legitimate point of view - however it's easy to get a program out of the opening with a few non-standard but not ridiculous moves in which case the program will be on it's own and will need to be able to play the opening well.

I think that if you get the program out of book by non standard moves you do not lose the game but you clearly get relatively worse position in most cases so testing with very short opening books does not give a good estimate what the program can practically do in tournaments when both sides use books(if you choose to give komodo a book with non standard moves then maybe komodo is going to play the opening relatively better than the opponent but it also is going to start from slightly worse position).

If you start from non-standard positions it does not have to be inferior positions but even if it is slightly inferior, that is easily compensated for by the fact that your program is superior in the opening and you can force him into using clock time to do what YOU do better. It's stupid to let him cruise into the part of the game he does best without even taking a time hit.

I am also not sure if komodo is better with very short book at long time control.

I'm not sure either. That is a working hypothesis and I don't believe until I can prove with some experiment. However I do think it's very likely and Larry has a huge amount of experience with Komodo at very long thinking time in his opening book analysis projects and strongly believes Komodo is superior to every other program for this purpose. (which address you next point since this is not short time control stuff.)

Being relatively better in evaluating the opening stage does not help much at long time control when the program already search middle game positions even in the early opening stage.

But I'm talking only about long time controls. I don't know how Komodo does at short time controls with the opening. Maybe it's superior there too?

Mathematically however you have fallen prey to a common fallacy which is that at long time controls the program is thinking 10 times deeper. No, it doesn't work that way. It may be thinking 10 times LONGER but it's not searching 10 times deeper. In the first couple of seconds of search Komodo and any other program has completely most of the depth it's going to unless you are playing correspondence chess, and even then it probably is searching at least half the depth in the first 2 or 3 seconds.

A good place to pose this question to you. And no, it's not a trick question with a trap. It is a serious question:

We all know if you have 3 exes: a normal, an SSE and a popcnt- it is best to run them in console mode to determine the fastest at nps- then use it (even tho that is supposed to be popcnt anyway)

Point being you want the fastest. All other things being equal, you want your engine to be considered "fast". Averaging 1700kns is better than 950kns- even tho an extreme example.

Am I wrong when I say the faster your engine runs, the more it would matter and help in a 2 to 5 min. blitz game. Whereas at 40/40 repeating and slower controls, the speed factor takes on less importance. Because at those controls, time is not that much of a factor- and it is less important how quick you get there- but rather what you do when you do arrive.

Let me add a "part B" to the question. In the pgns for a game, it shows the average kns both engines ran at, and the average depth to which each thought. Why often do I see the engine that is running at much slower kns repeatedly thinking 1 to 2.5 ply depths deeper than the faster engine?

Actually these 2 questions, or 1 question with 2 parts- depending on your POV, is one(s) whose answers are very important to me.



Thanks in advance-

george

It is certainly true that in general, the shorter the time limit, the greater the elo difference that would result from a given speedup. It is probably also true that in general, if you trade speed for more chess knowledge, it is more likely to be a good tradeoff with longer time limits, although in many cases it won't be a good tradeoff at any practical time limit.
As for your second question, if you are talking about unrelated engines, the depth reached depends on other factors than just NPS. Stockfish will reach higher depths than other programs because it prunes more, even if it does less NPS than some other programs.

Does this answer your questions?

That takes care of it Larry. Thanks. One thing- when looking at the average kns and ply depth each engine averaged in a game, more often than the slower one going to the deeper depths that I stated, usually for instance Komodo will go to the same depth or maybe one-half to 1 ply depth deeper than Houdini. Actually no matter the nps- they both pretty much average the same depth. The 2 ply depth diff. by the slower one is really much more rare than I stated.


Thanks again,

george


PS: One important thing to consider about the use of generic books that is being discussed and I never see it mentioned: Even tho I always set the maximum moves allowed in the opening to 12- some of Sedat's books don't go that deep and others of his will. Either is fine with me. As long as they come out of the opening in basically an equal position, the book has done it's job for me. (Tho I always set the slider where there will never be enough variation to let it play the rogue openings that ALL generic books and most others have.)

But the point is you pretty much never see it at a limit of 8 or 9 moves- but if your generic book goes 12 move deep- each game should be checked to make SURE that neither engine came out of the opening +1.00 or higher. It is ok if it does and later drops back below +.80 or whatever you have decided is ok and then goes back up- I let that go. But if it comes out at +1.00 or higher, and keeps climbing and wins, I void the game. The +1 figure is not written in stone- just decide something reasonable and stick with it.

You would be surprised at the number of testers that have told me they never thought to check for that.


gts

[lkaufman]

Some comments:
1. Komodo should outsearch Houdini but not Stockfish because Komodo is in between them (assuming Houdini is same as Ivanhoe in this matter) in how much we reduce in general.
2. Use of a score of one pawn as a threshold seems way too high. In some programs such a score means over 90% chance to win.
3. Use of any score to cancel games is a potential bias, because it depends on which program reports the score. A given score has different meanings in different programs. Scores run nearly double in Stockfish compared to Houdini. As in most things, Komodo is in the middle.
4. It is far better to test with "testsuites" or opening books designed for such tests, rather than generic ones. Then you needn't worry about the score out of book, this has already been done for you.