Komodo 10.4 released

[lkaufman]

KRfB&W is one of my favorite chess books. Will you please consider releasing it in a digital format as well? Ideally a database of some kind, but even an ebook would be more searchable.
-Carl

That would be a decision for New in Chess to make; I would be fine with whatever they decide.

[lkaufman]

Hello Larry, thanks for the new Komodo release. In your opinion, what are the top 3 current chess engines with contrasting styles? This would be useful information for game analysis to get different "points of view".
Thanks,
Gerald

Since Stockfish, Komodo, and Houdini 5 are rated far above all other engines and are significantly different from each other, they would be the obvious choice. I know that some other engines are less similar to these three than they are to each other, but this cannot compensate for a hundred or more elo points. That said, I always liked Shredder for a different opinion.

[lkaufman]

Using our own tester in Linux at the same tc you used, I ran pairs of games using our normal opening book and compared the pairs to see if the games were the same or not. With Komodo 10.4 using zero Variety I always got a different game (until I got bored checking), with the deviation usually taking place around moves ten to twenty. This is probably consistent with your result where you only used one position. However with SF8 I also always got a different game, which is very much NOT like your result. Both SF and Komodo played identical games when I ran on fixed 11 ply depth. So for me the conclusion is that even without using Variety, at these super-fast levels the games diverge with both engines, but I have no idea why your Stockfish result was so different in this respect. Perhaps the fact that I run many more games at once than you do (due to big hardware) has something to do with it, but I don't see why that would affect Stockfish so differently from Komodo.

The full game of in average 120 plies is pretty poor indicator of randomness. It's natural that even SF will give different games. I used 5-mover games, 10 plies, to see the branching factor of the book. For Komodo, having 240 different final positions in 400 games in 5 moves is equivalent to having a book with branching factor about 1.7-1.8. For each position, there are on average 1.7-1.8 (close to 2) next positions in the tree. And we can say that on average Komodo games diverge after 5 moves (10 plies). With SF, the branching factor is about 1.2, and we can say that on average SF games diverge after 15-20 moves. With full 120 ply games both Komodo and SF games will diverge. But that a 120 ply game is different is not enough for variety. Branching factor close to 2 is. That's a pretty solid book, if the moves are serious. And with Komodo either Var=0 or Var=20, without restarting engine in Cutechess-Cli, the in-built Komodo randomization gives serious moves even at 10''+0.1''. Probably better than my initial in this thread book building with SF depth=7 2-threads. The randomization is better there (340/400), but the quality of moves is worse. I will probably use non-restarted 1-threaded Komodo Var=20 or Var=0 for book building from 1 position. It seems, non-restarted 1-threaded Komodo in Cutechess-Cli, can almost play from 1-position fairly representative matches (although some lines will be favored, so a book building of unique positions is recommended).

Sorry, I missed that you were only talking about a handful of moves out. Anyway, I think you are implying that Stockfish changes its mind WAY less often than Komodo does. Assuming this is so, the question is why? I don't expect you to know the answer, perhaps no one does, but I'd be interested in any theories about this. Of course Komodo and Stockfish are different programs, but they are not so drastically different as to easily explain this phenomenon. I have a feeling that this is at the heart of why Stockfish is so strong, so any chess programmer who has studied SF code a lot might have some ideas about this.

I don't think there is something very deep there. I re-did the test with restart=on in Cutechess-Cli, the results for Komodo are drastically different, and more similar to Stockfish. Same conditions:

Komodo:
Var 0: 16/400
Var 20: 84/400

Stockfish: 6/400

So, "Variety" does add variety, but not clearing hash on "ucinewgame", not resetting history tables, king and pawn hashes, material hashes, is adding even more variety (more than 200/400 different positions on 5-movers). It seems Stockfish resets everything on "ucinewgame", independent of restart=on or =off in Cutechess-Cli.

Basically, I probably missed for quite a lot of time that I can randomize games using Komodo, due to its more lax implementation of "ucinewgame".

OK, thanks. Sixteen out of 400 vs 6 out of 400 is still a significant difference, but not a ridiculous one. So it still seems that Stockfish changes its "mind" less than Komodo, but perhaps this can be explained by differences in lmr, null move, eval, etc. Still I suspect something more. Stockfish seems to prune/reduce more than Komodo even if we duplicate SF pruning and reduction rules, and I don't know why.

[cdani]

Stockfish seems to prune/reduce more than Komodo even if we duplicate SF pruning and reduction rules, and I don't know why.

My latest theory is the equilibrium between evaluation parameters, that tend to find some moves and stick with them without much changes. So move ordering related to evaluation.

[cdani]

Stockfish seems to prune/reduce more than Komodo even if we duplicate SF pruning and reduction rules, and I don't know why.

My latest theory is the equilibrium between evaluation parameters, that tend to find some moves and stick with them without much changes. So move ordering related to evaluation.

I think that sometimes one changes the value of a parameter and its a win, not because the evaluation merits it, but because it differentiates a set of positions that get found quicker, so this helps move order.

[Jesse Gersenson]

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/QNBRRBNK w - - 0 1

White's easily winning in that position.

[Nordlandia]

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/QNBRRBNK w - - 0 1

White's easily winning in that position.

Unfortunately yes. An better try is to mirror that position so Komodo go first against H4 Pro.

[d]qnbrrbnk/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR b - - 0 1

[lkaufman]

Stockfish seems to prune/reduce more than Komodo even if we duplicate SF pruning and reduction rules, and I don't know why.

My latest theory is the equilibrium between evaluation parameters, that tend to find some moves and stick with them without much changes. So move ordering related to evaluation.

I think that sometimes one changes the value of a parameter and its a win, not because the evaluation merits it, but because it differentiates a set of positions that get found quicker, so this helps move order.

Yes, that is quite plausible, although the difference in depth reached comparing SF to a modified Komodo with SF-like search is larger than I would expect from your explanation. Any idea what part of the SF eval is especially suited for efficient pruning?

[MikeB]

We have just released Komodo 10.4 at komodochess.com. It is about 20 elo stronger than Komodo 10.3 (at blitz levels) and has one new UCI option, "variety". Improvements were in both search and eval. Linux users will see an even larger elo gain. In my testing it leads by 13 elo against Houdini 5.01 at 2' + 1" after 1350 games on four threads. This will probably be the last of the 10.x series. As usual, it is free for subscribers and available for a 20% discount for buyers of Komodo 9 or later version.

tc 1 min with 1 sec inc

Code: Select all

200 rounds and 2000 games completed...
time control: 6000+100
Date: 03/16/17 : 16:18:30
2000 game(s) loaded
Rank Name                      Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
---------------------------------------------------------------------------------------------------------
   1 McBrain 2017 v1.3 64 POP   3130   0.0   11   11   800  444.5  55.6  201  112  487  25.1  60.9  3093 
   2 Stockfish 080317 64 POPC   3128   2.3   11   11   800  441.0  55.1  187  105  508  23.4  63.5  3093 
   3 Komodo 10.4 64-bit         3089  38.5   11   11   800  384.5  48.1  143  174  483  17.9  60.4  3103 
   4 McBrain v2.0 64 POPCNT     3082   7.2   11   11   800  371.5  46.4  133  190  477  16.6  59.6  3105 
   5 Komodo 10.3 64-bit         3071  10.4   11   11   800  358.5  44.8  128  211  461  16.0  57.6  3107 
---------------------------------------------------------------------------------------------------------

ResultSet-EloRating>los
                          Mc St Ko Mc Ko
McBrain 2017 v1.3 64 POP     60 99 99 99
Stockfish 080317 64 POPC  39    99 99 99
Komodo 10.4 64-bit         0  0    79 97
McBrain v2.0 64 POPCNT     0  0 20    88
Komodo 10.3 64-bit         0  0  2 11   
ResultSet-EloRating>


clearly better than 10.3 with a 97% confident level and better against SF in bullet- interestingly  McBrain-v2.0 played well against these same opponents game 15 sec with .25 increment - one of those cases where micro testing did not scale to longer tc games 

100 rounds and 1000 games completed...
time control: 1500+25
Date: 03/15/17 : 21:41:06
1000 game(s) loaded
Rank Name                      Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
---------------------------------------------------------------------------------------------------------
   1 McBrain v2.0 64 POPCNT     3194   0.0   19   19   400  261.5  65.4  178   55  167  44.5  41.8  3077 
   2 McBrain 2017 v1.3 64 POP   3181  12.5   18   18   400  256.5  64.1  154   41  205  38.5  51.2  3080 
   3 Stockfish 080317 64 POPC   3109  72.0   18   18   400  205.5  51.4  112  101  187  28.0  46.8  3098 
   4 Komodo 10.4 64-bit         3023  86.2   19   19   400  149.0  37.2   64  166  170  16.0  42.5  3119 
   5 Komodo 10.3 64-bit         2993  30.1   19   19   400  127.5  31.9   43  188  169  10.8  42.2  3127 
---------------------------------------------------------------------------------------------------------

[cdani]

I think that sometimes one changes the value of a parameter and its a win, not because the evaluation merits it, but because it differentiates a set of positions that get found quicker, so this helps move order.

Yes, that is quite plausible, although the difference in depth reached comparing SF to a modified Komodo with SF-like search is larger than I would expect from your explanation. Any idea what part of the SF eval is especially suited for efficient pruning?

In chess there are many moves that lose, many than draw, and just a few that win.

Stockfish seems specially efficient at not losing, so it discards fast losing moves. This points to threat stuff and of course king safety. But is also required to put well the pieces, on contrary one generates a lot of losing possibilities. So other positional stuff is also key. Yes, I know I'm being little specific But I cannot be very specific, as is the combined effect that gives the result, obviously.

Stockfish is quickly satisfied with some not losing or narrowly losing move, discarding other stuff. For me this points somehow to the efficiency of having fewer eval parameters. I mean that other engines with more ellaborated eval can doubt between various non losing moves, thus losing unnecesarily time. This Stockfish effect is good for not losing, but not for generating winning possibilities. The second in Stockfish is more given with the increased depth.