Komodo 10.4 released

[lkaufman]

Sure.

"Variety" modifies something in the evaluation function. A user can enter a number from 0 (no variety--always play the best move) to 20 (most variety, play more different moves). The value of "Variety" defines the magnitude of this evaluation change. At the start of each new game, Komodo randomly selects a value in the range of 0...Value, and modifies this evaluation term. For the rest of the game it uses this new value in computing the evaluation. These smallish changes are enough to have Komodo play a variety of different games, without it changing elo much.

This means a human play will get some different move choices (not one forced moves but in places where several moves have a close eval). It is possible it is even useful in computer match play, since games will vary instead of Komodo always choosing the same move.

We were surprised to find that even the higher Variety settings do not hurt elo by much. In fact, some test matches did a little better than default. But for best play Variety=0 is probably best.

One last use: prevent someone with a copy of Komodo from "booking up" against your Komodo.

Mark

I fooled around with this "Variety" UCI option, and saw something surprising: Komodo 10.4 Variety=0 already randomizes quite a lot on 1 thread at 10''+0.1'' TC. Very different from Stockfish on 1 thread, where I found almost no randomization. I must have missed an important aspect of Komodo, or is it something new with the default Komodo? Here are my results for the number of different final positions got in 5 moves from basic KID Saemisch at 10''+0.1'' in 400 self-play games:

Starting position:
[d]rnbqk2r/ppp1ppbp/3p1np1/8/2PPP3/2N2P2/PP4PP/R1BQKBNR b KQkq - 0 5

Different final positions:

Code: Select all

Kom Var 20: 229/400
Kom Var  0: 218/400
SF:           6/400

Looking at the distribution of openings in the resulting EPD files of different positions, it seems that Komodo Var 20 or Var 0 cannot yet play randomly reliably, some openings in 400 games (on 5 moves) are repeated 20+ times, while the majority are single occurrences. Still, Komodo Var 20 is probably the best tool to build opening "books" for a position we are interested in.

Using our own tester in Linux at the same tc you used, I ran pairs of games using our normal opening book and compared the pairs to see if the games were the same or not. With Komodo 10.4 using zero Variety I always got a different game (until I got bored checking), with the deviation usually taking place around moves ten to twenty. This is probably consistent with your result where you only used one position. However with SF8 I also always got a different game, which is very much NOT like your result. Both SF and Komodo played identical games when I ran on fixed 11 ply depth. So for me the conclusion is that even without using Variety, at these super-fast levels the games diverge with both engines, but I have no idea why your Stockfish result was so different in this respect. Perhaps the fact that I run many more games at once than you do (due to big hardware) has something to do with it, but I don't see why that would affect Stockfish so differently from Komodo.

[mjlef]

Cutechess-Cli, doesn't reload, in a 2 engine match, the engines are loaded once for entire match.

Doesn't this option work?

Code: Select all

  restart=MODE	
	Set the restart mode to MODE which can be:
			'auto': the engine decides whether to restart (default)
			'on': the engine is always restarted between games
			'off': the engine is never restarted between games
			Setting this option does not prevent engines from being
			restarted between rounds in a tournament featuring more
         than two engines.

I actually use restart=off, as it seemed to me faster and more reliable in hyper-fast games.

But even if engines clear the main Hash on ucinewgame, they often do not reset things like history tables, king and pawn hashes, material hashes which can effect move choice. I suggest repeating the test with restart on. Our tester always does a full restart, and when we do in fixed depth we get exact game matches.

[Laskos]

Using our own tester in Linux at the same tc you used, I ran pairs of games using our normal opening book and compared the pairs to see if the games were the same or not. With Komodo 10.4 using zero Variety I always got a different game (until I got bored checking), with the deviation usually taking place around moves ten to twenty. This is probably consistent with your result where you only used one position. However with SF8 I also always got a different game, which is very much NOT like your result. Both SF and Komodo played identical games when I ran on fixed 11 ply depth. So for me the conclusion is that even without using Variety, at these super-fast levels the games diverge with both engines, but I have no idea why your Stockfish result was so different in this respect. Perhaps the fact that I run many more games at once than you do (due to big hardware) has something to do with it, but I don't see why that would affect Stockfish so differently from Komodo.

The full game of in average 120 plies is pretty poor indicator of randomness. It's natural that even SF will give different games. I used 5-mover games, 10 plies, to see the branching factor of the book. For Komodo, having 240 different final positions in 400 games in 5 moves is equivalent to having a book with branching factor about 1.7-1.8. For each position, there are on average 1.7-1.8 (close to 2) next positions in the tree. And we can say that on average Komodo games diverge after 5 moves (10 plies). With SF, the branching factor is about 1.2, and we can say that on average SF games diverge after 15-20 moves. With full 120 ply games both Komodo and SF games will diverge. But that a 120 ply game is different is not enough for variety. Branching factor close to 2 is. That's a pretty solid book, if the moves are serious. And with Komodo either Var=0 or Var=20, without restarting engine in Cutechess-Cli, the in-built Komodo randomization gives serious moves even at 10''+0.1''. Probably better than my initial in this thread book building with SF depth=7 2-threads. The randomization is better there (340/400), but the quality of moves is worse. I will probably use non-restarted 1-threaded Komodo Var=20 or Var=0 for book building from 1 position. It seems, non-restarted 1-threaded Komodo in Cutechess-Cli, can almost play from 1-position fairly representative matches (although some lines will be favored, so a book building of unique positions is recommended).

[Laskos]

Cutechess-Cli, doesn't reload, in a 2 engine match, the engines are loaded once for entire match.

Doesn't this option work?

Code: Select all

  restart=MODE	
	Set the restart mode to MODE which can be:
			'auto': the engine decides whether to restart (default)
			'on': the engine is always restarted between games
			'off': the engine is never restarted between games
			Setting this option does not prevent engines from being
			restarted between rounds in a tournament featuring more
         than two engines.

I actually use restart=off, as it seemed to me faster and more reliable in hyper-fast games.

But even if engines clear the main Hash on ucinewgame, they often do not reset things like history tables, king and pawn hashes, material hashes which can effect move choice. I suggest repeating the test with restart on. Our tester always does a full restart, and when we do in fixed depth we get exact game matches.

Yes, I may re-do the test, but I am not against randomization. If I am able in Cutechess-Cli, without re-starting engines, get desired strong-move randomization of Komodo, I am pretty happy. Building books with 2-threaded SF, at fixed depth (Lazy SMP seems to randomize very much), I somehow missed the fact that Komodo even 1-thread, without restarting, randomizes heavily at fixed time.

[lkaufman]

Using our own tester in Linux at the same tc you used, I ran pairs of games using our normal opening book and compared the pairs to see if the games were the same or not. With Komodo 10.4 using zero Variety I always got a different game (until I got bored checking), with the deviation usually taking place around moves ten to twenty. This is probably consistent with your result where you only used one position. However with SF8 I also always got a different game, which is very much NOT like your result. Both SF and Komodo played identical games when I ran on fixed 11 ply depth. So for me the conclusion is that even without using Variety, at these super-fast levels the games diverge with both engines, but I have no idea why your Stockfish result was so different in this respect. Perhaps the fact that I run many more games at once than you do (due to big hardware) has something to do with it, but I don't see why that would affect Stockfish so differently from Komodo.

The full game of in average 120 plies is pretty poor indicator of randomness. It's natural that even SF will give different games. I used 5-mover games, 10 plies, to see the branching factor of the book. For Komodo, having 240 different final positions in 400 games in 5 moves is equivalent to having a book with branching factor about 1.7-1.8. For each position, there are on average 1.7-1.8 (close to 2) next positions in the tree. And we can say that on average Komodo games diverge after 5 moves (10 plies). With SF, the branching factor is about 1.2, and we can say that on average SF games diverge after 15-20 moves. With full 120 ply games both Komodo and SF games will diverge. But that a 120 ply game is different is not enough for variety. Branching factor close to 2 is. That's a pretty solid book, if the moves are serious. And with Komodo either Var=0 or Var=20, without restarting engine in Cutechess-Cli, the in-built Komodo randomization gives serious moves even at 10''+0.1''. Probably better than my initial in this thread book building with SF depth=7 2-threads. The randomization is better there (340/400), but the quality of moves is worse. I will probably use non-restarted 1-threaded Komodo Var=20 or Var=0 for book building from 1 position. It seems, non-restarted 1-threaded Komodo in Cutechess-Cli, can almost play from 1-position fairly representative matches (although some lines will be favored, so a book building of unique positions is recommended).

Sorry, I missed that you were only talking about a handful of moves out. Anyway, I think you are implying that Stockfish changes its mind WAY less often than Komodo does. Assuming this is so, the question is why? I don't expect you to know the answer, perhaps no one does, but I'd be interested in any theories about this. Of course Komodo and Stockfish are different programs, but they are not so drastically different as to easily explain this phenomenon. I have a feeling that this is at the heart of why Stockfish is so strong, so any chess programmer who has studied SF code a lot might have some ideas about this.

[Nordlandia]

Odds of the castled king + no castling rights for black

Houdini 4 Pro vs Komodo 10.4 | K10.4 is not allowed to castle | Odds is equivalent to 1.50

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/QNBRRBNK w - - 0 1

[Laskos]

Using our own tester in Linux at the same tc you used, I ran pairs of games using our normal opening book and compared the pairs to see if the games were the same or not. With Komodo 10.4 using zero Variety I always got a different game (until I got bored checking), with the deviation usually taking place around moves ten to twenty. This is probably consistent with your result where you only used one position. However with SF8 I also always got a different game, which is very much NOT like your result. Both SF and Komodo played identical games when I ran on fixed 11 ply depth. So for me the conclusion is that even without using Variety, at these super-fast levels the games diverge with both engines, but I have no idea why your Stockfish result was so different in this respect. Perhaps the fact that I run many more games at once than you do (due to big hardware) has something to do with it, but I don't see why that would affect Stockfish so differently from Komodo.

The full game of in average 120 plies is pretty poor indicator of randomness. It's natural that even SF will give different games. I used 5-mover games, 10 plies, to see the branching factor of the book. For Komodo, having 240 different final positions in 400 games in 5 moves is equivalent to having a book with branching factor about 1.7-1.8. For each position, there are on average 1.7-1.8 (close to 2) next positions in the tree. And we can say that on average Komodo games diverge after 5 moves (10 plies). With SF, the branching factor is about 1.2, and we can say that on average SF games diverge after 15-20 moves. With full 120 ply games both Komodo and SF games will diverge. But that a 120 ply game is different is not enough for variety. Branching factor close to 2 is. That's a pretty solid book, if the moves are serious. And with Komodo either Var=0 or Var=20, without restarting engine in Cutechess-Cli, the in-built Komodo randomization gives serious moves even at 10''+0.1''. Probably better than my initial in this thread book building with SF depth=7 2-threads. The randomization is better there (340/400), but the quality of moves is worse. I will probably use non-restarted 1-threaded Komodo Var=20 or Var=0 for book building from 1 position. It seems, non-restarted 1-threaded Komodo in Cutechess-Cli, can almost play from 1-position fairly representative matches (although some lines will be favored, so a book building of unique positions is recommended).

Sorry, I missed that you were only talking about a handful of moves out. Anyway, I think you are implying that Stockfish changes its mind WAY less often than Komodo does. Assuming this is so, the question is why? I don't expect you to know the answer, perhaps no one does, but I'd be interested in any theories about this. Of course Komodo and Stockfish are different programs, but they are not so drastically different as to easily explain this phenomenon. I have a feeling that this is at the heart of why Stockfish is so strong, so any chess programmer who has studied SF code a lot might have some ideas about this.

I don't think there is something very deep there. I re-did the test with restart=on in Cutechess-Cli, the results for Komodo are drastically different, and more similar to Stockfish. Same conditions:

Komodo:
Var 0: 16/400
Var 20: 84/400

Stockfish: 6/400

So, "Variety" does add variety, but not clearing hash on "ucinewgame", not resetting history tables, king and pawn hashes, material hashes, is adding even more variety (more than 200/400 different positions on 5-movers). It seems Stockfish resets everything on "ucinewgame", independent of restart=on or =off in Cutechess-Cli.

Basically, I probably missed for quite a lot of time that I can randomize games using Komodo, due to its more lax implementation of "ucinewgame".

[clumma]

KRfB&W is one of my favorite chess books. Will you please consider releasing it in a digital format as well? Ideally a database of some kind, but even an ebook would be more searchable.

-Carl

[Amstaff]

Hello Larry, thanks for the new Komodo release. In your opinion, what are the top 3 current chess engines with contrasting styles? This would be useful information for game analysis to get different "points of view".
Thanks,
Gerald

[mjlef]

I privately sent a message to Jon to get more details. Note the screen has castle rights turned off (seem to be the Fritz interface). And this is not a Chess 960 starting position since that requires same pieces on the same file. Komodo was designed with this Chess 960 requirement. But if there is interest in unbalance scrambled positions with castling rights, I can see what we can do to make Komodo handle this.

Mark