komodo 12.1.1 MCTS vs Texel 1.07

[lkaufman]

After considerable trial and error, I finally found a close match for Komodo 12.1.1 MCTS. After 110 single thread games at 5 min plus 4 sec against the latest version Texel (1.07) (CCRL blitz 3117), the score was tied, 55 to 55. My estimated CCRL blitz rating in the readme file was 3120.

[Werewolf]

Impressive. I'd love to see the match repeated at a higher thread count.

[cc2150dx]

And a longer time control

[lkaufman]

Impressive. I'd love to see the match repeated at a higher thread count.

First, small correction. It was 5' + 3", not 4" (just a typo). I'm now rerunning the same match on four threads but half the time limit; so far Texel is slightly ahead (112 to 105). More tests to come. These are run on my I7 quad laptop, in Windows. I could run a lot more games on our Linux tester, but I don't know if using Linux would favor Komodo or Texel.

[Werewolf]

I could run a lot more games on our Linux tester, but I don't know if using Linux would favor Komodo or Texel.

Interesting comment.

[lkaufman]

And a longer time control

OK, I ran two more single thread matches. First I ran at half the time limit of the 5' + 3" match, Komodo losing by 93.5 to 118.5, about 41 elo. Then I ran at double the time of the 5' + 3" match, this time with Komodo winning by 88 to 66, which is fifty elo. So based on these results, each doubling of the time limit favors Komodo by 45 elo points, which is incredible. Of course some of this may be sampling error, but it does suggest that at some very long time control Komodo MCTS might become competitive with normal Komodo. But it's not practical to test at levels like an hour per move. The scaling with more threads appears to be positive but much less so than the scaling with time, meaning that the MP is not really so great with the MCTS version but the effective quadrulpling of the time limit with four threads helps a lot. I'm starting some Linux tests to clarify further. I hope that the testing groups test MCTS at 40/20 and 40/40, it should do better on those lists than on the blitz lists.

[lkaufman]

More results. At 10' + 6" single thread, MCTS is tied with Stockfish 3 after 50 games. Just six more SF versions to catch up with!
Against the final Critter, 1.6a, which is somewhat higher rated than Texel 1.07 and SF3, I'm seeing similar scaling behavior. At 200" + 2" (this time on Linux) MCTS lost by 93 elo after 414 games. At 400" + 4" MCTS lost by 39 elo after 696 games. At 800" + 8" MCTS is down by 19 elo after 293 games. One more doubling might close the gap entirely, I'll try that overnight. On four threads (at 100" + 1") Critter won by just 7 elo. So the overall scaling was again very good, similar to the Texel matches, but so far the scaling with more threads was better than with more time against Critter. Anyway it looks like Komodo MCTS will get around 3100 on CCRL 40/40 while the normal Komdo 12.1.1 should be around 3340, both on single thread. So 240 elo to go at that level to catch base Komodo.

[Modern Times]

Anyway it looks like Komodo MCTS will get around 3100 on CCRL 40/40 while the normal Komdo 12.1.1 should be around 3340, both on single thread. So 240 elo to go at that level to catch base Komodo.

Testing is underway on 4 threads with MCTS for 40/40. I'd say it is at least 3200 Elo (consistent with your 3100 estimate on 1 thread) but it is very early days. Let's see if I am right.

[lkaufman]

Anyway it looks like Komodo MCTS will get around 3100 on CCRL 40/40 while the normal Komdo 12.1.1 should be around 3340, both on single thread. So 240 elo to go at that level to catch base Komodo.

Testing is underway on 4 threads with MCTS for 40/40. I'd say it is at least 3200 Elo (consistent with your 3100 estimate on 1 thread) but it is very early days. Let's see if I am right.

Against SF3 on one thread Komodo at 10' + 6" Komodo MCTS pulled ahead by 129 to 119, which would give a CCRL 40/40 rating of about 3110 on one thread. Against Critter 1.60a at 1600" + 16" Komodo lost by 19 elo which brings down the combined result to a few points shy of 3100, but on four threads at 800" + 8" we beat Critter by 25 elo (243 games) for a 3190 CCRL 40/40 performance. So basically I agree with your comment, 3200 for quad looks about right. My estimates may be a bit low as I've noticed that MCTS tends to underperform against IPPO derivatives such as Gull, Critter, and Houdini.

[ThatsIt]

Sounds interesting.
I will try to activate our "long time testers"

Best wishes,
G.S. (CEGT team)