komodo 12.1.1 MCTS vs Texel 1.07

Discussion of computer chess matches and engine tournaments.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
lkaufman
Posts: 3837
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

komodo 12.1.1 MCTS vs Texel 1.07

Post by lkaufman » Tue Jun 05, 2018 4:18 pm

After considerable trial and error, I finally found a close match for Komodo 12.1.1 MCTS. After 110 single thread games at 5 min plus 4 sec against the latest version Texel (1.07) (CCRL blitz 3117), the score was tied, 55 to 55. My estimated CCRL blitz rating in the readme file was 3120.
Komodo rules!

Werewolf
Posts: 1199
Joined: Thu Sep 18, 2008 8:24 pm

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by Werewolf » Tue Jun 05, 2018 4:35 pm

Impressive. I'd love to see the match repeated at a higher thread count.

cc2150dx
Posts: 176
Joined: Sat Nov 30, 2013 8:51 am
Location: Canada

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by cc2150dx » Tue Jun 05, 2018 5:48 pm

And a longer time control :)
Komodo tester

lkaufman
Posts: 3837
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by lkaufman » Tue Jun 05, 2018 8:40 pm

Werewolf wrote:
Tue Jun 05, 2018 4:35 pm
Impressive. I'd love to see the match repeated at a higher thread count.
First, small correction. It was 5' + 3", not 4" (just a typo). I'm now rerunning the same match on four threads but half the time limit; so far Texel is slightly ahead (112 to 105). More tests to come. These are run on my I7 quad laptop, in Windows. I could run a lot more games on our Linux tester, but I don't know if using Linux would favor Komodo or Texel.
Komodo rules!

Werewolf
Posts: 1199
Joined: Thu Sep 18, 2008 8:24 pm

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by Werewolf » Tue Jun 05, 2018 9:22 pm

lkaufman wrote:
Tue Jun 05, 2018 8:40 pm

I could run a lot more games on our Linux tester, but I don't know if using Linux would favor Komodo or Texel.
Interesting comment.

lkaufman
Posts: 3837
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by lkaufman » Wed Jun 06, 2018 6:38 pm

cc2150dx wrote:
Tue Jun 05, 2018 5:48 pm
And a longer time control :)
OK, I ran two more single thread matches. First I ran at half the time limit of the 5' + 3" match, Komodo losing by 93.5 to 118.5, about 41 elo. Then I ran at double the time of the 5' + 3" match, this time with Komodo winning by 88 to 66, which is fifty elo. So based on these results, each doubling of the time limit favors Komodo by 45 elo points, which is incredible. Of course some of this may be sampling error, but it does suggest that at some very long time control Komodo MCTS might become competitive with normal Komodo. But it's not practical to test at levels like an hour per move. The scaling with more threads appears to be positive but much less so than the scaling with time, meaning that the MP is not really so great with the MCTS version but the effective quadrulpling of the time limit with four threads helps a lot. I'm starting some Linux tests to clarify further. I hope that the testing groups test MCTS at 40/20 and 40/40, it should do better on those lists than on the blitz lists.
Komodo rules!

lkaufman
Posts: 3837
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by lkaufman » Thu Jun 07, 2018 2:57 am

More results. At 10' + 6" single thread, MCTS is tied with Stockfish 3 after 50 games. Just six more SF versions to catch up with!
Against the final Critter, 1.6a, which is somewhat higher rated than Texel 1.07 and SF3, I'm seeing similar scaling behavior. At 200" + 2" (this time on Linux) MCTS lost by 93 elo after 414 games. At 400" + 4" MCTS lost by 39 elo after 696 games. At 800" + 8" MCTS is down by 19 elo after 293 games. One more doubling might close the gap entirely, I'll try that overnight. On four threads (at 100" + 1") Critter won by just 7 elo. So the overall scaling was again very good, similar to the Texel matches, but so far the scaling with more threads was better than with more time against Critter. Anyway it looks like Komodo MCTS will get around 3100 on CCRL 40/40 while the normal Komdo 12.1.1 should be around 3340, both on single thread. So 240 elo to go at that level to catch base Komodo.
Komodo rules!

Modern Times
Posts: 2451
Joined: Thu Jun 07, 2012 9:02 pm

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by Modern Times » Thu Jun 07, 2018 1:13 pm

lkaufman wrote:
Thu Jun 07, 2018 2:57 am
Anyway it looks like Komodo MCTS will get around 3100 on CCRL 40/40 while the normal Komdo 12.1.1 should be around 3340, both on single thread. So 240 elo to go at that level to catch base Komodo.
Testing is underway on 4 threads with MCTS for 40/40. I'd say it is at least 3200 Elo (consistent with your 3100 estimate on 1 thread) but it is very early days. Let's see if I am right.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

lkaufman
Posts: 3837
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by lkaufman » Thu Jun 07, 2018 6:27 pm

Modern Times wrote:
Thu Jun 07, 2018 1:13 pm
lkaufman wrote:
Thu Jun 07, 2018 2:57 am
Anyway it looks like Komodo MCTS will get around 3100 on CCRL 40/40 while the normal Komdo 12.1.1 should be around 3340, both on single thread. So 240 elo to go at that level to catch base Komodo.
Testing is underway on 4 threads with MCTS for 40/40. I'd say it is at least 3200 Elo (consistent with your 3100 estimate on 1 thread) but it is very early days. Let's see if I am right.

Against SF3 on one thread Komodo at 10' + 6" Komodo MCTS pulled ahead by 129 to 119, which would give a CCRL 40/40 rating of about 3110 on one thread. Against Critter 1.60a at 1600" + 16" Komodo lost by 19 elo which brings down the combined result to a few points shy of 3100, but on four threads at 800" + 8" we beat Critter by 25 elo (243 games) for a 3190 CCRL 40/40 performance. So basically I agree with your comment, 3200 for quad looks about right. My estimates may be a bit low as I've noticed that MCTS tends to underperform against IPPO derivatives such as Gull, Critter, and Houdini.
Komodo rules!

ThatsIt
Posts: 925
Joined: Thu Mar 09, 2006 1:11 pm
Contact:

Re: komodo 12.1.1 MCTS vs Texel 1.07

Post by ThatsIt » Fri Jun 08, 2018 9:05 am

Sounds interesting.
I will try to activate our "long time testers" ;-)

Best wishes,
G.S. (CEGT team)

Post Reply