Komodo 12 and MCTS

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
JohnS
Posts: 135
Joined: Sun Feb 24, 2008 1:08 am

Re: Komodo 12 and MCTS

Post by JohnS » Mon May 14, 2018 11:59 pm

Tdunbug wrote:
Mon May 14, 2018 11:42 pm
I don't speak much here but I do want to say that MCTS is a great idea and look forward to it being improved. At the very least, it is a fresh new idea that can be tested.
Agreed. I don't post much either but I'm finding Lczero a fascinating development. It's great to see fresh ideas to challenge the dominant A-B approach. Even if they come up short, it's still good to try new things. Maybe a hybrid approach as some suggest could be better. Who knows, but it will be interesting to watch how things pan out.

I also hope the discussions don't generate into name calling and abuse. No one has perfect knowledge of this or any other field and so a little humility and restraint can go a long way.

shrapnel
Posts: 1248
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: Komodo 12 and MCTS

Post by shrapnel » Tue May 15, 2018 4:55 am

Daniel Shawul wrote:
Mon May 14, 2018 7:53 pm
There is so much bullshit here. The authors are throwing around ideas to benefit from the A0 hype. E.g.
a) We use averaging => style is much better like A0's
b) MCTS better against humans. I bet standard komodo is much better against computers, humans and cockroaches
c) MCTS better in multi-pv mode, I highly doubt it
d) "Although tactical strength is lower with MCTS, positional play and judgment may well be better in many positions". Pure bullshit as they
don't have NN evaluation which account for A0's positional strength.
In general, MCTS maybe better strategically but not at the level they are suggesting without the NN
e) "Evals don't fluctuate so suddenly or wildly." So what? That is infact bad since you will have to spend many more simulations to realize
you are screwed with some shallow tactics.

Daniel
Awesome Post. Ruthless Dissection.
Game, Set and Match ! :lol:
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis

lkaufman
Posts: 3756
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo 12 and MCTS

Post by lkaufman » Tue May 15, 2018 5:33 am

shrapnel wrote:
Tue May 15, 2018 4:55 am
Daniel Shawul wrote:
Mon May 14, 2018 7:53 pm

b) MCTS better against humans. I bet standard komodo is much better against computers, humans and cockroaches
c) MCTS better in multi-pv mode, I highly doubt it

Daniel
Awesome Post. Ruthless Dissection.
Game, Set and Match ! :lol:
Just to address two points (many of the others were just speculation), I agree that standard Komodo is stronger against humans, but I would bet that Komodo MCTS is stronger against humans than similarly rated alpha-beta engines such as (assuming my 3000 CCRL 40/40 estimate is accurate) Gull 2 or Arasan 20.5. It is surely a better strategy against fallible humans not to assume they will see things exactly as the engine does. Maybe Komodo MCTS has to get within 100 or 150 elo of normal Komodo before it will be tougher for humans to beat. On the second point, again we don't claim that Komodo MCTS is currently superior to normal Komodo in multi-pv mode, only that the elo gap shrinks drastically as the number of moves analyzed increases, and may vanish entirely with all moves analyzed. So here too, when and if we get within a couple hundred elo of normal Komodo, the MCTS version should be stronger with enough moves being analyzed.

I just want to add that I'm surprised that Daniel appears to be so negative about MCTS, when I will freely state that his relative success in making a Scorpio MCTS not too far in strength from normal Scorpio was a significant factor in our decision to devote a month to this project. Would it really be so surprising if we also got Komodo MCTS up to say 100 elo below the level of normal Komodo in a few months?
Komodo rules!

Hurnavich
Posts: 722
Joined: Wed Jun 19, 2013 6:14 pm

Re: Komodo 12 and MCTS

Post by Hurnavich » Tue May 15, 2018 6:45 am

Hi,

A little confused here will this run as normal Komodo and are you saying ran as normal it is about 5 elo stronger as none MCTS?
or is it better to stick with 11.3.1 for normal play?

thanks in advance
"May your next game be your best"

mjlef
Posts: 1428
Joined: Thu Mar 30, 2006 12:08 pm
Contact:

Re: Komodo 12 and MCTS

Post by mjlef » Tue May 15, 2018 11:39 am

SzG wrote:
Tue May 15, 2018 8:16 am
Hurnavich wrote:
Tue May 15, 2018 6:45 am
Hi,

A little confused here will this run as normal Komodo and are you saying ran as normal it is about 5 elo stronger as none MCTS?
or is it better to stick with 11.3.1 for normal play?

thanks in advance
5 Elos better than 11.3.1. It will surely show after you have played 1,000,000 games with them.
Komodo 12 is about 5 elo stronger than Komodo 11.3.1 in normal (non MCTS) mode. It takes a few thousand games to prove this, not one million.

Mark

mar
Posts: 2013
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: Komodo 12 and MCTS

Post by mar » Tue May 15, 2018 12:01 pm

mjlef wrote:
Tue May 15, 2018 11:39 am
It takes a few thousand games to prove this, not one million.
Perhaps you meant tens of thousands because just error bars for 10k games are ~5 elo... So you can't reliably measure 5 elo improvement in 10k games.
Martin Sedlak

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: Komodo 12 and MCTS

Post by Joerg Oster » Tue May 15, 2018 12:02 pm

lkaufman wrote:
Tue May 15, 2018 5:33 am
shrapnel wrote:
Tue May 15, 2018 4:55 am
Daniel Shawul wrote:
Mon May 14, 2018 7:53 pm

b) MCTS better against humans. I bet standard komodo is much better against computers, humans and cockroaches
c) MCTS better in multi-pv mode, I highly doubt it

Daniel
Awesome Post. Ruthless Dissection.
Game, Set and Match ! :lol:
Just to address two points (many of the others were just speculation), I agree that standard Komodo is stronger against humans, but I would bet that Komodo MCTS is stronger against humans than similarly rated alpha-beta engines such as (assuming my 3000 CCRL 40/40 estimate is accurate) Gull 2 or Arasan 20.5. It is surely a better strategy against fallible humans not to assume they will see things exactly as the engine does. Maybe Komodo MCTS has to get within 100 or 150 elo of normal Komodo before it will be tougher for humans to beat. On the second point, again we don't claim that Komodo MCTS is currently superior to normal Komodo in multi-pv mode, only that the elo gap shrinks drastically as the number of moves analyzed increases, and may vanish entirely with all moves analyzed. So here too, when and if we get within a couple hundred elo of normal Komodo, the MCTS version should be stronger with enough moves being analyzed.

I just want to add that I'm surprised that Daniel appears to be so negative about MCTS, when I will freely state that his relative success in making a Scorpio MCTS not too far in strength from normal Scorpio was a significant factor in our decision to devote a month to this project. Would it really be so surprising if we also got Komodo MCTS up to say 100 elo below the level of normal Komodo in a few months?
Well, I wouldn't be surprised. :)
Just like plain Alpha-Beta search without any pruning/reduction applied is tremendously weaker,
there are almost certainly many improvements/refinements for MCTS waiting to be found.
At least, it seems like a reasonable consideration.

However, previous attempts with MCTS for chess weren't very successful afaik.
So some sort of scepticism out there is very understandable.
Jörg Oster

mjlef
Posts: 1428
Joined: Thu Mar 30, 2006 12:08 pm
Contact:

Re: Komodo 12 and MCTS

Post by mjlef » Tue May 15, 2018 3:51 pm

SzG wrote:
Tue May 15, 2018 12:46 pm
mjlef wrote:
Tue May 15, 2018 11:39 am
SzG wrote:
Tue May 15, 2018 8:16 am


5 Elos better than 11.3.1. It will surely show after you have played 1,000,000 games with them.
Komodo 12 is about 5 elo stronger than Komodo 11.3.1 in normal (non MCTS) mode. It takes a few thousand games to prove this, not one million.

Mark
Well, one million may have been an exaggeration but if you study the 40/4 CCRL list you can find that it requires about 12,000 games to bring down the error margins to about 6 (in the case of SlowChess), which by square adding them is still almost 9 for two engines. To prove a 5 Elo difference the error margin of each engine must stay below about 3, so I am sure that the required number of games will be close to 100,000 for each engine.
People can certainly disagree about what constitutes "proof". In most statistics I have seen people accept a 2 SD error margin, although in some fields, they us a much higher number of SD. I do not disagree with your numbers for the higher SD, but I was speaking about a 2 SD result (so 95%). In any case, we have a lot more games than needed to say with reasonable confidence K 12 is about 5 elo above 11.3.1, depending on the time control in standard search mode. The nice thing about "about" is it lets you get away with some inaccuracy!

Jesse Gersenson
Posts: 575
Joined: Sat Aug 20, 2011 7:43 am
Contact:

Re: Komodo 12 and MCTS

Post by Jesse Gersenson » Tue May 15, 2018 7:58 pm

is 2 SD 95% or 95.45%?

User avatar
CMCanavessi
Posts: 844
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: Komodo 12 and MCTS

Post by CMCanavessi » Tue May 15, 2018 8:21 pm

@mark and larry, now that K12 is out will you make 10 free?
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Post Reply