Tdunbug wrote: ↑Tue May 15, 2018 1:42 am
I don't speak much here but I do want to say that MCTS is a great idea and look forward to it being improved. At the very least, it is a fresh new idea that can be tested.
Agreed. I don't post much either but I'm finding Lczero a fascinating development. It's great to see fresh ideas to challenge the dominant A-B approach. Even if they come up short, it's still good to try new things. Maybe a hybrid approach as some suggest could be better. Who knows, but it will be interesting to watch how things pan out.
I also hope the discussions don't generate into name calling and abuse. No one has perfect knowledge of this or any other field and so a little humility and restraint can go a long way.
Daniel Shawul wrote: ↑Mon May 14, 2018 9:53 pm
There is so much bullshit here. The authors are throwing around ideas to benefit from the A0 hype. E.g.
a) We use averaging => style is much better like A0's
b) MCTS better against humans. I bet standard komodo is much better against computers, humans and cockroaches
c) MCTS better in multi-pv mode, I highly doubt it
d) "Although tactical strength is lower with MCTS, positional play and judgment may well be better in many positions". Pure bullshit as they
don't have NN evaluation which account for A0's positional strength. In general, MCTS maybe better strategically but not at the level they are suggesting without the NN
e) "Evals don't fluctuate so suddenly or wildly." So what? That is infact bad since you will have to spend many more simulations to realize
you are screwed with some shallow tactics.
Daniel
Awesome Post. Ruthless Dissection.
Game, Set and Match !
Daniel Shawul wrote: ↑Mon May 14, 2018 9:53 pm
b) MCTS better against humans. I bet standard komodo is much better against computers, humans and cockroaches
c) MCTS better in multi-pv mode, I highly doubt it
Daniel
Awesome Post. Ruthless Dissection.
Game, Set and Match !
Just to address two points (many of the others were just speculation), I agree that standard Komodo is stronger against humans, but I would bet that Komodo MCTS is stronger against humans than similarly rated alpha-beta engines such as (assuming my 3000 CCRL 40/40 estimate is accurate) Gull 2 or Arasan 20.5. It is surely a better strategy against fallible humans not to assume they will see things exactly as the engine does. Maybe Komodo MCTS has to get within 100 or 150 elo of normal Komodo before it will be tougher for humans to beat. On the second point, again we don't claim that Komodo MCTS is currently superior to normal Komodo in multi-pv mode, only that the elo gap shrinks drastically as the number of moves analyzed increases, and may vanish entirely with all moves analyzed. So here too, when and if we get within a couple hundred elo of normal Komodo, the MCTS version should be stronger with enough moves being analyzed.
I just want to add that I'm surprised that Daniel appears to be so negative about MCTS, when I will freely state that his relative success in making a Scorpio MCTS not too far in strength from normal Scorpio was a significant factor in our decision to devote a month to this project. Would it really be so surprising if we also got Komodo MCTS up to say 100 elo below the level of normal Komodo in a few months?
A little confused here will this run as normal Komodo and are you saying ran as normal it is about 5 elo stronger as none MCTS?
or is it better to stick with 11.3.1 for normal play?
A little confused here will this run as normal Komodo and are you saying ran as normal it is about 5 elo stronger as none MCTS?
or is it better to stick with 11.3.1 for normal play?
thanks in advance
5 Elos better than 11.3.1. It will surely show after you have played 1,000,000 games with them.
Komodo 12 is about 5 elo stronger than Komodo 11.3.1 in normal (non MCTS) mode. It takes a few thousand games to prove this, not one million.
Daniel Shawul wrote: ↑Mon May 14, 2018 9:53 pm
b) MCTS better against humans. I bet standard komodo is much better against computers, humans and cockroaches
c) MCTS better in multi-pv mode, I highly doubt it
Daniel
Awesome Post. Ruthless Dissection.
Game, Set and Match !
Just to address two points (many of the others were just speculation), I agree that standard Komodo is stronger against humans, but I would bet that Komodo MCTS is stronger against humans than similarly rated alpha-beta engines such as (assuming my 3000 CCRL 40/40 estimate is accurate) Gull 2 or Arasan 20.5. It is surely a better strategy against fallible humans not to assume they will see things exactly as the engine does. Maybe Komodo MCTS has to get within 100 or 150 elo of normal Komodo before it will be tougher for humans to beat. On the second point, again we don't claim that Komodo MCTS is currently superior to normal Komodo in multi-pv mode, only that the elo gap shrinks drastically as the number of moves analyzed increases, and may vanish entirely with all moves analyzed. So here too, when and if we get within a couple hundred elo of normal Komodo, the MCTS version should be stronger with enough moves being analyzed.
I just want to add that I'm surprised that Daniel appears to be so negative about MCTS, when I will freely state that his relative success in making a Scorpio MCTS not too far in strength from normal Scorpio was a significant factor in our decision to devote a month to this project. Would it really be so surprising if we also got Komodo MCTS up to say 100 elo below the level of normal Komodo in a few months?
Well, I wouldn't be surprised.
Just like plain Alpha-Beta search without any pruning/reduction applied is tremendously weaker,
there are almost certainly many improvements/refinements for MCTS waiting to be found.
At least, it seems like a reasonable consideration.
However, previous attempts with MCTS for chess weren't very successful afaik.
So some sort of scepticism out there is very understandable.
5 Elos better than 11.3.1. It will surely show after you have played 1,000,000 games with them.
Komodo 12 is about 5 elo stronger than Komodo 11.3.1 in normal (non MCTS) mode. It takes a few thousand games to prove this, not one million.
Mark
Well, one million may have been an exaggeration but if you study the 40/4 CCRL list you can find that it requires about 12,000 games to bring down the error margins to about 6 (in the case of SlowChess), which by square adding them is still almost 9 for two engines. To prove a 5 Elo difference the error margin of each engine must stay below about 3, so I am sure that the required number of games will be close to 100,000 for each engine.
People can certainly disagree about what constitutes "proof". In most statistics I have seen people accept a 2 SD error margin, although in some fields, they us a much higher number of SD. I do not disagree with your numbers for the higher SD, but I was speaking about a 2 SD result (so 95%). In any case, we have a lot more games than needed to say with reasonable confidence K 12 is about 5 elo above 11.3.1, depending on the time control in standard search mode. The nice thing about "about" is it lets you get away with some inaccuracy!
