First Win by Leela Chess Zero against Stockfish dev

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
yanquis1972
Posts: 1762
Joined: Tue Jun 02, 2009 10:14 pm

Re: First Win by Leela Chess Zero against Stockfish dev

Post by yanquis1972 » Thu Jun 07, 2018 11:34 pm

Milos wrote:
Thu Jun 07, 2018 8:36 pm
duncan wrote:
Thu Jun 07, 2018 8:20 pm
Milos wrote:
Thu Jun 07, 2018 8:18 pm

Yes it does. But CUDA version is roughly 10x faster.
are you surprised with the win over stockfish ?
Not at all, when you force SF to start with opening of -1.00 or worse (we don't even know since there is 0 information about actual game beside the moves we don't know what is SF depth, how much time it used, etc, it is so easy to cripple it with some settings) and allow LC0 to freely play the moves, plus huge hardware advantage (more than 10x difference in hardware price) ofc that SF would loose once in a blue moon.
Once LC0 or A0 or any NN program is actually able to beat it in TCEC conditions (while using even dual Titan-V) than we can talk about something significant.
right answer, but reasons given point more to personal insecurities than reality. mathematically the elo disparity between SF on 4 cores & leela is narrow enough that leela can & thus will win games...that said, milo is correct that in both wins SF was at a significant disadvantage by the time it was allowed to make its own moves.

User avatar
CMCanavessi
Posts: 835
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: First Win by Leela Chess Zero against Stockfish dev

Post by CMCanavessi » Thu Jun 07, 2018 11:54 pm

Interesting how the "haters" opinions have varied over time:

When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

shrapnel
Posts: 1160
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: First Win by Leela Chess Zero against Stockfish dev

Post by shrapnel » Fri Jun 08, 2018 2:56 am

CMCanavessi wrote:
Thu Jun 07, 2018 11:54 pm
Interesting how the "haters" opinions have varied over time:

When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
Typical of Milos :)
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Asus ROG Strix OC 11 GB Geforce 2080 Ti

duncan
Posts: 10202
Joined: Mon Jul 07, 2008 8:50 pm

Re: First Win by Leela Chess Zero against Stockfish dev

Post by duncan » Fri Jun 08, 2018 4:10 pm

Milos wrote:
Thu Jun 07, 2018 8:36 pm
duncan wrote:
Thu Jun 07, 2018 8:20 pm
Milos wrote:
Thu Jun 07, 2018 8:18 pm

Yes it does. But CUDA version is roughly 10x faster.
are you surprised with the win over stockfish ?
Not at all, when you force SF to start with opening of -1.00 or worse (we don't even know since there is 0 information about actual game beside the moves we don't know what is SF depth, how much time it used, etc, it is so easy to cripple it with some settings) and allow LC0 to freely play the moves, plus huge hardware advantage (more than 10x difference in hardware price) ofc that SF would loose once in a blue moon.
Once LC0 or A0 or any NN program is actually able to beat it in TCEC conditions (while using even dual Titan-V) than we can talk about something significant.
can a program which is 150 elo below stockfish defeat it in any setting ?

yanquis1972
Posts: 1762
Joined: Tue Jun 02, 2009 10:14 pm

Re: First Win by Leela Chess Zero against Stockfish dev

Post by yanquis1972 » Fri Jun 08, 2018 7:51 pm

maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1

so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: First Win by Leela Chess Zero against Stockfish dev

Post by Milos » Fri Jun 08, 2018 8:57 pm

yanquis1972 wrote:
Fri Jun 08, 2018 7:51 pm
maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1

so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.
Calculators are pretty useless at this kind of difference. 150Elo is 21%, meaning score would be 71%:29%. Player B can achieve its 29% by all draws (highly improbable) or by half or more wins (also improbable). How many wins will depend on draw rate. Draw rate depends mainly on TC. If we fix draw rate at 50% player B would get 4% of its score through wins. With 56% draw rate there would be only 1% of wins by player B. However, since white has some advantage this further increases percentage of wins of player B.
One thing is highly probable though, that player B should have at least 1 win in 100 games match.
Similarly it is quite easy to mathematically prove that chance of AlphaZero score against SF8 (no wins for SF8) is statistically highly improbable and that match games were either not iid random variables (i.e. there were many duplicates) or they were simply cherry-picked.

David Xu
Posts: 47
Joined: Mon Oct 31, 2016 8:45 pm

Re: First Win by Leela Chess Zero against Stockfish dev

Post by David Xu » Fri Jun 08, 2018 11:50 pm

Milos wrote:
Fri Jun 08, 2018 8:57 pm
yanquis1972 wrote:
Fri Jun 08, 2018 7:51 pm
maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1

so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.
Calculators are pretty useless at this kind of difference. 150Elo is 21%, meaning score would be 71%:29%. Player B can achieve its 29% by all draws (highly improbable) or by half or more wins (also improbable). How many wins will depend on draw rate. Draw rate depends mainly on TC. If we fix draw rate at 50% player B would get 4% of its score through wins. With 56% draw rate there would be only 1% of wins by player B. However, since white has some advantage this further increases percentage of wins of player B.
One thing is highly probable though, that player B should have at least 1 win in 100 games match.
Similarly it is quite easy to mathematically prove that chance of AlphaZero score against SF8 (no wins for SF8) is statistically highly improbable and that match games were either not iid random variables (i.e. there were many duplicates) or they were simply cherry-picked.
Or that the draw rate for the match was sufficiently high that it compressed the strength difference between AlphaZero and Stockfish. Such a hypothesis is a bit too important to neglect mentioning entirely, wouldn't you say?

Werewolf
Posts: 1191
Joined: Thu Sep 18, 2008 8:24 pm

Re: First Win by Leela Chess Zero against Stockfish dev

Post by Werewolf » Sat Jun 09, 2018 2:45 pm

CMCanavessi wrote:
Thu Jun 07, 2018 11:54 pm
Interesting how the "haters" opinions have varied over time:

When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
It's not clear to me where we've got to. There's a 3000 elo + lc0 (provided you've got a decent card) but it's still a long way behind SF even on a single core. Then the project seemed to plateau. Now there's a new fresh start but it's early days.

On equally priced hardware (CPU vs GPU) I'd say Leela has a way to go yet with no certain future. But maybe you disagree, and I hope you're right if you do...

User avatar
CMCanavessi
Posts: 835
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: First Win by Leela Chess Zero against Stockfish dev

Post by CMCanavessi » Sat Jun 09, 2018 4:08 pm

Werewolf wrote:
Sat Jun 09, 2018 2:45 pm
CMCanavessi wrote:
Thu Jun 07, 2018 11:54 pm
Interesting how the "haters" opinions have varied over time:

When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
It's not clear to me where we've got to. There's a 3000 elo + lc0 (provided you've got a decent card) but it's still a long way behind SF even on a single core. Then the project seemed to plateau. Now there's a new fresh start but it's early days.

On equally priced hardware (CPU vs GPU) I'd say Leela has a way to go yet with no certain future. But maybe you disagree, and I hope you're right if you do...
I don't disagree at all, what you say is absolutely true. We have to see what lc0 can do with a bug-free network started from scratch, and remember that we never tested a network size of 256 yet (which is what A0 used). There's still a lot to gain, but there are so many things and tests going at the moment that the perception by the community might well be that nothing is happening and that progress has stalled, which is completely understandable.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

User avatar
Ozymandias
Posts: 1102
Joined: Sun Oct 25, 2009 12:30 am

Re: First Win by Leela Chess Zero against Stockfish dev

Post by Ozymandias » Sat Jun 09, 2018 7:15 pm

CMCanavessi wrote:
Sat Jun 09, 2018 4:08 pm
the perception by the community might well be that nothing is happening and that progress has stalled, which is completely understandable.
My perception is that maybe too much is happening at once, and that probably because of that, progress has nearly stalled.

Post Reply