right answer, but reasons given point more to personal insecurities than reality. mathematically the elo disparity between SF on 4 cores & leela is narrow enough that leela can & thus will win games...that said, milo is correct that in both wins SF was at a significant disadvantage by the time it was allowed to make its own moves.Milos wrote: ↑Thu Jun 07, 2018 10:36 pmNot at all, when you force SF to start with opening of -1.00 or worse (we don't even know since there is 0 information about actual game beside the moves we don't know what is SF depth, how much time it used, etc, it is so easy to cripple it with some settings) and allow LC0 to freely play the moves, plus huge hardware advantage (more than 10x difference in hardware price) ofc that SF would loose once in a blue moon.
Once LC0 or A0 or any NN program is actually able to beat it in TCEC conditions (while using even dual Titan-V) than we can talk about something significant.
First Win by Leela Chess Zero against Stockfish dev
Moderators: hgm, Rebel, chrisw
-
- Posts: 1766
- Joined: Wed Jun 03, 2009 12:14 am
Re: First Win by Leela Chess Zero against Stockfish dev
-
- Posts: 1142
- Joined: Thu Dec 28, 2017 4:06 pm
- Location: Argentina
Re: First Win by Leela Chess Zero against Stockfish dev
Interesting how the "haters" opinions have varied over time:
When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls
-
- Posts: 1339
- Joined: Fri Nov 02, 2012 9:43 am
- Location: New Delhi, India
Re: First Win by Leela Chess Zero against Stockfish dev
Typical of MilosCMCanavessi wrote: ↑Fri Jun 08, 2018 1:54 am Interesting how the "haters" opinions have varied over time:
When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis
-
- Posts: 12038
- Joined: Mon Jul 07, 2008 10:50 pm
Re: First Win by Leela Chess Zero against Stockfish dev
can a program which is 150 elo below stockfish defeat it in any setting ?Milos wrote: ↑Thu Jun 07, 2018 10:36 pmNot at all, when you force SF to start with opening of -1.00 or worse (we don't even know since there is 0 information about actual game beside the moves we don't know what is SF depth, how much time it used, etc, it is so easy to cripple it with some settings) and allow LC0 to freely play the moves, plus huge hardware advantage (more than 10x difference in hardware price) ofc that SF would loose once in a blue moon.
Once LC0 or A0 or any NN program is actually able to beat it in TCEC conditions (while using even dual Titan-V) than we can talk about something significant.
-
- Posts: 1766
- Joined: Wed Jun 03, 2009 12:14 am
Re: First Win by Leela Chess Zero against Stockfish dev
maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1
so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.
so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: First Win by Leela Chess Zero against Stockfish dev
Calculators are pretty useless at this kind of difference. 150Elo is 21%, meaning score would be 71%:29%. Player B can achieve its 29% by all draws (highly improbable) or by half or more wins (also improbable). How many wins will depend on draw rate. Draw rate depends mainly on TC. If we fix draw rate at 50% player B would get 4% of its score through wins. With 56% draw rate there would be only 1% of wins by player B. However, since white has some advantage this further increases percentage of wins of player B.yanquis1972 wrote: ↑Fri Jun 08, 2018 9:51 pm maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1
so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.
One thing is highly probable though, that player B should have at least 1 win in 100 games match.
Similarly it is quite easy to mathematically prove that chance of AlphaZero score against SF8 (no wins for SF8) is statistically highly improbable and that match games were either not iid random variables (i.e. there were many duplicates) or they were simply cherry-picked.
-
- Posts: 47
- Joined: Mon Oct 31, 2016 9:45 pm
Re: First Win by Leela Chess Zero against Stockfish dev
Or that the draw rate for the match was sufficiently high that it compressed the strength difference between AlphaZero and Stockfish. Such a hypothesis is a bit too important to neglect mentioning entirely, wouldn't you say?Milos wrote: ↑Fri Jun 08, 2018 10:57 pmCalculators are pretty useless at this kind of difference. 150Elo is 21%, meaning score would be 71%:29%. Player B can achieve its 29% by all draws (highly improbable) or by half or more wins (also improbable). How many wins will depend on draw rate. Draw rate depends mainly on TC. If we fix draw rate at 50% player B would get 4% of its score through wins. With 56% draw rate there would be only 1% of wins by player B. However, since white has some advantage this further increases percentage of wins of player B.yanquis1972 wrote: ↑Fri Jun 08, 2018 9:51 pm maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1
so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.
One thing is highly probable though, that player B should have at least 1 win in 100 games match.
Similarly it is quite easy to mathematically prove that chance of AlphaZero score against SF8 (no wins for SF8) is statistically highly improbable and that match games were either not iid random variables (i.e. there were many duplicates) or they were simply cherry-picked.
-
- Posts: 1796
- Joined: Thu Sep 18, 2008 10:24 pm
Re: First Win by Leela Chess Zero against Stockfish dev
It's not clear to me where we've got to. There's a 3000 elo + lc0 (provided you've got a decent card) but it's still a long way behind SF even on a single core. Then the project seemed to plateau. Now there's a new fresh start but it's early days.CMCanavessi wrote: ↑Fri Jun 08, 2018 1:54 am Interesting how the "haters" opinions have varied over time:
When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
On equally priced hardware (CPU vs GPU) I'd say Leela has a way to go yet with no certain future. But maybe you disagree, and I hope you're right if you do...
-
- Posts: 1142
- Joined: Thu Dec 28, 2017 4:06 pm
- Location: Argentina
Re: First Win by Leela Chess Zero against Stockfish dev
I don't disagree at all, what you say is absolutely true. We have to see what lc0 can do with a bug-free network started from scratch, and remember that we never tested a network size of 256 yet (which is what A0 used). There's still a lot to gain, but there are so many things and tests going at the moment that the perception by the community might well be that nothing is happening and that progress has stalled, which is completely understandable.Werewolf wrote: ↑Sat Jun 09, 2018 4:45 pmIt's not clear to me where we've got to. There's a 3000 elo + lc0 (provided you've got a decent card) but it's still a long way behind SF even on a single core. Then the project seemed to plateau. Now there's a new fresh start but it's early days.CMCanavessi wrote: ↑Fri Jun 08, 2018 1:54 am Interesting how the "haters" opinions have varied over time:
When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...
On equally priced hardware (CPU vs GPU) I'd say Leela has a way to go yet with no certain future. But maybe you disagree, and I hope you're right if you do...
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls
-
- Posts: 1534
- Joined: Sun Oct 25, 2009 2:30 am
Re: First Win by Leela Chess Zero against Stockfish dev
My perception is that maybe too much is happening at once, and that probably because of that, progress has nearly stalled.CMCanavessi wrote: ↑Sat Jun 09, 2018 6:08 pmthe perception by the community might well be that nothing is happening and that progress has stalled, which is completely understandable.