First Win by Leela Chess Zero against Stockfish dev

[yanquis1972]

Yes it does. But CUDA version is roughly 10x faster.

are you surprised with the win over stockfish ?

Not at all, when you force SF to start with opening of -1.00 or worse (we don't even know since there is 0 information about actual game beside the moves we don't know what is SF depth, how much time it used, etc, it is so easy to cripple it with some settings) and allow LC0 to freely play the moves, plus huge hardware advantage (more than 10x difference in hardware price) ofc that SF would loose once in a blue moon.
Once LC0 or A0 or any NN program is actually able to beat it in TCEC conditions (while using even dual Titan-V) than we can talk about something significant.

right answer, but reasons given point more to personal insecurities than reality. mathematically the elo disparity between SF on 4 cores & leela is narrow enough that leela can & thus will win games...that said, milo is correct that in both wins SF was at a significant disadvantage by the time it was allowed to make its own moves.

[CMCanavessi]

Interesting how the "haters" opinions have varied over time:

When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...

[shrapnel]

Interesting how the "haters" opinions have varied over time:

When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...

Typical of Milos

[duncan]

Yes it does. But CUDA version is roughly 10x faster.

are you surprised with the win over stockfish ?

Not at all, when you force SF to start with opening of -1.00 or worse (we don't even know since there is 0 information about actual game beside the moves we don't know what is SF depth, how much time it used, etc, it is so easy to cripple it with some settings) and allow LC0 to freely play the moves, plus huge hardware advantage (more than 10x difference in hardware price) ofc that SF would loose once in a blue moon.
Once LC0 or A0 or any NN program is actually able to beat it in TCEC conditions (while using even dual Titan-V) than we can talk about something significant.

can a program which is 150 elo below stockfish defeat it in any setting ?

[yanquis1972]

maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1

so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.

[Milos]

maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1

so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.

Calculators are pretty useless at this kind of difference. 150Elo is 21%, meaning score would be 71%:29%. Player B can achieve its 29% by all draws (highly improbable) or by half or more wins (also improbable). How many wins will depend on draw rate. Draw rate depends mainly on TC. If we fix draw rate at 50% player B would get 4% of its score through wins. With 56% draw rate there would be only 1% of wins by player B. However, since white has some advantage this further increases percentage of wins of player B.
One thing is highly probable though, that player B should have at least 1 win in 100 games match.
Similarly it is quite easy to mathematically prove that chance of AlphaZero score against SF8 (no wins for SF8) is statistically highly improbable and that match games were either not iid random variables (i.e. there were many duplicates) or they were simply cherry-picked.

[David Xu]

maybe some here can elaborate on corrections applicable to 3000+ engine-engine matches (if any) but this site popped up when i googled 'elo calculator': https://wismuth.com/elo/calculator.html ... &best_of=1

so to just throw some numbers out there, according to this site if player A is rated 3500 & B 3350 (-150 elo), player B has a 2% chance of winning a single game. expected result of a 100 game match would be 42W-2L-56D.

Calculators are pretty useless at this kind of difference. 150Elo is 21%, meaning score would be 71%:29%. Player B can achieve its 29% by all draws (highly improbable) or by half or more wins (also improbable). How many wins will depend on draw rate. Draw rate depends mainly on TC. If we fix draw rate at 50% player B would get 4% of its score through wins. With 56% draw rate there would be only 1% of wins by player B. However, since white has some advantage this further increases percentage of wins of player B.
One thing is highly probable though, that player B should have at least 1 win in 100 games match.
Similarly it is quite easy to mathematically prove that chance of AlphaZero score against SF8 (no wins for SF8) is statistically highly improbable and that match games were either not iid random variables (i.e. there were many duplicates) or they were simply cherry-picked.

Or that the draw rate for the match was sufficiently high that it compressed the strength difference between AlphaZero and Stockfish. Such a hypothesis is a bit too important to neglect mentioning entirely, wouldn't you say?

[Werewolf]

Interesting how the "haters" opinions have varied over time:

When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...

It's not clear to me where we've got to. There's a 3000 elo + lc0 (provided you've got a decent card) but it's still a long way behind SF even on a single core. Then the project seemed to plateau. Now there's a new fresh start but it's early days.

On equally priced hardware (CPU vs GPU) I'd say Leela has a way to go yet with no certain future. But maybe you disagree, and I hope you're right if you do...

[CMCanavessi]

Interesting how the "haters" opinions have varied over time:

When Leela started: don't dream, it will never even get to 2000 elo
Couple of days later: don't dream, it will never get to 2500 elo, it will eventually stall
Couple of days later: don't dream, it will never get to 3000 elo and it needs MASSIVE hardware to be competitive. Right now it's a 2100 elo engine (sic)
Couple of days later: don't dream, it may win a couple of games on massive hardware, but it will never ever get to Stockfish level
Couple of days later: don't dream, it only beat Stockfish a couple of times on an expensive GPU, against a cheap CPU
Couple of days in the future: who knows...

It's not clear to me where we've got to. There's a 3000 elo + lc0 (provided you've got a decent card) but it's still a long way behind SF even on a single core. Then the project seemed to plateau. Now there's a new fresh start but it's early days.

On equally priced hardware (CPU vs GPU) I'd say Leela has a way to go yet with no certain future. But maybe you disagree, and I hope you're right if you do...

I don't disagree at all, what you say is absolutely true. We have to see what lc0 can do with a bug-free network started from scratch, and remember that we never tested a network size of 256 yet (which is what A0 used). There's still a lot to gain, but there are so many things and tests going at the moment that the perception by the community might well be that nothing is happening and that progress has stalled, which is completely understandable.

[Ozymandias]

the perception by the community might well be that nothing is happening and that progress has stalled, which is completely understandable.

My perception is that maybe too much is happening at once, and that probably because of that, progress has nearly stalled.