Lc0 51010

[jp]

Whether chess is played out depends on how you define the question. It seems pretty clear that if you take the strongest Stockfish and the strongest NN engine on TCEC type hardware, and have them play a long match at TCEC time controls, the results will depend heavily on the openings. If you give each side a really good, deep opening book and have them play only the optimum or near optimum openings, nearly every game will end in a draw.
-------
Anyway none of this seems relevant to me to the question of why computers beat humans. No matter how correct our beliefs about chess might be, we cannot compete with something that searches a million times faster than we can.

Yep.

[lkaufman]

The situation has now become even more dramatic. I tested version 51058 (on my RTX 2080), which is self-rated slightly below 1700, against Fire 7.1 on 4 very fast threads, which is rated 2337 on CEGT blitz, 3425 CCRL blitz. The two day old Lc0 won by 12 to 8 (+70 elo), putting it over 3400 CEGT blitz and nearly 3500 CCRL blitz!! So I guess we won't be hearing that the self-play ratings are inflated anymore, although it remains to be seen whether further gains in self-play ratings continue to translate to real gains. One question: does anyone know roughly how many 2080s would be needed to duplicate the training that this 51xxx series has averaged so far? It doesn't mean much to say that it has trained for two days without stating what the average resources used for the training were. I imagine that they were just a tiny percentage of the resources used to train AlphaZero in 9 hours or so.

[jp]

One question: does anyone know roughly how many 2080s would be needed to duplicate the training that this 51xxx series has averaged so far? It doesn't mean much to say that it has trained for two days without stating what the average resources used for the training were. I imagine that they were just a tiny percentage of the resources used to train AlphaZero in 9 hours or so.

NN 51010 has had 559441 games. The NN size is smaller (10x128) than the A0 one and 41xxx NNs (20x256), but the training param is visits=10000 (visits=800 for 41xxx).

I'll let others say what 2080 time that works out to.

[Guenther]

The situation has now become even more dramatic. I tested version 51058 (on my RTX 2080), which is self-rated slightly below 1700, against Fire 7.1 on 4 very fast threads, which is rated 2337 on CEGT blitz, 3425 CCRL blitz. The two day old Lc0 won by 12 to 8 (+70 elo), putting it over 3400 CEGT blitz and nearly 3500 CCRL blitz!! So I guess we won't be hearing that the self-play ratings are inflated anymore, although it remains to be seen whether further gains in self-play ratings continue to translate to real gains. One question: does anyone know roughly how many 2080s would be needed to duplicate the training that this 51xxx series has averaged so far? It doesn't mean much to say that it has trained for two days without stating what the average resources used for the training were. I imagine that they were just a tiny percentage of the resources used to train AlphaZero in 9 hours or so.

May be tradeoff for speed is better than expected for 10b nets with less information?
There are no tests about best ratio of max size of NNs vs. speed.

[Laskos]

One question: does anyone know roughly how many 2080s would be needed to duplicate the training that this 51xxx series has averaged so far? It doesn't mean much to say that it has trained for two days without stating what the average resources used for the training were. I imagine that they were just a tiny percentage of the resources used to train AlphaZero in 9 hours or so.

NN 51010 has had 559441 games. The NN size is smaller (10x128) than the A0 one and 41xxx NNs (20x256), but the training param is visits=10000 (visits=800 for 41xxx).

I'll let others say what 2080 time that works out to.

I will try to estimate (writing on the phone, I am on a vacation ). About 0.2s/move on 2080, meaning games in about 20s. About 200 games per hour, so half a million games need 2500 hours, or about 100 days needed for training the NN 51010 on a single 2080. Or 100x 2080 GPUs needed to train it in one day.
The main unknown is the time to get 10000 visits, I took it as 0.2s on 2080. So this calculation is just for the order of magnitude.

[Uri Blass]

Over 3000 Elo at only half a million games? Wow, that's insane.

It is insane. Imagine poor Larry, all he learned in dozens of years is well surclassed by this dirt quick net using dirt basic monte carlo search.

This is not a sad thing. This is what many people in chess have strive to achieve for many reasons. As I chess player. I know we are flawed in our thinking when it comes to chess. And are assumptions are our worst enemy in chess play. This only makes us stronger for every class of chess player.

I don't think that flawed thinking and wrong assumptions are the reason I (or any strong player) lose to 3000+ rated engines. The reason is that I struggle to search even one node per second! Of course my thinking and assumptions are far from perfect, I just don't think that they are worse than what engines "think" or "assume". Although I'm not sure I could beat Lc0 in a serious match even with it set to use policy network only, i.e. one node per move, so maybe my last sentence only applies to A/B engines.

I think we make many wrong assumptions in chess. Lets start with the biggest one.

Chess is played out....

I do not understand.
What is the assumption that you mean in this post?

Have you ever asked yourself why Lc0 and A0 have stirred so much excitement with Titled players. It is because Lc0 and A0 have show us what we did not expect.

Or Lc0 and A0 would be like any other chess programs. Humans play chess with certain assumptions, and a thinking of the way chess should be played. Chess programs of the past, and now like Lc0 and A0 have show us things we never considered before.

Uri, you have been on CCC as long as I can remember. How many times over the years have you seen people say X chess program is rated X. And this is almost perfect play. I can remember this many times, and today those chess programs stand no chance against the best chess programs of today. And today's engine will stand no change against tomorrows.

Its like when you argued that chess could be solved, and not grasping the meaning of what 10 ^120 means in terms of solving chess.

Remember it was not that many years ago, players like Capablanca, Fischer, Seirawan... and other thought chess was played out. And needed new variants to keep chess alive. Like Capablanca Chess, Fischer random chess, Seirawan chess...

Chess programs have shown that many of our human assumptions about chess are wrong. And are assumptions are many, but all are subsets of Chess is played out...

I agree that the assumption that we are close to perfect chess was clearly wrong in the past.
I simply did not understand that it is what you meant by chess is played out and this was the reason for my question.

The question if we are close to perfect chess today is a different question.
It is clear that at blitz today we are not close to perfect chess.
The only question is about long time control like TCEC time control.
The claim is that TCEC used some opening book to reduce the number of draws and with good opening book we are going to get almost 100% draws.

larry's words about it:
"It seems pretty clear that if you take the strongest Stockfish and the strongest NN engine on TCEC type hardware, and have them play a long match at TCEC time controls, the results will depend heavily on the openings. If you give each side a really good, deep opening book and have them play only the optimum or near optimum openings, nearly every game will end in a draw."

I am not sure about it and I have 2 problems:
1)How do you define if a book is a good deep opening book?
If you do not get the expected draw result in a game you can claim that the book was not good enough or was not deep enough so you need some definition that is not dependent on the result.

2)How do you define if an opening is the optimum or near the optimum opening?
It is possible that the best move is a move that the engines do not like because they do not see deep enough.

claim 2 of larry:
"The only way the stronger engine can win games is to play suboptimal lines with White to exit from book early and hope to win despite the poor opening moves."

Based on what he claims that a line is suboptimal line?
A line that the best engines do not suggest does not have to be a suboptimal line.

There should be a basis to claim that a line is suboptimal(for example if the engine that beat another engine lose against itself in the same line)
I guess larry means to lines that white has equality instead of minimal advantage and in this case I doubt if it is correct to call it a suboptimal line because the line does not lose and can help to win against weaker engines.

[jp]

larry's words about it:
"It seems pretty clear that if you take the strongest Stockfish and the strongest NN engine on TCEC type hardware, and have them play a long match at TCEC time controls, the results will depend heavily on the openings. If you give each side a really good, deep opening book and have them play only the optimum or near optimum openings, nearly every game will end in a draw."

I am not sure about it and I have 2 problems:
1)How do you define if a book is a good deep opening book?
If you do not get the expected draw result in a game you can claim that the book was not good enough or was not deep enough so you need some definition that is not dependent on the result.

2)How do you define if an opening is the optimum or near the optimum opening?
It is possible that the best move is a move that the engines do not like because they do not see deep enough.

Your two problems are more problems with how to test the claim than with the claim itself.
If the claim was the opposite ("...every game will end in a win") your two problems would still be there.

The result of the games A0 vs crippled SF8 + book gives some support to the claim, though we cannot be certain without, as you say, defining our "good deep opening book" and testing.

[Ozymandias]

The claim is that TCEC used some opening book to reduce the number of draws and with good opening book we are going to get almost 100% draws.

larry's words about it:
"It seems pretty clear that if you take the strongest Stockfish and the strongest NN engine on TCEC type hardware, and have them play a long match at TCEC time controls, the results will depend heavily on the openings. If you give each side a really good, deep opening book and have them play only the optimum or near optimum openings, nearly every game will end in a draw."

I am not sure about it and I have 2 problems:
1)How do you define if a book is a good deep opening book?
If you do not get the expected draw result in a game you can claim that the book was not good enough or was not deep enough so you need some definition that is not dependent on the result.

2)How do you define if an opening is the optimum or near the optimum opening?
It is possible that the best move is a move that the engines do not like because they do not see deep enough.

claim 2 of larry:
"The only way the stronger engine can win games is to play suboptimal lines with White to exit from book early and hope to win despite the poor opening moves."

Based on what he claims that a line is suboptimal line?
A line that the best engines do not suggest does not have to be a suboptimal line.

There should be a basis to claim that a line is suboptimal(for example if the engine that beat another engine lose against itself in the same line)
I guess larry means to lines that white has equality instead of minimal advantage and in this case I doubt if it is correct to call it a suboptimal line because the line does not lose and can help to win against weaker engines.

I'm surprised to see this topic still being debated after so long, maybe you missed the draw death of Freestyle chess. The answer to your questions is there.

[Eduard]

The situation has now become even more dramatic. I tested version 51058 (on my RTX 2080), which is self-rated slightly below 1700, against Fire 7.1 on 4 very fast threads, which is rated 2337 on CEGT blitz, 3425 CCRL blitz. The two day old Lc0 won by 12 to 8 (+70 elo), putting it over 3400 CEGT blitz and nearly 3500 CCRL blitz!! [----]

Forgive me for these lines, but I'm a computer chess fan and today is Sunday! I have a lot of fun today. Unbelievable how this new NN plays.

[Albert Silver]

NN 51010 has had 559441 games. The NN size is smaller (10x128) than the A0 one and 41xxx NNs (20x256), but the training param is visits=10000 (visits=800 for 41xxx).

That is inaccurate. It is averaging around 800 visits per move too.