LCZero: Progress and Scaling. Relation to CCRL Elo

[jp]

With chess, what is it that we don't know? How to play like God, that's what. But if I want to study opening XYZ, of what interest is that when the program thinks that's a wrong opening in the first place? You want to use an engine to analyze a wrong opening, that's fine.
This is extremely wrong. The engine is not playing like God. Not even close. The program think's it's wrong is not the same as it is wrong.

I disagree. It is the same thing because it is the goal, training to play better chess than humans or A/B searchers have ever seen, and that means allowing it to play all moves. Then that is what needs to be looked-for in testing, how well does it play chess, not how good is it as a cripple-bot utility, landing it in positions it might otherwise avoid.

It would be like forcing Capablanca to play the Dutch defense all the time and then deciding his Elo based on that. Completely not logical nor an accurate appraisal of strength. Capablanca varied his openings very little. But if he were a machine, you lot would abuse him most abominably, and you cannot deny it.

It's okay to test it playing its chosen opening with its opponent loaded with whatever opening book (and tablebases) its author wants. What I disagree with is when you blurred the line between what it thinks is wrong and what is wrong.

I wouldn't abuse Capablanca. I would protest if anyone said any opening was "wrong" just because he didn't play it. No one would talk about his play and "playing like God" in the same breath. Many would say Capablanca was limited in his openings, which is not "abusing him most abominably".

[mhull]

It's okay to test it playing its chosen opening with its opponent loaded with whatever opening book (and tablebases) its author wants. What I disagree with is when you blurred the line between what it thinks is wrong and what is wrong.

Current testing method isn't measuring the "wrong/right" opening effect on Elo. That's the only point here.

I wouldn't abuse Capablanca. I would protest if anyone said any opening was "wrong" just because he didn't play it. No one would talk about his play and "playing like God" in the same breath. Many would say Capablanca was limited in his openings, which is not "abusing him most abominably".

Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.

[jp]

Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.

If people knew it was a perfect player they would not complain. If the player always won they would not complain.
That's the problem. We know Leela is nowhere near perfect and does not even avoid losing a lot of games. So it's completely different. Talk about perfect play is irrelevant.

[mhull]

Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.

If people knew it was a perfect player they would not complain. If the player always won they would not complain.
That's the problem. We know Leela is nowhere near perfect and does not even avoid losing a lot of games. So it's completely different. Talk about perfect play is irrelevant.

The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.

If we calculated human Elo using games composed partly of computer moves, we would call that cheating.

[Laskos]

Sudden Elo jump compared to AB engine with the latest tested, ID292. The red lines are one standard deviation lines.

[jp]

The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.
If we calculated human Elo using games composed partly of computer moves, we would call that cheating.

No current computer chess program tells us anything about the asymptotic approach to best play, because they are all so far below best play.

Most people would not agree that testing it with a suite of given positions is "ALWAYS interjecting their moves into the test".

Not all game testing is forcing it to play certain opening moves. CCRL doesn't force opening moves. The testing that does is like only 2 moves. If 2 reasonable moves is enough to derail it, then that's a sign it's not a very good player.

If humans played a match and reveal afterwards they'd both followed opening lines suggested to them by computer analysis, we wouldn't call that cheating. We just say it's opening preparation.

[mhull]

The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.
If we calculated human Elo using games composed partly of computer moves, we would call that cheating.

No current computer chess program tells us anything about the asymptotic approach to best play, because they are all so far below best play.

With all due respect, you wouldn't know. Human assessment of how close programs (which are hundreds of Elo better players than them) are approaching best play is likely of no value.

Most people would not agree that testing it with a suite of given positions is "ALWAYS interjecting their moves into the test".

Of course they don't agree. It is human nature to protect the huge investment in the current method instead of admitting they've traveled far down the wrong road.

Not all game testing is forcing it to play certain opening moves. CCRL doesn't force opening moves. The testing that does is like only 2 moves. If 2 reasonable moves is enough to derail it, then that's a sign it's not a very good player.

If humans played a match and reveal afterwards they'd both followed opening lines suggested to them by computer analysis, we wouldn't call that cheating. We just say it's opening preparation.

The difference is that humans ultimately decide if they like the preparation. Programs are forced to play human preparation whether they like it or not. This is a wrong design for accurate Elo calculation. We don't do it with humans, we should not do it with programs.

[JJJ]

The good news is Leela is back on track. Only 25 elo below his max ! And progress are coming back very fast.

[Laskos]

The good news is Leela is back on track. Only 25 elo below his max ! And progress are coming back very fast.

Don't look at their official self-play rating, it is only of some guidance. Look at matches against varied AB opposition. LC0 now is the strogest ever.

[JJJ]

The good news is Leela is back on track. Only 25 elo below his max ! And progress are coming back very fast.

Don't look at their official self-play rating, it is only of some guidance. Look at matches against varied AB opposition. LC0 now is the strogest ever.

Indeed, Lczero is reaching 3000 elo anytime soon now against various opponent :
https://docs.google.com/spreadsheets/d/ ... edit#gid=0