LCZero: Progress and Scaling. Relation to CCRL Elo

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
jp
Posts: 749
Joined: Mon Apr 23, 2018 5:54 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by jp » Sun May 13, 2018 6:35 am

mhull wrote:
Sun May 13, 2018 3:17 am
jp wrote:
Sat May 12, 2018 10:16 pm
mhull wrote:
Sat May 12, 2018 9:57 pm
With chess, what is it that we don't know? How to play like God, that's what. But if I want to study opening XYZ, of what interest is that when the program thinks that's a wrong opening in the first place? You want to use an engine to analyze a wrong opening, that's fine.
This is extremely wrong. The engine is not playing like God. Not even close. The program think's it's wrong is not the same as it is wrong.
I disagree. It is the same thing because it is the goal, training to play better chess than humans or A/B searchers have ever seen, and that means allowing it to play all moves. Then that is what needs to be looked-for in testing, how well does it play chess, not how good is it as a cripple-bot utility, landing it in positions it might otherwise avoid.

It would be like forcing Capablanca to play the Dutch defense all the time and then deciding his Elo based on that. Completely not logical nor an accurate appraisal of strength. Capablanca varied his openings very little. But if he were a machine, you lot would abuse him most abominably, and you cannot deny it. :wink:
It's okay to test it playing its chosen opening with its opponent loaded with whatever opening book (and tablebases) its author wants. What I disagree with is when you blurred the line between what it thinks is wrong and what is wrong.

I wouldn't abuse Capablanca. I would protest if anyone said any opening was "wrong" just because he didn't play it. No one would talk about his play and "playing like God" in the same breath. Many would say Capablanca was limited in his openings, which is not "abusing him most abominably".

User avatar
mhull
Posts: 12378
Joined: Wed Mar 08, 2006 8:02 pm
Location: Dallas, Texas
Full name: Matthew Hull

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by mhull » Mon May 14, 2018 3:38 am

jp wrote:
Sun May 13, 2018 6:35 am
It's okay to test it playing its chosen opening with its opponent loaded with whatever opening book (and tablebases) its author wants. What I disagree with is when you blurred the line between what it thinks is wrong and what is wrong.
Current testing method isn't measuring the "wrong/right" opening effect on Elo. That's the only point here.
jp wrote:
Sun May 13, 2018 6:35 am
I wouldn't abuse Capablanca. I would protest if anyone said any opening was "wrong" just because he didn't play it. No one would talk about his play and "playing like God" in the same breath. Many would say Capablanca was limited in his openings, which is not "abusing him most abominably".
Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.
Matthew Hull

jp
Posts: 749
Joined: Mon Apr 23, 2018 5:54 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by jp » Mon May 14, 2018 6:07 am

mhull wrote:
Mon May 14, 2018 3:38 am
Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.
If people knew it was a perfect player they would not complain. If the player always won they would not complain.
That's the problem. We know Leela is nowhere near perfect and does not even avoid losing a lot of games. So it's completely different. Talk about perfect play is irrelevant.

User avatar
mhull
Posts: 12378
Joined: Wed Mar 08, 2006 8:02 pm
Location: Dallas, Texas
Full name: Matthew Hull

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by mhull » Mon May 14, 2018 1:39 pm

jp wrote:
Mon May 14, 2018 6:07 am
mhull wrote:
Mon May 14, 2018 3:38 am
Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.
If people knew it was a perfect player they would not complain. If the player always won they would not complain.
That's the problem. We know Leela is nowhere near perfect and does not even avoid losing a lot of games. So it's completely different. Talk about perfect play is irrelevant.
The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.

If we calculated human Elo using games composed partly of computer moves, we would call that cheating.
Matthew Hull

User avatar
Laskos
Posts: 9408
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos » Mon May 14, 2018 10:43 pm

Sudden Elo jump compared to AB engine with the latest tested, ID292. The red lines are one standard deviation lines.
Image

jp
Posts: 749
Joined: Mon Apr 23, 2018 5:54 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by jp » Tue May 15, 2018 9:39 am

mhull wrote:
Mon May 14, 2018 1:39 pm
The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.
If we calculated human Elo using games composed partly of computer moves, we would call that cheating.
No current computer chess program tells us anything about the asymptotic approach to best play, because they are all so far below best play.

Most people would not agree that testing it with a suite of given positions is "ALWAYS interjecting their moves into the test".

Not all game testing is forcing it to play certain opening moves. CCRL doesn't force opening moves. The testing that does is like only 2 moves. If 2 reasonable moves is enough to derail it, then that's a sign it's not a very good player.

If humans played a match and reveal afterwards they'd both followed opening lines suggested to them by computer analysis, we wouldn't call that cheating. We just say it's opening preparation.

User avatar
mhull
Posts: 12378
Joined: Wed Mar 08, 2006 8:02 pm
Location: Dallas, Texas
Full name: Matthew Hull

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by mhull » Tue May 15, 2018 2:08 pm

jp wrote:
Tue May 15, 2018 9:39 am
mhull wrote:
Mon May 14, 2018 1:39 pm
The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.
If we calculated human Elo using games composed partly of computer moves, we would call that cheating.
No current computer chess program tells us anything about the asymptotic approach to best play, because they are all so far below best play.
With all due respect, you wouldn't know. Human assessment of how close programs (which are hundreds of Elo better players than them) are approaching best play is likely of no value.
jp wrote:
Tue May 15, 2018 9:39 am
Most people would not agree that testing it with a suite of given positions is "ALWAYS interjecting their moves into the test".
Of course they don't agree. It is human nature to protect the huge investment in the current method instead of admitting they've traveled far down the wrong road.
jp wrote:
Tue May 15, 2018 9:39 am
Not all game testing is forcing it to play certain opening moves. CCRL doesn't force opening moves. The testing that does is like only 2 moves. If 2 reasonable moves is enough to derail it, then that's a sign it's not a very good player.

If humans played a match and reveal afterwards they'd both followed opening lines suggested to them by computer analysis, we wouldn't call that cheating. We just say it's opening preparation.
The difference is that humans ultimately decide if they like the preparation. Programs are forced to play human preparation whether they like it or not. This is a wrong design for accurate Elo calculation. We don't do it with humans, we should not do it with programs.
Matthew Hull

JJJ
Posts: 1285
Joined: Sat Apr 19, 2014 11:47 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by JJJ » Tue May 15, 2018 2:57 pm

The good news is Leela is back on track. Only 25 elo below his max ! And progress are coming back very fast.

User avatar
Laskos
Posts: 9408
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos » Tue May 15, 2018 3:18 pm

JJJ wrote:
Tue May 15, 2018 2:57 pm
The good news is Leela is back on track. Only 25 elo below his max ! And progress are coming back very fast.
Don't look at their official self-play rating, it is only of some guidance. Look at matches against varied AB opposition. LC0 now is the strogest ever.

JJJ
Posts: 1285
Joined: Sat Apr 19, 2014 11:47 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by JJJ » Tue May 15, 2018 3:26 pm

Laskos wrote:
Tue May 15, 2018 3:18 pm
JJJ wrote:
Tue May 15, 2018 2:57 pm
The good news is Leela is back on track. Only 25 elo below his max ! And progress are coming back very fast.
Don't look at their official self-play rating, it is only of some guidance. Look at matches against varied AB opposition. LC0 now is the strogest ever.
Indeed, Lczero is reaching 3000 elo anytime soon now against various opponent :
https://docs.google.com/spreadsheets/d/ ... edit#gid=0

Post Reply