mhull wrote: ↑Sun May 13, 2018 5:17 amIt's okay to test it playing its chosen opening with its opponent loaded with whatever opening book (and tablebases) its author wants. What I disagree with is when you blurred the line between what it thinks is wrong and what is wrong.jp wrote: ↑Sun May 13, 2018 12:16 amI disagree. It is the same thing because it is the goal, training to play better chess than humans or A/B searchers have ever seen, and that means allowing it to play all moves. Then that is what needs to be looked-for in testing, how well does it play chess, not how good is it as a cripple-bot utility, landing it in positions it might otherwise avoid.mhull wrote: ↑Sat May 12, 2018 11:57 pm With chess, what is it that we don't know? How to play like God, that's what. But if I want to study opening XYZ, of what interest is that when the program thinks that's a wrong opening in the first place? You want to use an engine to analyze a wrong opening, that's fine.
This is extremely wrong. The engine is not playing like God. Not even close. The program think's it's wrong is not the same as it is wrong.
It would be like forcing Capablanca to play the Dutch defense all the time and then deciding his Elo based on that. Completely not logical nor an accurate appraisal of strength. Capablanca varied his openings very little. But if he were a machine, you lot would abuse him most abominably, and you cannot deny it.
I wouldn't abuse Capablanca. I would protest if anyone said any opening was "wrong" just because he didn't play it. No one would talk about his play and "playing like God" in the same breath. Many would say Capablanca was limited in his openings, which is not "abusing him most abominably".
LCZero: Progress and Scaling. Relation to CCRL Elo
Moderators: hgm, Rebel, chrisw
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: LCZero: Progress and Scaling. Relation to CCRL Elo
-
- Posts: 13447
- Joined: Wed Mar 08, 2006 9:02 pm
- Location: Dallas, Texas
- Full name: Matthew Hull
Re: LCZero: Progress and Scaling. Relation to CCRL Elo
Current testing method isn't measuring the "wrong/right" opening effect on Elo. That's the only point here.
Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.jp wrote: ↑Sun May 13, 2018 8:35 am I wouldn't abuse Capablanca. I would protest if anyone said any opening was "wrong" just because he didn't play it. No one would talk about his play and "playing like God" in the same breath. Many would say Capablanca was limited in his openings, which is not "abusing him most abominably".
Matthew Hull
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: LCZero: Progress and Scaling. Relation to CCRL Elo
If people knew it was a perfect player they would not complain. If the player always won they would not complain.mhull wrote: ↑Mon May 14, 2018 5:38 am Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.
That's the problem. We know Leela is nowhere near perfect and does not even avoid losing a lot of games. So it's completely different. Talk about perfect play is irrelevant.
-
- Posts: 13447
- Joined: Wed Mar 08, 2006 9:02 pm
- Location: Dallas, Texas
- Full name: Matthew Hull
Re: LCZero: Progress and Scaling. Relation to CCRL Elo
The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.jp wrote: ↑Mon May 14, 2018 8:07 amIf people knew it was a perfect player they would not complain. If the player always won they would not complain.mhull wrote: ↑Mon May 14, 2018 5:38 am Sure, many would say his openings were limited but his results suggest that nobody was able to take advantage of it or even that it was a disadvantage in the first place. If God only played one opening as White and always won, isn't it true that people would complain his opening repertoire was limited? But it wouldn't be relevant, would it.
That's the problem. We know Leela is nowhere near perfect and does not even avoid losing a lot of games. So it's completely different. Talk about perfect play is irrelevant.
If we calculated human Elo using games composed partly of computer moves, we would call that cheating.
Matthew Hull
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: LCZero: Progress and Scaling. Relation to CCRL Elo
No current computer chess program tells us anything about the asymptotic approach to best play, because they are all so far below best play.mhull wrote: ↑Mon May 14, 2018 3:39 pm The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.
If we calculated human Elo using games composed partly of computer moves, we would call that cheating.
Most people would not agree that testing it with a suite of given positions is "ALWAYS interjecting their moves into the test".
Not all game testing is forcing it to play certain opening moves. CCRL doesn't force opening moves. The testing that does is like only 2 moves. If 2 reasonable moves is enough to derail it, then that's a sign it's not a very good player.
If humans played a match and reveal afterwards they'd both followed opening lines suggested to them by computer analysis, we wouldn't call that cheating. We just say it's opening preparation.
-
- Posts: 13447
- Joined: Wed Mar 08, 2006 9:02 pm
- Location: Dallas, Texas
- Full name: Matthew Hull
Re: LCZero: Progress and Scaling. Relation to CCRL Elo
With all due respect, you wouldn't know. Human assessment of how close programs (which are hundreds of Elo better players than them) are approaching best play is likely of no value.jp wrote: ↑Tue May 15, 2018 11:39 amNo current computer chess program tells us anything about the asymptotic approach to best play, because they are all so far below best play.mhull wrote: ↑Mon May 14, 2018 3:39 pm The idea of computer chess is the asymptotic approach to best play. You can't measure that approach if much weaker humans are ALWAYS interjecting their moves into the test. Any resulting Elo measures are contaminated with human moves.
If we calculated human Elo using games composed partly of computer moves, we would call that cheating.
Of course they don't agree. It is human nature to protect the huge investment in the current method instead of admitting they've traveled far down the wrong road.
The difference is that humans ultimately decide if they like the preparation. Programs are forced to play human preparation whether they like it or not. This is a wrong design for accurate Elo calculation. We don't do it with humans, we should not do it with programs.jp wrote: ↑Tue May 15, 2018 11:39 am Not all game testing is forcing it to play certain opening moves. CCRL doesn't force opening moves. The testing that does is like only 2 moves. If 2 reasonable moves is enough to derail it, then that's a sign it's not a very good player.
If humans played a match and reveal afterwards they'd both followed opening lines suggested to them by computer analysis, we wouldn't call that cheating. We just say it's opening preparation.
Matthew Hull
-
- Posts: 1346
- Joined: Sat Apr 19, 2014 1:47 pm
Re: LCZero: Progress and Scaling. Relation to CCRL Elo
The good news is Leela is back on track. Only 25 elo below his max ! And progress are coming back very fast.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
-
- Posts: 1346
- Joined: Sat Apr 19, 2014 1:47 pm
Re: LCZero: Progress and Scaling. Relation to CCRL Elo
Indeed, Lczero is reaching 3000 elo anytime soon now against various opponent :
https://docs.google.com/spreadsheets/d/ ... edit#gid=0