LCZero: Progress and Scaling. Relation to CCRL Elo

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

Albert Silver
Posts: 3019
Joined: Wed Mar 08, 2006 9:57 pm
Location: Rio de Janeiro, Brazil

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Albert Silver »

yanquis1972 wrote: Wed May 16, 2018 11:24 am do they say "this graph is bogus btw" in their own paper, or would i have to read the dozens of articles to find that?

i guess the main problem is, while chess & chess engines fascinate me & are an on/off hobby, i have no knowledge in the sciences. did i miss a forum rule? is there one particular degree i should pursue or will i need a couple? is post-grad at least recommended? would appreciate any further advice on what i need to do before posting on the internet.
Why would you need a degree in sciences to look up an article and read it? It's like 5 pages long. You can skim over the mathematics and just read the results and then you would know what everyone else here knows instead of guessing or stumbling. If something fascinates you, doesn't it make sense to try to know things as opposed to guessing? I expect the stumbling block here is that for me being interested in something means reading and trying to learn about these things. I find it incomprehensible that anyone who declares themselves interested at first, much less fascinated, would make excuses to not try to inform themselves at all. Oh, and for the record, I studied French Lit, not math or computer science.

https://en.chessbase.com/post/the-futur ... arns-chess
https://en.chessbase.com/post/leela-che ... for-the-pc
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

Albert Silver wrote: Tue May 15, 2018 7:14 pm
Laskos wrote: Tue May 15, 2018 6:02 pm
Albert Silver wrote: Tue May 15, 2018 5:50 pm

You mean as opposed to the normal builds, or are you referring to the normal builds? The LC0-cudnn builds are indeed the strongest, though they only run in machines equipped with Nvidia GPUs.

The self-attributed ratings for the NN are unreliable IMHO. I ran a 300-game match with v10 between NN223 and NN253, and they were about equal (facing each other). NN223 actually pulled fractionally ahead (+8 Elo) but well within the error margins obviously.
I just posted in this thread the result for ID292, it is the strongest ever (the standard v0.10 CPU and GPU build on master).
If confirmed, that will be very promising as it will clearly indicates that not only was the rut the neural network was in caused by the bug but that it is also past it and finally making genuine progress.
Yes, it was outside error margins. Since ID292, again something fishy happens, ID300 in their self-games is about 100 Elo points stronger than ID292, but against AB engine in my test, I see no improvement.
yanquis1972
Posts: 1766
Joined: Wed Jun 03, 2009 12:14 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by yanquis1972 »

Albert Silver wrote: Wed May 16, 2018 4:29 pm
yanquis1972 wrote: Wed May 16, 2018 11:24 am do they say "this graph is bogus btw" in their own paper, or would i have to read the dozens of articles to find that?

i guess the main problem is, while chess & chess engines fascinate me & are an on/off hobby, i have no knowledge in the sciences. did i miss a forum rule? is there one particular degree i should pursue or will i need a couple? is post-grad at least recommended? would appreciate any further advice on what i need to do before posting on the internet.
Why would you need a degree in sciences to look up an article and read it? It's like 5 pages long. You can skim over the mathematics and just read the results and then you would know what everyone else here knows instead of guessing or stumbling. If something fascinates you, doesn't it make sense to try to know things as opposed to guessing? I expect the stumbling block here is that for me being interested in something means reading and trying to learn about these things. I find it incomprehensible that anyone who declares themselves interested at first, much less fascinated, would make excuses to not try to inform themselves at all. Oh, and for the record, I studied French Lit, not math or computer science.

https://en.chessbase.com/post/the-futur ... arns-chess
https://en.chessbase.com/post/leela-che ... for-the-pc
i can be interested & not care to be an expert, or enjoy being informed but not wish to be consumed. i have several other things that interest be more, & for my own mental health (not a shot at anyone here, i'm talking about me) i can't devote hours to abstractions right now. i don't see what it matters, & i assumed there would be no point in a layman reading the AZ paper. that's my mistake & i'll take your advice & check it out.
jp
Posts: 1470
Joined: Mon Apr 23, 2018 7:54 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by jp »

yanquis1972 wrote: Wed May 16, 2018 7:07 pm
Albert Silver wrote: Wed May 16, 2018 4:29 pm
yanquis1972 wrote: Wed May 16, 2018 11:24 am do they say "this graph is bogus btw" in their own paper, or would i have to read the dozens of articles to find that?
It's like 5 pages long. You can skim over the mathematics and just read the results and then you would know what everyone else here knows instead of guessing or stumbling.
i can be interested & not care to be an expert, or enjoy being informed but not wish to be consumed. i have several other things that interest be more, & for my own mental health (not a shot at anyone here, i'm talking about me) i can't devote hours to abstractions right now. i don't see what it matters, & i assumed there would be no point in a layman reading the AZ paper. that's my mistake & i'll take your advice & check it out.
Yeah, read it to get the idea but don't swallow everything it claims. Its defects are in those dozens of articles...
e.g.
https://medium.com/@josecamachocollados ... 66ae1c84f2
Jesse Gersenson
Posts: 593
Joined: Sat Aug 20, 2011 9:43 am

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Jesse Gersenson »

Laskos wrote: Wed May 16, 2018 6:19 pm ...against AB engine in my test, I see no improvement....
What are the conditions of the test?
Albert Silver
Posts: 3019
Joined: Wed Mar 08, 2006 9:57 pm
Location: Rio de Janeiro, Brazil

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Albert Silver »

jp wrote: Wed May 16, 2018 7:56 pm
yanquis1972 wrote: Wed May 16, 2018 7:07 pm
Albert Silver wrote: Wed May 16, 2018 4:29 pm
It's like 5 pages long. You can skim over the mathematics and just read the results and then you would know what everyone else here knows instead of guessing or stumbling.
i can be interested & not care to be an expert, or enjoy being informed but not wish to be consumed. i have several other things that interest be more, & for my own mental health (not a shot at anyone here, i'm talking about me) i can't devote hours to abstractions right now. i don't see what it matters, & i assumed there would be no point in a layman reading the AZ paper. that's my mistake & i'll take your advice & check it out.
Yeah, read it to get the idea but don't swallow everything it claims. Its defects are in those dozens of articles...
e.g.
https://medium.com/@josecamachocollados ... 66ae1c84f2
I didn't really see anything particular pertinent. Complaints about hardware differences don't make much sense since the design requires different hardware to work. His argument about their having access to special hardware to do it in so little time is correct, but then: so what? He also mistakenly says that the 7-piece tablebases were done with much less computing power, which is hard to measure, and most likely dead wrong. They were done over a period of months, and a lot of computer time on the Lomonosov Supercomputer. That is neither weak, nor accessible to the Average Joe, no matter how deep their pockets. The rest is just a lot of talk on the match conditions, which is to completely miss the point: the novel approach and the self-learning setup. The match was just designed to test it out, and show how it performed, however it was also completely incidental.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

Jesse Gersenson wrote: Wed May 16, 2018 8:34 pm
Laskos wrote: Wed May 16, 2018 6:19 pm ...against AB engine in my test, I see no improvement....
What are the conditions of the test?
1s/move on 1 CPU thread against 1 weak AB engine. Master v0.10 engine. Not very sound conditions, but I usually saw my results later confirmed by more soud gauntlets. I will soon have my GTX 1060, so the conditions will change dramatically.
Werewolf
Posts: 1795
Joined: Thu Sep 18, 2008 10:24 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Werewolf »

Laskos wrote: Wed May 16, 2018 6:19 pm
Since ID292, again something fishy happens, ID300 in their self-games is about 100 Elo points stronger than ID292, but against AB engine in my test, I see no improvement.
Does this still hold true for ID 302? The self play elo has really gone up now.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

Werewolf wrote: Wed May 16, 2018 10:27 pm
Laskos wrote: Wed May 16, 2018 6:19 pm
Since ID292, again something fishy happens, ID300 in their self-games is about 100 Elo points stronger than ID292, but against AB engine in my test, I see no improvement.
Does this still hold true for ID 302? The self play elo has really gone up now.
Yes, the same for ID302 compared to ID292, no improvement (well, within error margins, so there is maybe at most 20 Elo points improvement). They see 130 Elo points improvement in self-games. Either something is wrong with my testing, or again something is fishy in their framework.
David Xu
Posts: 47
Joined: Mon Oct 31, 2016 9:45 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by David Xu »

jp wrote: Wed May 16, 2018 7:56 pm
yanquis1972 wrote: Wed May 16, 2018 7:07 pm
Albert Silver wrote: Wed May 16, 2018 4:29 pm
It's like 5 pages long. You can skim over the mathematics and just read the results and then you would know what everyone else here knows instead of guessing or stumbling.
i can be interested & not care to be an expert, or enjoy being informed but not wish to be consumed. i have several other things that interest be more, & for my own mental health (not a shot at anyone here, i'm talking about me) i can't devote hours to abstractions right now. i don't see what it matters, & i assumed there would be no point in a layman reading the AZ paper. that's my mistake & i'll take your advice & check it out.
Yeah, read it to get the idea but don't swallow everything it claims. Its defects are in those dozens of articles...
e.g.
https://medium.com/@josecamachocollados ... 66ae1c84f2
I have a question for you: do you have any idea what you're talking about when you comment in these threads?