Something goes wrong with lc0 since yesterday?

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Something goes wrong with lc0 since yesterday?

Post by Laskos »

Werewolf wrote: Thu Jun 28, 2018 10:01 am
yanquis1972 wrote: Thu Jun 28, 2018 12:24 am didn't AZ tend towards quiet positions? i don't think it was a tactical mastermind, but it understood won & lost positions middlegame positions better than stockfish.
This was my observation too. The impressive thing about AZ showcase moves wasn't the quality of the combination, but its evaluation of the resulting position.
Still, I blunder checked all ten games, some positions were pretty tactical, I am pretty sure LC0 would have moderately blundered in 3-4 of those games, but A0 did not. In fact it was SF8 which blundered seriously tactically once.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Something goes wrong with lc0 since yesterday?

Post by Laskos »

Werewolf wrote: Thu Jun 28, 2018 9:59 am
Laskos wrote: Wed Jun 27, 2018 11:27 pm
Werewolf wrote: Wed Jun 27, 2018 11:07 pm Haven’t yet found answers on the forums, but do we know why the relatively large network jump and the Elo reset?
Seems random network MCTS games as baseline, and the goal with this 20x256 bignet is to have a very strong engine, maybe above SF.
15 or so times slower in NPS than 6x64 smallnet used up to now, therefore 15 times slower games, so one new ID will appear in a matter of a half or a full day (as an order of magnitude) with the current number of users contributing. May take several weeks to overcome the current strongest networks (bignet ID395 from mainserver and smallnet ID9155 from testserver) and another several weeks to reach SF level.
I thought training games from earlier nets were usable, so I'm surprised elo has gone to zero again. I wonder what was achieved with the 6x64 net that made them proceed to the 20x256. And why such a big network jump? Anyway, it's underway and we just have to be patient.
They probably want to build from scratch without re-using the 6x64 nets. More time-consuming, but cleaner, as in A0 paper. I am not following their discussions, so you might ask more informed people.
yanquis1972
Posts: 1766
Joined: Wed Jun 03, 2009 12:14 am

Re: Something goes wrong with lc0 since yesterday?

Post by yanquis1972 »

Laskos wrote: Thu Jun 28, 2018 9:49 am
yanquis1972 wrote: Thu Jun 28, 2018 12:24 am didn't AZ tend towards quiet positions? i don't think it was a tactical mastermind, but it understood won & lost positions middlegame positions better than stockfish. some blunders could explain the questioned draw rate.

lcz is also many millions of games away from AZ, & if 390/strongest net available has been simulated to those match conditions (as far as we know them) i'm unaware of it.

it is a little surprising to me though, if you're correct, that lcz won't be able to intuit endgames just as well as opening/early middlegames. are you (or anyone else) able to articulate a possible reason? it's just counterintuitive at a very primitive level; i.e. how can a computer know what's winning in the beginning if it doesn't understand the end? and why are opening themes -- a wide variety -- already grasped so well but endgame themes elusive?
First, most games aren't decided in endgames even with standard engines. With LC0 in self-games, even less so. So, LC0 is trained mostly in undecided openings and midgames, and less so in undecided endgames. Skewed endgames are easy even for LC0. Second, in endgames long sequences can occur of unique optimal moves (like ladders in Go), and either they are hit very rarely in order to have a solid learning statistic, or patterns to learn are hard to fathom. In fact yesterday night I was surprised to see that in 100 KBNK won positions, LC0 ID395 won all of the games against an AB engine of similar strength in my conditions, Arasan 20.5. It didn't miss a single KBNK win out of 100. So, LC0 CAN learn these sorts of things. How did it learn that is unclear to me, as to hit a correct sequence form a random network has a very low probability, and the pattern is not very clear. But with 5-men easy wins (though not very skewed for one side), Arasan 20.5 missed 1/100 wins, LC0 ID395 missed 22/100 wins. With 6-men easy wins, Arasan 20.5 missed 3/100 wins, LC0 ID395 missed 48/100 wins. So, the difficulty in endgames is obvious even comparing with not that strong an AB engine which has some endgame knowledge encoded in eval. SF, for example, didn't miss any win out of 100 with those 5- and 6-men easy won positions (even more knowledge in the eval and better search). Let's see this new monster they are building now with 20x256 nets, the first step was a pretty high jump and it takes about half a day for a new ID in current hardware conditions. At this pace, in several weeks we will see something very strong, if they won't have major bugs or botch network building.
i don't think i have the site link anymore, but someone charted various basic known-result 5 piece endings. leela 'solved' (100% conversion rate) the KBN mate fairly smoothly, iirc, although it obviously took some time. KQRkq was unsolved & (again from memory) showing little progress at best. with 2 queens on the board tho, that one should prove extremely difficult/impossible. perhaps the result will be that she will learn to avoid queen endings (& the long-sequence types you mention) whenever possible in match play. did AZ demonstrate a long-sequence string of only-moves?
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Something goes wrong with lc0 since yesterday?

Post by Laskos »

yanquis1972 wrote: Thu Jun 28, 2018 10:27 am
Laskos wrote: Thu Jun 28, 2018 9:49 am
yanquis1972 wrote: Thu Jun 28, 2018 12:24 am didn't AZ tend towards quiet positions? i don't think it was a tactical mastermind, but it understood won & lost positions middlegame positions better than stockfish. some blunders could explain the questioned draw rate.

lcz is also many millions of games away from AZ, & if 390/strongest net available has been simulated to those match conditions (as far as we know them) i'm unaware of it.

it is a little surprising to me though, if you're correct, that lcz won't be able to intuit endgames just as well as opening/early middlegames. are you (or anyone else) able to articulate a possible reason? it's just counterintuitive at a very primitive level; i.e. how can a computer know what's winning in the beginning if it doesn't understand the end? and why are opening themes -- a wide variety -- already grasped so well but endgame themes elusive?
First, most games aren't decided in endgames even with standard engines. With LC0 in self-games, even less so. So, LC0 is trained mostly in undecided openings and midgames, and less so in undecided endgames. Skewed endgames are easy even for LC0. Second, in endgames long sequences can occur of unique optimal moves (like ladders in Go), and either they are hit very rarely in order to have a solid learning statistic, or patterns to learn are hard to fathom. In fact yesterday night I was surprised to see that in 100 KBNK won positions, LC0 ID395 won all of the games against an AB engine of similar strength in my conditions, Arasan 20.5. It didn't miss a single KBNK win out of 100. So, LC0 CAN learn these sorts of things. How did it learn that is unclear to me, as to hit a correct sequence form a random network has a very low probability, and the pattern is not very clear. But with 5-men easy wins (though not very skewed for one side), Arasan 20.5 missed 1/100 wins, LC0 ID395 missed 22/100 wins. With 6-men easy wins, Arasan 20.5 missed 3/100 wins, LC0 ID395 missed 48/100 wins. So, the difficulty in endgames is obvious even comparing with not that strong an AB engine which has some endgame knowledge encoded in eval. SF, for example, didn't miss any win out of 100 with those 5- and 6-men easy won positions (even more knowledge in the eval and better search). Let's see this new monster they are building now with 20x256 nets, the first step was a pretty high jump and it takes about half a day for a new ID in current hardware conditions. At this pace, in several weeks we will see something very strong, if they won't have major bugs or botch network building.
i don't think i have the site link anymore, but someone charted various basic known-result 5 piece endings. leela 'solved' (100% conversion rate) the KBN mate fairly smoothly, iirc, although it obviously took some time. KQRkq was unsolved & (again from memory) showing little progress at best. with 2 queens on the board tho, that one should prove extremely difficult/impossible. perhaps the result will be that she will learn to avoid queen endings (& the long-sequence types you mention) whenever possible in match play. did AZ demonstrate a long-sequence string of only-moves?
Not that I observed one, but I cannot be sure. Usually, endgames were already pretty disbalanced in these 10 published games. It might be interesting to pit LC0 from these won endgames which A0 converted easily against SF8, and see how LC0 converts them.
Werewolf
Posts: 2039
Joined: Thu Sep 18, 2008 10:24 pm

Re: Something goes wrong with lc0 since yesterday?

Post by Werewolf »

Laskos wrote: Wed Jun 27, 2018 11:27 pm
15 or so times slower in NPS than 6x64 smallnet used up to now, therefore 15 times slower games, so one new ID will appear in a matter of a half or a full day (as an order of magnitude) with the current number of users contributing. May take several weeks to overcome the current strongest networks (bignet ID395 from mainserver and smallnet ID9155 from testserver) and another several weeks to reach SF level.
It will get frustrating at 3000 elo. Then the "rapid" (you know what I mean) progress will have stopped and we'll get +12 elo one day, -5 elo the next +20 elo the day after etc.
Laskos wrote: Wed Jun 27, 2018 11:27 pm Although I don't understand how AlphaZero Chess was so strong tactically and in endgames (checked recently for blunders the published ten games)
And it's not just a speed thing? 80,000 nps on a 256 net is a lot. I wonder what the tactical strength of the best Leela would be with 100x more time.
yanquis1972
Posts: 1766
Joined: Wed Jun 03, 2009 12:14 am

Re: Something goes wrong with lc0 since yesterday?

Post by yanquis1972 »

FYI: assume anything on the testserver is a test (unless someone with authority in the project has or can say otherwise). that might sound apparent, but it seems like most people think otherwise. it's going to be reset until they're satisfied with parameters. starting at 256 seems like a clear example of experimentation (but if i'm wrong & that is the final plan, please let me know!).

in the meantime the main network continues to make progress relatively rapidly.

https://blog.lczero.org/2018/06/26/project-update/
The rollout of lc0 and the new test pipeline will not happen this week or next.
yanquis1972
Posts: 1766
Joined: Wed Jun 03, 2009 12:14 am

Re: Something goes wrong with lc0 since yesterday?

Post by yanquis1972 »

Werewolf wrote: Thu Jun 28, 2018 4:27 pm
Laskos wrote: Wed Jun 27, 2018 11:27 pm
15 or so times slower in NPS than 6x64 smallnet used up to now, therefore 15 times slower games, so one new ID will appear in a matter of a half or a full day (as an order of magnitude) with the current number of users contributing. May take several weeks to overcome the current strongest networks (bignet ID395 from mainserver and smallnet ID9155 from testserver) and another several weeks to reach SF level.
It will get frustrating at 3000 elo. Then the "rapid" (you know what I mean) progress will have stopped and we'll get +12 elo one day, -5 elo the next +20 elo the day after etc.
Laskos wrote: Wed Jun 27, 2018 11:27 pm Although I don't understand how AlphaZero Chess was so strong tactically and in endgames (checked recently for blunders the published ten games)
And it's not just a speed thing? 80,000 nps on a 256 net is a lot. I wonder what the tactical strength of the best Leela would be with 100x more time.
curious as well, except it's not just 100x time but averaging 5M nodes/position (for comparison to match conditions, anyway). would have to check if they reported it stronger at 1s/move or not.

3000 elo will still see substantive gains every day (i.e. raw elo gained might = what a top A/B engine gains in a year, on any given day). judging by the AZ paper, it's the last 100 elo or so that will be painstaking. but perhaps there'll be a major hardware influx by that time.
Werewolf
Posts: 2039
Joined: Thu Sep 18, 2008 10:24 pm

Re: Something goes wrong with lc0 since yesterday?

Post by Werewolf »

yanquis1972 wrote: Thu Jun 28, 2018 5:50 pm
curious as well, except it's not just 100x time but averaging 5M nodes/position (for comparison to match conditions, anyway). would have to check if they reported it stronger at 1s/move or not.
I meant to compensate for AZ massive hardware advantage (I'm guessing an Nvidia 1060 will be at around 800 nps on a 256 net) - we'd have to give Leela 100x more time to produce the same number of nodes.
yanquis1972
Posts: 1766
Joined: Wed Jun 03, 2009 12:14 am

Re: Something goes wrong with lc0 since yesterday?

Post by yanquis1972 »

right, but i meant that whatever that # is, it has to be multiplied by 60 to reach AZ match-level simulation. ie hella time consuming.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Something goes wrong with lc0 since yesterday?

Post by Laskos »

yanquis1972 wrote: Thu Jun 28, 2018 5:34 pm FYI: assume anything on the testserver is a test (unless someone with authority in the project has or can say otherwise). that might sound apparent, but it seems like most people think otherwise. it's going to be reset until they're satisfied with parameters. starting at 256 seems like a clear example of experimentation (but if i'm wrong & that is the final plan, please let me know!).

in the meantime the main network continues to make progress relatively rapidly.

https://blog.lczero.org/2018/06/26/project-update/
The rollout of lc0 and the new test pipeline will not happen this week or next.
I don't know what rapid progress is making the main network, against AB engines at short TC it improves very slowly, and ID395 still seems the best in my tests, although ID450 is very close and within error margins. If the lc0 branch will start resetting even this 20x256 network, I think they are not eager to have a very strong net and are just setting the framework. I don't know why would they do that, this 20x256 net needs big hardware effort to build. Someone more informed about their affairs could answer this.