Something goes wrong with lc0 since yesterday?

[yanquis1972]

huge elo jump (nearly +30). looks to be above 390 in selfplay too, a match is being run now. http://lczero.org/matches

[Guenther]

huge elo jump (nearly +30). looks to be above 390 in selfplay too, a match is being run now. http://lczero.org/matches

Nope. They could not resist and tested before 10 iterations were over and the result was that ID390 still is better (but inside of error margins).

[yanquis1972]

ah, thanks, i must've misread.

[Werewolf]

The ID 10000 series looks very interesting

[Werewolf]

Haven’t yet found answers on the forums, but do we know why the relatively large network jump and the Elo reset?

[Laskos]

Haven’t yet found answers on the forums, but do we know why the relatively large network jump and the Elo reset?

Seems random network MCTS games as baseline, and the goal with this 20x256 bignet is to have a very strong engine, maybe above SF.
15 or so times slower in NPS than 6x64 smallnet used up to now, therefore 15 times slower games, so one new ID will appear in a matter of a half or a full day (as an order of magnitude) with the current number of users contributing. May take several weeks to overcome the current strongest networks (bignet ID395 from mainserver and smallnet ID9155 from testserver) and another several weeks to reach SF level. Although I don't understand how AlphaZero Chess was so strong tactically and in endgames (checked recently for blunders the published ten games), this seems almost impossible to achieve in the current framework of Leela Chess. Let's see. Leela might still become stronger than SF on GTX 1060 against SF on i7 CPU in certain conditions, say playing from early regular opening positions, as tactical complications are not that common. But in other conditions, say playing from MES1258 endgame tactical suite as openings, I doubt Leela Chess as it is currently built will ever be stronger than SF.

[yanquis1972]

didn't AZ tend towards quiet positions? i don't think it was a tactical mastermind, but it understood won & lost positions middlegame positions better than stockfish. some blunders could explain the questioned draw rate.

lcz is also many millions of games away from AZ, & if 390/strongest net available has been simulated to those match conditions (as far as we know them) i'm unaware of it.

it is a little surprising to me though, if you're correct, that lcz won't be able to intuit endgames just as well as opening/early middlegames. are you (or anyone else) able to articulate a possible reason? it's just counterintuitive at a very primitive level; i.e. how can a computer know what's winning in the beginning if it doesn't understand the end? and why are opening themes -- a wide variety -- already grasped so well but endgame themes elusive?

[Laskos]

didn't AZ tend towards quiet positions? i don't think it was a tactical mastermind, but it understood won & lost positions middlegame positions better than stockfish. some blunders could explain the questioned draw rate.

lcz is also many millions of games away from AZ, & if 390/strongest net available has been simulated to those match conditions (as far as we know them) i'm unaware of it.

it is a little surprising to me though, if you're correct, that lcz won't be able to intuit endgames just as well as opening/early middlegames. are you (or anyone else) able to articulate a possible reason? it's just counterintuitive at a very primitive level; i.e. how can a computer know what's winning in the beginning if it doesn't understand the end? and why are opening themes -- a wide variety -- already grasped so well but endgame themes elusive?

First, most games aren't decided in endgames even with standard engines. With LC0 in self-games, even less so. So, LC0 is trained mostly in undecided openings and midgames, and less so in undecided endgames. Skewed endgames are easy even for LC0. Second, in endgames long sequences can occur of unique optimal moves (like ladders in Go), and either they are hit very rarely in order to have a solid learning statistic, or patterns to learn are hard to fathom. In fact yesterday night I was surprised to see that in 100 KBNK won positions, LC0 ID395 won all of the games against an AB engine of similar strength in my conditions, Arasan 20.5. It didn't miss a single KBNK win out of 100. So, LC0 CAN learn these sorts of things. How did it learn that is unclear to me, as to hit a correct sequence form a random network has a very low probability, and the pattern is not very clear. But with 5-men easy wins (though not very skewed for one side), Arasan 20.5 missed 1/100 wins, LC0 ID395 missed 22/100 wins. With 6-men easy wins, Arasan 20.5 missed 3/100 wins, LC0 ID395 missed 48/100 wins. So, the difficulty in endgames is obvious even comparing with not that strong an AB engine which has some endgame knowledge encoded in eval. SF, for example, didn't miss any win out of 100 with those 5- and 6-men easy won positions (even more knowledge in the eval and better search). Let's see this new monster they are building now with 20x256 nets, the first step was a pretty high jump and it takes about half a day for a new ID in current hardware conditions. At this pace, in several weeks we will see something very strong, if they won't have major bugs or botch network building.

[Werewolf]

Haven’t yet found answers on the forums, but do we know why the relatively large network jump and the Elo reset?

Seems random network MCTS games as baseline, and the goal with this 20x256 bignet is to have a very strong engine, maybe above SF.
15 or so times slower in NPS than 6x64 smallnet used up to now, therefore 15 times slower games, so one new ID will appear in a matter of a half or a full day (as an order of magnitude) with the current number of users contributing. May take several weeks to overcome the current strongest networks (bignet ID395 from mainserver and smallnet ID9155 from testserver) and another several weeks to reach SF level.

I thought training games from earlier nets were usable, so I'm surprised elo has gone to zero again. I wonder what was achieved with the 6x64 net that made them proceed to the 20x256. And why such a big network jump? Anyway, it's underway and we just have to be patient.

[Werewolf]

didn't AZ tend towards quiet positions? i don't think it was a tactical mastermind, but it understood won & lost positions middlegame positions better than stockfish.

This was my observation too. The impressive thing about AZ showcase moves wasn't the quality of the combination, but its evaluation of the resulting position.