Something goes wrong with lc0 since yesterday?

[yanquis1972]

and in between 10016 showed +11 elo (relatively speaking, nothing). when i saw that last night i was hoping it signified a consolidation of some kind...no idea if it actually did or this is a coincidence, but i hope someone smart puts some effort into decoding the significance of self-play elo.

[Laskos]

and in between 10016 showed +11 elo (relatively speaking, nothing). when i saw that last night i was hoping it signified a consolidation of some kind...no idea if it actually did or this is a coincidence, but i hope someone smart puts some effort into decoding the significance of self-play elo.

I don't know what they are decoding. Again, some erratic behavior. ID10021 shows 140 Elo points progress in their self-games compared to ID10017, but it comes 30 Elo points weaker in my gauntlet against AB engines (almost within error margins). No progress at all, that's almost sure. And when progress appears, it is unrelated to their self-games results. Something odd is going on.

[yanquis1972]

to me it looks like google stalled out, or, at least, the gains decreased significantly for a period at around the same elo. so it may just be as simple as dropping the learning rate. by selfplay elo at least, that seems to be working on the mainserver (only 2 nets so far).

[George Tsavdaris]

Someone on Leela discord channel had a very impressive result even though games are too few.

All engines except Leela with 1 CPU of 3.2 GHz Intel Xeon W.
Leela uses a Nvidia Tesla V100 GPU!
Leela uses a 20(blocks)x256(filters) net called kb3 20b that is experimental from an independent user and it is based on generated games from smaller nets i think and not based on selfplay training with itself(games of 20x256 net).
Time control was 40moves/40 second repeating.

I've set the base ELO of Stockfish 050618 to 3381.0 CCRL 40/40 ELO(i had to use 40/4 but anyway doesn't change much) even though that was the ELO the Stockfish 9 had on CCRL and not latest SF 050618.

Code: Select all

#  PLAYER            :  RATING   ERROR  POINTS   PLAYED   (%)
1 Houdini 6          :  3433.7  ±158.6    10.5      20    53
2 Leela kb3 20b      :  3416.2  ±126.1    46.5      80    58
3 Fire 7.1           :  3381.0  ±164.6     9.0      20    45
4 Stockfish 050618   :  3381.0   ----      9.0      20    45
5 komodo10           :  3223.6  ±159.5     5.0      20    25

Leela kb3 20b        : 80 (+ 26,= 41,- 13), 58.1 %
results against:
Stockfish 050618     : 20 (+  5,= 12,-  3), 55.0 %
Houdini 6            : 20 (+  4,= 11,-  5), 47.5 %
komodo10             : 20 (+ 11,=  8,-  1), 75.0 %
Fire 7.1             : 20 (+  6,= 10,-  4), 55.0 %

[yanquis1972]

well that's rather incredible...this may be link to net in question i'm not 100% sure.

edit: i just saw the that GPU costs $8000...maybe not so incredible. if performance matches price that'd be like me giving 20:1 time odds.

[Laskos]

well that's rather incredible...this may be link to net in question i'm not 100% sure.

edit: i just saw the that GPU costs $8000...maybe not so incredible. if performance matches price that'd be like me giving 20:1 time odds.

That net performs pretty badly at very fast time control. Maybe it scales well to longer tc. Can you find out if the link is to the correct net?

On the other hand, the new mainserver ID462 performed a bit better than ID395, but within error margins. Maybe the new nets from mainserver will become the strongest from now on.

[Werewolf]

On the other hand, the new mainserver ID462 performed a bit better than ID395, but within error margins. Maybe the new nets from mainserver will become the strongest from now on.

How much elo below the best from the main server is the 10000 series against Alpha Betas?

The 10000 is still improving quite fast I'd just point out...

[yanquis1972]

depends how much the latest 5-6 nets have gained. hopefully 2700+ CCRL, but i really have no idea. from the blog post yesterday i think they expect it to pass mainserver in 1-3 weeks.

New pipeline won’t be next week, and probably not week after next, so more cooking on the existing main net. On the other hand, it could happen next week.

if the mainserver is progressing w/ the new lower learning rate, frankly i'd keep it going in parallel, either at the same block size or increase it when it saturates. the smaller size has advantages, as far as i'm aware, & (arbitrary #) 90% of resources are dedicated to the testserver already anyway.

again just assuming this by eyeballing the AZ 'graph' but it looks like progress crawls once elo is -100 to -50 from SF8 [CCRL 40/40, 4CPU appears to be the rating they used, which is 3386. rated 3500 CCRL 4/40, which is typically used w/r/t leela, but in theory that's a less advantageous TC for LZ]

[Henk]

Watching the graphic. Looks like they need another 4 or 8 million games to get a new high. Or is any conclusion too early.
That means waiting time of 4 or 8 weeks.

[yanquis1972]

selfplay elo on the testserver progressing well & as of 10025 is backed up with results against A/B engines per
2700+ CCRL

mainserver: selfplay elo regressed but 460, 461 & 464 all did well in matches at the previous link & here at the same time 390 crushed 464 (+30 elo) in selfplay

3100+(??) CCRL