Something goes wrong with lc0 since yesterday?

[yanquis1972]

there's some solid evidence of a spike in elo gain post-460, when the learning rate was reduced.

http://training.lczero.org./#scalars
EDIT -- sorry guys previous link was apparently to a copy i gained ownership of when accessing the spreadsheet, try this link:
https://drive.google.com/drive/folders/ ... sp=sharing [gauntlet results going back to 3xx]

in watching some games against SF8 with latest net (482) i feel like the biggest difference between it and several nets ago is opening play; wondering if anyone can link me to the openings 200 testsuite mentioned or if kai'd want to re-test.

[Werewolf]

The 10000 series went up a bit in self play but it's still getting crushed by my Magnus app on my iPhone (2850 elo), so I think it's about 2650 FIDE elo.

I hope it hasn't saturated - progress seems very slow now.

[yanquis1972]

i don't know enough about the play magnus app; my only hope is that if 2850 is the top level it could be an unlocked (if modified) SF9+, which would be terrifically strong on a current iphone. i saw a couple matches yesterday that seemed promising (one linked in this thread; it's from a list that seemed quite inflated at the time according to kai's results & my observations of a few games) one a (statistically) short match against komodo 9.42 4CPU where it picked up a few wins, both suggesting ~3000elo (ccrl, not fide). but re-reading the blog post from yesterday more closely, it sounds to me like they're still genuinely doing testing, not training a replacement a net. i'm not in the loop, but they reached 3000elo+ in 1-2 days, so they may not be worried about the initial phase, just want to be as sure as possible it's set up to knock it out of the park.

What’s stopping us from rolling out lc0 to main net?

The network compression is possibly the single most important outstanding thing. There are attempts to reduce precision to fp16 (worked well), and int8 (didn’t work well so far, ~150 Elo drop on same nodes).
Merging both of @Cyanogenoid’s training PRs.
We’d like to see cpuct 1.7 or 1.8, and perhaps resign-playthru set to 20% instead of 10%.
Given the above changes, we’d like to see one more test run that’s really really short, to verify that the previous at least work correctly. It doesn’t even need to reach first LR drop, only a couple of days is necessary
Then promote!

[Laskos]

there's some solid evidence of a spike in elo gain post-460, when the learning rate was reduced.

http://training.lczero.org./#scalars
EDIT -- sorry guys previous link was apparently to a copy i gained ownership of when accessing the spreadsheet, try this link:
https://drive.google.com/drive/folders/ ... sp=sharing [gauntlet results going back to 3xx]

in watching some games against SF8 with latest net (482) i feel like the biggest difference between it and several nets ago is opening play; wondering if anyone can link me to the openings 200 testsuite mentioned or if kai'd want to re-test.

In my tests at 10s/position on GTX 1060 GPU, the results are for positional Openings200 and tactical WAC200

ID482
Pos: 120/200
Tact: 160/200

ID462
Pos: 115/200
Tact: 167/200

ID395
Post: 132/200
Tact: 156/200

So, ID482 came positionally a bit stronger than ID462 and tactically a bit weaker. As strength goes against AB engines, there seem to not be much difference between ID482 and ID462, both the levels of ID395 (but only 200 games gauntlets).

As for bignet 10060s nets level, the improvement against AB engines seems very slow. 10060s are about 400 Elo points weaker than smallnet 9060s nets, about 2800 CCRL 40/4' Elo level. And the improvement seems to be only about 200 Elo points compared to early bignet ID10017 against AB engines. At this pace, I am very worried about the potential of the current bignet run.

[yanquis1972]

did you have 395 at over 3350 (CCRL 40/4 scale)? i have 482 looking scary close to SF8 atm, but nowhere near 200 games.

[yanquis1972]

As for bignet 10060s nets level, the improvement against AB engines seems very slow. 10060s are about 400 Elo points weaker than smallnet 9060s nets, about 2800 CCRL 40/4' Elo level. And the improvement seems to be only about 200 Elo points compared to early bignet ID10017 against AB engines. At this pace, I am very worried about the potential of the current bignet run.

thanks -- i actually checked the google spreadsheet rating list a bit ago & for whatever reason your ratings aren't so far apart now (100 elo +/- error margins, if you've both tested one of the same top nets). just going off of it, testserver nets after 10k45 are all over the place; i assume it's due to experimentation (i don't speak the language of the bit i quoted, but it sounds like they're going to do another reset, & must be very confident of fast growth if they're going to put it on main before it's even passed it in strength...)

[Laskos]

did you have 395 at over 3350 (CCRL 40/4 scale)? i have 482 looking scary close to SF8 atm, but nowhere near 200 games.

I had ID395 at about 3300 CCRL 40/4' conditions. I am usually testing at shorter than 40/4' time control, and then extrapolate to 40/4'. What time control are you testing at? And still, 200 games each gauntlet is not much as Elo errors go, especially with low draw rate.

[Laskos]

As for bignet 10060s nets level, the improvement against AB engines seems very slow. 10060s are about 400 Elo points weaker than smallnet 9060s nets, about 2800 CCRL 40/4' Elo level. And the improvement seems to be only about 200 Elo points compared to early bignet ID10017 against AB engines. At this pace, I am very worried about the potential of the current bignet run.

thanks -- i actually checked the google spreadsheet rating list a bit ago & for whatever reason your ratings aren't so far apart now (100 elo +/- error margins, if you've both tested one of the same top nets). just going off of it, testserver nets after 10k45 are all over the place; i assume it's due to experimentation (i don't speak the language of the bit i quoted, but it sounds like they're going to do another reset, & must be very confident of fast growth if they're going to put it on main before it's even passed it in strength...)

They will reset in the same conditions with this 20x256 100xx bignet run? It seems a problematic run, unlike those 6x64 smallnet xx, 40xx and 90xx runs. May be it's better, to use increasingly bigger nets, like the initial LCZero? 6x64 achieved a 3250 CCRL Elo points in several days, I guess that incrementally bigger nets will perform very well very soon.

[crem]

As for bignet 10060s nets level, the improvement against AB engines seems very slow. 10060s are about 400 Elo points weaker than smallnet 9060s nets, about 2800 CCRL 40/4' Elo level. And the improvement seems to be only about 200 Elo points compared to early bignet ID10017 against AB engines. At this pace, I am very worried about the potential of the current bignet run.

Unlike previous training attempts where learning rate was reduced frequently (every 20 nets or so) and slowly (by dividing by 3), test10 tries to replicate what is believed DeepMind (change LR 2 times in total, by dividing by 10) did. With that in mind test10 did not reduce LR yet at all.

The first LR reduction will happen around network id10098 (probably testing will restart or moved to main server before that though). After LR change, progress should be fast again.

In general squeezing everything from one LR before switching to a next one is known to improve final quality of the next (at the cost of training speed).

[crem]

(change LR 2 times in total, by dividing by 10)

Correction:
(change LR 3 times in total, by dividing by 10)