Scaling of Lc0 at high Leela Ratio

[yanquis1972]

I get dramatically better results going from eg 0+15s to 0+60s (should work out to very roughly 100kn searches/move vs 400kn on my hardware), this with test30 and prior to the 19.1 update. Since your hardware is considerably faster I’d be curious about the results vs SF8 with the deepmind opening set. I think it should be about = at the longer TC, but haven’t tested yet and it takes considerable time to generate games.

[Laskos]

That +2 -0 =2 for SF10 at real LTC in these games with diverse developments, the double failure of Lc0 to convert, indicate again that scaling to real LTC might be not that good for Lc0.

Which Lc0 version did you use? The most recent one v0.19.1 (it's just a version rename of v0.19.1-rc2 from 3 days ago so rc2 is fine too) has Cpuct growing with time (taken from AlphaZero paper) and it should help with LTC scalability.

I just performed the first shorter tests with the v19.1, with the same ID11261.

0.25m + 0.25s
Score of lc0_v191_11261 vs SF10: 21 - 18 - 61 [0.515] 100
Elo difference: 10.43 +/- 42.67
Finished match

1m + 1s
Score of lc0_v191_11261 vs SF10: 21 - 17 - 62 [0.520] 100
Elo difference: 13.90 +/- 42.10
Finished match

The longer TC later, it will take time.

But I have a question about these test30 nets. They seem to exhibit a bit weird NPS behavior (with both v19 and v19.1), and seems this behavior is systematic one with newer and newer nets.
I took an older net, ID31240 and a new net, ID31941 (700 nets difference, 20+ million games). The difference between them is some 70 real Elo points at short time control. But look at these NPS numbers from Initial Board position on my RTX 2070 (Lc0 v0.19.1 used):

Usual short time controls used in tests:

ID31240:
info depth 12 seldepth 41 time 1180 nodes 17793 score cp 27 hashfull 2 nps 15078
info depth 13 seldepth 45 time 2585 nodes 45929 score cp 29 hashfull 6 nps 17767

ID31941:
info depth 9 seldepth 30 time 1090 nodes 17879 score cp 54 hashfull 2 nps 16402
info depth 10 seldepth 33 time 2515 nodes 47949 score cp 51 hashfull 6 nps 19065

So, at these usual short time controls, newer ID are 8-10% faster.

But at real LTC, things reverse pretty drastically:

ID31240:
info depth 16 seldepth 53 time 150668 nodes 6162321 score cp 29 hashfull 358 nps 40899
info depth 18 seldepth 55 time 358737 nodes 15040358 score cp 32 hashfull 760 nps 41925

ID31941:
info depth 18 seldepth 43 time 151858 nodes 5454084 score cp 42 hashfull 385 nps 35915
info depth 20 seldepth 49 time 363744 nodes 12928381 score cp 42 hashfull 874 nps 35542

At real LTC, newer nets are 14-18% slower.

A total of 25 % or so slowdown from short time control to long time control for newer nets compared to older nets. Doesn't this affect the scaling too of the test30 nets, older nets scaling better to real LTC?