How to find your Leela Ratio - for Windows users w/ GPU(s)

[jp]

I think the whole concept of "Leela Ratio" is useless. You'll take your best buy anyway.

Useful only for comparing Lc with A0 (but then you'd also need to reproduce their other conditions and settings, which you probably can only guess at).
Bad name, too. It's got nothing to do with Leela.

[dkappe]

If you want to return to the dark ages of “hey, leela just beat stockfish on my computer,” be my guest. If you have a better benchmark, let’s see it. It’s easy to throw stones.

[sincerelybrent]

... The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters...

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.

You can change Leela Ratio settings to match conditions you would use for different time controls, as I show in the doc, if you’d like. Bullet vs TCEC time controls, for example. But the changes to nncache, like you mention, won’t affect the ratio much, unless you’re choking Leela by giving way too little. The biggest factor in changing the ratio’s value is backend, as I’m sure you’re aware. As long as that’s correct, you’ll be close. You mentioned a 10-15% margin in other test groups for CPU testing being acceptable. If we can get our GPU to CPU ratios within that margin, why not? That’s a lot better than a 1.5x - 3.8x difference in my rig.

Without some benchmark, two variables are in the test: GPU-to-CPU difference and the engines. If we can make the GPU to CPU ratio as close to 1:1 as we can, then we are testing the engines only, without there being a great disparity between GPU and CPU. Was A0 truly better? Is it that much better than Stockfish? We don’t know, because A0 had such an advantage in “time.” Set the ratio to 1:1 and have the match again. What would the result be?

How would you solve the problem of “leveling the playing field” for GPU vs CPU engines?

[sincerelybrent]

If you want to return to the dark ages of “hey, leela just beat stockfish on my computer,” be my guest. If you have a better benchmark, let’s see it. It’s easy to throw stones.

Exactly.

As an example, when I set my time controls to make the Leela Ratio 1:1, Stockfish and Leela were dead even after 100 games. When I don’t adjust the time and just let Leela rip, she spanks Stockfish and scores almost quadruple the number of wins.

[Laskos]

I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.

I have found that when running on a GPU there is significant startup overhead and the NPS increases very rapidly until about 2 seconds (and slowly continues to increase after that until the nncache fills). This is on a 1070. Accordingly, although the games take more time, I no longer test at faster time controls than with an increment of 2 seconds. I have seen match results completely reversed at faster TCs vs 1+2, even well within the margin of error. Leela NPS is also much more variable depending on the position than the NPS for CPU engines. Of course, CPUs have practically no overhead, so the ultra-rapid TC testing is fine. Moreover, the Leela Ratio only applies to nets with a specific size/architecture, as has been pointed out.

Just noticed more info in this post:
viewtopic.php?f=7&t=67347

Thanks for the link.

On RTX 2070 it is even worse than on 1070, from 2 seconds NPS still increases further to some maybe 30% higher at 20 seconds and maybe another 20% until the cache fills in to say 50%, which for large cache can be 200 seconds or more. At short times like 0.6s compared to 0.2s, the NPS could be a factor of 2 higher, the things are uncontrollable there as anything so called "Leela Ratio" goes due to overhead. Averaging over in-game positions at any TC works badly, LittleBlitzer does that, with stupid results. Suddenly on an in-game position, cache hits seem to dominate, giving some five fold sudden increase in speed, a thing never happening with CPU regular engines. Averaging with these spikes gives pretty meaningless results. Also the speed varies pretty significantly from net to net even inside the same run, say T40 run. Some of the nets there are say 20-30% slower than other using the same engine (same run, same net size, architecture, etc.)

But it's still good to have this benchmark in order to have some idea about hardware configuration. For now it's easy to form an idea, say you have a strong RTX GPU and a quad- or six-core CPU, means pretty high so called "Leela Ratio" in say Blitz or longer TC. But more GPUs will appear, from AMD too, possibly with tensor cores, and this benchmark will be somehow useful to give some impression.

[mwyoung]

I think the whole concept of "Leela Ratio" is useless. You'll take your best buy anyway.

I agree it is meaningless. My ratio is 2950x vs 2080 ti.

All you need to know is gpu and settings. And as long as you test against known ratings of the other engines. You know the true strength of Lc0 on a given gpu.