Usefulness of "Leela ratio"

[corres]

There is a dispute on the forum of lczero.org about Leela ratio.
There are peoples who think its a useful idea and tool to compare the different systems what are used to run matches between AB engines (like Stockfish) and NN engines (like Leela) for getting comparable test results.
And there are peoples who think Leela ratio is a superfluous restriction for testing because usage of Leela ratio does not enhance the comparability of different tests and different systems.
What is your opinion about Leela ratio?

[Laskos]

There is a dispute on the forum of lczero.org about Leela ratio.
There are peoples who think its a useful idea and tool to compare the different systems what are used to run matches between AB engines (like Stockfish) and NN engines (like Leela) for getting comparable test results.
And there are peoples who think Leela ratio is a superfluous restriction for testing because usage of Leela ratio does not enhance the comparability of different tests and different systems.
What is your opinion about Leela ratio?

Sure it is useful. But one has to be careful. Leela ratio on an RTX GPU at 60'' + 0.6'' is smaller than at 60' + 36''. There are "curves" of Leela ratios function of hardware, time control, cache size, etc.

[Eduard]

ChessBase likes it and compares Fat Fritz to AlphaZero. I think it's not perfect, but very good (Ratio 0.7 - 1,3 is OK). I need it for my own tests!

[AndrewGrant]

Useful for naive comparison of testing configurations.
Useless as a basis for determining what is balanced.

[Raphexon]

Useless and with caveats.

[MikeB]

There is a dispute on the forum of lczero.org about Leela ratio.
There are peoples who think its a useful idea and tool to compare the different systems what are used to run matches between AB engines (like Stockfish) and NN engines (like Leela) for getting comparable test results. That is fantasy
And there are peoples who think Leela ratio is a superfluous restriction for testing because usage of Leela ratio does not enhance the comparability of different tests and different systems. Probably true
What is your opinion about Leela ratio?

We are probably maxing out on the usefulness of the number of cores - saying going from a 128 core system to 256 core system. On the other hand, NN are attracting a lot more interest and not just for chess - and big money is spending. It's really not going out on a limb, but overtime in the near future, you will see a lot bigger gains in the effectiveness of GPUs on a cost basis. So whatever ratio one believes in today will not be relevant in the future. CCRL did Chessbase a great favor by including Fat Fritz in their testing - which I am not pro or against - CCRL is an independent volunteer organization and they can test what they please.

[the_real_greco]

The true "Leela ratio" is whatever your home hardware can give you.

Other uses involve complaining about the hardware in a tournament, adding ambiguity for the acronym "LR", and as an arbitrary setting so that AlphaZero can beat Stockfish.

[dkappe]

Useful for naive comparison of testing configurations.
Useless as a basis for determining what is balanced.

Bingo!

When I came up with the “leela ratio” (a name that I now regret), the whole focus was around beating A0. As t10 started getting good, people started posting breathless match results of leela on a 1080 destroying sf9 on one CPU. I came up with a naive formula to peg 1.0 to the node ratio between A0 and SF8. It wasn’t terribly precise, but it was a bit of an antidote to ridiculous testing conditions.

Now it is naive and outdated, but one does need a way of comparing performance between GPU and CPU, just as one would if two engines were running on different CPUs. I look forward to the successor of the Leela Ratio, so I can stop defending it for what it is and, more often, what it isn’t.

[corres]

Useful for naive comparison of testing configurations.
Useless as a basis for determining what is balanced.

Bingo!
When I came up with the “leela ratio” (a name that I now regret), the whole focus was around beating A0. As t10 started getting good, people started posting breathless match results of leela on a 1080 destroying sf9 on one CPU. I came up with a naive formula to peg 1.0 to the node ratio between A0 and SF8. It wasn’t terribly precise, but it was a bit of an antidote to ridiculous testing conditions.
Now it is naive and outdated, but one does need a way of comparing performance between GPU and CPU, just as one would if two engines were running on different CPUs. I look forward to the successor of the Leela Ratio, so I can stop defending it for what it is and, more often, what it isn’t.

After the above I think it is enough writing from me about "Leela Ratio":
I never used it.

[Ozymandias]

I support testing that will bring cost (TCO) to the spotlight, abandoning the Leela ratio. This won't give you the reliability of a formula (LR), because prices fluctuate over time and depend on where you live, but at any given time and place, some basics can be stablished.

Some components will be needed to operate either an AB engine or a NN. In both cases you need: monitor, keyboard, mouse, case, PSU, HDD and RAM. These components are of no consequence when comparing. The motherboard on the other hand, should be given extra attention.

For AB engines, you want a strong processor, which means a good MOBO with quality VRM. If you plan on buying with a NN in mind, you can save money on this front, as a low power processor will do the job just fine and the graphics card's VRM, will take care of cleaning up the current delivered to the GPU.

Of course, you finally have to look at prices for CPU and GPU, but they aren't exclusive. The NN will also use CPU cores (although not as many as you need for AB), but inversely, the AB engine will also need a GPU, even if just for 2D output. The real comparison should be done like this:

(Cheap CPU + cheap MOBO + expensive GPU) vs (expensive CPU + expensive MOBO + cheap GPU)