Usefulness of "Leela ratio"

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
corres
Posts: 1761
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Usefulness of "Leela ratio"

Post by corres » Mon Dec 02, 2019 1:29 pm

There is a dispute on the forum of lczero.org about Leela ratio.
There are peoples who think its a useful idea and tool to compare the different systems what are used to run matches between AB engines (like Stockfish) and NN engines (like Leela) for getting comparable test results.
And there are peoples who think Leela ratio is a superfluous restriction for testing because usage of Leela ratio does not enhance the comparability of different tests and different systems.
What is your opinion about Leela ratio?

User avatar
Laskos
Posts: 9498
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Usefulness of "Leela ratio"

Post by Laskos » Mon Dec 02, 2019 1:44 pm

corres wrote:
Mon Dec 02, 2019 1:29 pm
There is a dispute on the forum of lczero.org about Leela ratio.
There are peoples who think its a useful idea and tool to compare the different systems what are used to run matches between AB engines (like Stockfish) and NN engines (like Leela) for getting comparable test results.
And there are peoples who think Leela ratio is a superfluous restriction for testing because usage of Leela ratio does not enhance the comparability of different tests and different systems.
What is your opinion about Leela ratio?
Sure it is useful. But one has to be careful. Leela ratio on an RTX GPU at 60'' + 0.6'' is smaller than at 60' + 36''. There are "curves" of Leela ratios function of hardware, time control, cache size, etc.

User avatar
Eduard
Posts: 224
Joined: Fri Oct 26, 2018 10:58 pm
Location: Germany
Full name: Eduard Nemeth
Contact:

Re: Usefulness of "Leela ratio"

Post by Eduard » Mon Dec 02, 2019 1:54 pm

ChessBase likes it and compares Fat Fritz to AlphaZero. :) I think it's not perfect, but very good (Ratio 0.7 - 1,3 is OK). I need it for my own tests!

AndrewGrant
Posts: 494
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: Usefulness of "Leela ratio"

Post by AndrewGrant » Mon Dec 02, 2019 2:00 pm

Useful for naive comparison of testing configurations.
Useless as a basis for determining what is balanced.

Raphexon
Posts: 114
Joined: Sun Mar 17, 2019 11:00 am
Full name: Henk Drost

Re: Usefulness of "Leela ratio"

Post by Raphexon » Mon Dec 02, 2019 2:58 pm

Useless and with caveats.

MikeB
Posts: 3542
Joined: Thu Mar 09, 2006 5:34 am
Location: Pen Argyl, Pennsylvania

Re: Usefulness of "Leela ratio"

Post by MikeB » Mon Dec 02, 2019 7:38 pm

corres wrote:
Mon Dec 02, 2019 1:29 pm
There is a dispute on the forum of lczero.org about Leela ratio.
There are peoples who think its a useful idea and tool to compare the different systems what are used to run matches between AB engines (like Stockfish) and NN engines (like Leela) for getting comparable test results. That is fantasy
And there are peoples who think Leela ratio is a superfluous restriction for testing because usage of Leela ratio does not enhance the comparability of different tests and different systems. Probably true
What is your opinion about Leela ratio?
We are probably maxing out on the usefulness of the number of cores - saying going from a 128 core system to 256 core system. On the other hand, NN are attracting a lot more interest and not just for chess - and big money is spending. It's really not going out on a limb, but overtime in the near future, you will see a lot bigger gains in the effectiveness of GPUs on a cost basis. So whatever ratio one believes in today will not be relevant in the future. CCRL did Chessbase a great favor by including Fat Fritz in their testing - which I am not pro or against - CCRL is an independent volunteer organization and they can test what they please.
Last edited by MikeB on Mon Dec 02, 2019 7:47 pm, edited 1 time in total.

the_real_greco
Posts: 8
Joined: Thu Sep 19, 2019 3:55 am
Full name: Andy Erly

Re: Usefulness of "Leela ratio"

Post by the_real_greco » Mon Dec 02, 2019 7:43 pm

The true "Leela ratio" is whatever your home hardware can give you.

Other uses involve complaining about the hardware in a tournament, adding ambiguity for the acronym "LR", and as an arbitrary setting so that AlphaZero can beat Stockfish.

dkappe
Posts: 272
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: Usefulness of "Leela ratio"

Post by dkappe » Mon Dec 02, 2019 9:42 pm

AndrewGrant wrote:
Mon Dec 02, 2019 2:00 pm
Useful for naive comparison of testing configurations.
Useless as a basis for determining what is balanced.
Bingo!

When I came up with the “leela ratio” (a name that I now regret), the whole focus was around beating A0. As t10 started getting good, people started posting breathless match results of leela on a 1080 destroying sf9 on one CPU. I came up with a naive formula to peg 1.0 to the node ratio between A0 and SF8. It wasn’t terribly precise, but it was a bit of an antidote to ridiculous testing conditions.

Now it is naive and outdated, but one does need a way of comparing performance between GPU and CPU, just as one would if two engines were running on different CPUs. I look forward to the successor of the Leela Ratio, so I can stop defending it for what it is and, more often, what it isn’t.

corres
Posts: 1761
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Usefulness of "Leela ratio"

Post by corres » Mon Dec 02, 2019 10:27 pm

dkappe wrote:
Mon Dec 02, 2019 9:42 pm
AndrewGrant wrote:
Mon Dec 02, 2019 2:00 pm
Useful for naive comparison of testing configurations.
Useless as a basis for determining what is balanced.
Bingo!
When I came up with the “leela ratio” (a name that I now regret), the whole focus was around beating A0. As t10 started getting good, people started posting breathless match results of leela on a 1080 destroying sf9 on one CPU. I came up with a naive formula to peg 1.0 to the node ratio between A0 and SF8. It wasn’t terribly precise, but it was a bit of an antidote to ridiculous testing conditions.
Now it is naive and outdated, but one does need a way of comparing performance between GPU and CPU, just as one would if two engines were running on different CPUs. I look forward to the successor of the Leela Ratio, so I can stop defending it for what it is and, more often, what it isn’t.
After the above I think it is enough writing from me about "Leela Ratio":
I never used it.

User avatar
Ozymandias
Posts: 1137
Joined: Sun Oct 25, 2009 12:30 am

Re: Usefulness of "Leela ratio"

Post by Ozymandias » Tue Dec 03, 2019 12:51 pm

I support testing that will bring cost (TCO) to the spotlight, abandoning the Leela ratio. This won't give you the reliability of a formula (LR), because prices fluctuate over time and depend on where you live, but at any given time and place, some basics can be stablished.

Some components will be needed to operate either an AB engine or a NN. In both cases you need: monitor, keyboard, mouse, case, PSU, HDD and RAM. These components are of no consequence when comparing. The motherboard on the other hand, should be given extra attention.

For AB engines, you want a strong processor, which means a good MOBO with quality VRM. If you plan on buying with a NN in mind, you can save money on this front, as a low power processor will do the job just fine and the graphics card's VRM, will take care of cleaning up the current delivered to the GPU.

Of course, you finally have to look at prices for CPU and GPU, but they aren't exclusive. The NN will also use CPU cores (although not as many as you need for AB), but inversely, the AB engine will also need a GPU, even if just for 2D output. The real comparison should be done like this:

(Cheap CPU + cheap MOBO + expensive GPU) vs (expensive CPU + expensive MOBO + cheap GPU)

Post Reply