Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio
If you don't want to go through all of that, maybe this will give you a reference:
My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz
My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84
My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
How to find your Leela Ratio - for Windows users w/ GPU(s)
Moderators: hgm, Rebel, chrisw
-
- Posts: 47
- Joined: Fri Feb 15, 2019 11:41 pm
- Full name: Brent M
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio
If you don't want to go through all of that, maybe this will give you a reference:
My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz
My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84
My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
-
- Posts: 1631
- Joined: Tue Aug 21, 2018 7:52 pm
- Full name: Dietrich Kappe
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
You misunderstand the purpose of the leela ratio. It is a measure of gpu vs cpu performance, not of chess performance. As such one should use sf9 and net 11248 (or 11258) from the start position. It's much like the crafty benchmark used by CCRL. Yes, that particular version of crafty will perform differently in different positions, but as a CPU benchmark, it works just fine.Laskos wrote: ↑Thu Feb 28, 2019 12:10 amConstant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio
If you don't want to go through all of that, maybe this will give you a reference:
My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz
My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84
My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.dkappe wrote: ↑Thu Feb 28, 2019 1:10 amYou misunderstand the purpose of the leela ratio. It is a measure of gpu vs cpu performance, not of chess performance. As such one should use sf9 and net 11248 (or 11258) from the start position. It's much like the crafty benchmark used by CCRL. Yes, that particular version of crafty will perform differently in different positions, but as a CPU benchmark, it works just fine.Laskos wrote: ↑Thu Feb 28, 2019 12:10 amConstant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio
If you don't want to go through all of that, maybe this will give you a reference:
My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz
My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84
My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.
-
- Posts: 1631
- Joined: Tue Aug 21, 2018 7:52 pm
- Full name: Dietrich Kappe
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
It is semi-arbitrary (based on the reported nps ratio between a0 and sf8), yet predictive of performance at various tc and positions. I assume you want a much more complex benchmark? I’m eager to see it.Laskos wrote: ↑Thu Feb 28, 2019 6:04 am
Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.dkappe wrote: ↑Thu Feb 28, 2019 7:09 amIt is semi-arbitrary (based on the reported nps ratio between a0 and sf8), yet predictive of performance at various tc and positions. I assume you want a much more complex benchmark? I’m eager to see it.Laskos wrote: ↑Thu Feb 28, 2019 6:04 am
Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.
This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.
-
- Posts: 200
- Joined: Fri Oct 12, 2012 12:48 pm
- Location: Seville (SPAIN)
- Full name: Javier Ros
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
I think Leela Ratio is very useful to get an idea of the relative speed of the GPU and the CPU.sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio
If you don't want to go through all of that, maybe this will give you a reference:
My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz
My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84
My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
But I don't understand why the Leela Ratio grows with LeelaFish.
Is taking into account the nodes of the auxiliary engine?
How?
-
- Posts: 1631
- Joined: Tue Aug 21, 2018 7:52 pm
- Full name: Dietrich Kappe
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
You shouldn’t use leelafish to compute the Leela Ratio. Use lc0 with 11248 vs sf9.Javier Ros wrote: ↑Thu Feb 28, 2019 9:35 amI think Leela Ratio is very useful to get an idea of the relative speed of the GPU and the CPU.sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio
If you don't want to go through all of that, maybe this will give you a reference:
My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz
My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84
My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
But I don't understand why the Leela Ratio grows with LeelaFish.
Is taking into account the nodes of the auxiliary engine?
How?
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
-
- Posts: 536
- Joined: Thu Mar 09, 2006 3:01 pm
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
I have found that when running on a GPU there is significant startup overhead and the NPS increases very rapidly until about 2 seconds (and slowly continues to increase after that until the nncache fills). This is on a 1070. Accordingly, although the games take more time, I no longer test at faster time controls than with an increment of 2 seconds. I have seen match results completely reversed at faster TCs vs 1+2, even well within the margin of error. Leela NPS is also much more variable depending on the position than the NPS for CPU engines. Of course, CPUs have practically no overhead, so the ultra-rapid TC testing is fine. Moreover, the Leela Ratio only applies to nets with a specific size/architecture, as has been pointed out.Laskos wrote: ↑Thu Feb 28, 2019 8:19 am I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.
This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.
Just noticed more info in this post:
viewtopic.php?f=7&t=67347
-
- Posts: 1480
- Joined: Thu Mar 09, 2006 5:33 am
Re: How to find your Leela Ratio - for Windows users w/ GPU(s)
I think the whole concept of "Leela Ratio" is useless. You'll take your best buy anyway.
Regards, Mike