Daniel Mehrmann wrote:I think 2950 CCRL elo is a way to high.
I tested Leela id 234 versus Naum 4.6, Deep Shredder 11 and ProDeo. After my results 10+10/game, she has a rating of around 2850 CCRL elo.
Depends on your graphics card
While he mentions timecontrol (which is necessary), the result also depends on his CPU speed. And GPU speed (as you pointed out). And also about which CCRL list we are talking about. I wonder why typically none bothers to mention those things.
As far as I can tell the only reasonable way of comparing lc0 strength to CCRL is the following:
1.) select CCRL list - for example 40/4
2.) select CPU engine against which you will be comparing lc0. This engine needs to play at conditions equivalent to CCRL 40/4 so you will adjust the time control for the CPU engine depending on your CPU speed - relative to CCRL reference CPU which is Athlon 64 X2 4600+ (2.4 GHz)
3.) Select time control for leela-zero - for example 40 moves in 2 minutes (it should be independent of CPU engine time control calibrated in step 2.) It's at this time control you will be comparing lc0's stength relative to CCRL 40/4.
4.) Run the match: lc0 at timecontrol decided in step 3. vs CPU engine at timecontrol calibrated in step 2.
5.) Caclulate lc0's performance from the match given the known elo of her opposition. Then you can say:
LC0's estimated CCRL 40/4 elo at time-control decided in step 3. and on my GPU = calculated lc0's performance.
So for example:
"LC0 at 40 moves per 2 minutes on GTX 1060 has estimated CCRL 40/4 elo = 2900."
While without the timecontrol at which LC0 was playing or without proper CPU engine timecontrol calibration or without the GPU speed it's quite meaningless I am afraid.
PS: you could make LC0 and the CPU engine run at the same time control.
Then you would skip step 2 and go directly into step 3. So you select timecontrol for LC0 and then you have to estimate the CPU engine's CCRL 40/4 rating based on how it's scaling at that given timecontrol on your CPU = less precise (different engines my scale differently, especially if it's many times more or less than would be the calibrated time control for the CPU engine on your CPU)
In other words the goal should be to determine how LC0 compares to CCRL 40/4 at various timecontrols on given GPU (and independent of CPU speed). So if you know that for example at 40/2 and GTX 1060 it's roughly 2900 you can then also say that on GTX 1080 it will be roughly 2900 at 40/1 etc.
Also for given time control and system with given CPU and given GPU you can then estimate how LC0 would compare (on that system) to some CPU engine of known CCRL 40/4 strength.