SPCC: Testrun of LC Zero Net 512 finished

[pohl4711]

Long thinking-time testrun of LC0 CUDA v0.16.0 Net 512 finished. A huge step forward (+34 Elo (!)) to the testrun before. But it is the first result with my parameter-change of the really awful time-management of LC0 (I changed "Scale thinking time" from 1.95 to 1.55). This improves the time-management a lot and should give a measureable Elo-gain. So it is not clear, how much Elo-gain the Net 512 has given...

The problem is, that the thinking-time ratio in my testruns (12'+5'' = 720''+5'' = 144:1) is very close to the TCEC ratio (30'+10'' = 1800''+10'' = 180:1). So, if the default setting of Leelas awful time-management will stay as it is now, a lot of Elo will be lost in TCEC!



http://www.sp-cc.de

(Perhaps you have to clear your browsercache or reload the website)


In the CEGT-ratinglist, LC0 would have passed Gull 3 (Elo 3078) with her performance in my latest testrun (Elo 3087) and would have climbed to position #11: http://www.cegt.net/40_40%20Rating%20Li ... liste.html

[jorose]

LC0 is certainly very strong, but is it not misleading to claim it is number 11 on a single core rating list when you let it use a GPU? I understand letting it use a GPU, but then the other engines shouldn't be restricted to a single core for a comparison. Think about it, essentially you are restricting the other engines to technology which was available 15 years ago (2Ghz single core) but are letting LC0 use a GPU.

[pohl4711]

First of all, my notebookCPU has 2.6 GHz, not 2. And the GPU is a mobileGPU GTX 950M, which is much slower than a normal, modern GPU (gtx 1070 or 1080). The Leela-Ratio on my machine is 1.37, which is pretty close to the Google AlphaZero vs. Stockfish match.
But it is definitly very hard to compare Leela on a GPU with normal engines. That is a real problem.

[carldaman]

A GPU standard could be useful for testers, in the same way of 1CPU, 4CPU, etc.
What is the most common GPU available at a reasonable price?
Maybe the community can agree on a common-denominator GPU for testing Leela-like entities ?!

[pohl4711]

The Leela-Ratio is a good possibility to measure the GPU/CPU(-cores) speed relationship on a machine.

https://blog.lczero.org/2018/07/04/leela-ratio

A value in the range 0.5 to 1.5 seems quite “fair“, when testing Leela against “normal“-CPU-Engines, I believe.

[Guenther]

The Leela-Ratio is a good possibility to measure the GPU/CPU(-cores) speed relationship on a machine.

https://blog.lczero.org/2018/07/04/leela-ratio

A value in the range 0.5 to 1.5 seems quite “fair“, when testing Leela against “normal“-CPU-Engines, I believe.

I think giving a bench and the used hardware should be sufficient for all users to judge on their own.
(the so called alphazero ratio is completely arbitrary IMHO and relies on data which cannot be verified and probably never will,
because google never published enough data, thus the L-ratio only uses averaged guess-numbers of a few games)

[Milos]

A GPU standard could be useful for testers, in the same way of 1CPU, 4CPU, etc.
What is the most common GPU available at a reasonable price?
Maybe the community can agree on a common-denominator GPU for testing Leela-like entities ?!

A common denominator should be cost. 1060 costs like very fast 6 cores or slow 8 cores CPU.
So basically running 1060 vs single core for fairness one should give at least 6x time odds to CPU.

[Nay Lin Tun]

Oh well, Leela fans already hyped that Leela is 4th best chess engine in the world.
Check out!!
https://groups.google.com/forum/#!topic ... ibaDc3dpeg

[JJJ]

Oh well, Leela fans already hyped that Leela is 4th best chess engine in the world.
Check out!!
https://groups.google.com/forum/#!topic ... ibaDc3dpeg

Maybe she is if she plays with a good GPU card against 1CPU engine. But of course it is not fair for the others engines.

[Nay Lin Tun]

Oh well, Leela fans already hyped that Leela is 4th best chess engine in the world.
Check out!!
https://groups.google.com/forum/#!topic ... ibaDc3dpeg

Maybe she is if she plays with a good GPU card against 1CPU engine. But of course it is not fair for the others engines.

If we assume 2x1080ti vs 43 cores as a fair match in TCEC, I am pretty sure that Leela will stand around 7th or 8th position in premier division, 4th is too optimistic though.
There were a series of leela games already on bonus matches and her performance was minimally lower than Andascacs.