Leela Chess Zero and what is the "Best" Graphics Card.

[mwyoung]

Just completed a test match playing Lc0 Cudnn fp-16 vs Lc0 Cudnn. At blitz time controls. And the results have some implications if more testing confirms the results.

This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.

In terms of Price and performance, with Lc0 being your only consideration.

You would be better off buying a RTX 2060 ($350) or two.

RTX.jpg

Results.jpg

[Milos]

This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.

This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.

[mwyoung]

This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.

This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.

Results.jpg

[mwyoung]

This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.

This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.

Let see how Stockfish scales. Testing same time controls, Stockfish 090319 - 6 CPU, 6 Threads vs Stockfish 090319 - 16 CPU, 16 Threads.

[Laskos]

This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.

This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.

I am not sure of these results. NPS has to be checked literally in games at respective time control, etc. I can say that from my GTX 1060 to RTX 2070, which is 5-6x times faster at close to peak speed, at short TC I got several hundreds Elo points. But I too haven't checked the actual NPS in games. Conventional AB engines on CPU don't suffer such NPS variability with TC, backend, GPU unit. Also, I see now, only 30 games played.

[mwyoung]

This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.

This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.

I am not sure of these results. NPS has to be checked literally in games at respective time control, etc. I can say that from my GTX 1060 to RTX 2070, which is 5-6x times faster at close to peak speed, at short TC I got several hundreds Elo points. But I too haven't checked the actual NPS in games. Conventional AB engines on CPU don't suffer such NPS variability with TC, backend, GPU unit. Also, I see now, only 30 games played.

My guess here is that we are seeing the law of diminishing returns. It is not just about NPS differences. But where along the line of diminishing returns are you measuring the NPS difference.

[Modern Times]

This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.

This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.

Does not surprise me either.

[corres]

The Elo of an NN engine is determined mainly by the measure of NN and the quality of learning process.
Supposing optimal learning process the main factor remains the measure of NN. Inside of NN the main bearing of the knowledge of an NN engine is the value head. If the two GPU have such a high power what is enough to exhaust the knowledge stored in value head the play out will give the surplus ElO what is relative smaller than the Elo from value head. If the weaker GPU can not exhaust the knowledge of value head but the stronger GPU can do it, the difference in Elo will be greater for the benefit of the stronger GPU.
Now the used nets for Leela are relative small nets - even the 32000 series too.
If we will use larger net - let's say 60x256 - the Elo difference between RTX 2080 Ti and RTX 2060 will be higher.

[Laskos]

I dislike the sloppiness of the test and the sweeping conclusions derived from it. In their paper DeepMind indeed showed a bad scaling of A0 (worse than SF8), but on large hardware and going from rapid TC to LTC, not Blitz TC or shorter. And it might be due to not choosing correct MCTS parameters for longer TC. In TCEC, Lc0 used different Cpuct parameters from default values, suitable for longer TC TCEC uses, and the result was very good.

TCEC S14 and DeepMind paper had very similar hardware: both about 60-65M NPS for Stockfish and 60-65k NPS for Leela/A0 (in the middlegame).

We know the TCEC result of 50.5:49.5, or +3 Elo points for SF_dev against Lc0 T32930 at 120min + 15s time control, practically equal in strength. One can approximate this hardware * TC with 1.2 billion nodes/move for SF_dev against 1.2 million nodes/move for Lc0.

I used the same SF_dev, the same Lc0 T32930, the same TCEC openings with both engines at 400x times lesser node count per move: SF_dev at 3 million nodes/move against Lc0 at 3 thousand nodes/move.

The result is a sweeping 71:29 win for SF_dev:

Score of lc0_32930 vs SF_dev: +11 -53 =36 [0.290] 100
Elo difference: -155.54 +/- 56.99
Finished match

+155 Elo points for SF_dev at 400x less nodes per move than in TCEC (both engines). So, in 8-9 doublings of nodes per move, Lc0 gained 150 Elo points more than SF_dev, or more than 15 Elo points per doubling better scaling of Lc0 compared to the scaling of SF_dev. This result is unambiguous and outside error margins.


The scaling can be directly compared, although error margins in 100 games here are large when comparing the two outcomes:


Lc0 at 3k nodes/move versus Lc0 at 1k nodes/move (what the OP seem to try to imply, but IMO fails):

Score of lc0_32930_3k vs lc0_32930_1k: +53 -3 =44 [0.750] 100
Elo difference: 190.85 +/- 51.12
Finished match


SF_dev at 3 million nodes/move versus SF_dev at 1 million nodes/move:

Score of SF_dev_3M vs SF_dev_1M: +60 -4 =36 [0.780] 100
Elo difference: 219.87 +/- 57.61
Finished match


They are in the same ballpark as scaling goes, within even 1SD error margins.

It's important to keep in mind of what node count per move (or hardware x TC) one talks when comparing Lc0 scaling with a regular engine scaling, and here I used TCEC and DeepMind paper ratios, which are fairly balanced even on a domestic PC.

[JJJ]

So lczero is at same strenght than Stockfish only at very long time control with a big hardware. Your test with lower condition show the exact same result than mine , Stockfish being 155 elo stronger than the best net. And I have a 1060 GTI and 4 core only.