Leela Chess Zero and what is the "Best" Graphics Card.

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
mwyoung
Posts: 1561
Joined: Wed May 12, 2010 8:00 pm

Leela Chess Zero and what is the "Best" Graphics Card.

Post by mwyoung » Sat Mar 09, 2019 9:47 pm

Just completed a test match playing Lc0 Cudnn fp-16 vs Lc0 Cudnn. At blitz time controls. And the results have some implications if more testing confirms the results.

This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.

In terms of Price and performance, with Lc0 being your only consideration.

You would be better off buying a RTX 2060 ($350) or two.
The attachment RTX.jpg is no longer available
RTX.jpg
RTX.jpg (110.85 KiB) Viewed 1355 times
Professing themselves to be wise, they became fools,
Take on me. foes 0

Milos
Posts: 3372
Joined: Wed Nov 25, 2009 12:47 am

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by Milos » Sat Mar 09, 2019 10:58 pm

mwyoung wrote:
Sat Mar 09, 2019 9:47 pm
This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.
This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.

mwyoung
Posts: 1561
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by mwyoung » Sat Mar 09, 2019 11:09 pm

Milos wrote:
Sat Mar 09, 2019 10:58 pm
mwyoung wrote:
Sat Mar 09, 2019 9:47 pm
This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.
This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.
Results.jpg
Results.jpg (79.18 KiB) Viewed 1323 times
Professing themselves to be wise, they became fools,
Take on me. foes 0

mwyoung
Posts: 1561
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by mwyoung » Sat Mar 09, 2019 11:31 pm

Milos wrote:
Sat Mar 09, 2019 10:58 pm
mwyoung wrote:
Sat Mar 09, 2019 9:47 pm
This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.
This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.
Let see how Stockfish scales. Testing same time controls, Stockfish 090319 - 6 CPU, 6 Threads vs Stockfish 090319 - 16 CPU, 16 Threads.
Professing themselves to be wise, they became fools,
Take on me. foes 0

User avatar
Laskos
Posts: 9242
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by Laskos » Sat Mar 09, 2019 11:36 pm

Milos wrote:
Sat Mar 09, 2019 10:58 pm
mwyoung wrote:
Sat Mar 09, 2019 9:47 pm
This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.
This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.
I am not sure of these results. NPS has to be checked literally in games at respective time control, etc. I can say that from my GTX 1060 to RTX 2070, which is 5-6x times faster at close to peak speed, at short TC I got several hundreds Elo points. But I too haven't checked the actual NPS in games. Conventional AB engines on CPU don't suffer such NPS variability with TC, backend, GPU unit. Also, I see now, only 30 games played.

mwyoung
Posts: 1561
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by mwyoung » Sat Mar 09, 2019 11:53 pm

Laskos wrote:
Sat Mar 09, 2019 11:36 pm
Milos wrote:
Sat Mar 09, 2019 10:58 pm
mwyoung wrote:
Sat Mar 09, 2019 9:47 pm
This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.
This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.
I am not sure of these results. NPS has to be checked literally in games at respective time control, etc. I can say that from my GTX 1060 to RTX 2070, which is 5-6x times faster at close to peak speed, at short TC I got several hundreds Elo points. But I too haven't checked the actual NPS in games. Conventional AB engines on CPU don't suffer such NPS variability with TC, backend, GPU unit. Also, I see now, only 30 games played.
My guess here is that we are seeing the law of diminishing returns. It is not just about NPS differences. But where along the line of diminishing returns are you measuring the NPS difference.
Professing themselves to be wise, they became fools,
Take on me. foes 0

Modern Times
Posts: 2346
Joined: Thu Jun 07, 2012 9:02 pm

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by Modern Times » Sun Mar 10, 2019 6:10 am

Milos wrote:
Sat Mar 09, 2019 10:58 pm
mwyoung wrote:
Sat Mar 09, 2019 9:47 pm
This match is a worse match up in terms of NPS (2.5x to 3x NPS advantage) then matching a RTX 2080 ti vs RTX 2060. This NPS advantage netted Lc0 with only a +47 elo advantage. And generally you will see the most advantage with shorter time controls.
This is totally expected coz Lc0 scaling is worse than any strong A/B engine. For example SF on 48 cores vs. SF on 16 cores is certainly more than 50 Elo difference (not to mention SF on 3 cores vs SF on 1 core) in TC you did the test.
Does not surprise me either.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

corres
Posts: 1407
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by corres » Sun Mar 10, 2019 10:14 am

The Elo of an NN engine is determined mainly by the measure of NN and the quality of learning process.
Supposing optimal learning process the main factor remains the measure of NN. Inside of NN the main bearing of the knowledge of an NN engine is the value head. If the two GPU have such a high power what is enough to exhaust the knowledge stored in value head the play out will give the surplus ElO what is relative smaller than the Elo from value head. If the weaker GPU can not exhaust the knowledge of value head but the stronger GPU can do it, the difference in Elo will be greater for the benefit of the stronger GPU.
Now the used nets for Leela are relative small nets - even the 32000 series too.
If we will use larger net - let's say 60x256 - the Elo difference between RTX 2080 Ti and RTX 2060 will be higher.

User avatar
Laskos
Posts: 9242
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by Laskos » Sun Mar 10, 2019 11:03 am

I dislike the sloppiness of the test and the sweeping conclusions derived from it. In their paper DeepMind indeed showed a bad scaling of A0 (worse than SF8), but on large hardware and going from rapid TC to LTC, not Blitz TC or shorter. And it might be due to not choosing correct MCTS parameters for longer TC. In TCEC, Lc0 used different Cpuct parameters from default values, suitable for longer TC TCEC uses, and the result was very good.

TCEC S14 and DeepMind paper had very similar hardware: both about 60-65M NPS for Stockfish and 60-65k NPS for Leela/A0 (in the middlegame).

We know the TCEC result of 50.5:49.5, or +3 Elo points for SF_dev against Lc0 T32930 at 120min + 15s time control, practically equal in strength. One can approximate this hardware * TC with 1.2 billion nodes/move for SF_dev against 1.2 million nodes/move for Lc0.

I used the same SF_dev, the same Lc0 T32930, the same TCEC openings with both engines at 400x times lesser node count per move: SF_dev at 3 million nodes/move against Lc0 at 3 thousand nodes/move.

The result is a sweeping 71:29 win for SF_dev:

Score of lc0_32930 vs SF_dev: +11 -53 =36 [0.290] 100
Elo difference: -155.54 +/- 56.99
Finished match

+155 Elo points for SF_dev at 400x less nodes per move than in TCEC (both engines). So, in 8-9 doublings of nodes per move, Lc0 gained 150 Elo points more than SF_dev, or more than 15 Elo points per doubling better scaling of Lc0 compared to the scaling of SF_dev. This result is unambiguous and outside error margins.


The scaling can be directly compared, although error margins in 100 games here are large when comparing the two outcomes:


Lc0 at 3k nodes/move versus Lc0 at 1k nodes/move (what the OP seem to try to imply, but IMO fails):

Score of lc0_32930_3k vs lc0_32930_1k: +53 -3 =44 [0.750] 100
Elo difference: 190.85 +/- 51.12
Finished match


SF_dev at 3 million nodes/move versus SF_dev at 1 million nodes/move:

Score of SF_dev_3M vs SF_dev_1M: +60 -4 =36 [0.780] 100
Elo difference: 219.87 +/- 57.61
Finished match


They are in the same ballpark as scaling goes, within even 1SD error margins.

It's important to keep in mind of what node count per move (or hardware x TC) one talks when comparing Lc0 scaling with a regular engine scaling, and here I used TCEC and DeepMind paper ratios, which are fairly balanced even on a domestic PC.

JJJ
Posts: 1277
Joined: Sat Apr 19, 2014 11:47 am

Re: Leela Chess Zero and what is the "Best" Graphics Card.

Post by JJJ » Sun Mar 10, 2019 3:35 pm

So lczero is at same strenght than Stockfish only at very long time control with a big hardware. Your test with lower condition show the exact same result than mine , Stockfish being 155 elo stronger than the best net. And I have a 1060 GTI and 4 core only.

Post Reply