Leela Chess 0.10 64-bit w323 Gauntlet for CCRL 40/40

[Nay Lin Tun]

These questions, designed to ridicule the ones quoted before, are unwarranted.

I thought they were delightful and meant no offense. All work and no play, etc.

For many years, the recurrent "is there an engine that can use my GPU?" popped in computer chess forums every now and then; now that we have the option to finally use that (previously untapped) HW resource, it'd be nice to see traditional rating lists reacting a little more promptly to the new situation, instead of just keeping testing it as a CPU bound engine.

For many years the recurrent "is there a bicycle that could use an internal combustion engine?" question, no doubt, popped up in many cyclists' minds, yet we do not see motorcycles taking part in Tour de France.

Ridicule (verb): to make fun of. What sort of delight did it bring you? The offence is in trying to be funny, when there was no other desire in the original questions, other than to obtain information.

The analogy you propose isn't appropriate. The first Tour de France took place in 1903, almost a decade after the mass production of the motorcycles, had already started. It was decided from the get go, to leave motorcycles out of the race. Besides, that's not my complain, my point is that a "motorcycle" is participating, but it's being pedaled.

Leela runs on GPU. If you let leela run on CPU, it is like you are asking an aeroplane to run on the ground. (your logic might be that both competitors must compete in exactly the same condition for fairness).

But people would be interested in seeing which one is the best(fastest), A formula 1 car running on the road vs an aeroplane flying in the air.

I mean it would be fair and square if you use $250 4 core CPU vs $250 GPU like 1060 GTX or similar hardware.

Of course , CCRL can stick on their own rule, but people always like "fair competition". (if people feel like that there is no fairness, they will follow a different rating list)

[IanO]

Just as the CCRL can have multiple entries for an engine (one core vs. four core) why not allow multiple entries for Leela (1xCPU, 4xCPU, GPU a, GPU b, etc)? It might be nice to annotate the entries with the approximate nps for each architecture. Bonus: it gives extra data points for the scalability of Leela's unique MCTS search strategy.

[Ozymandias]

it would be fair and square if you use $250 4 core CPU vs $250 GPU like 1060 GTX or similar hardware

Yes, from the user's POW, equal cost is fair, This is what it could look like in the future, once testers move to newer/faster GPUs:

Time control: Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz), about 15 minutes on a modern Intel CPU, about 15 minutes on a GeForce GTX 1050 2GB, about x minutes on a modern nVidia card.

Because Intel changed the i3 to 4 real cores, the GeForce GTX 1050 2GB is the closest to the price point of the i3 8100.

Just as the CCRL can have multiple entries for an engine (one core vs. four core) why not allow multiple entries for Leela (1xCPU, 4xCPU, GPU a, GPU b, etc)?

Considering how lame Leela CPU performance is, I don't see a need for any CPU testing at all, but I guess that's up to the individual testers, if they feel like it, no one can tell them not to. As for different GPU configurations, that wouldn't be reflected on the list, if you follow their current procedure. There would be a GPU in the description at the top of the page (the imaginary quoted text above) and then different cards would be used by different testers with different adjusted time controls, just as they do now. The only difference would be that engines running on a CPU would use one TC and GPU engines would use another one.

[Milos]

Considering how lame Leela CPU performance is, I don't see a need for any CPU testing at all, but I guess that's up to the individual testers, if they feel like it, no one can tell them not to. As for different GPU configurations, that wouldn't be reflected on the list, if you follow their current procedure. There would be a GPU in the description at the top of the page (the imaginary quoted text above) and then different cards would be used by different testers with different adjusted time controls, just as they do now. The only difference would be that engines running on a CPU would use one TC and GPU engines would use another one.

You can't just adjust TC for GPU because nps is a strong function of TC, search parameters and network used. So you'd just get one huge mess. A/B engine on CPU is a different world to LC0 on GPU. Apples and oranges simply. Assuming logic you use in one would work in the other is plain wrong.

[Ozymandias]

Considering how lame Leela CPU performance is, I don't see a need for any CPU testing at all, but I guess that's up to the individual testers, if they feel like it, no one can tell them not to. As for different GPU configurations, that wouldn't be reflected on the list, if you follow their current procedure. There would be a GPU in the description at the top of the page (the imaginary quoted text above) and then different cards would be used by different testers with different adjusted time controls, just as they do now. The only difference would be that engines running on a CPU would use one TC and GPU engines would use another one.

You can't just adjust TC for GPU because nps is a strong function of TC, search parameters and network used. So you'd just get one huge mess. A/B engine on CPU is a different world to LC0 on GPU. Apples and oranges simply. Assuming logic you use in one would work in the other is plain wrong.

The only logic being initially applied is that of cost. If a modern CPU that costs x is being used at given tc, then a similarly priced GPU should get a similar tc. If one tester uses a CPU with a different coefficient, they'll keep using whatever tc was applicable to their CPU, based on CPU to CPU calculations. If they also use a GPU, the engine running on it should have a different tc, based on a GPU to GPU comparison.

[Milos]

You can't just adjust TC for GPU because nps is a strong function of TC, search parameters and network used. So you'd just get one huge mess. A/B engine on CPU is a different world to LC0 on GPU. Apples and oranges simply. Assuming logic you use in one would work in the other is plain wrong.

The only logic being initially applied is that of cost. If a modern CPU that costs x is being used at given tc, then a similarly priced GPU should get a similar tc. If one tester uses a CPU with a different coefficient, they'll keep using whatever tc was applicable to their CPU, based on CPU to CPU calculations. If they also use a GPU, the engine running on it should have a different tc, based on a GPU to GPU comparison.

Again you either didn't read my comment or didn't understand it.
You can't easily make GPU to GPU comparison and adjust TC accordingly as it is now done for CPUs because

nps is a strong function of TC, search parameters and network used.

while on CPU nps scaling is much more linear.
Just look at that excel sheet of LC0 "benchmark" you have differences of more than 2x of LC0-cudnn running on the same GPU.

[Ozymandias]

You can't just adjust TC for GPU because nps is a strong function of TC, search parameters and network used. So you'd just get one huge mess. A/B engine on CPU is a different world to LC0 on GPU. Apples and oranges simply. Assuming logic you use in one would work in the other is plain wrong.

The only logic being initially applied is that of cost. If a modern CPU that costs x is being used at given tc, then a similarly priced GPU should get a similar tc. If one tester uses a CPU with a different coefficient, they'll keep using whatever tc was applicable to their CPU, based on CPU to CPU calculations. If they also use a GPU, the engine running on it should have a different tc, based on a GPU to GPU comparison.

Again you either didn't read my comment or didn't understand it.
You can't easily make GPU to GPU comparison and adjust TC accordingly as it is now done for CPUs because

nps is a strong function of TC, search parameters and network used.

while on CPU nps scaling is much more linear.
Just look at that excel sheet of LC0 "benchmark" you have differences of more than 2x of LC0-cudnn running on the same GPU.

I never said it'd be a simple nps-based comparison. I'm aware of those excel sheets, so obviously, the first thing to do is agree on what parameters should be used, in order to have some sort of standardised performance for GPUs. Only from that point forward, can you start to measure their relative speed.