How do you know if your cpu is equal to your gpu?

[Ovyron]

Getting an equal outcome (e.g. 50% score in a match) when the engines are not equal in strength is exactly what the OP does not want.

Getting a 50% performance against each other MEANS they're playing equal in strength. That's why if you entered their games into any software you'd get back exactly the same elo.

[jp]

He does not want that. He does not want different strength software to have hardware handicapping so the combination of sofware and hardware is the same ELO. e.g. he does not want to adjust the hardware so that Crafty 1.0 gets a 50% score against Lc0, or to adjust the hardware so that an untrained NN gets a 50% score against SF-dev.

It's possible that might give an entertaining match, but that's not what he wants. He wants some sort of "pure" comparison of software strength.

[Ovyron]

You adjust the hardware to emulate slower hardware and see how'd they perform to make the graph. After it is done if someone's CPU runs at max Stockfish 120000kn/s and their GPU runs at max Leela 20000 nps, you check their ratio on the graph and you know for who is it unfair, you don't need to do any handicapping yourself, it was done to create the graph.

[jp]

Please give an example of what you mean. Just make up numbers for imaginary engines 1 and 2. Tell me what exactly your graph is, and then what you do afterwards.

It sounds like you're saying something that is not correct, but maybe you mean something different.

[Ovyron]

Please give an example of what you mean. Just make up numbers for imaginary engines 1 and 2. Tell me what exactly your graph is, and then what you do afterwards.

Okay, so here's an entirely made up graph with fake data with the idea:


(click if it's too small)

So in this fantasy universe it was found that Stockfish with 50000 knodes/move plays with the same strength of Leela at 1 minute and 10 seconds per game. Let's say that if the MAXIMUM time a CPU of someone could provide gives 50000 knodes/move then if they get a 50% performance against a Leela at 1:10 per game, then it is fair.

If in YOUR computer you play 50000 knodes/move against Leela at 1:10/game and Stockfish wins, then that's unfair to Leela, so the ratio is below 1.0. If Leela wins, then that's unfair to Stockfish, so the ratio is above 1.0.

You should be able to play both at maximum strength in your hardware and figure out if it's unfair to one or the other by comparing it to the graph and if you end with a ratio of 1.0, then your cpu is equal to your gpu.

[jp]

If in YOUR computer you play 50000 knodes/move against Leela at 1:10/game and Stockfish wins, then that's unfair to Leela, so the ratio is below 1.0. If Leela wins, then that's unfair to Stockfish, so the ratio is above 1.0.

Thanks for the fantasy example. I don't think this has anything to do with the idea of "fairness". All you're doing is saying whether "your" (some fantasy user's) computer is offering weaker or stronger handicapping than the benchmark computer's (the computer that generated the handicapping of 1:10/g vs. 50k npm).

e.g. in the main horse race, the officials may handicap horse 1 with 10 stone weight more than horse 2. They may then get their hope of a dead heat between horse 1 and horse 2. That is not meant to be fair. It's a handicap to make the race the way they want it. In your horse race, your officials may give 5 or 15 stone weight handicap to horse 1. If they give 5 stone handicap and horse 1 wins, that is not unfair to horse 2. It's still unfair to horse 1. All of 5, 10, 15 stone are unfair to horse 1; 5 stone and 15 stone are just weaker and stronger handicapping of horse 1 than the main race's 10 stone. The fair race condition is zero weight handicap given to horse 1.

(Of course back to computers, it's disputed what is "zero weight handicap" for horses.)

[Ovyron]

All you're doing is saying whether "your" (some fantasy user's) computer is offering weaker or stronger handicapping than the benchmark computer's (the computer that generated the handicapping of 1:10/g vs. 50k npm).

You're just looking at the single point where the lines cross. For any average Stockfish knodes per second (say, higher than 50k) where will be some Leela time that plays at that strength. For every Leela that plays at a longer time control there will be some Stockfish knodes per second that plays at that strength. All those have a ratio of 1.0.

So, if your Stockfish runs at a MAXIMUM of something you could look at the graph and see what Leela time is needed to match it. If you match it and it's not enough for Stockfish to win, your ratio is below 1.0 and you know your GPU is better than your CPU. If it's not enough for Leela to win, your ratio is above 1.0 and you know your CPU is better than your GPU. If you get a 50 performance, you know they're equal.

Please note that for this comparison with the graph no handicap is happening, you run your strongest Stockfish and limit Leela by time per game, the graph is supposed to cover all cases, if not it wasn't built properly (say, some gal's Stockfish runs at some 20M knodes per move, the graph is useless to them as this isn't included. We'd need a taller graph.)

[jp]

Well, your 1.0 point is just like the weight handicap of horse 1 that makes the horses dead heat in a race. (Your first paragraph is just saying there's always some handicap of the horses that gets a dead heat.)

I don't know why you regard that as "equality" of the cpu and gpu. That's like saying the weights on the different horses are equal. The horses themselves are not equal unless they are identical twins trained in exactly the same way. Your second paragraph is like saying the 5 stone weight on horse 1 in my example (please re-read it) is actually a handicap of horse 2 because horse 1 then wins the race. Of course it isn't. The horses are not equal racers.

If you look for the condition where unequal horses dead heat a race, that is handicapping. Perhaps what you're thinking (in your third paragraph) is that it's not deliberate handicapping of the hardware. It doesn't have to be deliberate for one horse to be handicapped. We are trying to find out how much your hardware "unintentionally handicaps" the horses.

[Ovyron]

Okay, so let's talk about horses. Except, horses can be compared 1 to 1, because they're horses, just like one CPU and another CPU can be compared, or a GPU and another GPU can be compared.

Instead, let's talk about... I don't know, the cheetah and the black marlin? One is the fastest at running, and the other is the fastest at swimming. A black marlin can't run just like Stockfish can't use a GPU, and a cheetah would be a very slow swimmer just like CPU Leela is very slow compared to the GPU one.

You want to know what would you need to do to have a fair race between them. To make the analogy match, let's suppose both animals have very different acceleration (the nodes it can search for the black marlin, or the time needed to run for the cheetah), and thus you can "fix" the match by choosing the distance they have to travel. The distance point at which both animals have fully accelerated is their Ratio 1.0. A shorter distance is a Ratio <1.0 which means the black marlin doesn't have time to accelerate and the cheetah will always win, a longer distance is a Ratio >1.0 and it means the marlin catches up with the cheetah and always wins. This is like ensuring Stockfish always wins with more knodes/move or Leela always wins with more time/game.

You get their ratio and produce a graph.

So you have a cheetah and a black marlin at home, and you wish to know which one is faster, but you want a fair match. So first you run a race to the known distance in the graph that matches your black marlin's distance with the one on there, and see who wins. If the black marlin wins this is equivalent to the point in the graph with Ratio >1.0, so you know your CPU is faster than your GPU. If the cheeta wins this is equivalent to the point in the graph with Ratio <1.0, so you know your GPU is better.

Once you know this, you put weights in the animal that wins, so they perform with a Ratio 1.0 (they reach the same distance at the same time) so you are able to know by how much one is faster than the other (say, the black marlin wins, so you know your CPU is better, if to make it and the cheetah reach the same distance at the same time, you'd need to put enough weight on the black marlin to cut its speed in half, this way you know its ratio is 2.0. If you need to put weights on the cheetah to make her reach that point at the same time with half her speed, that's a Ratio 0.5.)

[Guenther]

Well, your 1.0 point is just like the weight handicap of horse 1 that makes the horses dead heat in a race. (Your first paragraph is just saying there's always some handicap of the horses that gets a dead heat.)

I don't know why you regard that as "equality" of the cpu and gpu. That's like saying the weights on the different horses are equal. The horses themselves are not equal unless they are identical twins trained in exactly the same way. Your second paragraph is like saying the 5 stone weight on horse 1 in my example (please re-read it) is actually a handicap of horse 2 because horse 1 then wins the race. Of course it isn't. The horses are not equal racers.

If you look for the condition where unequal horses dead heat a race, that is handicapping. Perhaps what you're thinking (in your third paragraph) is that it's not deliberate handicapping of the hardware. It doesn't have to be deliberate for one horse to be handicapped. We are trying to find out how much your hardware "unintentionally handicaps" the horses.

Why don't you use a chat channel? This is already the third or fourth time in a few weeks you both have a stubborn 'dispute' (childish version of it),
which is completely boring and useless for the whole talkchess community...
Of course 'ovyron' aka X aka Y likes it, because he is addicted for posting emptiness and vanity in its most shining form, you should be wiser at least.