How do you know if your cpu is equal to your gpu?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: Harvey Williamson, bob, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
dkappe
Posts: 542
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How do you know if your cpu is equal to your gpu?

Post by dkappe » Wed Jan 29, 2020 5:09 am

You asked a controversial question: when are cpu and gpu “equal?” At best it’s difficult if not impossible to answer. How do you define “equal?”

A much more useful question would be “how do I know if matches run on two different machines between gpu and cpu engines are comparable?” A primitive approach has been to determine the nps ratio from the start position for sf8 and 11248 running on lc0 v18. The hypothesis is that if those ratios are the same, then the results would be comparable.

Not everyone has those engines available to them, so it might be both primitive and impractical.

jp
Posts: 1404
Joined: Mon Apr 23, 2018 5:54 am

Re: How do you know if your cpu is equal to your gpu?

Post by jp » Wed Jan 29, 2020 8:44 am

dkappe wrote:
Wed Jan 29, 2020 5:09 am
The hypothesis is that if those ratios are the same, then the results would be comparable.
That of course depends on the curves (performance vs. time or nodes per move) being the same shape, so again there's need for those curves.

And comparable results can still be dangerous, because invariably people will think of them as "comparably fair", not "equally unfair".

dkappe
Posts: 542
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How do you know if your cpu is equal to your gpu?

Post by dkappe » Wed Jan 29, 2020 3:37 pm

jp wrote:
Wed Jan 29, 2020 8:44 am

That of course depends on the curves (performance vs. time or nodes per move) being the same shape, so again there's need for those curves.
There was an attempt to create a better “leela ratio” maybe a month ago. It turned out not to be successful. Instead they decided to peg a ratio at that of the current TCEC hardware (which will famously change this season). I’ve tested the ratio hypothesis — that the same ratio on different hardware leads to similar results — on a 1060, a 1070 and a 2070. I’d need many more thousands of games, but I think it’s good enough to upgrade it from a hypothesis to a “thesis.”
jp wrote:
Wed Jan 29, 2020 8:44 am

And comparable results can still be dangerous, because invariably people will think of them as "comparably fair", not "equally unfair".
Meh. Now you’re getting into the region of religion. The point may be moot when CPU’s begin to pick up the characteristics of GPU’s.

jp
Posts: 1404
Joined: Mon Apr 23, 2018 5:54 am

Re: How do you know if your cpu is equal to your gpu?

Post by jp » Wed Jan 29, 2020 6:36 pm

dkappe wrote:
Wed Jan 29, 2020 3:37 pm
There was an attempt to create a better “leela ratio” maybe a month ago. It turned out not to be successful.
In what way was it unsuccessful?

Why hasn't there been more interest in Cscuile's ratio? Obviously it takes some effort to find (if it changes, and if it doesn't you could say that means no real progress), but where an engine goes flatline is what actually matters, not hardware.

jp wrote:
Sat Jan 25, 2020 9:01 am
jp wrote:
Fri Jan 24, 2020 8:53 am
Identify the points at which flatlining starts and use that ratio. That way, it's independent of hardware capabilities, etc.
https://groups.google.com/forum/#!msg/l ... oJrjDgBQAJ

dkappe
Posts: 542
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How do you know if your cpu is equal to your gpu?

Post by dkappe » Wed Jan 29, 2020 11:01 pm

jp wrote:
Wed Jan 29, 2020 6:36 pm
dkappe wrote:
Wed Jan 29, 2020 3:37 pm
There was an attempt to create a better “leela ratio” maybe a month ago. It turned out not to be successful.
In what way was it unsuccessful?
Too complex and unworkable vs a simple benchmark.
jp wrote:
Wed Jan 29, 2020 6:36 pm
Why hasn't there been more interest in Cscuile's ratio? Obviously it takes some effort to find (if it changes, and if it doesn't you could say that means no real progress), but where an engine goes flatline is what actually matters, not hardware.
Probably for the same reasons -- too complex and unworkable. Also, the concept it not fully fleshed out and there's no evidence to far that its useful for comparing results across hardware.

jp
Posts: 1404
Joined: Mon Apr 23, 2018 5:54 am

Re: How do you know if your cpu is equal to your gpu?

Post by jp » Thu Jan 30, 2020 6:27 am

dkappe wrote:
Wed Jan 29, 2020 11:01 pm
Also, the concept it not fully fleshed out and there's no evidence to far that its useful for comparing results across hardware.
If one ratio "works" (i.e. allows comparison) across different time controls and hardware, as you suggested before, then another more meaningful ratio should also work.


Leela maxing out its performance at 200000 npm in Cscuile's testing back then is not good.
Testing that just avoids this issue is also not good.

dkappe
Posts: 542
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How do you know if your cpu is equal to your gpu?

Post by dkappe » Thu Jan 30, 2020 4:55 pm

jp wrote:
Thu Jan 30, 2020 6:27 am
If one ratio "works" (i.e. allows comparison) across different time controls and hardware, as you suggested before, then another more meaningful ratio should also work.
More meaningful? Ratio = C * leela nps / sf nps. The only thing that changed was the C. Meh.

jp
Posts: 1404
Joined: Mon Apr 23, 2018 5:54 am

Re: How do you know if your cpu is equal to your gpu?

Post by jp » Thu Jan 30, 2020 8:50 pm

Yes, more meaningful. Cscuile's ratio is not arbitrary and has something to do with chess and computer chess. The other ratio is arbitrary and has almost nothing to do with chess or computer chess.

jp wrote:
Thu Jan 30, 2020 6:27 am
Leela maxing out its performance at 200000 npm in Cscuile's testing back then is not good.
Testing that just avoids this issue is also not good.

dkappe
Posts: 542
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How do you know if your cpu is equal to your gpu?

Post by dkappe » Thu Jan 30, 2020 9:05 pm

Well, I’m glad you have a favorite ratio. Cscuile has had so many ratios with so many formulas and acronyms that I couldn’t keep track of them all. But his heart was in the right place.

jp
Posts: 1404
Joined: Mon Apr 23, 2018 5:54 am

Re: How do you know if your cpu is equal to your gpu?

Post by jp » Thu Jan 30, 2020 9:15 pm

I don't know what all Cscuile's ideas were, so I can't judge them all, just this one.
Belittling him won't change the merits of this ratio, nor will it increase the merits of a largely meaningless one.

What matters most is the (shape of the) curves of performance vs. computation time.
The question is whether Leela (or other engines) still hits the wall at 200000npm.

Post Reply