ELO "inflation" due to hardware improvements

[Peteshnick]

I imagine there are posts all over this forum relating to this, but I was wondering if there is a simple functional form that can describe the relationship between ELO and hardware specs for a given engine. For example, perhaps it's ELO = a+b log(FLOPS), so one such situation would be a doubling of processor speed results in an increase in ELO by say, 50 points. Has such a functional form been estimated?

If so, it'd be interesting to take the SSDF list and "inflate" or "deflate" it so that all engines have scores based on similar hardware. For example, this would allow one to see what Rybka would be rated on hardware from the 1980's, or some early chess program's rating on today's hardware (maybe this has already been done?). Again, I don't think this is some novel idea, I'm just curious to hear a summary of what the thinking on this issue is.

[Le Fou numerique]

Hi Peter,

The inflation is mainly due to the addition of new engines or engines update (if the old version still in the rating list).

The calculation of the rating system Elo is not a measure of playing strength but a comparison of the relative playing strength.

So it's no important that Fishsteack has 4500 'coconuts Elo' but the difference between Fishsteack and Romsteack.

Regards,
Patrick

[George Tsavdaris]

I imagine there are posts all over this forum relating to this, but I was wondering if there is a simple functional form that can describe the relationship between ELO and hardware specs for a given engine. For example, perhaps it's ELO = a+b log(FLOPS), so one such situation would be a doubling of processor speed results in an increase in ELO by say, 50 points. Has such a functional form been estimated?

For Rybka 3 the following formula if my memory is correct holds true up to 16 cores:

ELO(n) : ELO gain for n cores relatively to 1 core.
ELO(n)=X·log2((20-3·log2(n))·n/20)

Where X is most probably around 70 and its definition is:
X: the gain of ELO when we double the speed.
Perhaps it is 50 as the 40/4 list of CCRL shows as an average, before some years(i don't know if this still holds true).

[Peteshnick]

Hi Peter,

The inflation is mainly due to the addition of new engines or engines update (if the old version still in the rating list).

The calculation of the rating system Elo is not a measure of playing strength but a comparison of the relative playing strength.

So it's no important that Fishsteack has 4500 'coconuts Elo' but the difference between Fishsteack and Romsteack.

Regards,
Patrick

Thanks for the response Patrick. I understand that Elo is not an absolute measure; the formula I gave as an example was a statistical model, not a literal description. The inflation I'm talking about comes from the fact that the old programs on the list were tested on much slower hardware than the new programs, giving the new programs an upward bias.

Best,
Pete

[Bo Persson]

If so, it'd be interesting to take the SSDF list and "inflate" or "deflate" it so that all engines have scores based on similar hardware. For example, this would allow one to see what Rybka would be rated on hardware from the 1980's, or some early chess program's rating on today's hardware (maybe this has already been done?). Again, I don't think this is some novel idea, I'm just curious to hear a summary of what the thinking on this issue is.

Such a comparison is not as fair as it might first sound. To make Rybka run on 1980's hardware, you would first have to remove some of the code, or it would just be way too slow.

Similarly, a 1980's chess program would run extremely fast on current hardware, but would become "unbalanced" in that the increase in nps would perhaps be better spent on an improved evaluation. Some of the things that were just too expensive then, are quite feasible on current hardware.

[Peteshnick]

Such a comparison is not as fair as it might first sound. To make Rybka run on 1980's hardware, you would first have to remove some of the code, or it would just be way too slow.

Similarly, a 1980's chess program would run extremely fast on current hardware, but would become "unbalanced" in that the increase in nps would perhaps be better spent on an improved evaluation. Some of the things that were just too expensive then, are quite feasible on current hardware.

Bo,

Those are interesting points. The latter, about the 1980's program, I definitely agree with, but I think it'd still be interesting (I'm sure many in this forum would be delighted to somehow have Chessmaster 2000 play in one of our engine tournaments!)

As to the first point, let's assume the computer is such that somehow the only bottleneck is the processor, running at 8mhz, which is what I recall my old 8088 speed demon used to manage. Wouldn't Rybka running slow amount to the equivalent of reaching the same depth as in say, a one minute game, but in classical time controls? That doesn't seem so unreasonable for Rybka. Come to think of it, that'd be an interesting way to do this "deflation".

For example, say you are interested in classical time controls. By giving Rybka a restriction of 40 moves in 1 minute, etc, you would essentially get the same result as running it on 120x slower hardware (ignoring pondering by either side for the moment). Right? Or are there large "fixed costs" involved that would preclude this?

[Edmund]

That is still not quite right, considering that not only the clock-speed of the processors has improved over the years.

For example in the early days of computerchess there wouldn't have been the storage resources for things like large material imbalance tables, movegeneration-lookuptables (magic-movegeneration) and endgametablebases. Another factor coming to my mind is that most current engines are 64bit based - that also is a significant speed boost which wouldn't have been available to early programs.

So it is not so easy to measure the sole improvement of algorithmic changes.