ELO progression by year, period 2006-2021

[Rebel]

ELO progression by year, period 2006-2021

http://rebel13.nl/misc/stats.html

[Guenther]

ELO progression by year, period 2006-2021

http://rebel13.nl/misc/stats.html

I created this graph once (data from GURL3 2019-01) - was just too lazy to add more programs and their data points.
(ratings are spread more due to using ordo)

[lkaufman]

ELO progression by year, period 2006-2021

http://rebel13.nl/misc/stats.html

I created this graph once (data from GURL3 2019-01) - was just too lazy to add more programs and their data points.
(ratings are spread more due to using ordo)

The point about Ordo is becoming increasingly important. As I understand it, the extent of rating contraction using BayesElo (as CCRL does) when compared with Ordo (normal Elo formula) depends on the draw percentage, the contraction becoming more severe with more draws. Now that the top NNUE engines are drawing so many games with each other, the contraction due to BayesElo is quite severe, and the elo gains in the last couple years are much less with BayesElo than with Ordo. This wasn't a big deal a few years ago, maybe a 10% contraction, but now the gains from new top versions may be understated by a lot, possibly even cut in half (?) with BayesElo. Continued reliance on BayesElo will make it appear that progress in computer chess is faltering, whereas with the real Elo system (Ordo) it seems to be quite healthy.

[hgm]

What I would like to see is the progression of the Elo measured by playing against a pool of LC0-type engines (several different networks, and different time-odds factors). To test whether these Elo increases genuinely give you stronger play, or whether it is just a case of "the emperor's new Elo". It could be that the conventionally reported Elo just measures how well the engine does on just a single trick (tactics), which might not be so important in the grander scheme of things.

[lkaufman]

What I would like to see is the progression of the Elo measured by playing against a pool of LC0-type engines (several different networks, and different time-odds factors). To test whether these Elo increases genuinely give you stronger play, or whether it is just a case of "the emperor's new Elo". It could be that the conventionally reported Elo just measures how well the engine does on just a single trick (tactics), which might not be so important in the grander scheme of things.

I actually did run many tests like this vs. Lc0 several months ago with that same thought, I reported them in Talkchess. The basic conclusion was that rating differences as measured by normal engine testing (CCRL, CEGT, FastGM) shrink by about a third (to about 2/3 of the value on lists) or so when measured against Lc0. So the progress is real but overstated.

[hgm]

Ah, I missed that. Good job! Was this for NNUE versions against LC0 or for old-fashioned HCE?

[dangi12012]

I dont know why you would limit to 4 Cores.
The computational power for chess increases with the standard 1000 dollar machine too! (An interesting metric would also be Elo/Watt)

Thats the fun part. CPU gets better, RAM gets better, Compilers get better, Softwareengineers have better tools and also get better.
Thats a general rule in all of technology as it seems.

The infinite layers of dimishing S curves are in aggregate an exponential. You can tweak every layer of technology a few % more performance per year and that since 1950.

[lkaufman]

Ah, I missed that. Good job! Was this for NNUE versions against LC0 or for old-fashioned HCE?

Both. I ran programs ranging from about 2000 elo (CCRL) up to the latest SF and Dragon.

[Rebel]

ELO progression by year, period 2006-2021

http://rebel13.nl/misc/stats.html

I created this graph once (data from GURL3 2019-01) - was just too lazy to add more programs and their data points.
(ratings are spread more due to using ordo)

The point about Ordo is becoming increasingly important. As I understand it, the extent of rating contraction using BayesElo (as CCRL does) when compared with Ordo (normal Elo formula) depends on the draw percentage, the contraction becoming more severe with more draws. Now that the top NNUE engines are drawing so many games with each other, the contraction due to BayesElo is quite severe, and the elo gains in the last couple years are much less with BayesElo than with Ordo. This wasn't a big deal a few years ago, maybe a 10% contraction, but now the gains from new top versions may be understated by a lot, possibly even cut in half (?) with BayesElo. Continued reliance on BayesElo will make it appear that progress in computer chess is faltering, whereas with the real Elo system (Ordo) it seems to be quite healthy.

CEGT uses Ordo, if someone has an up-to-date 40/20 database I could do the same statistic for CEGT.

[CMCanavessi]

It is very weird that we don't see the NNUE jump in elo. It's almost linear in that progression.