Seven years of AB chess engine progress

[Alayan]

To make this, I made use of FastGM bullet ratings, which have the best sample sizes of all rating lists along with great consistency in the methodology. I tracked down the release dates of engines in various ways, but the Chess Programming Wiki was a great help.

Of course, engines that struggle at bullet TC like Komodo and Andscacs look worse than they would on a longer TC comparison, but not that much.


(Click to display it bigger)

Zoomed-in version :


This list contains all engines stronger than Rybka 4.1 in FastGM bullet conditions except :
- Engines that only have a dev version stronger than it, but not an official release, or only a very recent release (e.g. Demolito, SlowChess)
- Engines that haven't a proper testing history because they mostly stayed private (Ginkgo, Chiron).
- Engines that are missing from the rating list (Defenchess)
- Some unoriginal derivatives (e.g. Equinox, Roc)
- Shredder

Results of SF dev-builds tests are not included in the graph, I included Xiphos 0.5.6 results however, and I'd include results for Laser 190324 if they were available.

I think there are a lot of interesting takeaways from this graph, but I'll refrain from commenting for now, as I'm interested to see what people here think of it.

[mmt]

I wonder where Houdini would be now with active development...

[Nay Lin Tun]

I wonder where Houdini would be now with active development...

-10 / -30 elo of latest SF.

[Ovyron]

Maybe not. It's possible Houdart quit because he set an ELO target of improvement for a Houdini 7 release, and could never get anywhere close to it, so he abandoned computer chess.

[Alayan]

I'd guess if he had kept working on it, he'd be somewhere between Stockfish 9 and Stockfish 10.

One thing I noticed while doing this is that while Stockfish is now alone at the top the level of second-tier engines has improve at a great pace over the last 2 years.

For example, in 2016, one engine crossed the 3000 fastGM bullet elo barrier, in 2017 another one, in 2018 four, and in 2019 three more (if we include Defenchess that would be four).

There has been a big gap between the "big 3" and the rest of the field for years, but this was not so much seen as stagnation when Komdo and/or Houdini made progress and there was competition at the top.

Ethereal and Xiphos are the first new engines in years to get that close to Stockfish ; Fizbo and Andy at their peak were farther away.

[Vinvin]

Nice graph !
I would like to do the same with top engines (top 3 or top 5) from 1990 to 2010.
Data are here : http://www.talkchess.com/forum3/viewtop ... 08#p701908
But hardware changed also in the meantime.

[Guenther]

To make this, I made use of FastGM bullet ratings, which have the best sample sizes of all rating lists along with great consistency in the methodology. I tracked down the release dates of engines in various ways, but the Chess Programming Wiki was a great help.

Of course, engines that struggle at bullet TC like Komodo and Andscacs look worse than they would on a longer TC comparison, but not that much.

...

This list contains all engines stronger than Rybka 4.1 in FastGM bullet conditions except :
- Engines that only have a dev version stronger than it, but not an official release, or only a very recent release (e.g. Demolito, SlowChess)
- Engines that haven't a proper testing history because they mostly stayed private (Ginkgo, Chiron).
- Engines that are missing from the rating list (Defenchess)
- Some unoriginal derivatives (e.g. Equinox, Roc)
- Shredder

Results of SF dev-builds tests are not included in the graph, I included Xiphos 0.5.6 results however, and I'd include results for Laser 190324 if they were available.

I think there are a lot of interesting takeaways from this graph, but I'll refrain from commenting for now, as I'm interested to see what people here think of it.

Did something similar for top programs only and one long time opensource program for comparison on GURL3 data a year ago.
May be I should update it again and add more programs, but this would need a recalculation of GURL or going back to plain CCRL data,
because I am only interested in a longer time span.

http://talkchess.com/forum3/viewtopic.p ... rl#p790342


(the spreadsheet gives name/release date/rating as mouseonover additional data, compared to the picture below)

[Alayan]

Nice graph !
I would like to do the same with top engines (top 3 or top 5) from 1990 to 2010.
Data are here : http://www.talkchess.com/forum3/viewtop ... 08#p701908
But hardware changed also in the meantime.

Getting good quality data with decent sample size is a serious problem for old engines. SSDF might be the best for pre-2005 data, but when I look at more recent engines, it's completely missing many of the strongest ones.

Variable hardware also complicate making a meaningful graph out of this data.

[Alayan]

Another visualization, with the elo difference to the strongest engine :

[the_real_greco]

My takeaway from this thread is Guenther's graph, which clearly shows that the Arasan era is nigh.