Seven years of AB chess engine progress

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Alayan
Posts: 130
Joined: Tue Nov 19, 2019 7:48 pm
Full name: Alayan Feh

Seven years of AB chess engine progress

Post by Alayan » Mon Feb 10, 2020 10:04 pm

To make this, I made use of FastGM bullet ratings, which have the best sample sizes of all rating lists along with great consistency in the methodology. I tracked down the release dates of engines in various ways, but the Chess Programming Wiki was a great help.

Of course, engines that struggle at bullet TC like Komodo and Andscacs look worse than they would on a longer TC comparison, but not that much.

Image
(Click to display it bigger)

Zoomed-in version :
Image

This list contains all engines stronger than Rybka 4.1 in FastGM bullet conditions except :
- Engines that only have a dev version stronger than it, but not an official release, or only a very recent release (e.g. Demolito, SlowChess)
- Engines that haven't a proper testing history because they mostly stayed private (Ginkgo, Chiron).
- Engines that are missing from the rating list (Defenchess)
- Some unoriginal derivatives (e.g. Equinox, Roc)
- Shredder

Results of SF dev-builds tests are not included in the graph, I included Xiphos 0.5.6 results however, and I'd include results for Laser 190324 if they were available.

I think there are a lot of interesting takeaways from this graph, but I'll refrain from commenting for now, as I'm interested to see what people here think of it.

mmt
Posts: 141
Joined: Sun Aug 25, 2019 6:33 am
Full name: Jim Fields

Re: Seven years of AB chess engine progress

Post by mmt » Tue Feb 11, 2020 2:51 am

I wonder where Houdini would be now with active development...

Nay Lin Tun
Posts: 552
Joined: Mon Jan 16, 2012 5:34 am

Re: Seven years of AB chess engine progress

Post by Nay Lin Tun » Tue Feb 11, 2020 4:10 am

mmt wrote:
Tue Feb 11, 2020 2:51 am
I wonder where Houdini would be now with active development...
-10 / -30 elo of latest SF. :mrgreen:

User avatar
Ovyron
Posts: 3384
Joined: Tue Jul 03, 2007 2:30 am

Re: Seven years of AB chess engine progress

Post by Ovyron » Tue Feb 11, 2020 7:48 am

Maybe not. It's possible Houdart quit because he set an ELO target of improvement for a Houdini 7 release, and could never get anywhere close to it, so he abandoned computer chess.

Alayan
Posts: 130
Joined: Tue Nov 19, 2019 7:48 pm
Full name: Alayan Feh

Re: Seven years of AB chess engine progress

Post by Alayan » Tue Feb 11, 2020 2:08 pm

I'd guess if he had kept working on it, he'd be somewhere between Stockfish 9 and Stockfish 10.

One thing I noticed while doing this is that while Stockfish is now alone at the top the level of second-tier engines has improve at a great pace over the last 2 years.

For example, in 2016, one engine crossed the 3000 fastGM bullet elo barrier, in 2017 another one, in 2018 four, and in 2019 three more (if we include Defenchess that would be four).

There has been a big gap between the "big 3" and the rest of the field for years, but this was not so much seen as stagnation when Komdo and/or Houdini made progress and there was competition at the top.

Ethereal and Xiphos are the first new engines in years to get that close to Stockfish ; Fizbo and Andy at their peak were farther away.

Vinvin
Posts: 4457
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: Seven years of AB chess engine progress

Post by Vinvin » Tue Feb 11, 2020 2:41 pm

Nice graph !
I would like to do the same with top engines (top 3 or top 5) from 1990 to 2010.
Data are here : http://www.talkchess.com/forum3/viewtop ... 08#p701908
But hardware changed also in the meantime.

Guenther
Posts: 3278
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: Seven years of AB chess engine progress

Post by Guenther » Tue Feb 11, 2020 2:59 pm

Alayan wrote:
Mon Feb 10, 2020 10:04 pm
To make this, I made use of FastGM bullet ratings, which have the best sample sizes of all rating lists along with great consistency in the methodology. I tracked down the release dates of engines in various ways, but the Chess Programming Wiki was a great help.

Of course, engines that struggle at bullet TC like Komodo and Andscacs look worse than they would on a longer TC comparison, but not that much.

...

This list contains all engines stronger than Rybka 4.1 in FastGM bullet conditions except :
- Engines that only have a dev version stronger than it, but not an official release, or only a very recent release (e.g. Demolito, SlowChess)
- Engines that haven't a proper testing history because they mostly stayed private (Ginkgo, Chiron).
- Engines that are missing from the rating list (Defenchess)
- Some unoriginal derivatives (e.g. Equinox, Roc)
- Shredder

Results of SF dev-builds tests are not included in the graph, I included Xiphos 0.5.6 results however, and I'd include results for Laser 190324 if they were available.

I think there are a lot of interesting takeaways from this graph, but I'll refrain from commenting for now, as I'm interested to see what people here think of it.
Did something similar for top programs only and one long time opensource program for comparison on GURL3 data a year ago.
May be I should update it again and add more programs, but this would need a recalculation of GURL or going back to plain CCRL data,
because I am only interested in a longer time span.

viewtopic.php?f=2&t=69546&p=785563&hilit=gurl#p790342

https://docs.google.com/spreadsheets/d/ ... 1866892607
(the spreadsheet gives name/release date/rating as mouseonover additional data, compared to the picture below)

Image
https://rwbc-chess.de/chronology.htm
--------------------------------------------------
The troll explosion at talkchess:
https://docs.google.com/spreadsheets/d/ ... KSptBx9AUs

Alayan
Posts: 130
Joined: Tue Nov 19, 2019 7:48 pm
Full name: Alayan Feh

Re: Seven years of AB chess engine progress

Post by Alayan » Wed Feb 12, 2020 12:15 am

Vinvin wrote:
Tue Feb 11, 2020 2:41 pm
Nice graph !
I would like to do the same with top engines (top 3 or top 5) from 1990 to 2010.
Data are here : http://www.talkchess.com/forum3/viewtop ... 08#p701908
But hardware changed also in the meantime.
Getting good quality data with decent sample size is a serious problem for old engines. SSDF might be the best for pre-2005 data, but when I look at more recent engines, it's completely missing many of the strongest ones.

Variable hardware also complicate making a meaningful graph out of this data.

Alayan
Posts: 130
Joined: Tue Nov 19, 2019 7:48 pm
Full name: Alayan Feh

Re: Seven years of AB chess engine progress

Post by Alayan » Thu Feb 13, 2020 10:48 pm

Another visualization, with the elo difference to the strongest engine :

Image

the_real_greco
Posts: 19
Joined: Thu Sep 19, 2019 3:55 am
Full name: Andy Erly

Re: Seven years of AB chess engine progress

Post by the_real_greco » Fri Feb 14, 2020 4:43 pm

My takeaway from this thread is Guenther's graph, which clearly shows that the Arasan era is nigh.

Post Reply