CCRL broken down by ECO code

[dkappe]

Looking at CCRL results, it seemed to me that their balance of openings by eco codes was a bit peculiar. I pulled down the games for sf12 1 CPU and ran them through ordo. Two thing that struck me were the over abundance of ECO A games (800) where SF12 really over performed, and how SF12 really underperformed in ECO B (one could really use some more games there).


660 B games (top 10), 77.4%:

Code: Select all

    # PLAYER                                :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 Stockfish 12 64-bit 8CPU              :      91     52    32.0      51  62.7      69   14   36    1  70.6
   2 Leelenstein 11.1 RTX2080              :      62    105     7.0      12  58.3      57    2   10    0  83.3
   3 Fat Fritz w471 RTX2080                :      46    129     4.5       8  56.2      57    3    3    2  37.5
   4 Lc0 0.25.1 t40-1541 RTX2080           :      31    103     6.5      12  54.2      72    3    7    2  58.3
   5 Stockfish 12 64-bit                   :       0   ----   511.0     660  77.4      93  376  244   27  37.7
   6 Dragon by Komodo 64-bit               :     -52     70    13.0      31  41.9      63    1   24    6  77.4
   7 Nemorino 6.00 64-bit                  :     -74     98     5.5      15  36.7      56    0   11    4  73.3
   8 Stoofvlees II a14 RTX2080             :     -83     86     7.0      18  38.9      54    1   12    5  66.7
   9 Komodo 14 64-bit 8CPU                 :     -94    186     1.5       4  37.5      68    0    3    1  75.0
  10 Allie 0.5 nn49921 RTX2080             :    -144     99     5.0      16  31.2      61    0   10    6  62.5

If people are picking SF because it is “the best,” they might be surprised to find out that that’s because it really kills in the Polish opening (83.8% in ECO A). I suppose it would still come out top of the list, but maybe lc0 should be your choice for analysis in the Sicilian?

I’d love to see a CCRL that allowed one to correct for ECO codes.

[dkappe]

Looking at non-dup games of the top 4 engines from vanilla CCRL, restricting to ECO B, we get a somewhat different picture.

Code: Select all

   # PLAYER                                :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L 
   1 Stoofvlees II a16 RTX2080             :    3747     50   152.0     260  58.5      53   71  162   27  62.3
   2 Lc0 0.26.3 t40-1541 RTX2080           :    3745     55   102.5     165  62.1      64   52  101   12  61.2
   3 Stockfish 12 64-bit                   :    3692   ----   510.5     659  77.5      95  376  243   27  37.6
   4 Dragon by Komodo 64-bit               :    3655     44   308.5     393  78.5      72  235  147   11  37.4

[peter]

If people are picking SF because it is “the best,” they might be surprised to find out that that’s because it really kills in the Polish opening (83.8% in ECO A). I suppose it would still come out top of the list, but maybe lc0 should be your choice for analysis in the Sicilian?

Interesting question, Dietrich.
But what I'd be interested in even more, as a consequence of your findings, are you already working on an pure Sicilian NNUE for SF?

Combined with a Queens Pawn opening net, it would be even more interesting as for eng-eng-matches with a book restricted to
1.d4 and to 1.e4 (played only passively) 1...c5 (played out acively), wouldn`t it?
I'd love to see SF play with a net trained on positions after these three half- moves and a corresponding book regards

[Graham Banks]

CCRL members use a variety of good quality opening books, and the games played are all valid.

The book that Ray used is based on 11,000 IM games and he sets the opening depth to no more than 8 moves, often less.

We do our best.

It would be interesting to see a similar ECO distribution breakdown of popular online databases like the Chessbase, ChessOK, PlaychessInfinity, ICCF, IECG, ICOfy, etc.

[Modern Times]

The book that Ray used is based on 11,000 IM games and he sets the opening depth to no more than 8 moves, often less.

None of those are my games. That is the blitz list.

[Graham Banks]

The book that Ray used is based on 11,000 IM games and he sets the opening depth to no more than 8 moves, often less.

None of those are my games. That is the blitz list.

You used a 4 moves per side book, each line being unique, so that's okay too.
Some lines may have transposed into certain positions eventually, but that's just the way it is.

[Modern Times]

The book that Ray used is based on 11,000 IM games and he sets the opening depth to no more than 8 moves, often less.

None of those are my games. That is the blitz list.

You used a 4 moves per side book, each line being unique, so that's okay too.
Some lines may have transposed into certain positions eventually, but that's just the way it is.

I repeat - the screen shots above are the Blitz list. I did not play any of those games. I do not contribute to the blitz list. Ask Gabor and Sergio.

[Graham Banks]

The book that Ray used is based on 11,000 IM games and he sets the opening depth to no more than 8 moves, often less.

None of those are my games. That is the blitz list.

You used a 4 moves per side book, each line being unique, so that's okay too.
Some lines may have transposed into certain positions eventually, but that's just the way it is.

I repeat - the screen shots above are the Blitz list. I did not play any of those games. I do not contribute to the blitz list. Ask Gabor and Sergio.

I know that.
You ran some games for 40/15, which are 100% valid.

[dkappe]

We do our best.

It would be interesting to see a similar ECO distribution breakdown of popular online databases like the Chessbase, ChessOK, PlaychessInfinity, ICCF, IECG, ICOfy, etc.

Graham,

I know there is a tendency on this forum to go to the wall and pillory people who should be praised and thanked instead. I definitely appreciate CCRL’s work over the years.

Certainly it’s become more difficult to differentiate engines with the huge number of draws. I really appreciate that you make the games public so that people like me can mine for interesting nuggets like this.

[Ferdy]

Looking at CCRL results, it seemed to me that their balance of openings by eco codes was a bit peculiar. I pulled down the games for sf12 1 CPU and ran them through ordo. Two thing that struck me were the over abundance of ECO A games (800) where SF12 really over performed, and how SF12 really underperformed in ECO B (one could really use some more games there).

Looks normal as A has a wider coverage of good openings.
B has pirc (e4 d6), alekhine (e4 Nf6) and modern (e4 g6) which probably does not deserve more coverage for engine vs engine matches.