Chess.com 2018 computer chess championship

[Javier Ros]

This is the table after some cleaning up (not only the unterminated, but also the wrong game headers)
Game 1 which was still testing and was played with no opening moves at all still counts...
Andscacs instead of 1. e4 c5 2. Nf3 h6 3. c3 vs. SF played from the start position and the testing game
was elevated to a real stage 2 game later.

The normally reversed opening pair for that game:

[Event "TESTING for Stage 2"]
[Site "Chess.com"]
[Date "2018.09.17"]
[Round "?"]
[White "Andscacs 0.94"]
[Black "Stockfish 220818"]
[Result "0-1"]
[WhiteElo "3244"]
[BlackElo "3300"]
[ECO "C01"]
[TimeControl "900+5"]
[Time "10:23:16"]
[Termination "normal"]
[PlyCount "118"]

1. e4 {+0.20/26 21}
1... e6 {-0.15/33 50}
2. d4 {+0.29/29 29}
2... d5 {-0.11/32 8}
3. exd5 {+0.27/29 67}

[Event "CCCC 1: Rapid Rumble Stage 2"]
[Site "Chess.com"]
[Date "2018.09.18"]
[Round "?"]
[White "Stockfish 220818"]
[Black "Andscacs 0.94"]
[Result "1-0"]
[WhiteElo "3439"]
[BlackElo "3300"]
[ECO "B27"]
[TimeControl "900+5"]
[Time "08:39:57"]
[Termination "normal"]
[PlyCount "147"]

1. e4 c5 2. Nf3 h6 3. c3 {+0.77/32 50}

Code: Select all

CCCC 1: Rapid Rumble (15|5) Stage 2
Chess.com, 2018.09.17 - 2018.09.25
Average Rating: 3430  (Category 48)
                     Rtng    Score              1          2          3          4          5          6          7          8   Perf Chg
------------------------------------------------------------------------------------------------------------------------------------------
 1: Stockfish 220818 3439  51.0 / 70   XXXXXXXXXX =1=1=1=1=1 ===1=====1 ====1===== 1=1=1=1=11 11=11=1=1= =1=111===1 11=11111==   3596 +140  (+32 -0 =38)
 2: Houdini 6.03     3439  42.0 / 69   =0=0=0=0=0 XXXXXXXXXX 1==1==01=. ========1= 1==1=1==== 11=11=1==1 =1===1=1=1 1=1=1===1=   3501 +55  (+21 -6 =42)
 3: Komodo 2118.00   3439  38.5 / 69   ===0=====0 0==0==10=. XXXXXXXXXX ======0=01 1====1===1 1==101==== 1==1==1==1 1==1=1=1==   3465 +21  (+16 -8 =45)
 4: Lc0 17.11089     3439  37.5 / 70   ====0===== ========0= ======1=10 XXXXXXXXXX ========10 =111=0==== 11====1=== 10====101=   3451  +7  (+12 -7 =51)
 5: Ethereal 10.88   3404  31.0 / 70   0=0=0=0=00 0==0=0==== 0====0===0 ========01 XXXXXXXXXX ==0=0====1 ======1=== 1=1=10=11=   3386 -14  (+8 -16 =46)
 6: Fire 7.1         3404  29.0 / 70   00=00=0=0= 00=00=0==0 0==010==== =000=1==== ==1=1====0 XXXXXXXXXX ===1=1=1== =====0=1==   3364 -35  (+8 -20 =42)
 7: Booot 6.3.1      3439  25.5 / 70   =0=000===0 =0===0=0=0 0==0==0==0 00====0=== ======0=== ===0=0=0== XXXXXXXXXX ====1=====   3321 -119  (+1 -20 =49)
 8: Andscacs 0.94    3439  24.5 / 70   00=00000== 0=0=0===0= 0==0=0=0== 01====010= 0=0=01=00= =====1=0== ====0===== XXXXXXXXXX   3310 -126  (+4 -25 =41)
------------------------------------------------------------------------------------------------------------------------------------------
279 games: +79 -23 =177

I want to suggest the elo of LC0 somehow manages to match the elo of whatever it is playing against. Well, that's one "explanation" of its rather curious behaviour:

If one looks at the loss count of the second program, Houdini against the final ranking of its opponents, we get, as would be expected, an decreasing gradient: 5,1,0,0,0,0,0
Komodo gets: 2,3,2,0,1,0,0
Ethereal: 6,3,3,1,2,0,1
Fire: 6,6,3,3,1,0,1
Booot: 5,4,4,3,1,3,0
Andsacs: 7,4,4,3,5,1,1

Lc0 is different: 1,1,1,1,1,0,2, almost irrelevent who the opponent is, the loss rate remains almost constant.

Obviously, "non-losses", counting wins and draws together, shows the same pattern in reverse. Which suggests, well, to me, that LC0 doesn't really have an elo that can be mapped onto any particular opponent. It's not behaving itself properly according to the laws of elo ratings.

This constant loss rate is mainly due to the tactical weakness of lc0 that, when it appears against any alpha-beta program always leads to defeat.
On the other hand, this tactical weakness is practically irrelevant against humans, unable to detect it and take advantage of it in most cases.

So, perhaps we should say that lc0 is not behaving itself according to the laws of elo of the classical alpha-beta chess programs.

[jdart]

Are games from the completed parts of the tourney available somewhere?

--Jon

[Guenther]

Are games from the completed parts of the tourney available somewhere?

--Jon

Official CCCC archives stage 1+2
https://cccfiles.chess.com/archive/ccc1 ... /games.pgn
https://cccfiles.chess.com/archive/ccc1 ... /games.pgn

Inofficial current stage 3 (final) games
http://newman.chess.com/games.pgn

Note that stage 1 and stage 2 are not cleaned up files and adjudication results are missing for the
unterminated games. Also one game is missing per file.

[Guenther]

Code: Select all

   # PLAYER              :   RATING   ERROR  POINTS  PLAYED   (%)
   1 Stockfish 220818    :  3663.79   52.89    90.0     116  77.6
   2 Houdini 6.03        :  3584.63   41.68    79.0     115  68.7
   3 Komodo 2118.00      :  3565.40   46.24    76.5     115  66.5
   4 Lc0 17.11089        :  3513.30   47.34    68.5     115  59.6
   5 Fire 7.1            :  3465.80   43.57    62.0     116  53.4
   6 Ethereal 10.88      :  3459.26   41.68    61.0     116  52.6
   7 Booot 6.3.1         :  3416.64   44.68    54.5     116  47.0
   8 Andscacs 0.94       :  3400.08   44.39    52.0     116  44.8
   9 Shredder 13         :  3370.30   62.61    26.5      46  57.6
  10 Xiphos 0.3.17       :  3324.13   63.92    24.0      46  52.2
  11 Fritz 16.10         :  3324.13   60.07    24.0      46  52.2
  12 Texel 1.07          :  3305.80   63.57    23.0      46  50.0
  13 Laser 1.6           :  3289.81   60.37    21.0      44  47.7
  14 Pedone 1.8          :  3269.06   64.72    21.0      46  45.7
  15 Vajolet 2.6         :  3269.06   62.75    21.0      46  45.7
  16 Arasan CCCC-2018    :  3250.55   61.77    20.0      46  43.5
  17 Fizbo 1.9           :  3241.23   68.62    19.5      46  42.4
  18 Gull 3.syz          :  3222.42   70.70    18.5      46  40.2
  19 Wasp 3.25           :  3164.08   72.15    15.5      46  33.7
  20 Nemorino 5.00       :  3143.72   70.10    14.5      46  31.5
  21 Ivanhoe 999946h     :  3112.00   72.30    13.0      46  28.3
  22 Senpai 2.0          :  3042.52   80.28    10.0      46  21.7
  23 Nirvana 2.4         :  2989.46   90.39     8.0      46  17.4
  24 Crafty 25.2         :  2932.85  104.85     6.0      45  13.3

White advantage = 61.91 +/- 8.17
Draw rate (equal opponents) = 72.56 % +/- 2.60

(based on average rating of 3305 see above)

Actualised after 72 games in the final, with the same calculation method and assumptions:

Code: Select all

   # PLAYER              :   RATING   ERROR  POINTS  PLAYED   (%)
   1 Stockfish 220818    :  3657.20   41.36   132.0     188  70.2
   2 Houdini 6.03        :  3589.18   40.50   109.0     187  58.3
   3 Komodo 2118.00      :  3564.86   42.28    76.5     115  66.5
   4 Lc0 17.11089        :  3512.94   45.25    68.5     115  59.6
   5 Fire 7.1            :  3465.54   37.90    62.0     116  53.4
   6 Ethereal 10.88      :  3459.02   42.84    61.0     116  52.6
   7 Booot 6.3.1         :  3416.49   40.92    54.5     116  47.0
   8 Andscacs 0.94       :  3399.97   40.10    52.0     116  44.8
   9 Shredder 13         :  3370.24   62.11    26.5      46  57.6
  10 Xiphos 0.3.17       :  3324.15   63.11    24.0      46  52.2
  11 Fritz 16.10         :  3324.15   62.58    24.0      46  52.2
  12 Texel 1.07          :  3305.86   65.03    23.0      46  50.0
  13 Laser 1.6           :  3289.88   59.45    21.0      44  47.7
  14 Vajolet 2.6         :  3269.19   62.39    21.0      46  45.7
  15 Pedone 1.8          :  3269.19   60.36    21.0      46  45.7
  16 Arasan CCCC-2018    :  3250.71   65.01    20.0      46  43.5
  17 Fizbo 1.9           :  3241.40   63.69    19.5      46  42.4
  18 Gull 3.syz          :  3222.63   64.67    18.5      46  40.2
  19 Wasp 3.25           :  3164.39   67.80    15.5      46  33.7
  20 Nemorino 5.00       :  3144.07   65.87    14.5      46  31.5
  21 Ivanhoe 999946h     :  3112.39   77.37    13.0      46  28.3
  22 Senpai 2.0          :  3043.02   74.59    10.0      46  21.7
  23 Nirvana 2.4         :  2990.04   83.16     8.0      46  17.4
  24 Crafty 25.2         :  2933.47  102.07     6.0      45  13.3

White advantage = 59.74 +/- 7.87
Draw rate (equal opponents) = 74.87 % +/- 2.32

[corres]

I think the issue is that LC0 plays very well, except for a glaring hole that almost any opponent can capitalize on. Other engines have their own holes, but they are subtle, so weaker engines are less likely to discover them.

So we can say LC0 capitalize the weaker positional knowledge of AB engines and AB engines capitalize the weaker tactical knowledge of AB engines.
Where is the engine with positional knowledge of LC0 and tactical knowledge of AB engines?

[yanquis1972]

Not far off. Will continue to be shocked there isn’t an adequate if not necessarily legal jerryrigged hybrid floating around. If lc0 + 3rd party doesn’t do it my money would be on Komodo or Houdini making a splash commercially.

[ernest]

What was the final result of the CCCC final : 120-80, 119-81 ?
Could find it nowhere...

[Paloma]

ernest ,

i found this Link: https://www.chess.com/news/view/stockfi ... s-3rd-3887
120 : 80

[ernest]

ernest ,

i found this Link: https://www.chess.com/news/view/stockfi ... s-3rd-3887
120 : 80

Thanks ! How did you find that link ?
I had an older link, which never got updated...

[jp]

On chess.com: "CCC2 is on hold until we can figure out the Lc0 timeout issue, and why some engines are going to 1 thread."

What is the Lc0 timeout issue??