CCRL 40/40, 40/4 and FRC lists updated (28th October 2017)

[Graham Banks]

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.

40/40 testing this week that I'm aware of will include (with live broadcast port for TLCV noted where applicable):

The Brawl Tournament (continuing 16001)
66th Amateur Series Division 7 Tournament (continuing 16002)
66th Amateur Series Division 8 Tournament (continuing 16053)
Monolith 0.3 64-bit Gauntlet (continuing 16063)
Isa 2.0.48 64-bit Gauntlet (continuing 16064)
RomiChess P3n 64-bit Gauntlet (starting Monday 16065)
Texel 1.07 64-bit Gauntlet (finishing Monday 16066)
Devel 1.8090 Gauntlet (continuing)
Amoeba 2.6 64-bit Gauntlet (finishing)
Andscacs 0.92 64-bit Gauntlet (finishing)
SugaR XPrO 1.3 64-bit Gauntlet (finishing)
The Baron 3.41 64-bit Gauntlet (starting)
Marvin 2.2.0 64-bit Gauntlet (starting)
Jumbo 0.5.3 64-bit Gauntlet (starting)
Houdini 6 64-bit 4CPU Gauntlet (continuing 16083)
Fire 6.1 64-bit 4CPU Gauntlet (continuing 16084)
Texel 1.07 64-bit 4CPU Gauntlet (being run by Charles)
Andscacs 0.92 64-bit 4CPU Gauntlet (being run by Charles)

40/4 testing since the last update has included:

Little Wing 0.3.0 64-bit
Texel 1.07 64-bit
Demolito 2017-08-26 64-bit
Magnum 4.0
Devel 1.8090
Arasan 20.2 64-bit
Monolith 0.3 64-bit
ChessBrainVB 3.50
Anatoli 0.35k
Fire 6.1 64-bit
tomitankChess 1.4 64-bit
Orion 0.4.0 64-bit
Counter 2.0.2 64-bit
Amoeba 2.6 64-bit
Zurichess Neuchatel 64-bit
RomiChess P3n 64-bit
ECE X3 64-bit
Galjoen 0.36.0.1 64-bit
Hoichess 0.21.0 64-bit
K2 0.87
Zevra 1.6 r512 64-bit
The Baron 3.41 64-bit
CT800 1.12 64-bit
Soldat 4.0 64-bit
Pupsi2 0.09
Drosophila 1.4 64-bit
RuyDos 1.0.27 64-bit
Oberon 0.04
FireFly 2.7.2 64-bit
Dimitri 3.81
Sergio's Swiss Tournaments
Fire 6.1 64-bit 4CPU

40/4 FRC testing since the last update has included:

Nemorino 3.04 64-bit

[Ras]

I'm surprised to see a drop of more than 40 Elo from CT800 1.11 to 1.12, that didn't show up during testing. Some questions:

1) do you use a standardised hash size (via the appropriate WB/UCI commands)? If no hash size is given, 1.12 will use the default of only 8 MB.

2) is the opening book handled in a different way for WB and UCI engines?

[Graham Banks]

I'm surprised to see a drop of more than 40 Elo from CT800 1.11 to 1.12, that didn't show up during testing. Some questions:

1) do you use a standardised hash size (via the appropriate WB/UCI commands)? If no hash size is given, 1.12 will use the default of only 8 MB.

Hash size is set in the UCI options when creating the engine.

2) is the opening book handled in a different way for WB and UCI engines?

I use UCI, WB and Java engines and they all use the book similarly.
We use generic books.

[Ras]

Hash size is set in the UCI options when creating the engine.

I see, but what is the value of the hash size (in MB)?

I use UCI, WB and Java engines and they all use the book similarly.
We use generic books.

Ok, then I guess WB engines get the GUI opening book moves transmitted in forced mode so that it's pretty much the same as for UCI, logically seen.

[Guenther]

I'm surprised to see a drop of more than 40 Elo from CT800 1.11 to 1.12, that didn't show up during testing. Some questions:

from CCRL 40/4 full list:

Code: Select all

CT800 1.11 64-bit       2240    +26     -26     45.8%   +32.7   19.7%   532     54.7%
CT800 1.12 64-bit       2194    +33     -33     43.2%   +49.8   19.7%   330     49.4%
NG-Play 9.86 64-bit     2187    +18     -18     50.5%   -3.1    22.2%   1079    53.5%

The error bars look still very high? Did you expect version 1.11 that much
above NG-Play? May be 1.11 was just a bit lucky so far and 1.12 will rise.

Of course a check of the games won't hurt.

Guenther

Edit: It seems the good result of 1.11 is primarily based on an extreme outsider result vs. Devel? (which produced strange outsider results so far en masse, if I see it right)

[Guenther]

http://www.computerchess.org.uk/ccrl/40 ... vel_1_8090

There seem to be extreme fluctuations in the results of Devel 1.8090? (especially very bad performances which seem beyond statistical noise?)
Also the author estimated it above 2600?
http://www.develchessengine.no/

A pity that the games download per player for 40/4 only contains stripped off pgn.
I checked one game against CT800 1.11 and Devel seemed to play from a certain point on only depth 5 moves (according to my analysis) and lost in 26 moves.

CT800 1.11 also won several other games up to move 30 against Devel!

[pgn][Event "CCRL 40/4"]
[Site "CCRL"]
[Date "2017.10.10"]
[Round "395.5.611"]
[White "Devel 1.8090"]
[Black "CT800 1.11 64-bit"]
[Result "0-1"]
[ECO "D43"]
[Opening "QGD semi-Slav"]
[PlyCount "51"]
[WhiteElo "2457"]
[BlackElo "2240"]

1. d4 Nf6 2. c4 c6 3. Nf3 d5 4. Nc3 e6 5. Bg5 h6 6. Bxf6 Qxf6 7. e3 Nd7 8. cxd5
exd5 9. Bd3 Be7 10. O-O O-O 11. Qe2 Re8 12. e4 dxe4 13. Qxe4 Nf8 14. Ne5 Bd6
15. Rae1 Be6 16. a3 Rad8 17. Re3 Bc7 18. Rf3 Qe7 19. Qe3 c5 20. Nb5 cxd4 21.
Qf4 Bb8 22. Rg3 f6 23. Qxh6 fxe5 24. Qh5 a6 25. Nxd4 exd4 26. f4 0-1[/pgn]

Guenther

[Modern Times]

http://www.computerchess.org.uk/ccrl/40 ... vel_1_8090

There seem to be extreme fluctuations in the results of Devel 1.8090? (especially very bad performances which seem beyond statistical noise?)

Yes that looks odd.

[Ras]

The error bars look still very high?

That's also in there, yes.

Did you expect version 1.11 that much above NG-Play?

No, and you're right, that's also astonishing. I remember 1.11 at 2228, and somehow, it has risen to 2240. The playing style has changed considerably from NG-Play to CT800, but not the raw strength. At least not against engines as CT800 is designed to play humans.

Of course a check of the games won't hurt.

I took a look at the current Amateur Division 8 where 1.12 only scored 0.5/4 in the first 4 rounds. Against Dorpsgek Dillinger, I've run extensive testing, and the usual score is around 50%. My direct 100 game match between 1.11 and 1.12 also ended up at around 50%.

The other spectacular loss with a king attack is to be expected, that has always been a weak side already in NG-Play, and if the CT doesn't get enough counterplay in the centre, it will be toast. Usually, the opening book avoids lines with blocked centre to cover this weakness, but if the internal book isn't used, that's the problem. The embedded version doesn't have this issue unless the opening book is disabled.

It seems the good result of 1.11 is primarily based on an extreme outsider result vs. Devel? (which produced strange outsider results so far en masse, if I see it right)

That would be a comforting explanation.. there's no big drop because there had been no big rise either.

[carldaman]

The other spectacular loss with a king attack is to be expected, that has always been a weak side already in NG-Play, and if the CT doesn't get enough counterplay in the centre, it will be toast. Usually, the opening book avoids lines with blocked centre to cover this weakness, but if the internal book isn't used, that's the problem. The embedded version doesn't have this issue unless the opening book is disabled.

Hi Rasmus,

Very interesting project you have there, developing an engine with anti-human play in mind. Have you considered making changes (such as king safety, pawn storms, etc) that improve its play in flank attacks and closed positions (even though that may cost Elo - after all it is the style that matters here)?

Avoiding closed games only skirts the issue and does not resolve it. A human player would greatly appreciate an engine opponent that can put up a good fight in closed positions, too.

Most developers stumble over the issue of Elo gains, leaving stylistic considerations on the back burner, with the end result being an engine that plays like a machine. If the goal is NOT to produce the strongest engine possible, then Elo should matter less than style, as long as the level of play remains adequately high (at least candidate master, let's say).

Interested to know your thoughts on this matter.
Cheers,
CL

[Ras]

Very interesting project you have there, developing an engine with anti-human play in mind.

The cornerstones are: 1) preventing closed centres, both via the opening book and via the asymmetric pawn eval that scores blocked centres negative for the computer side. 2) in the root position, the material balance is recorded, and if piece exchanges happen somewhere down the tree, it is scored slightly negative for the engine. The idea is to keep the board full so that tactical complications arise, and of course this isn't helpful against other engines.

Have you considered making changes (such as king safety, pawn storms, etc) that improve its play in flank attacks and closed positions

Yes, I've tried that for opposite castling, but I couldn't make it work. The only real effect was slowing down the search. Well, from what I've read here in the forum, even the top-engines don't perform too well in these positions, and e.g. Stockfish is made by much more capable chess programmers than I will ever be. That's how I got the idea to cover the weaknesses instead of resolving them. The engine knows what positions it cannot handle and tries to avoid them.

One thing that I think I might still add is better king safety. Things like how many squares around the king are attacked/defended by how many pieces, that could work.

then Elo should matter less than style, as long as the level of play remains adequately high (at least candidate master, let's say).

The PC version is mostly a testbed and also a nice goodie, but the "real" core is the embedded version running with a Cortex-M4 microcontroller and no operating system. Candidate master level is not quite in range with that hardware, but it should be a challenge for amateur players.

I think the good old Mephisto Vancouver is a bit stronger on average, but the CT800 is a dangerous opponent. Here a game at 30 minutes per side:
[pgn][White "CT800"]
[Black "Mephisto Vancouver 020"]
[Result "1-0"]

1. d4 d5 2. Nf3 Nf6 3. c4 dxc4 4. e3 e6 5. Bxc4 c5 6. O-O
a6 7. Qe2 b5 8. Bd3 cxd4 9. exd4 Bb7 10. a4 bxa4 11. Rxa4
Be7 12. Nc3 O-O 13. Be3 Qd6 14. Rfa1 Ng4 15. h3 Nf6 16. Bc4
Rc8 17. Ne5 Nfd7 18. Nxd7 Qxd7 19. Qg4 Rd8 20. Ne4 Qc6
21. Nc5 g6 22. Nxe6 fxe6 23. Bxe6+ Kg7 24. d5 Qd6 25. Qd4+
Bf6 26. Bh6+ Kxh6 27. Qxf6 Qh2+ 28. Kxh2 Rxd5 29. Bxd5 Bxd5
30. Rh4# 1-0[/pgn]