CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Graham Banks
Posts: 32493
Joined: Sun Feb 26, 2006 9:52 am
Location: Auckland, NZ

CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by Graham Banks » Sat May 04, 2019 11:22 pm

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.
Also, please note that the 40/40 and 40/4 lists are constructed from totally different databases and not comparable to each other.

Thanks to all of our currently active testers (Graham, Ray, Tirsa, Gabor and Sergio), plus to Chessdom.com for providing an online server to assist with resources.

40/40 testing this week that I'm aware of will include (with live broadcast port for TLCV noted where applicable):

74th Amateur Series Division 1 Tournament (continuing 16002)
74th Amateur Series Division 2 Tournament (starting today 16053)
Stalwarts v Newcomers Teams Match (finishing today 16001)
New Teams Match (starting today 16001)
Houdini 1.5a 64-bit Gauntlet (continuing 16063)
Topple 0.5.0 64-bit Gauntlet (finishing Wednesday 16064)
Komodo 11.3.1 64-bit Gauntlet (finishing Tuesday 16065)
Komodo 12.2.2 64-bit Gauntlet (starting Tuesday 16065)
Gull 3 64-bit Gauntlet (starting Wednesday 16066)
ShashChess 8.0 64-bit Gauntlet (continuing 16091)
Arasan 21.3 64-bit Gauntlet (continuing 16092)
Wasp 3.60 64-bit Gauntlet (continuing 16093)
RofChade 2.1 64-bit Gauntlet (continuing 16094)
The Sway 4CPU Tournament (continuing 16083)
ShashChess 8.0 64-bit 4CPU Gauntlet (continuing 16084)
Winter 0.5 64-bit 4CPU Gauntlet (being run by Tirsa)
Various 1CPU Gauntlets (being run by Tirsa)
Bagatur 1.7 64-bit Gauntlet (being run by Gabor)
Various 1CPU Tournaments (being run by Sergio)

40/4 testing since the last update report has included (thanks to Gabor, Sergio, Ray and Tirsa):

Cinnamon 2.2a 64-bit
SpaceDog 0.97.7 64-bit
RofChade 2.1 64-bit
Sapeli 1.02 64-bit RND
Raven 0.30
ChessBrainVB 3.72
Pupsi2 0.09 64-bit
RubiChess 1.4 64-bit
Dimitri 4.90
Tunguska 1.1 64-bit
Lc0 0.21.1 JH.T6.532 GPU
Laser 1.7 64-bit 4CPU
Minic 0.47 64-bit 4CPU

40/4 FRC testing since the last update report has included (thanks to Ray):

Sapeli 1.02 64-bit
My email addresses:
gbanksnz at gmail.com
gbanksnz at yahoo.co.nz

User avatar
xr_a_y
Posts: 534
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by xr_a_y » Sun May 05, 2019 6:33 am

Thanks for this 4cores Minic testing, very interesting results ! 2882 :D

Necromancer
Posts: 33
Joined: Wed Nov 23, 2016 12:30 am
Location: Brazil

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by Necromancer » Sun May 05, 2019 7:47 pm

Big thanks for testing Tunguska 1.1, got 2509 (previous version is 2134). The funny thing is that I fixed a lot of problem with my qsearch, which is giving me 55% against the github version :shock:
The 40/4 list still showing the 1.0 version though (and I'm Brazilian, not Spanish) :mrgreen:
The truth comes from inside.
https://github.com/fernandotenorio/Tunguska

flok
Posts: 98
Joined: Tue Jul 03, 2018 8:19 am
Full name: Folkert van Heusden
Contact:

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by flok » Mon May 06, 2019 12:00 pm

Hi Graham (and the other CCRL people!),

Is CCRL willing to test older versions as well?
I've put 2.0.0 on-line and more recently 2.0.5.
Now from CCRL I found that 2.0.5 plays much worse.
So I would like to put 2.0.1 on-line as well for testing.
My own testing showed that 2.0.5 was better than 2.0.0 but CCRL makes me doubt that.

https://vanheusden.com/Embla/files/Embl ... -win64.zip


regards
www.vanheusden.com: Micah / Embla / PuppetMaster / DeepBrutePos / Pos / Feeks

tpoppins
Posts: 918
Joined: Tue Nov 24, 2015 8:11 pm
Location: upstate

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by tpoppins » Fri May 10, 2019 12:09 am

I could do a blitz test for v2.0.1 if you like, Folkert, but the exec you provided crashes instantly on startup here.


Stack dump:

Code: Select all

Exception: STATUS_ILLEGAL_INSTRUCTION at rip=001004142A7
rax=0000000000000001 rbx=00000000FFFFC830 rcx=0000000600063ED0
rdx=00000000FFFFCE00 rsi=0000000600063ED0 rdi=0000000000000000
r8 =0000000600028490 r9 =0000000000000001 r10=0000000100000000
r11=000000010042E4C4 r12=00000000FFFFCC10 r13=00000000FFFFC7B0
r14=00000000FFFFC838 r15=00000000FFFFCB30
rbp=0000000600028488 rsp=00000000FFFFC710
program=C:\Program Files\ChessBase\Engines.x64\CCRL\d-j\embla-201\Embla.exe, pid 12788, thread main
cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B
Stack trace:
Frame        Function    Args
00600028488  001004142A7 (001800BAE1C, 00000000000, 7FEFC8A10AC, 006000640D8)
00600028488  0010042E48A (00180145E79, 001802FCBE8, 00000000030, 0010043B000)
000FFFFCCB0  0010043ACD2 (0010043AE20, 00000000001, 000FFFFCCB0, 000FFFFCC10)
000FFFFCCB0  0010043AD39 (001800D6214, 00000000000, 000FFFFCA00, 000FFFFCA00)
000FFFFCCB0  001800461A5 (001800D709E, 00000000000, 00180000000, 001802FCBE8)
000FFFFCCB0  00100438325 (00000000020, FF0700010302FF00, 00180047B51, 00000000000)
000FFFFCCB0  00180047BC2 (00000000000, 00000000000, 00000000000, 00000000000)
00000000000  00180045863 (00000000000, 00000000000, 00000000000, 00000000000)
000FFFFFFF0  00180045914 (00000000000, 00000000000, 00000000000, 00000000000)
End of stack trace
I've tested Embla since v1.0.1 and this is the first time I see it crash this way. This is a dual Xeon, Westmere-EX, 12-core running Win 7 x64 Pro.

You don't specify what kind of tests you run. If it's test suits like STS then v2.0.5 may well be better at them than v2.0.0 yet play worse. Similarly, if you run games at sudden-death and/or hyper-bullet TCs your results may bear no resemblance to ours.

As to your doubts, I hope a close look at the numbers can remove them easily enough. v2.0.0 is 51 Elo over v2.0.5 on the 40/40 list; the combined error margins are nearly half that at +/-26 Elo. The LOS is 100%.


BTW, what happened to Syzygy support? v2.0.5 doesn't offer the SyzygyPath UCI option anymore.

BTW2, the SMP functionality introduced in v2.0.0 (IIRC) ... errm ... doesn't appear to function as expected. Normally, one would expect an engine configured to use four threads use 400% of the CPU time used by the same engine configured to use one thread. Embla v2.0.0 and v2.0.5 here use 150%. It gets curiousier still -- here are some stats from the starting position:

Embla v2.0.5 one thread, 512 MB hash, 40/40:
depth=15, time=43s, 373 kN/s

Embla v2.0.5 4CPU, 1024 MB hash, 40/40:
depth=16, time=42s, 274 kN/s
Tirsa Poppins
CCRL

flok
Posts: 98
Joined: Tue Jul 03, 2018 8:19 am
Full name: Folkert van Heusden
Contact:

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by flok » Fri May 10, 2019 7:28 am

tpoppins wrote:
Fri May 10, 2019 12:09 am
I could do a blitz test for v2.0.1 if you like, Folkert, but the exec you provided crashes instantly on startup here.
That's odd, I did not change anything in the compilerflags and I'm not using any asm.
I've recompiled it without -mtune=native and it runs on my pc at work (core i5).
You don't specify what kind of tests you run. If it's test suits like STS then v2.0.5 may well be better at them than v2.0.0 yet play worse. Similarly, if you run games at sudden-death and/or hyper-bullet TCs your results may bear no resemblance to ours.
Usually I let it play > 10k games against dorpsgek. Dorpsgek is a bit stronger so it should be a good match.
Timecontrol is something like tc=40/40+0.1
BTW, what happened to Syzygy support? v2.0.5 doesn't offer the SyzygyPath UCI option anymore.
It is still in the soruce but I recompiled without it. Tests showed that embla plays worse with the syzygy-code in it.
BTW2, the SMP functionality introduced in v2.0.0 (IIRC) ... errm ... doesn't appear to function as expected. Normally, one would expect an engine configured to use four threads use 400% of the CPU time used by the same engine configured to use one thread.
Ah, maybe that's the problem why 2.0.5 playse worse. That is; under windows because under linux (just tried it) it uses ~1134% cpu on a 12 threads system.
Embla v2.0.0 and v2.0.5 here use 150%. It gets curiousier still -- here are some stats from the starting position:

Embla v2.0.5 one thread, 512 MB hash, 40/40:
depth=15, time=43s, 373 kN/s

Embla v2.0.5 4CPU, 1024 MB hash, 40/40:
depth=16, time=42s, 274 kN/s
This is really strange, I wonder if it has to do with the windows scheduler.
www.vanheusden.com: Micah / Embla / PuppetMaster / DeepBrutePos / Pos / Feeks

flok
Posts: 98
Joined: Tue Jul 03, 2018 8:19 am
Full name: Folkert van Heusden
Contact:

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by flok » Sun May 12, 2019 8:53 am

flok wrote:
Fri May 10, 2019 7:28 am
tpoppins wrote:
Fri May 10, 2019 12:09 am
I could do a blitz test for v2.0.1 if you like, Folkert, but the exec you provided crashes instantly on startup here.
That's odd, I did not change anything in the compilerflags and I'm not using any asm.
I've recompiled it without -mtune=native and it runs on my pc at work (core i5).
So there's a new version (to prevent misinterpretations of what I wrote :-]).
www.vanheusden.com: Micah / Embla / PuppetMaster / DeepBrutePos / Pos / Feeks

Modern Times
Posts: 2276
Joined: Thu Jun 07, 2012 9:02 pm

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by Modern Times » Sun May 19, 2019 7:53 am

flok wrote:
Mon May 06, 2019 12:00 pm
Hi Graham (and the other CCRL people!),

Is CCRL willing to test older versions as well?
I've put 2.0.0 on-line and more recently 2.0.5.
Now from CCRL I found that 2.0.5 plays much worse.
So I would like to put 2.0.1 on-line as well for testing.
My own testing showed that 2.0.5 was better than 2.0.0 but CCRL makes me doubt that.

https://vanheusden.com/Embla/files/Embl ... -win64.zip


regards

Tirsa was kind enough to run 2.0.1

http://ccrl.chessdom.com/ccrl/404/cgi/c ... +opponents
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

flok
Posts: 98
Joined: Tue Jul 03, 2018 8:19 am
Full name: Folkert van Heusden
Contact:

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by flok » Tue May 21, 2019 7:19 am

Modern Times wrote:
Sun May 19, 2019 7:53 am
Tirsa was kind enough to run 2.0.1

http://ccrl.chessdom.com/ccrl/404/cgi/c ... +opponents
Thanks @Tirsa!

Strange though how different the results are compared to my own tests.

My tests:

Code: Select all

Rank Name               Elo    +    - games score oppo. draws
   6 Embla2.0.0 4CPU   1968   25   25   400   54%  1942   56%
   9 Embla1.0.3        1942   18   18   804   48%  1952   53%
  10 Embla2.0.0        1937   25   25   404   49%  1942   50%
  12 tscp181           1724   11   11  4428   67%  1577   18%
CCRL:

Code: Select all

Embla 1.0.3 64-bit	2115	 
Embla 2.0.0 64-bit 4CPU	1927
I test like this:

Code: Select all

cutechess-cli \
        -engine cmd=/home/folkert/t/4860_lazy-smp_branch_2.0.0/Embla4.sh proto=uci name=Embla2.0.0-4c \
        -engine cmd=/home/folkert/t/4817_1.0.3/Embla proto=uci name=Embla1.0.3 \
        -concurrency 3 \
        -each dir=. tc=40/10+0.1 book=/home/folkert/dc-3200.bin option.Hash=256 -rounds 200 -games 2 -repeat -bookmode disk -recover -pgnout test.pgn -site "hier"
www.vanheusden.com: Micah / Embla / PuppetMaster / DeepBrutePos / Pos / Feeks

tpoppins
Posts: 918
Joined: Tue Nov 24, 2015 8:11 pm
Location: upstate

Re: CCRL 40/40, 40/4 and FRC lists updated (4th May 2019)

Post by tpoppins » Thu May 23, 2019 1:42 am

1) Embla's SMP implementation is broken, at least under Windoze
2) your TC is about 10x shorter than our 40/4


Here are two commented PGNs of my recent Embla tests:

Embla 2.0.1 64-bit 404 (FEOBOSv201a-c3-6m).7z
Embla 2.0.5 64-bit 4040 Gauntlet (IM_4mvs).7z

Embla reports some bizarre evals, by the way; e.g. a 106 cp advantage in a KRKR endgame:

Tirsa Poppins
CCRL

Post Reply