Arasan Test Suite 12 - results - 16 engines

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

Martin Thoresen
Posts: 1833
Joined: Thu Jun 22, 2006 12:07 am

Arasan Test Suite 12 - results - 16 engines

Post by Martin Thoresen »

All engines are configured with 1024 MB Hash.
All engines are 64-bit.
1 Thread only (Single CPU).
Each position - 10 Seconds.
CPU: Core i7 920 @ 4101 MHz
Test suite link: http://www.arasanchess.org/testsuite.shtml

Results:
142 of 215 matching moves - Spark 0.4 x64
138 of 215 matching moves - Rybka 3 x64
135 of 215 matching moves - Houdini 1.0 x64
130 of 215 matching moves - Robbolito 009 x64
129 of 215 matching moves - Fire 1.2 x64
126 of 215 matching moves - Naum 4.2 x64
111 of 215 matching moves - Stockfish 1.7.1 x64
90 of 215 matching moves - Critter 0.60 x64
78 of 215 matching moves - Zappa Mexico II x64
65 of 215 matching moves - Crafty 23.2 x64
61 of 215 matching moves - Komodo 1.2 x64
53 of 215 matching moves - Onno 1.2.70 x64
52 of 215 matching moves - Shredder 12 x64
47 of 215 matching moves - Protector 1.3.4 x64
41 of 215 matching moves - Daydreamer 1.75 x64
30 of 215 matching moves - Gaviota 0.74.41 x64

Download:
Complete .zip file with all the data.
Top
Martin Thoresen
Posts: 1833
Joined: Thu Jun 22, 2006 12:07 am

Re: Arasan Test Suite 12 - results - 16 engines

Post by Martin Thoresen »

New engines added in a second batch:
Hiarcs 13.1, Toga II 3.2.1SE, Sjeng WC2008 x64, Loop 2007 x64, Scorpio 2.5 x64, Tornado 3.51a x64, N2 0.4 x64, Philou 3.1.4.1 and Brutus 8.05 x64.

Updated Results:
142 of 215 matching moves - Spark 0.4 x64
138 of 215 matching moves - Rybka 3 x64
135 of 215 matching moves - Houdini 1.0 x64
130 of 215 matching moves - Robbolito 009 x64
129 of 215 matching moves - Fire 1.2 x64
126 of 215 matching moves - Naum 4.2 x64
111 of 215 matching moves - Stockfish 1.7.1 x64
101 of 215 matching moves - Hiarcs 13.1
90 of 215 matching moves - Critter 0.60 x64
78 of 215 matching moves - Zappa Mexico II x64
75 of 215 matching moves - Toga II 3.2.1SE
75 of 215 matching moves - Sjeng WC2008 x64
66 of 215 matching moves - Loop 2007 x64
65 of 215 matching moves - Crafty 23.2 x64
61 of 215 matching moves - Komodo 1.2 x64
53 of 215 matching moves - Onno 1.2.70 x64
52 of 215 matching moves - Shredder 12 x64
50 of 215 matching moves - Brutus 8.05 x64
47 of 215 matching moves - Protector 1.3.4 x64
43 of 215 matching moves - Scorpio 2.5 x64
41 of 215 matching moves - Tornado 3.51a x64
41 of 215 matching moves - Daydreamer 1.75 x64
36 of 215 matching moves - N2 0.4 x64
30 of 215 matching moves - Gaviota 0.74.41 x64
16 of 215 matching moves - Philou 3.1.4.1

Download:
Complete .zip file with all the data from the first batch.
Complete .zip file with all the data from the second batch.
Top
jpqy
Posts: 550
Joined: Thu Apr 24, 2008 9:31 am
Location: Belgium

Re: Arasan Test Suite 12 - results - 16 engines

Post by jpqy »

Very interesting testing results Martin..thanks!
Top
jdart
Posts: 4366
Joined: Fri Mar 10, 2006 5:23 am
Location: http://www.arasanchess.org

Re: Arasan Test Suite 12 - results - 16 engines

Post by jdart »

FYI I have some additional positions I have been considering adding to the suite. Some were posted in this forum - some are from recent games:

6rk/p1r2qp1/1n3p1p/3pp3/p7/B1PP1Q2/B1P4R/6RK b - - bm e4; id "arasan2.216"; c0 "WATCHMEN-Arasan, ICC 2010";
6k1/1p1q4/p1rP2p1/5p1p/5Q2/1P5P/5PP1/3R2K1 w - h6 bm g4; id "arasan2.217"; c0 "Shredder-Arasan, freechess.org 2010";
1qrrbbk1/1p1nnppp/p3p3/4P3/2P5/1PN1N3/PB2Q1PP/1B2RR1K w - - bm Bxh7; id "arasan2.218"; c0 "Georgiev-Ionescu, UAE 1986";
r4rk1/pp5p/2p1PppB/3p4/2P2Q2/P1q2P2/6PP/n4R1K w - - bm Qd6; id "arasan2.219"; c0 "Nevio-Peunariemi, corr 2002";
r1r2nk1/1p1b1pbp/p4np1/P1Np2q1/1P1Pp3/B1N1P3/Q3BPPP/RR4K1 w - - bm N3a4; id "arasan2.220"; c0 "Stockfish 1.7.1-Deep Sjeng WC2008, CCRL Gauntlet 2010";
2q4k/pp4pp/2p1bpn1/P1P5/4r2r/2Q4P/2PBNR1K/1R3B2 w - - bm a6; id "arasan2.221"; c0 "Orcrist-Arasan, ICC 2010";
3r4/2kP1p2/p3r1p1/1p4Kp/2p5/P4P2/1P1R2P1/3R4 b - - bm Re7; id "arasan2.222"; c0 "Arasan-Orcrist, ICC 2010";
3r4/2q5/5pk1/p3n1p1/N3Pp1p/1PPr1P1P/2Q1R1P1/5R1K b - - bm g4; id "arasan2.223"; c0 "Critter 0.60-Deep Fritz 12, Martin Thoresen 2010";
r1b2r1k/ppqn2bp/3p2p1/3Q1p2/8/BP1BR2P/P1PP1PP1/4R1K1 w - - bm Re8; id "arasan2.224"; c0 "Nezhmetdinov-Kotkov, 17th RSFSR Ch 1957";
r1b4k/pp2qnr1/3p4/NP1Qp2p/P1B1PpP1/5P2/6P1/3R1RK1 b - - bm Qh4; id "arasan2.225"; c0 "Hovde-Timmerman, 25 Years Jubilee cr 1991";
Top
jarkkop
Posts: 198
Joined: Thu Mar 09, 2006 2:44 am
Location: Helsinki, Finland

Re: Arasan Test Suite 12 - results - 16 engines

Post by jarkkop »

Hi,

Can you say your computers Fritzmark so I can scale the time for my processor?

- Jarkko
Top
Martin Thoresen
Posts: 1833
Joined: Thu Jun 22, 2006 12:07 am

Re: Arasan Test Suite 12 - results - 16 engines

Post by Martin Thoresen »

jarkkop wrote:Hi,

Can you say your computers Fritzmark so I can scale the time for my processor?

- Jarkko
Hello,

It's 3097 with 1 thread and 12243 with 4 threads.
Top
metax
Posts: 344
Joined: Wed Sep 23, 2009 5:56 pm
Location: Germany

Re: Arasan Test Suite 12 - results - 16 engines

Post by metax »

ChessMind 0.82 got 29 of 215 right at 23sec/move (which is scaled according to the Fritz benchmark), which is almost as much as Gaviota had! :shock: :)
Top
Martin Thoresen
Posts: 1833
Joined: Thu Jun 22, 2006 12:07 am

Re: Arasan Test Suite 12 - results - 16 engines

Post by Martin Thoresen »

jdart wrote:FYI I have some additional positions I have been considering adding to the suite. Some were posted in this forum - some are from recent games:

6rk/p1r2qp1/1n3p1p/3pp3/p7/B1PP1Q2/B1P4R/6RK b - - bm e4; id "arasan2.216"; c0 "WATCHMEN-Arasan, ICC 2010";
6k1/1p1q4/p1rP2p1/5p1p/5Q2/1P5P/5PP1/3R2K1 w - h6 bm g4; id "arasan2.217"; c0 "Shredder-Arasan, freechess.org 2010";
1qrrbbk1/1p1nnppp/p3p3/4P3/2P5/1PN1N3/PB2Q1PP/1B2RR1K w - - bm Bxh7; id "arasan2.218"; c0 "Georgiev-Ionescu, UAE 1986";
r4rk1/pp5p/2p1PppB/3p4/2P2Q2/P1q2P2/6PP/n4R1K w - - bm Qd6; id "arasan2.219"; c0 "Nevio-Peunariemi, corr 2002";
r1r2nk1/1p1b1pbp/p4np1/P1Np2q1/1P1Pp3/B1N1P3/Q3BPPP/RR4K1 w - - bm N3a4; id "arasan2.220"; c0 "Stockfish 1.7.1-Deep Sjeng WC2008, CCRL Gauntlet 2010";
2q4k/pp4pp/2p1bpn1/P1P5/4r2r/2Q4P/2PBNR1K/1R3B2 w - - bm a6; id "arasan2.221"; c0 "Orcrist-Arasan, ICC 2010";
3r4/2kP1p2/p3r1p1/1p4Kp/2p5/P4P2/1P1R2P1/3R4 b - - bm Re7; id "arasan2.222"; c0 "Arasan-Orcrist, ICC 2010";
3r4/2q5/5pk1/p3n1p1/N3Pp1p/1PPr1P1P/2Q1R1P1/5R1K b - - bm g4; id "arasan2.223"; c0 "Critter 0.60-Deep Fritz 12, Martin Thoresen 2010";
r1b2r1k/ppqn2bp/3p2p1/3Q1p2/8/BP1BR2P/P1PP1PP1/4R1K1 w - - bm Re8; id "arasan2.224"; c0 "Nezhmetdinov-Kotkov, 17th RSFSR Ch 1957";
r1b4k/pp2qnr1/3p4/NP1Qp2p/P1B1PpP1/5P2/6P1/3R1RK1 b - - bm Qh4; id "arasan2.225"; c0 "Hovde-Timmerman, 25 Years Jubilee cr 1991";
Great Jon, let us know if you do. :)
Top
Carlos777
Posts: 1730
Joined: Sun Dec 13, 2009 6:09 pm

Re: Arasan Test Suite 12 - results - 16 engines

Post by Carlos777 »

Hi Martin,

Thanks for sharing these results. What kind of suite is this? Tactical? (I am too lazy to check all the positions :wink: )

Regards,
Carlos
Top
Martin Thoresen
Posts: 1833
Joined: Thu Jun 22, 2006 12:07 am

Re: Arasan Test Suite 12 - results - 16 engines

Post by Martin Thoresen »

Carlos777 wrote:Hi Martin,

Thanks for sharing these results. What kind of suite is this? Tactical? (I am too lazy to check all the positions :wink: )

Regards,
Carlos
Hello Carlos!

I guess it's mostly tactical after following most of the positions. And I guess that's why Spark does so well here.

New result just in:

116 of 215 matching moves - Critter 0.70 x64

That is 26 matching moves better than previous version, 0.60 !
This means that the list now looks like this:

Updated Results:
142 of 215 matching moves - Spark 0.4 x64
138 of 215 matching moves - Rybka 3 x64
135 of 215 matching moves - Houdini 1.0 x64
130 of 215 matching moves - Robbolito 009 x64
129 of 215 matching moves - Fire 1.2 x64
126 of 215 matching moves - Naum 4.2 x64
116 of 215 matching moves - Critter 0.70 x64
111 of 215 matching moves - Stockfish 1.7.1 x64
101 of 215 matching moves - Hiarcs 13.1
90 of 215 matching moves - Critter 0.60 x64
78 of 215 matching moves - Zappa Mexico II x64
75 of 215 matching moves - Toga II 3.2.1SE
75 of 215 matching moves - Sjeng WC2008 x64
66 of 215 matching moves - Loop 2007 x64
65 of 215 matching moves - Crafty 23.2 x64
61 of 215 matching moves - Komodo 1.2 x64
53 of 215 matching moves - Onno 1.2.70 x64
52 of 215 matching moves - Shredder 12 x64
50 of 215 matching moves - Brutus 8.05 x64
47 of 215 matching moves - Protector 1.3.4 x64
43 of 215 matching moves - Scorpio 2.5 x64
41 of 215 matching moves - Tornado 3.51a x64
41 of 215 matching moves - Daydreamer 1.75 x64
36 of 215 matching moves - N2 0.4 x64
30 of 215 matching moves - Gaviota 0.74.41 x64
16 of 215 matching moves - Philou 3.1.4.1
Top

Return to “Computer Chess Club: General Topics”