A harder tactical test

[Dann Corbit]

Your result is not surprising, since it is {crudely} (2.8+2.8)/2.2 = 2.5 times as fast. Given a bit of SMP loss, we can expect a 50 Elo improvement or so for your system.

[Vempele]

It seems probable that there is a problem either with Rybka or with Arena. I was using the current beta version of Arena to run the test..

Isn't FEN just about completely broken in the Arena beta? That was my experience when it adjudicated the game every time an engine castled. And IIRC it crashed when I tried the Arasan test set.

[Dann Corbit]

It seems probable that there is a problem either with Rybka or with Arena. I was using the current beta version of Arena to run the test..

Isn't FEN just about completely broken in the Arena beta? That was my experience when it adjudicated the game every time an engine castled. And IIRC it crashed when I tried the Arasan test set.

You can't use a setup position for tournaments (though you can use a PGN file). But EPD analysis works (or I thought that it did)> I am not sure if the bug is in Arena or in Rybka.

[Alessandro Scotti]

Isn't FEN just about completely broken in the Arena beta? That was my experience when it adjudicated the game every time an engine castled. And IIRC it crashed when I tried the Arasan test set.

I had problems with FEN and latest Arena, and I'm currently back to beta 3.

[Uri Blass]

Isn't FEN just about completely broken in the Arena beta? That was my experience when it adjudicated the game every time an engine castled. And IIRC it crashed when I tried the Arasan test set.

I had problems with FEN and latest Arena, and I'm currently back to beta 3.

I have rule not to use beta versions of arena because I assume that beta has bugs.

I use only arena1.1 for games that I play with unequal time control.

Uri

[Will Singleton]

all the "quick" positions at the end are malformed; there aren't quotes around the id's.

I thought, the quotes are only required if the id contains a space? That is why I used Quick-12 instead of "Quick 12". Interfaces like Arena, Shredder Classic, Fritz accept my file, IIRC WinBoard too.

You're right, of course. When I wrote my pgn-parser, I didn't look at the spec, only at typical test sets. I've never noticed the id field without quotes.

Amateur 2.86, core 2 duo 2.6ghz, gets 181 correct.

Will

[Dirt]

Quick-03 is a study starting with an amazing queen sac.
[d]3Q4/3p4/P2p4/N2b4/8/4P3/5p1p/5Kbk w - - 0 1

After several hours Toga 1.3.1 could not find the right move, but after forcing Qa8 the mate is seen in a few hundredths of a second. I wonder if this should be considered a bug?

[Mike S.]

I wonder if this should be considered a bug?

I am not sure... maybe it is just a RARE downside of pruning. In principle, Toga does not have problems with deep queen sacs, for example in the first QT. position:

1: Von_Witonsti,G - [+4884.76d1g8], tb13 1913
[d]1n1r1rk1/ppq2ppp/3p2b1/3B1NP1/4PB1R/bP2P2P/P1P5/3KQ1R1 w - - 0 1
Analysis by Toga II 1.3.1 (D945 3.4 GHz, 256 MB hash):

(...)

1.c4 Bb2 2.Kc2 Be5 3.Kb1 Nc6 4.Qf2 Bxf4 5.Qxf4 Rfe8 6.Bxc6 bxc6
± (1.26) Depth: 8/28 00:00:00 386kN
± (1.32) Depth: 10/30 00:00:02 1006kN
1.Qc3 Bxf5 2.Qxc7 Bg6 3.Qxb7 Bc5 4.b4 Bb6 5.Bc4 Rfe8 6.Bd3 Nd7 7.Qd5 Ne5 8.Bb5 Re7
+- (1.57) Depth: 10/32 00:00:02 1180kN
+- (8.94) Depth: 13/54 00:01:14 42077kN

Or 1...Qxc3 2.Ne7+ Kh8 3.Nxg6+ Kg8 (3...fxg6 4.Rxh7+ Kxh7 5.Rg4 Rxf4) 4.Ne7+ Kh8 5.g6 h6 6.Bxh6 gxh6 7.Rxh6+ Kg7 8.Nf5+ Kf6 9.gxf7+ Ke5 10.Re6#

Or this one (much simpler):

3: Fox - N.N., Antwerpen 1901
[d]r1bqr1k1/ppp1bppp/2n3n1/4N2R/2pP1PP1/2PQ4/PP5P/R1B2BK1 w - - 0 1
Analysis by Toga II 1.3.1:

(...)
1.Qxg6 hxg6 2.Nxg6 fxg6 3.Bxc4+ Qd5 4.Bxd5+ Be6 5.Bxe6+ Kf8 6.Rh8#
-+ (-1.93) Depth: 6/22 00:00:00 98kN
+- (#6) Depth: 9/26 00:00:01 1245kN

[Uri Blass]

Quick-03 is a study starting with an amazing queen sac.
[d]3Q4/3p4/P2p4/N2b4/8/4P3/5p1p/5Kbk w - - 0 1

After several hours Toga 1.3.1 could not find the right move, but after forcing Qa8 the mate is seen in a few hundredths of a second. I wonder if this should be considered a bug?

No

The reason is probably null move pruning.
Movei does not use null move pruning in this specific case but program who use always null move pruning will never find it(or need a very long time in case of having verification search(toga has it but I guess that some hours are not enough for it).

After changing null move pruning from always to never toga can find it in few minutes

New game
[d]3Q4/3p4/P2p4/N2b4/8/4P3/5p1p/5Kbk w - - 0 1

Analysis by Toga II 1.3 Beta1:

1.Qd8xd7 Bd5-g2+ 2.Kf1-e2 f2-f1Q+ 3.Ke2-d2 Qf1xa6
-+ (-5.19) Depth: 1/6 00:00:00
1.Kf1-e2
+- (4.15) Depth: 1/6 00:00:00
1.Qd8-g5
+- (4.36) Depth: 1/15 00:00:00
1.Qd8-g5 Bd5-e4
+- (4.62) Depth: 2/15 00:00:00
1.Qd8-g5 Bd5-e4 2.Kf1-e2
+- (4.27) Depth: 3/15 00:00:00
1.Qd8-g5 Bd5-e4 2.Kf1-e2 Be4-d3+ 3.Ke2xd3 f2-f1Q+ 4.Kd3-e4 Qf1xa6
-+ (-1.87) Depth: 4/15 00:00:00
1.Qd8-g5 Bd5-e4 2.Kf1-e2 Be4-g2 3.Qg5-f5 f2-f1Q+ 4.Qf5xf1 Bg2xf1+ 5.Ke2xf1 Bg1xe3
-+ (-1.53) Depth: 5/15 00:00:00
1.Qd8-f6 Bd5-g2+ 2.Kf1-e2 f2-f1R 3.Qf6xd6 Rf1-f5 4.Qd6xd7 Bg2-f3+ 5.Ke2-f1
-+ (-1.41) Depth: 5/16 00:00:00
1.Qd8-f6 Bd5-g2+ 2.Kf1-e2 f2-f1Q+ 3.Qf6xf1 Bg2xf1+ 4.Ke2xf1 Bg1xe3 5.Na5-b7 d6-d5 6.Nb7-d6 Be3-d4 7.Kf1-e2
-+ (-1.51) Depth: 6/20 00:00:00 8kN
1.Qd8-f6 Bd5-g2+ 2.Kf1-e2 f2-f1Q+ 3.Qf6xf1 Bg2xf1+ 4.Ke2xf1 Bg1xe3 5.Na5-b7 d6-d5 6.Nb7-d6 Be3-d4 7.Nd6-f5 Bd4-e5
µ (-1.14) Depth: 7/23 00:00:00 34kN
1.Qd8-f6 Bd5-e4 2.Kf1-e2 Kh1-g2 3.Qf6-g5+ Kg2-h1 4.a6-a7 Be4-d3+ 5.Ke2-d2 Bd3-e4 6.Qg5-f5
+- (4.55) Depth: 8/25 00:00:00 184kN
1.Qd8-f6 Bd5-e4 2.Kf1-e2 Kh1-g2 3.Qf6-g5+ Kg2-h3 4.Qg5-h6+ Kh3-g2 5.Qh6-g7+ Kg2-h1 6.a6-a7 Be4-d3+ 7.Ke2-d2 Bd3-e4
+- (4.36) Depth: 9/25 00:00:01 209kN
1.Qd8-f6 Bd5-e4 2.Kf1-e2 Kh1-g2 3.Qf6-g5+ Kg2-h3 4.Qg5-h6+ Kh3-g2 5.Qh6-g7+ Kg2-h3 6.Qg7xd7+ Kh3-g3 7.Qd7-g7+ Kg3-h3 8.Qg7-h6+ Kh3-g2 9.Qh6-g5+ Kg2-h3
+- (4.34) Depth: 10/29 00:00:01 443kN
1.Qd8-f6 Bd5-e4 2.Kf1-e2 Kh1-g2 3.Qf6-g5+ Kg2-h3 4.Qg5-h6+ Kh3-g2 5.Qh6-g7+ Kg2-h3 6.Qg7xd7+ Kh3-g3 7.Qd7-g7+ Kg3-h3 8.Qg7-h6+ Kh3-g2 9.Qh6-g5+ Kg2-h3
+- (4.34) Depth: 11/34 00:00:01 937kN
1.Qd8-f6 Bd5-e4 2.Kf1-e2 Kh1-g2 3.Qf6-g5+ Kg2-h3 4.Qg5-h6+ Kh3-g2 5.Qh6-g7+ Kg2-h3 6.Qg7xd7+ Kh3-g3 7.Qd7-g7+ Kg3-h3 8.Qg7-h6+ Kh3-g2 9.Qh6-g5+ Kg2-h3 10.Qg5-h5+ Kh3-g3
+- (4.10) Depth: 12/34 00:00:02 2177kN
1.Qd8-f6 Bd5-e4 2.Kf1-e2 Kh1-g2 3.Qf6-g5+ Kg2-h3 4.Qg5-h6+ Kh3-g2 5.Qh6-g7+ Kg2-h3 6.Qg7xd7+ Kh3-g3 7.Qd7-g7+ Kg3-h3 8.Qg7-h8+ Kh3-g3 9.Qh8-g8+ Kg3-h4 10.Qg8-d8+ Kh4-h3 11.Qd8-c8+ Kh3-g3
+- (4.01) Depth: 13/42 00:00:06 5602kN
1.Qd8-f6 Bd5-e4 2.Kf1-e2 Kh1-g2 3.Qf6-g5+ Kg2-h3 4.Qg5-h6+ Kh3-g2 5.Qh6-g7+ Kg2-h3 6.Qg7xd7+ Kh3-g3 7.Qd7-g7+ Kg3-h3 8.Qg7-h8+ Kh3-g3 9.Qh8-g8+ Kg3-h4 10.Qg8-d8+ Kh4-h3 11.Qd8-c8+ Kh3-g3 12.Qc8-g8+
= (0.00) Depth: 14/47 00:00:18 16609kN
1.Qd8-f6 Bd5-e4 2.Kf1-e2 Kh1-g2 3.Qf6-g5+ Kg2-h3 4.Qg5-h6+ Kh3-g2 5.Qh6-g7+ Kg2-h3 6.Qg7xd7+ Kh3-g3 7.Qd7-g7+ Kg3-h3 8.Qg7-h8+ Kh3-g3 9.Qh8-g8+ Kg3-h4 10.Qg8-d8+ Kh4-h3 11.Qd8-c8+ Kh3-g3 12.Qc8-g8+
= (0.00) Depth: 15/49 00:00:45 40591kN
1.Qd8-f6 Bd5-e4 2.Qf6xf2 Be4-d3+ 3.Qf2-e2 Bd3xe2+ 4.Kf1xe2 Kh1-g2 5.a6-a7 h2-h1Q 6.a7-a8Q+ d6-d5 7.Qa8xd5+ Kg2-h2 8.Qd5-e5+ Kh2-g2 9.Qe5-g5+ Kg2-h2 10.Qg5-h5+ Kh2-g2 11.Qh5-g6+ Kg2-h2 12.Qg6-h6+ Kh2-g2 13.Qh6-g7+ Kg2-h2 14.Qg7-h8+ Kh2-g2 15.Qh8-a8+ Kg2-h2 16.Qa8-b8+ Kh2-g2
² (0.58) Depth: 16/54 00:02:31 142315kN
1.Qd8-a8 Bd5xa8 2.Na5-b7 d6-d5 3.Nb7-d6 Ba8-c6 4.a6-a7 d5-d4 5.e3-e4 Bc6-b5+ 6.Nd6xb5 d4-d3 7.a7-a8Q d7-d5 8.Qa8-g8 d5xe4 9.Qg8-g2#
+- (#9) Depth: 16/54 00:03:07 175872kN
1.Qd8-a8 Bd5xa8 2.Na5-b7 d6-d5 3.Nb7-d6 Ba8-c6 4.a6-a7 d5-d4 5.e3-e4 Bc6-b5+ 6.Nd6xb5 d4-d3 7.a7-a8Q d7-d5 8.Qa8-g8 d5xe4 9.Qg8-g2#
+- (#9) Depth: 17/54 00:03:08 177176kN
1.Qd8-a8 Bd5xa8 2.Na5-b7 d6-d5 3.Nb7-d6 Ba8-c6 4.a6-a7 d5-d4 5.e3-e4 Bc6-b5+ 6.Nd6xb5 d4-d3 7.a7-a8Q d7-d5 8.Qa8-g8 d5xe4 9.Qg8-g2#
+- (#9) Depth: 18/54 00:03:11 183144kN

(, 21.11.2007)

[Roman Hartmann]

Dan,

How do you run that test set? Which GUI do you use, and do you somehow take all that text and convert it to a PGN file or something? Or, do I have to put the FEN in whatever the GUI is and save each one as its own "game?"

Erik

The test sets I run are EPD test sets.
I use Arena and then under arena "Automatic Analysis"
You can also use the EPD2WB tool, written by Bruce Moreland and modified by Thomas Mayer.
Chess Assistant can do it.

You could also use polglot with uci-engines at least. As I switched to linux recently I can't use the Chessbase GUI for epd-tests anymore and now I am using polyglot.

A command to run the arasan testsuite would look like: '/.polyglot epd-test -epd arasancorbit.epd -max-time 60'

Much safer to run testsuites with Polyglot than with any GUI I know of.

Roman