Hi,
Looking for a good positional test suite
in EPD format.
Thanks,
Stuart
positional test suite sought
Moderator: Ras
-
- Posts: 737
- Joined: Wed Mar 08, 2006 8:08 pm
- Location: Orange County California
- Full name: Stuart Cracraft
-
- Posts: 12781
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: positional test suite sought
I guess that this is the most famous one:
http://home.interact.se/~w100107/fentest.htm
You can also get a copy here:
http://cap.connx.com/EPD/tony-pos.epd.bz2
http://home.interact.se/~w100107/fentest.htm
You can also get a copy here:
http://cap.connx.com/EPD/tony-pos.epd.bz2
-
- Posts: 737
- Joined: Wed Mar 08, 2006 8:08 pm
- Location: Orange County California
- Full name: Stuart Cracraft
Re: positional test suite sought
Thanks/appreciated.
-
- Posts: 737
- Joined: Wed Mar 08, 2006 8:08 pm
- Location: Orange County California
- Full name: Stuart Cracraft
Re: positional test suite sought
Score=127 with a bare-bones PVS searcher, Fruit'sDann Corbit wrote:I guess that this is the most famous one:
http://home.interact.se/~w100107/fentest.htm
You can also get a copy here:
http://cap.connx.com/EPD/tony-pos.epd.bz2
pc/sq tables and material as sole terms in static
evaluation, and 5 seconds per position on a MacBook
(2.4ghz Intel Core 2 Duo) with a small hash table,
null move R=3, late move reduction, futility,
and the only major extension being if in check...
quiescence searches winning captures and all
promotions only. Maximum depth of search is
limited to 60 ply.
Not sure if the above positional test results table was
on super-slow hardware (Athlon?) or what but I don't
see why such a simple engine could get 127 otherwise.
Too simple a test positionally?
Stuart
-
- Posts: 12781
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: positional test suite sought
You must be doing something right. Here is a table of results from other engines:smcracraft wrote:Score=127 with a bare-bones PVS searcher, Fruit'sDann Corbit wrote:I guess that this is the most famous one:
http://home.interact.se/~w100107/fentest.htm
You can also get a copy here:
http://cap.connx.com/EPD/tony-pos.epd.bz2
pc/sq tables and material as sole terms in static
evaluation, and 5 seconds per position on a MacBook
(2.4ghz Intel Core 2 Duo) with a small hash table,
null move R=3, late move reduction, futility,
and the only major extension being if in check...
quiescence searches winning captures and all
promotions only. Maximum depth of search is
limited to 60 ply.
Not sure if the above positional test results table was
on super-slow hardware (Athlon?) or what but I don't
see why such a simple engine could get 127 otherwise.
Too simple a test positionally?
Stuart
Code: Select all
Program/Hardware 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Tot
Rybka 1.2 Athlon 1200 NEW 10 10 10 7 7 10 10 10 8 10 7 10 8 10 10 6 143
Shredder 9 UCI Athlon 1200 10 9 10 7 10 10 10 10 10 - 7 8 6 10 9 7 133
Hiarcs 8 Athlon 1200 10 9 10 7 7 10 10 5 10 8 5 7 8 5 10 7 128
Junior 7 Athlon 1200 9 6 10 8 7 10 8 10 10 - 7 10 8 8 9 7 127
Chess Tiger 14 CB Athlon 1200 10 9 10 7 10 10 7 5 8 7 10 7 10 - 9 7 126
Hiarcs 9 Athlon 1200 - 9 10 7 7 9 10 10 10 8 10 7 6 5 10 7 125
AnMon 5.07 AMD K6-2 450 10 9 10 8 7 10 7 10 9 10 10 8 - 10 - 7 125
Junior 9 Athlon 1200 10 10 10 7 7 10 10 10 - - 7 7 8 8 10 7 121
Deep Fritz 8 Athlon 1200 - 9 10 7 7 10 10 10 10 - 7 7 8 10 10 6 121
Chessmaster 8000 Athlon 1200 10 7 10 7 7 10 7 10 10 10 - 7 8 - 10 7 120
-
- Posts: 2684
- Joined: Sat Jun 14, 2008 9:17 pm
Re: positional test suite sought
With these positions Stockfish achieves a better score with lower depth!smcracraft wrote:
Too simple a test positionally?
Stuart
Below are two extreme examples when the _suggested_ best move is found almost immediately and then is lost going deep with thinking to the effect that the best move after more then 5 minutes thinking and 30M nodes searched is even not scored among the possible ones!
Code: Select all
POSITION 1:
Positional scores are: b5=10, Ngxe5=9, Ncxe5=9, Qc7=8, Na5=7, a6=6, h6=5, f6=3"; id "tony.pos.09";
Searching: 1r1q1rk1/pp1bbppp/2n1p1n1/4P3/2BpN3/3P1N2/PP2QPPP/R1B1R1K1 b - -
infinite: 1 ponder: 0 time: 0 increment: 0 moves to go: 0
2 +1.08 00:00 184 Ngxe5 Bf4
2 +1.20 00:00 557 b5 Bb3
3 +1.20 00:00 864 b5 Bb3 Ngxe5
4 +0.98 00:00 1624 b5 Bb3 Ngxe5 Bf4
5 +0.67 00:00 5956 b5 Bb3 Ngxe5 Bf4 Nxf3+ Qxf3
5 +0.76 00:00 13380 Ngxe5 Bf4 Qa5 Bxe5 Nxe5 Nxd4
6 +0.76 00:00 18654 Ngxe5 Bf4 Qa5 Bxe5 Nxe5 Nxd4 Rfc8
7 +1.08 00:00 30842 Ngxe5 Bf4 Qa5 Rac1 Nxf3+ Qxf3 Ne5 Qg3
8 +0.73 00:00 54149 Ngxe5 Bf4 Qa5 Nxe5 Nxe5 Ng3 f6 Nf5 Nxc4
9 +0.76 00:00 123727 Ngxe5 Nxe5 Nxe5 Bf4 Qa5 Ng3 Nxc4 Bxb8 Rxb8 dxc4
Bb4 Red1
10 +0.67 00:00 319060 Ngxe5 Nxe5 Nxe5 Bf4 Qa5 Ng3 Nxc4 Bxb8 Rxb8 dxc4
Bb4 Qe5 d3
10 +0.76 00:00 559085 Qc7 Bg5 Ngxe5 Bxe7 Nxf3+ Qxf3 Nxe7 Qh5 Rbc8 Rac1
Nd5 Ng5
11 +0.82 00:00 756804 Qc7 Bg5 Ngxe5 Rac1 Rbc8 Nxe5 Qxe5 Qh5 Qf5 Qh4 Bb4
Re2
12 +0.92 00:01 1574k Qc7 Bg5 Bb4 Rec1 h6 Bd2 Bxd2 Qxd2 Ngxe5 Nxe5 Qxe5
Rab1 Rbc8 Bb3
13 +0.90 00:02 2396k Qc7 Bg5 Bb4 Rec1 h6 Bd2 Bxd2 Qxd2 Ngxe5 Nxe5 Qxe5
Rab1 Ne7 Bb3 Nf5
14 +0.57 00:07 7241k Qc7 Ng3 Rfc8 Bd2 Na5 Rac1 Nxc4 Rxc4 Qb6 b3 Rxc4
bxc4 Bc6 Ne4 Qb2 Bg5
15 +0.45 00:18 18488k Qc7 Ng3 Rfc8 Bd2 Na5 Rac1 Nxc4 Rxc4 Qb6 b3 Rxc4
bxc4 Bc6 Ne4 Qb2 Bg5 Qxe2 Rxe2
16 +0.39 00:49 48925k Qc7 Ng3 Rfc8 Bd2 Na5 Rac1 Nxc4 Rxc4 Qb6 b3 Rxc4
bxc4 Bc6 Ne4 Rd8 Bg5 Bxg5 Nexg5 Nf4
16 +0.57 01:26 86094k Qb6 Bg5 Bxg5 Nexg5 h6 Nh3 Rfc8 Bb3 Nce7 Qe4 Nf5
Qg4 Bb5 Rad1 Rc5 Nf4 Nxf4
17 +0.51 01:43 102836k Qb6 Bg5 Bxg5 Nexg5 h6 Nh3 Nce7 a4 Bc6 a5 Qd8 Nd2
Qc7 Qh5 Nd5 Nf3 b5 Bxd5
18 +0.53 02:51 171468k Qb6 Bg5 Bxg5 Nexg5 h6 Nh3 Nce7 Qe4 Nf5 Rab1 Rbc8
Qg4 Bc6 Nf4 Nxf4 Qxf4 Rfd8 Rec1 Qa5 Bb3
Nodes: 298372057
Nodes/second: 994573
Best move: Qb6
Ponder move: Bg5
POSITION 2:
Positional scores are: d5=10, Bh6=8, Rae8=7, fxe4=5"; id "tony.pos.11";
Searching: r4rk1/p2qn1bp/1pnp2p1/2p2p2/4PP1N/2PPB3/PP2QN1P/R4RK1 b - -
infinite: 1 ponder: 0 time: 0 increment: 0 moves to go: 0
2 -0.04 00:00 298 d5 e5
2 -0.02 00:00 344 fxe4 dxe4
2 +0.00 00:00 446 Rae8 Nf3
2 +0.02 00:00 554 a6 Rae1
2 +0.18 00:00 702 Bf6 Nf3
3 -0.06 00:00 1079 Bf6 Nf3 d5
3 +0.22 00:00 1254 a6 Rae1 fxe4
4 -0.04 00:00 2803 a6 Nf3 Bh6 Rab1
4 +0.12 00:00 3367 fxe4 dxe4 d5 Rad1
5 +0.06 00:00 4827 fxe4 dxe4 d5 Rad1 Rae8
5 +0.14 00:00 7187 a6 Nf3 Bf6 Rae1 fxe4
6 +0.08 00:00 14660 a6 Nf3 Bf6 Qc2 Qe6 Rae1
6 +0.12 00:00 24551 Bf6 Nf3 Rae8 Rad1 Qe6 Rde1
6 +0.14 00:00 30564 Rae8 Nf3 fxe4 dxe4 d5 Rad1
7 +0.18 00:00 49427 Rae8 Nf3 Bf6 Rad1 a6 Rfe1 fxe4
8 +0.20 00:00 61432 Rae8 Nf3 Bf6 Rad1 a6 Rfe1 Qe6 a3
9 +0.16 00:00 102821 Rae8 Nf3 Bf6 Rad1 Qe6 c4 fxe4 dxe4 Nf5
10 +0.24 00:00 489090 Rae8 Nf3 Bf6 a3 fxe4 dxe4 d5 Rad1 Qe6 e5
11 +0.35 00:01 1003k Rae8 Nf3 Bh6 Nh4 Qe6 Qf3 fxe4 dxe4 Bg5 Qg3 Nf5
12 +0.45 00:02 2043k Rae8 Qc2 Na5 Qe2 Qe6 Nf3 Bh6 b4 fxe4 Nxe4 Nc4
dxc4
13 +0.59 00:04 4208k Rae8 Nf3 Bh6 Nh4 Nd8 Qf3 Ne6 e5 Nc6 Qd5 Rd8 exd6
Qxd6
14 +0.59 00:13 12415k Rae8 Rfd1 Nd8 Qc2 Bh6 Qb3+ Ne6 Nh3 Kh8 Ng2 Nc6
Re1 Rf7 Rad1 fxe4
15 +0.63 00:17 17362k Rae8 Rfd1 Nd8 Qc2 Ne6 Nh3 d5 Qb3 Rf7 Rac1 d4 cxd4
Nxd4 Bxd4 Qxd4+ Kg2
16 +0.55 00:42 40911k Rae8 Qf3 fxe4 dxe4 d5 Rad1 Qe6 exd5 Nxd5 Bxc5
Rxf4 Qxd5 bxc5 Rfe1 Be5 Qxe6+ Rxe6 Re4 Re8
16 +0.73 01:30 89168k Qe6 Rfe1 fxe4 dxe4 d5 Bd2 dxe4 Nxe4 Nd5 Qg2 Qd7
Qg5 Rae8 Rad1 a6 Be3 Rxe4 Rxd5
17 +0.78 02:21 141064k Qe6 Rfe1 fxe4 dxe4 d5 Bd2 Rad8 e5 d4 Ne4 d3 Qg2
Nd5 Qg3 c4 Rad1 Rxf4 Bxf4
18 +0.76 03:04 183865k Qe6 Rfe1 fxe4 dxe4 d5 Bd2 Rad8 e5 d4 Ne4 d3 Qg2
Nd5 Qg3 c4 Rad1 Rde8 Be3 Nxe3
Nodes: 300517336
Nodes/second: 1001674
Best move: Qe6
Ponder move: Rfe1
Marco
-
- Posts: 6401
- Joined: Thu Mar 09, 2006 8:30 pm
- Location: Chicago, Illinois, USA
Re: positional test suite sought
No, but they may tell you what an engine is lacking (or good at).mcostalba wrote:With these positions Stockfish achieves a better score with lower depth!smcracraft wrote:
Too simple a test positionally?
Stuart
Below are two extreme examples when the _suggested_ best move is found almost immediately and then is lost going deep with thinking to the effect that the best move after more then 5 minutes thinking and 30M nodes searched is even not scored among the possible ones!
I am not an expert of evaluation tuning but my first impression is that his kind of tests hardly substitute real games to validate a change.Code: Select all
POSITION 1: Positional scores are: b5=10, Ngxe5=9, Ncxe5=9, Qc7=8, Na5=7, a6=6, h6=5, f6=3"; id "tony.pos.09"; Searching: 1r1q1rk1/pp1bbppp/2n1p1n1/4P3/2BpN3/3P1N2/PP2QPPP/R1B1R1K1 b - - infinite: 1 ponder: 0 time: 0 increment: 0 moves to go: 0 2 +1.08 00:00 184 Ngxe5 Bf4 2 +1.20 00:00 557 b5 Bb3 3 +1.20 00:00 864 b5 Bb3 Ngxe5 4 +0.98 00:00 1624 b5 Bb3 Ngxe5 Bf4 5 +0.67 00:00 5956 b5 Bb3 Ngxe5 Bf4 Nxf3+ Qxf3 5 +0.76 00:00 13380 Ngxe5 Bf4 Qa5 Bxe5 Nxe5 Nxd4 6 +0.76 00:00 18654 Ngxe5 Bf4 Qa5 Bxe5 Nxe5 Nxd4 Rfc8 7 +1.08 00:00 30842 Ngxe5 Bf4 Qa5 Rac1 Nxf3+ Qxf3 Ne5 Qg3 8 +0.73 00:00 54149 Ngxe5 Bf4 Qa5 Nxe5 Nxe5 Ng3 f6 Nf5 Nxc4 9 +0.76 00:00 123727 Ngxe5 Nxe5 Nxe5 Bf4 Qa5 Ng3 Nxc4 Bxb8 Rxb8 dxc4 Bb4 Red1 10 +0.67 00:00 319060 Ngxe5 Nxe5 Nxe5 Bf4 Qa5 Ng3 Nxc4 Bxb8 Rxb8 dxc4 Bb4 Qe5 d3 10 +0.76 00:00 559085 Qc7 Bg5 Ngxe5 Bxe7 Nxf3+ Qxf3 Nxe7 Qh5 Rbc8 Rac1 Nd5 Ng5 11 +0.82 00:00 756804 Qc7 Bg5 Ngxe5 Rac1 Rbc8 Nxe5 Qxe5 Qh5 Qf5 Qh4 Bb4 Re2 12 +0.92 00:01 1574k Qc7 Bg5 Bb4 Rec1 h6 Bd2 Bxd2 Qxd2 Ngxe5 Nxe5 Qxe5 Rab1 Rbc8 Bb3 13 +0.90 00:02 2396k Qc7 Bg5 Bb4 Rec1 h6 Bd2 Bxd2 Qxd2 Ngxe5 Nxe5 Qxe5 Rab1 Ne7 Bb3 Nf5 14 +0.57 00:07 7241k Qc7 Ng3 Rfc8 Bd2 Na5 Rac1 Nxc4 Rxc4 Qb6 b3 Rxc4 bxc4 Bc6 Ne4 Qb2 Bg5 15 +0.45 00:18 18488k Qc7 Ng3 Rfc8 Bd2 Na5 Rac1 Nxc4 Rxc4 Qb6 b3 Rxc4 bxc4 Bc6 Ne4 Qb2 Bg5 Qxe2 Rxe2 16 +0.39 00:49 48925k Qc7 Ng3 Rfc8 Bd2 Na5 Rac1 Nxc4 Rxc4 Qb6 b3 Rxc4 bxc4 Bc6 Ne4 Rd8 Bg5 Bxg5 Nexg5 Nf4 16 +0.57 01:26 86094k Qb6 Bg5 Bxg5 Nexg5 h6 Nh3 Rfc8 Bb3 Nce7 Qe4 Nf5 Qg4 Bb5 Rad1 Rc5 Nf4 Nxf4 17 +0.51 01:43 102836k Qb6 Bg5 Bxg5 Nexg5 h6 Nh3 Nce7 a4 Bc6 a5 Qd8 Nd2 Qc7 Qh5 Nd5 Nf3 b5 Bxd5 18 +0.53 02:51 171468k Qb6 Bg5 Bxg5 Nexg5 h6 Nh3 Nce7 Qe4 Nf5 Rab1 Rbc8 Qg4 Bc6 Nf4 Nxf4 Qxf4 Rfd8 Rec1 Qa5 Bb3 Nodes: 298372057 Nodes/second: 994573 Best move: Qb6 Ponder move: Bg5 POSITION 2: Positional scores are: d5=10, Bh6=8, Rae8=7, fxe4=5"; id "tony.pos.11"; Searching: r4rk1/p2qn1bp/1pnp2p1/2p2p2/4PP1N/2PPB3/PP2QN1P/R4RK1 b - - infinite: 1 ponder: 0 time: 0 increment: 0 moves to go: 0 2 -0.04 00:00 298 d5 e5 2 -0.02 00:00 344 fxe4 dxe4 2 +0.00 00:00 446 Rae8 Nf3 2 +0.02 00:00 554 a6 Rae1 2 +0.18 00:00 702 Bf6 Nf3 3 -0.06 00:00 1079 Bf6 Nf3 d5 3 +0.22 00:00 1254 a6 Rae1 fxe4 4 -0.04 00:00 2803 a6 Nf3 Bh6 Rab1 4 +0.12 00:00 3367 fxe4 dxe4 d5 Rad1 5 +0.06 00:00 4827 fxe4 dxe4 d5 Rad1 Rae8 5 +0.14 00:00 7187 a6 Nf3 Bf6 Rae1 fxe4 6 +0.08 00:00 14660 a6 Nf3 Bf6 Qc2 Qe6 Rae1 6 +0.12 00:00 24551 Bf6 Nf3 Rae8 Rad1 Qe6 Rde1 6 +0.14 00:00 30564 Rae8 Nf3 fxe4 dxe4 d5 Rad1 7 +0.18 00:00 49427 Rae8 Nf3 Bf6 Rad1 a6 Rfe1 fxe4 8 +0.20 00:00 61432 Rae8 Nf3 Bf6 Rad1 a6 Rfe1 Qe6 a3 9 +0.16 00:00 102821 Rae8 Nf3 Bf6 Rad1 Qe6 c4 fxe4 dxe4 Nf5 10 +0.24 00:00 489090 Rae8 Nf3 Bf6 a3 fxe4 dxe4 d5 Rad1 Qe6 e5 11 +0.35 00:01 1003k Rae8 Nf3 Bh6 Nh4 Qe6 Qf3 fxe4 dxe4 Bg5 Qg3 Nf5 12 +0.45 00:02 2043k Rae8 Qc2 Na5 Qe2 Qe6 Nf3 Bh6 b4 fxe4 Nxe4 Nc4 dxc4 13 +0.59 00:04 4208k Rae8 Nf3 Bh6 Nh4 Nd8 Qf3 Ne6 e5 Nc6 Qd5 Rd8 exd6 Qxd6 14 +0.59 00:13 12415k Rae8 Rfd1 Nd8 Qc2 Bh6 Qb3+ Ne6 Nh3 Kh8 Ng2 Nc6 Re1 Rf7 Rad1 fxe4 15 +0.63 00:17 17362k Rae8 Rfd1 Nd8 Qc2 Ne6 Nh3 d5 Qb3 Rf7 Rac1 d4 cxd4 Nxd4 Bxd4 Qxd4+ Kg2 16 +0.55 00:42 40911k Rae8 Qf3 fxe4 dxe4 d5 Rad1 Qe6 exd5 Nxd5 Bxc5 Rxf4 Qxd5 bxc5 Rfe1 Be5 Qxe6+ Rxe6 Re4 Re8 16 +0.73 01:30 89168k Qe6 Rfe1 fxe4 dxe4 d5 Bd2 dxe4 Nxe4 Nd5 Qg2 Qd7 Qg5 Rae8 Rad1 a6 Be3 Rxe4 Rxd5 17 +0.78 02:21 141064k Qe6 Rfe1 fxe4 dxe4 d5 Bd2 Rad8 e5 d4 Ne4 d3 Qg2 Nd5 Qg3 c4 Rad1 Rxf4 Bxf4 18 +0.76 03:04 183865k Qe6 Rfe1 fxe4 dxe4 d5 Bd2 Rad8 e5 d4 Ne4 d3 Qg2 Nd5 Qg3 c4 Rad1 Rde8 Be3 Nxe3 Nodes: 300517336 Nodes/second: 1001674 Best move: Qe6 Ponder move: Rfe1
Marco
Miguel
-
- Posts: 6659
- Joined: Thu Mar 09, 2006 4:21 am
Re: positional test suite sought
Dann and I have been working on to build some strategical positional test suite, Chapter 1 Undermining/Pawn weakening is nearly getting finished. It consists of 100 set of problems in an EPD test suite - agreed on with by the Top 2 engines after longer analysis. It may get released in coming weeks, and then we will move on to creating the chapter 2.smcracraft wrote:Hi,
Looking for a good positional test suite
in EPD format.
Thanks,
Stuart
-
- Posts: 737
- Joined: Wed Mar 08, 2006 8:08 pm
- Location: Orange County California
- Full name: Stuart Cracraft
Re: positional test suite sought
Sounds great. Count me in as a beta-tester if you need any.swami wrote:Dann and I have been working on to build some strategical positional test suite, Chapter 1 Undermining/Pawn weakening is nearly getting finished. It consists of 100 set of problems in an EPD test suite - agreed on with by the Top 2 engines after longer analysis. It may get released in coming weeks, and then we will move on to creating the chapter 2.smcracraft wrote:Hi,
Looking for a good positional test suite
in EPD format.
Thanks,
Stuart
Re: positional test suite sought
I liked this testDann Corbit wrote:I guess that this is the most famous one:
http://home.interact.se/~w100107/fentest.htm
You can also get a copy here:
http://cap.connx.com/EPD/tony-pos.epd.bz2
At the 10 minutes per position my program scored 98.
After going back to the positions I was able to determine that my bishops did not want to go where they needed to. After looking at the middle game bishop square table I realized most of it was still a copy of the knights table!
So I corrected it for the bishops.
And I noticed my rooks did not work together very well. That was solved by changing some values in the rook eval routine. Again a few errors on my part.
At the 10 minutes per position my program now scored scored 123.
Chess Moron 10 7 - 7 7 10 10 8 10 8 7 10 10 10 9 - 123
I score nothing on the 3rd game only because I have no transposition tables (yet). I do not do well on endgames.
And the last game well after ply 4 it was q-d2 all the way with r-c1 (worth 10 points) as whites second move. Oh well.
Very helpful this test! Thanks!