building a good test suite to predict the rating of engines

[JBNielsen]

On my old HomePage I have written this:

From 1986 to 1991 I constructed a test based on more than 100 chess positions.
The goal was to be able to estimate the rating of the programs tested.
And use the test for improving the development of chess programs.

Many of the positions are too easy for the computers of today.

Perhaps someone makes a better test in the future...

The link is:
http://www.jens-musik.dk/skak.htm

Here you Can find more details: the positions, the weights and how they were adjusted, the results, a detailed article about this test etc.

In my old test each position had its own weight, as some positions are more important than others. The adjustment of weights ran automatically for hours, so the tests calculated rating for each chessengine came as close as possible to the 'real' rating determined by playing a lot of engine vs engine games.

Has anybody considered doing that with STS?

[yl84]

I'm interested but the link seems broken.
Best regards
Yves

[JBNielsen]

I'm interested but the link seems broken.
Best regards
Yves

It worked a few days ago, but it has been closed as I have missed a bill I should pay.
I hope it will be running soon!

[JBNielsen]

I'm interested but the link seems broken.
Best regards
Yves

It worked a few days ago, but it has been closed as I have missed a bill I should pay.
I hope it will be running soon!

The link is OK again.

[yl84]

I'm interested but the link seems broken.
Best regards
Yves

It worked a few days ago, but it has been closed as I have missed a bill I should pay.
I hope it will be running soon!

The link is OK again.

Thank you! The machines tested were between 1500 and 2100 elo; would you say that your test may be used better for programmes of this range?
Yves

[JBNielsen]

Thank you! The machines tested were between 1500 and 2100 elo; would you say that your test may be used better for programmes of this range?
Yves

Yes. The test is ca. 25 years old, and the chessprograms were much weaker back then.

A new, updated and better test could fx be formed of (a part of) the strategy test suite (STS), a tactical test and an endgame test.

I list the positions of my old test below.
Unfortunately there are ERRORS in some of the positions if you compare them with the diagrams.
Must be fixed some day...

2r2rk1/pp3ppp/4p3/7n/3P4/P3B1P1/1P3P1P/2R2RK1 w - - bm d5; id "Jens Bæk Nielsen.01";
r1b2rk1/ppppqppp/1b3n2/1B2p1Q1/4P3/2PPB3/PP3PPP/RN3RK1 b - - am Bxe3; id "Jens Bæk Nielsen.02";
2kr3r/p1qnbp2/1pn1p3/3pP1p1/2pP4/P1P2N2/1P1NQPPP/R1B2RK1 w - - bm b3; id "Jens Bæk Nielsen.03";
r2r2k1/p2bqpbp/2p1n1p1/Pp2p3/4P3/1NP1B3/1PQ1BPPP/R2R2K1 w - - bm Bg4; id "Jens Bæk Nielsen.04";
4b1k1/1q4p1/p1r1pp1p/P1Pp4/8/Q2BP2P/5PP1/2R3K1 w - - bm e4; id "Jens Bæk Nielsen.05";
2r2rk1/1p1bq3/p3p2p/3pPpp1/1P1Q4/P7/2P2PPP/2R1RBK1 b - - bm Bb5; id "Jens Bæk Nielsen.06";
1r1r2k1/2b1qp1p/b1p3p1/p1p1p3/2P1P3/1PN1BP2/P1Q3PP/R2R2K1 b - - bm Rd4; id "Jens Bæk Nielsen.07";
rn2k2r/pp1bbppp/1q2p3/3pP3/3P4/3B1N2/PP1B1PPP/1R1QK2R b Kkq - bm Bb5; id "Jens Bæk Nielsen.08";
r2q1rk1/p3pp1p/1p3np1/2p5/2P5/3P1NP1/P3PP1P/R2Q1RK1 w - - bm a4; id "Jens Bæk Nielsen.09";
r1b1k2r/pp1p1p1p/2p2p2/2b1p3/2B1P3/2NP4/PPP2PPP/R4RK1 w kq - bm Kh1; id "Jens Bæk Nielsen.10";
r4rk1/pppqn1b1/3p2pp/3Ppb2/2P1N3/3BBP2/PP4PP/R2QK2R w KQ - bm g4; id "Jens Bæk Nielsen.11";
r1q2rk1/ppnn2bp/3p2p1/2pPp3/2P1P1Q1/2N4P/PP1B1PB1/R3R1K1 b - - bm Rf7; id "Jens Bæk Nielsen.12";
rnbq1rk1/ppp1bpp1/4pn1p/3p4/2PP4/2N1P3/PP3PPP/2RQKBBR b K - am g5; id "Jens Bæk Nielsen.13";
r1bqk2r/pp2bppp/2nppn2/2p5/4P3/2NP1N2/PPP1BPPP/R1BQ1RK1 w kq - am Ng5; id "Jens Bæk Nielsen.14";
r1bqkb1r/1p1p2p1/1n3p1p/pN1P4/P3P3/5N1P/5P2/R2QKB1R w KQkq - bm d6; id "Jens Bæk Nielsen.15";
1k2r3/2pr2pp/ppp2p2/5R2/4P1P1/1P1P3P/P1P5/5RK1 w - - bm b4; id "Jens Bæk Nielsen.16";
2b5/4pkb1/p2p2p1/1p1P3p/4P3/1P2B1P1/P3K1BP/8 b - - am Bb7; id "Jens Bæk Nielsen.17";
q1rr1bk1/1b1n1p1p/pp1ppnp1/8/P1NPP3/1P4PP/1B1NQPB1/1R1R2K1 w - - bm d5; id "Jens Bæk Nielsen.18";
r2q1rk1/1ppnbppp/p2p1nb1/3Pp3/2P1P1P1/2N2N1P/PPB1QP2/R1B2RK1 b - - bm h5; id "Jens Bæk Nielsen.19";
r1bqk2r/pp2bppp/2p5/3pP3/P2Q1P2/2N1B3/1PP3PP/R4RK1 b kq - bm f6; id "Jens Bæk Nielsen.20";
3r1k2/4npp1/1ppr3p/p6P/P2PPPP1/1NR5/5K2/2R5 w - - bm d5; id "Jens Bæk Nielsen.21";
2r3k1/pppR1pp1/4p3/4P1P1/5P2/1P4K1/P1P5/8 w - - bm g6; id "Jens Bæk Nielsen.22";
2r1nrk1/p2q1ppp/bp1p4/n1pPp3/P1P1P3/2PBB1N1/4QPPP/R4RK1 w - - bm f4; id "Jens Bæk Nielsen.23";
r1bqk2r/pp1pppbp/2n2np1/8/2PN4/2N3P1/PP2PPBP/R1BQK2R b KQkq - am Na5; id "Jens Bæk Nielsen.24";
r3kb1r/1ppq1ppp/p2p1n2/8/2PQP3/2N5/PP3PPP/R1B1K2R b KQkq - am c5; id "Jens Bæk Nielsen.25";
r1b2rk1/p2n1ppp/1p3n2/2pp4/1q1P4/3BPN2/PPQN1PPP/R4RK1 w - - bm a3; id "Jens Bæk Nielsen.26";
r4rk1/1p1nq1pp/p1b1p3/4Pp2/3P1Q2/3B1N2/P4PPP/R1R3K1 w - - am Qg5; id "Jens Bæk Nielsen.27";
r1r3k1/1p6/p3n2p/3pPpp1/3P1P2/4KN2/P5PP/R1R5 w - - bm g3; id "Jens Bæk Nielsen.28";
r4rk1/4nppp/p2qb3/2p5/8/2PB1Q2/3P2PP/R1B2RK1 b - - am f5; id "Jens Bæk Nielsen.29";
r3k3/pp3p1p/6p1/8/8/6P1/PP3P1P/2R3K1 w - - bm Rc7; id "Jens Bæk Nielsen.30";
r1b2rk1/1pn1bpbp/p2p2p1/q1pP4/2P1P3/2NB4/PP1QNPPP/R4RK1 b - - bm b5; id "Jens Bæk Nielsen.31";
6k1/ppp4p/8/6P1/2P2p1R/1P1n3N/1r3PK1/8 w - - am Nxf4; id "Jens Bæk Nielsen.32";
8/ppp1k1p1/2n4p/3B4/4p3/2P4P/PP2K1P1/8 w - - bm Bxc6; id "Jens Bæk Nielsen.33";
r1bqr1k1/pp3ppp/2pb1nn1/3p2B1/1P1P4/2NBPN2/P1Q2PPP/1R3RK1 w - - bm b5; id "Jens Bæk Nielsen.34";
2r2rk1/3q1ppp/p3p3/1p1p4/1P1P4/P3P3/5PPP/1RQ2RK1 b - - bm Rc4; id "Jens Bæk Nielsen.35";
r1bqr1k1/ppp2pp1/2nb1n1p/3p4/3P1B2/2PB3P/PP1N1PP1/R2Q1RK1 w - - bm Be3; id "Jens Bæk Nielsen.36";
r2qr1k1/ppp2pp1/2nb1n1p/3p4/3P1B2/2PB1N1P/PP1N1PP1/R2Q1RK1 w - - bm Bxd6; id "Jens Bæk Nielsen.37";
4b3/p3kp2/6p1/3pP2p/2pP1P2/4K1P1/P3N2P/8 w - - bm f5; id "Jens Bæk Nielsen.38";
8/8/1P2P3/8/6p1/2r2kP1/7P/1R4K1 b - - bm Re3; id "Jens Bæk Nielsen.39";
r2q1rk1/ppp1ppbp/2np1np1/8/2PPP3/2NBBQ1P/PP3PP1/R3K2R b KQ - bm Nd7; id "Jens Bæk Nielsen.40";
N3Q3/3P2pk/5p2/8/2P1p1PK/1P1rP2P/5P2/2n4b b - - bm Ne2; id "Jens Bæk Nielsen.41";
2k5/1pp2p2/8/1P3b1n/8/1P3P2/P5PP/2K2B2 w - - bm g4; id "Jens Bæk Nielsen.42";
8/3p4/4p3/P7/5k2/8/4P3/5Kq1 w - - bm Kxg1; id "Jens Bæk Nielsen.43";
5k2/8/8/8/8/2P5/8/3K4 w - - bm Kc2; id "Jens Bæk Nielsen.44";
8/8/3k4/8/3K4/8/2P5/8 w - - am c4; id "Jens Bæk Nielsen.45";
8/p5kp/8/1P6/8/1K6/P7/8 w - - bm a4; id "Jens Bæk Nielsen.46";
8/2k5/1pP5/1P1K4/8/8/8/8 w - - bm Ke6; id "Jens Bæk Nielsen.47";
3k4/8/8/p5p1/P5P1/8/4K3/8 w - - bm Kd2; id "Jens Bæk Nielsen.48";
4k3/8/8/8/8/4Kp2/3RbP2/8 w - - bm Rxc2; id "Jens Bæk Nielsen.49";
8/8/8/5p2/5K2/8/3NkP2/8 b - - bm Kxf2; id "Jens Bæk Nielsen.50";
8/8/8/5p2/5K2/8/3BkP2/8 b - - bm Kxf2; id "Jens Bæk Nielsen.51";
8/rKn5/1n6/8/8/k7/8/8 w - - bm Kxa7; id "Jens Bæk Nielsen.52";
8/7N/7p/7p/7K/8/6kP/7N b - - bm Kxh2; id "Jens Bæk Nielsen.53";
4k3/5N1B/5K2/8/8/8/8/8 w - - bm Ne5; id "Jens Bæk Nielsen.54";
1KQ5/8/8/8/8/8/6p1/6k1 w - - bm Kc7; id "Jens Bæk Nielsen.55";
k7/2K5/P7/P7/8/6p1/6P1/8 w - - bm Kd6; id "Jens Bæk Nielsen.56";
1B5K/1n6/1p6/2k5/8/8/8/8 w - - bm Ba7; id "Jens Bæk Nielsen.57";
b1N4K/8/8/1p6/2k5/8/8/8 w - - bm Nd6+; id "Jens Bæk Nielsen.58";
r2n2k1/pp3ppp/1n6/N1rP4/1N2P1K1/PP1BP1b1/3RQ1Pq/2R5 w - - am Rxc5; id "Jens Bæk Nielsen.59";
5r1k/ppp4p/3p2p1/P2P4/1N2P1nq/2P4P/1P4P1/nQ2R2K w - - am Qxa1; id "Jens Bæk Nielsen.60";
r7/pp3bkp/q2p1pp1/3N4/Pp2P3/2N3PP/2PQ2BK/2B5 b - - am bxc3; id "Jens Bæk Nielsen.61";
6k1/K1ppp1p1/1p2p3/p3b1PP/7N/P2B3r/4P1Pr/2QR1R2 b - - am Rxh4; id "Jens Bæk Nielsen.62";
b1n2r1k/pqb2p1p/4pPp1/Pp4Q1/R4R2/2P4P/1P1P2p1/6K1 b - - am bxa4; id "Jens Bæk Nielsen.63";
1r5r/3Rbkpn/Bp2p1p1/5p2/4pP2/4PqP1/PP3N1P/1K1R4 b - - am Qxf2; id "Jens Bæk Nielsen.64";
2r2q2/2n4k/6p1/1Pp1p2p/2PpP2p/3P4/1KN2Q1P/R7 w - - bm Qxf8; id "Jens Bæk Nielsen.65";
3qnk2/1b3pp1/1np4p/2Pp4/PP6/B4BQP/3P1PPK/2R2N2 b - - am Nxa4; id "Jens Bæk Nielsen.66";
nbk1r3/ppq3p1/2p2pPp/2P5/B2Pb3/4P1P1/PP1Q2NP/K3R1N1 b - - bm Bxg2; id "Jens Bæk Nielsen.67";
r3k2r/pbn2ppp/8/1P1pP3/P1qP4/5B2/3Q1PPP/R3K2R w KQkq - bm Be2; id "Jens Bæk Nielsen.68";
3r2k1/ppq2pp1/4p2p/3n3P/3N2P1/2P5/PP2QP2/K2R4 b - - bm Nxc3; id "Jens Bæk Nielsen.69";
k7/8/8/1Q6/4Kb2/8/8/8 w - - bm Kd5; id "Jens Bæk Nielsen.70";
2nr1rk1/ppp3p1/n4pQ1/8/q6R/6P1/PPP2PNP/4R2K w - - bm Rh7; id "Jens Bæk Nielsen.71";
7k/3q2p1/4p2p/4Pp2/1r3P2/p1pR4/P1P3PP/3Q3K b - - bm Qxd3; id "Jens Bæk Nielsen.72";
r5k1/ppq2p1p/2p5/6r1/1P2pN2/P3PbP1/2Q2P2/R3R1K1 b - - bm Qxf4; id "Jens Bæk Nielsen.73";
3r2k1/5p2/6p1/4b3/1P2P3/1R2P2p/P1K1N3/8 b - - bm Rd2; id "Jens Bæk Nielsen.74";
r1b2rk1/2n1qppp/p1p1p3/1p2P3/4N3/3BP3/PP2QPPP/R4RK1 w - - bm Nf6+; id "Jens Bæk Nielsen.75";
q3rn1k/2QR4/pp2pp2/8/P1P5/1P4N1/6n1/6K1 w - - bm Nf5; id "Jens Bæk Nielsen.76";
r3rbk1/ppq2pp1/2p3P1/8/2n3P1/2N5/PPP2P2/1KQR3R w - - bm Rh8+; id "Jens Bæk Nielsen.77";
3r1rk1/pp1q1ppp/3pn3/2pN4/5PP1/P5PQ/1PP1B3/1K1R4 w - - bm Bb5; id "Jens Bæk Nielsen.78";
8/p5kp/1p3p2/6p1/8/1P3QPK/P2N4/6q1 b - - bm h5; id "Jens Bæk Nielsen.79";
8/5k2/pp6/5p2/6bP/P1Bp4/1P4RK/2r5 b - - bm Rxc3; id "Jens Bæk Nielsen.80";

[yl84]

Thank you. I have verified your test suite positions up to number 40 included. Below are the corrected fen positions. Only 3 positions were buggy compared to the diagrams. I like your test positions; you are right some of them could be in the STS, since they are pure positional and could help to tune engines. I also found a possible error in position 39. The best move could be 1...Rc8 but with a mate in 13 against blacks. At least for my engine most of positional tests are strong,but my engine is still weak. Last, I have not really understood your method for calculating elo ; maybe could have been simpler for guys like me.
Regards
Yves



2r2rk1/pp3ppp/4p3/7n/3P4/P3B1P1/1P3P1P/2R2RK1 w - - bm d5; id "Jens Bæk Nielsen.01";
r1b2rk1/ppppqppp/1b3n2/1B2p1Q1/4P3/2PPB3/PP3PPP/RN3RK1 b - - am Bxe3; id "Jens Bæk Nielsen.02";
2kr3r/p1qnbp2/1pn1p3/3pP1p1/2pP4/P1P2N2/1P1NQPPP/R1B2RK1 w - - bm b3; id "Jens Bæk Nielsen.03";
r2r2k1/p2bqpbp/2p1n1p1/Pp2p3/4P3/1NP1B3/1PQ1BPPP/R2R2K1 w - - bm Bg4; id "Jens Bæk Nielsen.04";
4b1k1/1q4p1/p1r1pp1p/P1Pp4/8/Q2BP2P/5PP1/2R3K1 w - - bm e4; id "Jens Bæk Nielsen.05";
2r2rk1/1p1bq3/p3p2p/3pPpp1/1P1Q4/P7/2P2PPP/2R1RBK1 b - - bm Bb5; id "Jens Bæk Nielsen.06";
1r1r2k1/2b1qp1p/b1p3p1/p1p1p3/2P1P3/1PN1BP2/P1Q3PP/R2R2K1 b - - bm Rd4; id "Jens Bæk Nielsen.07";
rn2k2r/pp1bbppp/1q2p3/3pP3/3P4/3B1N2/PP1B1PPP/1R1QK2R b Kkq - bm Bb5; id "Jens Bæk Nielsen.08";
r2q1rk1/p3pp1p/1p3np1/2p5/2P5/3P1NP1/P3PP1P/R2Q1RK1 w - - bm a4; id "Jens Bæk Nielsen.09";
r1b1k2r/pp1p1p1p/2p2p2/2b1p3/2B1P3/2NP4/PPP2PPP/R4RK1 w kq - bm Kh1; id "Jens Bæk Nielsen.10";
r4rk1/pppqn1b1/3p2pp/3Ppb2/2P1N3/3BBP2/PP4PP/R2QK2R w KQ - bm g4; id "Jens Bæk Nielsen.11";
r1q2rk1/ppnn2bp/3p2p1/2pPp3/2P1P1Q1/2N4P/PP1B1PB1/R3R1K1 b - - bm Rf7; id "Jens Bæk Nielsen.12";
rnbq1rk1/ppp1bpp1/4pn1p/3p4/2PP3B/2N1P3/PP3PPP/2RQKBNR b K - am g5; id "Jens Bæk Nielsen.13"; ==> modified
r1bqk2r/pp2bppp/2nppn2/2p5/4P3/2NP1N2/PPP1BPPP/R1BQ1RK1 w kq - am Ng5; id "Jens Bæk Nielsen.14";
r1bqkb1r/1p1p2p1/1n3p1p/pN1P4/P3P3/5N1P/5P2/R2QKB1R w KQkq - bm d6; id "Jens Bæk Nielsen.15";
1k2r3/2pr2pp/ppp2p2/5R2/4P1P1/1P1P3P/P1P5/5RK1 w - - bm b4; id "Jens Bæk Nielsen.16";
2b5/4pkb1/p2p2p1/1p1P3p/4P3/1P2B1P1/P3K1BP/8 b - - am Bb7; id "Jens Bæk Nielsen.17";
q1rr1bk1/1b1n1p1p/pp1ppnp1/8/P1NPP3/1P4PP/1B1NQPB1/1R1R2K1 w - - bm d5; id "Jens Bæk Nielsen.18";
r2q1rk1/1ppnbppp/p2p1nb1/3Pp3/2P1P1P1/2N2N1P/PPB1QP2/R1B2RK1 b - - bm h5; id "Jens Bæk Nielsen.19";
r1bqk2r/pp2bppp/2p5/3pP3/P2Q1P2/2N1B3/1PP3PP/R4RK1 b kq - bm f6; id "Jens Bæk Nielsen.20";
3r1k2/4npp1/1ppr3p/p6P/P2PPPP1/1NR5/5K2/2R5 w - - bm d5; id "Jens Bæk Nielsen.21";
2r3k1/pppR1pp1/4p3/4P1P1/5P2/1P4K1/P1P5/8 w - - bm g6; id "Jens Bæk Nielsen.22";
2r1nrk1/p2q1ppp/bp1p4/n1pPp3/P1P1P3/2PBB1N1/4QPPP/R4RK1 w - - bm f4; id "Jens Bæk Nielsen.23";
r1bqk2r/pp1pppbp/2n2np1/8/2PN4/2N3P1/PP2PPBP/R1BQK2R b KQkq - am Na5; id "Jens Bæk Nielsen.24";
r3kb1r/1ppq1ppp/p2p1n2/8/2PQP3/2N5/PP3PPP/R1B1K2R b KQkq - am c5; id "Jens Bæk Nielsen.25";
r1b2rk1/p2n1ppp/1p3n2/2pp4/1q1P4/3BPN2/PPQN1PPP/R4RK1 w - - bm a3; id "Jens Bæk Nielsen.26";
r4rk1/1p1nq1pp/p1b1p3/4Pp2/3P1Q2/3B1N2/P4PPP/R1R3K1 w - - am Qg5; id "Jens Bæk Nielsen.27";
r1r3k1/1p6/p3n2p/3pPpp1/3P1P2/4KN2/P5PP/R1R5 w - - bm g3; id "Jens Bæk Nielsen.28";
r4rk1/4nppp/p2qb3/2p5/8/2PB1Q2/3P2PP/R1B2RK1 b - - am f5; id "Jens Bæk Nielsen.29";
r3k3/pp3p1p/6p1/8/8/6P1/PP3P1P/2R3K1 w - - bm Rc7; id "Jens Bæk Nielsen.30";
r1b2rk1/1pn1ppbp/p2p2p1/q1pP4/2P1P3/2NB4/PP1QNPPP/R4RK1 b - - bm b5; id "Jens Bæk Nielsen.31"; ==> modified
6k1/ppp4p/8/6P1/2P2p1R/1P1n3N/1r3PK1/8 w - - am Nxf4; id "Jens Bæk Nielsen.32";
8/ppp1k1p1/2n4p/3B4/4p3/2P4P/PP2K1P1/8 w - - bm Bxc6; id "Jens Bæk Nielsen.33";
r1bqr1k1/pp3ppp/2pb1nn1/3p2B1/1P1P4/2NBPN2/P1Q2PPP/1R3RK1 w - - bm b5; id "Jens Bæk Nielsen.34";
2r2rk1/3q1ppp/p3p3/1p1p4/1P1P4/P3P3/5PPP/2RQ1RK1 b - - bm Rc4; id "Jens Bæk Nielsen.35"; ==> modified
r1bqr1k1/ppp2pp1/2nb1n1p/3p4/3P1B2/2PB3P/PP1N1PP1/R2Q1RK1 w - - bm Be3; id "Jens Bæk Nielsen.36";
r2qr1k1/ppp2pp1/2nb1n1p/3p4/3P1B2/2PB1N1P/PP1N1PP1/R2Q1RK1 w - - bm Bxd6; id "Jens Bæk Nielsen.37";
4b3/p3kp2/6p1/3pP2p/2pP1P2/4K1P1/P3N2P/8 w - - bm f5; id "Jens Bæk Nielsen.38";
8/8/1P2P3/8/6p1/2r2kP1/7P/1R4K1 b - - bm Re3; id "Jens Bæk Nielsen.39";
r2q1rk1/ppp1ppbp/2np1np1/8/2PPP3/2NBBQ1P/PP3PP1/R3K2R b KQ - bm Nd7; id "Jens Bæk Nielsen.40";

[JBNielsen]

Thank you very much.

Diagram 39+40+41 can be ignored. It is moves that set up a trap against humans. They won't work against computers.

Diagram 42+43 can also be ignored. It was a silly attempt to see if computers can play easy moves fast. But they don't, and it really doesn't matter very much.