"Poor Fish": very hard test suite

mcostalba · Post by **mcostalba** » Sun Sep 25, 2016 7:10 pm

Vinvin wrote:You can use my "hard set" too : http://www.talkchess.com/forum/viewtopic.php?t=60573

Thanks a lot!

Indeed I plan to build up a test suite, but actually it is not for testing, just to collect positions that humans but not engines where able to solve (in reasonable time).

So this test set will:

Code: Select all

1&#41; Contain only positions from human games or human studies &#40;with reference&#41;

2&#41; Be sound, so that between best move&#40;s&#41; and the others there is an important difference

3&#41; Be really difficult to solve for engines

I plan to publish somewhere on the net and keep a live leader-board with who found the solution / hardware and software used / pv line with depth an time.

So if you have some position you think that can qualify, please submit.

BTW I have redone the test 240 seconds per position, current SF, 3 threads on my QUAD (Fritzmark 12.19, 5.852Mnps)

Code: Select all

1. arasan19.16,  poor_fish   > 240s.
2. arasan19.23,  poor_fish   Solved in 29.98s/28; Solved&#58; 1
3. arasan19.42,  poor_fish   > 240s.
4. arasan19.44,  poor_fish   Solved in 76.77s/30; Solved&#58; 2
5. arasan19.115,  poor_fish   Solved in 60.56s/35; Solved&#58; 3
6. arasan19.116,  poor_fish   > 240s.
7. arasan19.137,  poor_fish   > 240s.
8. arasan19.154,  poor_fish   Solved in 13.34s/30; Solved&#58; 4
9. arasan19.155,  poor_fish   Solved in 183.01s/32; Solved&#58; 5
10. arasan19.157,  poor_fish   Solved in 22.87s/26; Solved&#58; 6
11. arasan19.159,  poor_fish   > 240s.
12. arasan19.188,  poor_fish   Solved in 43.23s/31; Solved&#58; 7
13. arasan19.179,  poor_fish   Solved in 108.51s/29; Solved&#58; 8
14. arasan19.189,  poor_fish   > 240s.

So I will drop positions that were found and this is what remain (at the moment):

Code: Select all

rn1q1rk1/2pbb3/pn2p3/1p1pPpp1/3P4/1PNBBN2/P1P1Q1PP/R4R1K w - - bm Nxg5; id "arasan19.16"; c0 "Dann Corbit, CCC 2015";
rn1qr1k1/1p2bppp/p3p3/3pP3/P2P1B2/2RB1Q1P/1P3PP1/R5K1 w - - bm Bxh7+; id "arasan19.42"; c0 "Polugaevsky-Torre, London 1984";
8/4nk2/1p3p2/1r1p2pp/1P1R1N1P/6P1/3KPP2/8 w - - bm Nd3; id "arasan19.116"; c0 "Grandelius-Raznikov, EU Youth Chess Ch B18, Albena 2011";
3r1r1k/pp5p/4b1pb/6q1/3P4/4p1BP/PP2Q1PK/3RRB2 b - - bm Qxg3+; id "arasan19.137"; c0 "Apicella-Lautier, Clichy 2001";
r1b2rk1/1pq1nppp/pbn1p3/8/3N4/3BBN2/PPP1QPPP/3R1RK1 w - - bm Bxh7+; id "arasan19.159"; c0 "Leu-Muck, DDR-ch H137 corr 1973";
r1b3r1/5p1k/p1n2P1p/P1qpp1P1/1p1p4/3P2Q1/BPPB2P1/R4RK1 w - - bm Kf2; id "arasan19.179"; c0 "Efremov-Fischer, Danube/4 ICCF 2012";
4K1k1/8/1p5p/1Pp3b1/8/1P3P2/P1B2P2/8 w - - bm f4; id "arasan19.189"; c0 "Šachová skladba 2011-12, 1st Prize";

I have not removed the 179 because I like it

Vinvin · Post by **Vinvin** » Mon Sep 26, 2016 4:53 am

mcostalba wrote:So this test set will:

Code: Select all

1&#41; Contain only positions from human games or human studies &#40;with reference&#41;

2&#41; Be sound, so that between best move&#40;s&#41; and the others there is an important difference

3&#41; Be really difficult to solve for engines

1) I plan to add references for the test set (around 150 are from human games).
2) It always has been my concern to have a CLEAR solution.
3) I plan to rerun the latest set next week.
The set with latest correction is here : http://home.scarlet.be/vincentlejeune/c ... 6beta6.epd

Until that you can find "14 solutions Stockfish should find 100 times faster" here : http://www.talkchess.com/forum/viewtopic.php?t=57674

Jouni · Post by **Jouni** » Mon Sep 26, 2016 4:41 pm

May be this is the most difficult/impossible to SF number 41:

[D]2b1rk2/5p2/p1P5/2p2P2/2p5/7B/P7/2KR4 w - - 0 1

Analysis by Houdini 3 x64 Tactical:

1.f6 Bxh3 2.c7 Kg8 3.Rd8 Rf8 4.Kc2 Bc8 5.Kc3 Be6 6.a3 Kh7 7.Rxf8 Kg6 8.c8Q Bxc8 9.Rxc8 Kxf6 10.Rxc5 Ke6 11.Rxc4 Kd5 12.Ra4 Ke6 13.Rxa6+ Ke7 14.Ra5 f6 15.Ra7+ Kf8 16.Ra6 f5 17.Ra5 f4 18.Rf5+ Ke7 19.Rxf4 Kd7 20.Rf6 Kc7 21.Ra6 Kd8 22.Ra5 Kd7 23.a4 Kd6 24.Kd3 Ke6 25.Kc3 Kf6 26.Ra6+ Ke5 27.Kc4 Ke4 28.a5 Ke3 29.Re6+ Kf3 30.a6 Kf4 31.a7
+- (9.67) Depth: 24/64 00:00:32 337mN, tb=54

Vinvin · Post by **Vinvin** » Mon Sep 26, 2016 4:47 pm

mcostalba wrote:So I will drop positions that were found and this is what remain (at the moment):

Code: Select all

rn1q1rk1/2pbb3/pn2p3/1p1pPpp1/3P4/1PNBBN2/P1P1Q1PP/R4R1K w - - bm Nxg5; id "arasan19.16"; c0 "Dann Corbit, CCC 2015";
rn1qr1k1/1p2bppp/p3p3/3pP3/P2P1B2/2RB1Q1P/1P3PP1/R5K1 w - - bm Bxh7+; id "arasan19.42"; c0 "Polugaevsky-Torre, London 1984";
8/4nk2/1p3p2/1r1p2pp/1P1R1N1P/6P1/3KPP2/8 w - - bm Nd3; id "arasan19.116"; c0 "Grandelius-Raznikov, EU Youth Chess Ch B18, Albena 2011";
3r1r1k/pp5p/4b1pb/6q1/3P4/4p1BP/PP2Q1PK/3RRB2 b - - bm Qxg3+; id "arasan19.137"; c0 "Apicella-Lautier, Clichy 2001";
r1b2rk1/1pq1nppp/pbn1p3/8/3N4/3BBN2/PPP1QPPP/3R1RK1 w - - bm Bxh7+; id "arasan19.159"; c0 "Leu-Muck, DDR-ch H137 corr 1973";
r1b3r1/5p1k/p1n2P1p/P1qpp1P1/1p1p4/3P2Q1/BPPB2P1/R4RK1 w - - bm Kf2; id "arasan19.179"; c0 "Efremov-Fischer, Danube/4 ICCF 2012";
4K1k1/8/1p5p/1Pp3b1/8/1P3P2/P1B2P2/8 w - - bm f4; id "arasan19.189"; c0 "Šachová skladba 2011-12, 1st Prize";

I have not removed the 179 because I like it

I ran an overnight analyze on this 7 positions.
Conditions : 10 min/positions, CPU : 6 cores 4 GHz, 8 GB HT, 6 pcs Syzygy, 4 runs.
Analyzing engine: Stockfish_160922_x64_modern_fast

Code: Select all

   Use seconds per move      &#58; 600 Seconds per move
   Analysis Lines            &#58; Minimum search depth=12
   Activate abort analysis   &#58; True. Minimal analysis depth 24. Abort, if the correct move is held 3 additional ply

                  1st run                     2nd run                3rd run                4th run          
 Nf3xg5         g2-g4                       Nf3xg5  * 37 Sec       g2-g4                  Nf3xg5  * 48 Sec 
 Bd3xh7         Bd3xh7  * 250 Sec           Bf4-e3                 Bd3xh7  * 195 Sec      Ra1-d1           
 Nf4-d3         h4xg5                       h4xg5                  h4xg5                  h4xg5            
 .. Qg5xg3      Qg5xg3  * 198 Sec           Qg5xg3  * 422 Sec      Qg5xg3  * 281 Sec      Qg5xg3  * 286 Sec
 Bd3xh7         Bd3xh7  * 77 Sec            Bd3xh7  * 6 Sec        Bd3xh7  * 15 Sec       Bd3xh7  * 8 Sec  
 Kg1-f2, ..     Kg1-f2  * 45 Sec            Kg1-f2  * 38 Sec       Kg1-f2  * 57 Sec       Kg1-f2  * 89 Sec 
 f3-f4          Ke8-d7                      Ke8-d7                 Ke8-d7                 Ke8-d7

mcostalba · Post by **mcostalba** » Mon Sep 26, 2016 6:49 pm

Vinvin wrote: I ran an overnight analyze on this 7 positions.

Great job!

I have just set-up a site under GitHub pages:

https://mcostalba.github.io/PoorFish/

It is stiill very primitive, and a lot of info regarding the positions is still missing. but should give a sketchy idea of what I have in mind.

It took my a while to setup this one because I didn't know Jekyll, the page generator used by GitHub pages and it comes with its full set of quirks

I have added the only 2 positions that at the moment seems to qualify.

peter · Post by **peter** » Mon Sep 26, 2016 7:14 pm

Hi!
And the original study from Smyslov starts even 4 moves earlier:

[D]1r3k2/5p2/p1P1b1p1/2p5/2p1PP2/8/P7/2KR1B2 w - - 0 1

1. f5 gxf5 2. Bh3 Re8 3. exf5 Bc8

lead to your test position with bm f6

[D]2b1rk2/5p2/p1P5/2p2P2/2p5/7B/P7/2KR4 w - - 0 4

mcostalba · Post by **mcostalba** » Mon Sep 26, 2016 8:55 pm

Thanks for the position and the explanation this is very interesting, indeed I would like to add as much info as possible to each position that qualifies to be in the set.

Only one position out of thousands will join this elite set, so it deserves a broader comment and historical notes.

Given a position, how it is possible to lookup the sources, there is some kind of big games database available that it is possible to query by fen string? A kind of super mega hash table of positions

peter · Post by **peter** » Mon Sep 26, 2016 9:17 pm

Hi Marco!

mcostalba wrote: Given a position, how it is possible to lookup the sources, there is some kind of big games database available that it is possible to query by fen string?

As for studies I don't have in my personal database already I always look into YACPDB at first:
http://www.yacpdb.org/#add
E.g. the one in question is to be found there too:
http://www.yacpdb.org/#search/MXIzazI1c ... LzEvMS8w/1

mcostalba · Post by **mcostalba** » Mon Sep 26, 2016 9:19 pm

I have found the full game of one of the 2 hard positions

http://chess-db.com/public/game.jsp?id= ... 1904.27155

Here is the pgn, the move of the puzzle is the move 50 Nd3:

[Event "ch-European Youth b18"]
[Site "?"]
[Date "2011.09.18"]
[Round "?"]
[White "Grandelius, Nils"]
[Black "Raznikov, Danny"]
[ECO "A30"]
[WhiteElo "2536"]
[BlackElo "2433"]
[Result "1-0"]

1. c4 c5 2. Nf3 Nc6 3. d4 cxd4 4. Nxd4 g6 5. g3 Bg7 6. Nc2 d6 7. Bg2 Be6 8. Ne3 Qd7 9. Nd5 Rc8 10. O-O Bh3 11. Nbc3 Bxg2 12. Kxg2 Ne5 13. b3 Nf6 14. Nxf6+ Bxf6 15. Nd5 Bg7 16. Bg5 Nc6 17. Rc1 h6 18. Be3 O-O 19. Qd2 Kh7 20. Rfd1 Rfd8 21. Qd3 e6 22. Nf4 Qe7 23. Qe4 Kg8 24. Bd2 a6 25. Bc3 d5 26. cxd5 exd5 27. Qxe7 Nxe7 28. Ba5 Rxc1 29. Rxc1 Rd7 30. Bb6 Nc6 31. Nd3 Re7 32. Kf1 h5 33. Rc5 Rd7 34. Nf4 Ne7 35. Rc2 Be5 36. Nd3 Bd6 37. Ke1 f6 38. Bc5 Bxc5 39. Rxc5 b6 40. Rc2 a5 41. Nf4 Kf7 42. h4 Ra7 43. Rd2 Rc7 44. Kd1 Rc5 45. Rd4 Ke8 46. Kd2 Kf7 47. a3 g5 48. b4 axb4 49. axb4 Rb5 50. Nd3 Nf5 51. e3 Nxd4 52. exd4 Kg6 53. Kc3 Kf5 54. f3 g4 55. Kb3 gxf3 56. Nf2 Kg6 57. Ka4 Rxb4+ 58. Kxb4 f5 59. Kb5 f4 60. gxf4 Kf5 61. Kxb6 Kxf4 62. Kc5 Kg3 63. Nd3 Kxh4 64. Kxd5 Kg3 65. Ke4 h4 66. Ke3

1-0

peter · Post by **peter** » Mon Sep 26, 2016 10:09 pm

Stockfish and komodo need the nullmove-pruning to be switched off for this position, Houdini gets it with default parameters too btw

"Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite

Re: "Poor Fish": very hard test suite