possible SF regression (testposition) ?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

Spliffjiffer
Posts: 416
Joined: Thu Aug 02, 2012 7:48 pm
Location: Germany

possible SF regression (testposition) ?

Post by Spliffjiffer »

As a notorious SF-tester (private) i noticed that since about 2 months the newest SF-dev.-versions have problems with the famous "Kasparov-Topalov"-game from 1999.
In the position below SF (Timestamp: 1390645772 from january 25.) "realized" quite quickly that after Re7 white is at least not worse
[D] b2r3r/k4p1p/p2q1np1/Np1P4/3p1Q2/P4PPB/1PP4P/1K2R3 w - - 0 2
bm Re7
Newest SF-versions getting the clue much slower (b4-move)
It seems to me that this began with the dev-version from 26.jan 2014 and is getting slightly worse until today.

Also the following position was "solved" about twice as fast by the version of jan 25. mentioned above compared to nearly all versions that followed up since then:
[D] 1rbq1r1k/pp2n1bp/3p1p2/2p1pPP1/P3B2Q/2NP3N/1P5P/n1B2RK1 b - - 0 16
bm Be6

To me these problems seem to be serious honestly and i hope this can help the further developement a little tiny bit
Deep respect for the SF-Team, i love you guys :-)

tested on winXP32 on a core2duo e6600 with th=2 and 256mb hash

regards from germany
Wahrheiten sind Illusionen von denen wir aber vergessen haben dass sie welche sind.
Top
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: possible SF regression (testposition) ?

Post by zullil »

Based on the dates, what you observe is likely related to changes in null-move pruning.
Top
Spliffjiffer
Posts: 416
Joined: Thu Aug 02, 2012 7:48 pm
Location: Germany

Re: possible SF regression (testposition) ?

Post by Spliffjiffer »

Thanks for your reply Louis :-)
Yes, thats right...there are much more positions (mainly midgame-positions) which are handled much better by SF-versions before 26. january !
Wahrheiten sind Illusionen von denen wir aber vergessen haben dass sie welche sind.
Top
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: possible SF regression (testposition) ?

Post by zullil »

Spliffjiffer wrote:As a notorious SF-tester (private) i noticed that since about 2 months the newest SF-dev.-versions have problems with the famous "Kasparov-Topalov"-game from 1999.
In the position below SF (Timestamp: 1390645772 from january 25.) "realized" quite quickly that after Re7 white is at least not worse
[D] b2r3r/k4p1p/p2q1np1/Np1P4/3p1Q2/P4PPB/1PP4P/1K2R3 w - - 0 2
bm Re7
Newest SF-versions getting the clue much slower (b4-move)
Here's what the latest SF does, using just 1 thread:

Code: Select all

Searching: b2r3r/k4p1p/p2q1np1/Np1P4/3p1Q2/P4PPB/1PP4P/1K2R3 w - - 0 2
infinite: 1 ponder: 0 time: 0 increment: 0 moves to go: 0
 1   -1.92   00:00      94  Qxd4+ Kb8 
 2   -1.82   00:00     361  Qxd4+ Kb8 c4 bxc4 Nxc4 
 3   -1.96   00:00     754  Qxd4+ Kb8 c4 Rhe8 
 4   -2.31   00:00    1513  Qxd4+ Kb8 c4 Rhe8 Rc1 
 5   -1.99   00:00    2604  Qxd4+ Kb8 c4 Rhe8 Rc1 Nd7 cxb5 axb5 
 6   -1.99   00:00    3266  Qxd4+ Kb8 c4 Rhe8 Rc1 Nd7 cxb5 axb5 
 7   -1.99   00:00    4708  Qxd4+ Kb8 c4 Rhe8 Rc1 Nd7 cxb5 axb5 
 8   -1.47   00:00   12201  Qxd4+ Qb6 Re7+ Nd7 Rxd7+ Rxd7 Qxh8 Rxd5 Nb3 Rd1+ Ka2 
                            Bxf3 Qxh7 
 9   -0.74   00:00   22916  Qxd4+ Qb6 Re7+ Nd7 Qc3 f5 Nc6+ Bxc6 dxc6 b4 axb4 
                            Qg1+ Ka2 
10   -1.59   00:00   68852  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ b3 f5 Rxh7 Kb6 
                            Rxh8 Rxh8 Qxh8 Kxa5 Qh4 Bxf3 
11   -2.21   00:00  141984  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ b3 Kb6 Rxd7 
                            Qc5 Qxc5+ Kxc5 Nb7+ Bxb7 Rxb7 Rd2 b4+ Kc6 Rxf7 Rxc2+ 
                            Kb3 Rxh2 
12   -2.03   00:00  155626  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ b3 Kb6 Rxd7 
                            Qc5 Qxc5+ Kxc5 Nb7+ Bxb7 Rxb7 Rd2 b4+ Kc6 Rxf7 Rxc2+ 
                            Kb3 Rxh2 Bd7+ 
13   -2.04   00:00  205695  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ b3 Kb6 Rxd7 
                            Qc5 Qxc5+ Kxc5 Nb7+ Bxb7 Rxb7 Rd2 b4+ Kc6 Rxf7 Rxc2+ 
                            Kb3 Rxh2 Bd7+ Kd6 Bg4 
14   -2.19   00:00  354074  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ Nb3 f5 Qc7+ 
                            Ka8 Rxd7 Rxd7 Qxd7 Qd1 Qc7 Qxf3 Qb6 Bxb3+ cxb3 Qb7 
                            Qf6 Rb8 
15   -2.25   00:00  476682  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ Nb3 f5 Qc7+ 
                            Ka8 Rxd7 Rxd7 Qxd7 Qd1 Qc7 Qxf3 Qb6 Bxb3+ cxb3 Qb7 
                            Qf6 Rc8 g4 Rc6 Qe5 
16   -2.53   00:00   1103K  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ b3 Kb6 Rxd7 
                            Qc5 Rb7+ Bxb7 Qxc5+ Kxc5 Nxb7+ Kc6 Nxd8+ Rxd8 Kb2 
                            Rd2 Bc8 a5 h4 
17   -2.73   00:01   1672K  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ Nb3 f5 Qc7+ 
                            Ka8 Rxd7 Rxd7 Qxd7 Qd1 Qc7 Qxf3 Qd6 Bb7 Qe7 Qd5 Bf1 
                            Qh1 Bd3 Qxh2 Nc5 Qh1 
18   -2.34   00:01   3200K  Qxd4+ Qb6 Re7+ Nd7 Qc3 Qg1+ Ka2 Bxd5+ b3 Kb6 Rxd7 
                            Qc5 Nc4+ bxc4 Qf6+ Kb5 a4+ Ka5 Rxd8 cxb3+ cxb3 Qc2+ 
                            Ka1 Qd1+ Kb2 Qxb3+ Kc1 Qc4+ Kd2 Rxd8 Qxd8+ Kxa4 
19   +0.00   00:02   3969K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ b4 axb4+ Qxb4 Qc7+ Ka4 b3+ 
                            Ka3 Qe5 Qc5 Bf1 Rc8 Qa1+ Kb4 Qa4+ Kc3 Qa1+ Kb4 
20   +0.00   00:02   4531K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 Rd7 
                            Rxd7 Bxd7 Qg1+ Ka2 Qc5 dxc6 Qc4+ Kb1 Qf1+ Ka2 Qc4+ 
21   -1.33   00:04   8794K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 f5 c7 Rc8 Re6+ Kxc7 Rxa6 Rhe8 c3 Kb7 
                            Rd6 Re2 f4 Rxh2 Bf1 Rh1 Rd1 Kb6 Kc2 Rh2+ Kb3 Re8 
22   -1.38   00:05  10969K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 f5 c7 Rc8 c4 Rxc7 Rxc7 Kxc7 cxb5 
                            axb5 Bf1 Kc6 Kc2 Kc5 f4 b4 Bg2 bxa3 bxa3 Re8 Kd3 
                            Rd8+ Ke3 Rb8 
23   -1.30   00:06  13330K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 f5 c7 Rc8 c4 Rxc7 Rxc7 Kxc7 cxb5 
                            axb5 Kc2 Re8 Kd2 Kb6 Bf1 Kc5 Bd3 b4 a4 Rd8 Ke3 Ra8 
                            Bb5 
24   -1.32   00:06  14233K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 f5 c7 Rc8 c4 Rxc7 Rxc7 Kxc7 cxb5 
                            axb5 Kc2 Re8 Kd2 Rd8+ Kc2 Kb6 Bf1 Kc5 Bd3 Rd4 Kc3 
                            b4+ axb4+ Rxb4 b3 Rb7 g4 fxg4 fxg4 
25   -1.53   00:12  27345K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 f5 c7 Rc8 c3 Rxc7 Re6+ Rc6 Re2 Rd8 
                            Kc2 Rcd6 f4 R8d7 Bf1 Rd1 Bh3 R7d6 Bg2 R1d3 Kc1 a5 
                            Kc2 a4 Bf1 h6 
26   -1.58   00:18  40528K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 f5 c7 Rc8 c3 Rxc7 Re6+ Rc6 Re2 Rd8 
                            Kc2 Rcd6 f4 R8d7 Bf1 Rd1 Bh3 a5 Bg2 R1d6 Bf1 a4 Bg2 
27   -1.78   00:34  76217K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 Rd1+ Ka2 f5 c7 Rc8 Rxh7 Rxc7 Rh6 Rc6 
                            c3 Rd2 g4 Rxc3 Rxg6+ Kc5 Rg5 Rxf3 Rxf5+ Rxf5 gxf5 
                            Rxh2 Bg4 Rf2 Kb3 Kd4 
28   -1.68   00:41  90818K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 Rd1+ Ka2 f5 g4 fxg4 Bxg4 Kxc6 h4 Rd2 
                            Kb3 h5 Be6 Rf2 Rg7 Rxf3+ c3 Rh6 Rf7 Rxf7 Bxf7 Kc5 
                            Kc2 a5 
29   -1.99   01:09 152874K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 Rd1+ Ka2 f5 g4 fxg4 Bxg4 Kxc6 f4 Rd2 
                            Bf3+ Kd6 Ra7 Rxh2 Rxa6+ Ke7 Be4 Rf8 Bd3 Rxf4 Kb3 g5 
                            Bxh7 g4 Bd3 g3 Rg6 g2 Bxb5 
30   -2.02   01:30 198938K  Re7+ Kb6 Qxd4+ Kxa5 Qc3+ Kb6 Qd4+ Qc5 Qxf6+ Bc6 
                            Qxc6+ Qxc6 dxc6 Rd1+ Ka2 f5 g4 fxg4 Bxg4 Kxc6 Kb3 
                            Rd2 Re6+ Rd6 Re7 h5 Bh3 Rf8 Bg2 Kb6 Re5 Rff6 Rg5 h4 
                            h3 a5 a4 bxa4+ Kxa4 Rf5 Rxf5 gxf5 
31   +0.00   01:54 250839K  Re7+ Kb6 Qxd4+ Kxa5 b4+ Ka4 Qc3 Qxd5 Ra7 Bb7 Rxb7 
                            Rhe8 Rb6 Ra8 Rxf6 Qc4 Qxc4 bxc4 Kb2 Re7 f4 Kb5 Rd6 
                            f5 Bg2 Rc8 Rd5+ Kb6 Rd6+ Kb5 
32   +0.00   02:01 265004K  Re7+ Kb6 Qxd4+ Kxa5 b4+ Ka4 Qc3 Qxd5 Ra7 Bb7 Rxb7 
                            Rhe8 Rb6 Ra8 Rxf6 Qc4 Qxc4 bxc4 Kb2 Re7 f4 Kb5 Rd6 
                            f5 Bg2 Rc8 Rd5+ Kb6 Rd6+ Kb5 
33   +0.00   02:05 272920K  Re7+ Kb6 Qxd4+ Kxa5 b4+ Ka4 Qc3 Qxd5 Ra7 Bb7 Rxb7 
                            Rhe8 Rb6 Ra8 Rxf6 Qc4 Qxc4 bxc4 Kb2 Re7 f4 Kb5 Rd6 
                            f5 Bg2 Rc8 Rd5+ Kb6 Rd6+ Kb5 
34   +0.00   02:22 310397K  Re7+ Kb6 Qxd4+ Kxa5 b4+ Ka4 Qc3 Qxd5 Ra7 Bb7 Rxb7 
                            Rhe8 Rb6 Ra8 Rxf6 Qc4 Qxc4 bxc4 Kb2 Re7 f4 Kb5 Rd6 
                            f5 Bg2 Rc8 Rd5+ Kb6 Rd6+ Kb5 
35   +0.39   02:43 353041K  Re7+ Kb6 Qxd4+ Kxa5 b4+ Ka4 Qc3 Qxd5 Ra7 Bb7 Rxb7 
                            Rhe8 Rb6 Ra8 Rc6 Re2 Rc5 Qd1+ Kb2 Rxc2+ Qxc2+ Qxc2+ 
                            Rxc2 Re8 Bf1 Nd5 Rc6 Re6 Rxe6 fxe6 Bh3 Ne3 Bxe6 Nd1+ 
                            Kc2 Nf2 Kc3 Kxa3 Bg8 h6 Bf7 Nd1+ Kd2 Nb2
Top
Spliffjiffer
Posts: 416
Joined: Thu Aug 02, 2012 7:48 pm
Location: Germany

Re: possible SF regression (testposition) ?

Post by Spliffjiffer »

yep....unfortunately i dont know how to post engine outputs but eg the version from 25. january by Joona is significantly faster here...
Wahrheiten sind Illusionen von denen wir aber vergessen haben dass sie welche sind.
Top
User avatar
Eelco de Groot
Posts: 4567
Joined: Sun Mar 12, 2006 2:40 am
Full name:   

Re: possible SF regression (testposition) ?

Post by Eelco de Groot »

It is a bit of an open door for anyone with some experience with computerchess, but just to mention it, it is a bit of a tradeoff; you do more nullmove pruning (in this case, the nullmove searches are made shorter so they take less time so you can search deeper) so get a better depth overall. The result of the change Variable null-move value based reduction is about 2.6 elo. But in some positions you will miss things. The depth of the verification search is also reduced so it will be slightly worse detecting any Zugzwangs.

Maybe a list of positions like this if it can be shown to be related to this change, could be helpful. Maybe some unrelated search improvement would help, but the weakness involving Zugzwang will not go away unless nullmove is totally disabled.

P.S.: engine output in most GUIs can be simply copied I think and then posted/pasted here in Talkchess. That should not be a problem?

Eelco
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
Top
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: possible SF regression (testposition) ?

Post by zullil »

Spliffjiffer wrote:
Also the following position was "solved" about twice as fast by the version of jan 25. mentioned above compared to nearly all versions that followed up since then:
[D] 1rbq1r1k/pp2n1bp/3p1p2/2p1pPP1/P3B2Q/2NP3N/1P5P/n1B2RK1 b - - 0 16
bm Be6
The very latest Stockfish (signature=7451319, Threads=16, Hash=1024) still finds this fairly quickly:

Code: Select all

info depth 29 seldepth 55 score cp 117 nodes 307014343 nps 14255866 time 21536 multipv 1 pv c8e6 f5e6 f6f5 g5g6 e7g6 h4d8 b8d8 e4f5 a1b3 c1g5 d8e8 c3b5 f8f5 f1f5 e8e6 f5f7 e5e4 d3e4 h7h6 f7b7 b3d4 g5d8 a7a6 b5c3 d4f3 g1g2 g6h4 d8h4 f3h4 g2g3 g7c3 b2c3 e6e4 h3f4 h8g8 a4a5 h4f5 g3f3 e4e3 f3g4
Top

Return to “Computer Chess Club: General Topics”