TT bug? or something else

Discussion of chess software programming and technical issues.

Moderator: Ras

amanjpro
Posts: 883
Joined: Sat Mar 13, 2021 1:47 am
Full name: Amanj Sherwany

TT bug? or something else

Post by amanjpro »

I witnessed Zahak (white) blow away a nice sequence of winning moves:

8 
7 
6 
5 
4 
3 
2 
1 
abcdefgh

6qQ/p7/1r4k1/5p2/4p2P/1P6/P4PP1/6K1 w - - 0 41

Here, Zahak played the dreadful Qe5. I would have said it is TT bug, but Zahak knew exactly how bad the move is, and guessed the new eval score, and guessed the opponent move, here is the PV of Zahak after the move:

Code: Select all

41. Qe5 {d=13, sd=14, pd=Qxg8, mt=11440, tl=785517, s=1313624, n=15028345, pv=Qe5 Qd8 g3 Qf6 Qf4 Ra6 g4 fxg4 Qxg4+ Kh6 a4 Rb6 Qxe4 Rxb3, tb=557391, h=99.9, ph=62.5, wv=-3.73, R50=49, Rd=-21, Rr=5, mb=+2+0+0-1+0,}
I wonder where could the bug be? I am sort of ruling out TT, because of PV and eval, but maybe I am wrong, and the bug is exactly there?

If interested here is the game:

amanjpro
Posts: 883
Joined: Sat Mar 13, 2021 1:47 am
Full name: Amanj Sherwany

Re: TT bug? or something else

Post by amanjpro »

Even stranger, black's move wasn't even a surprise to Zahak, here is Zahak's PV in the move prior to the blunder:

Code: Select all

 Rg8 d=14, sd=20, pd=Rc6, mt=29360, tl=794957, s=1393231, n=40905310, pv=Rg8+ Qxg8 Qxg8+ Kf6 h5 Ke5 Qg7+ Ke6 h6 Rb8 h7 Rd8 Qg8+ Kd7 Qf7+ Kc6 Qf6+ Kc7 h8=Q Rxh8, tb=3398469, h=99.9, ph=61.2, wv=12.00, R50=49, Rd=-21, Rr=5, mb=+2+0+0+0+0, 
 
User avatar
hgm
Posts: 28318
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: TT bug? or something else

Post by hgm »

The problem is not that the PV is wrong, but that it somehow doesn't see that Qxg8+ is (a full Queen) better. You would have to figure out what score that move had, and why. (This could have come from the TT.)

So just apply the standard method for debugging, of printing the move list with scores in every node from which the score propagated, or from the node that filled the TT entry where it came from.
amanjpro
Posts: 883
Joined: Sat Mar 13, 2021 1:47 am
Full name: Amanj Sherwany

Re: TT bug? or something else

Post by amanjpro »

hgm wrote: Mon Jun 14, 2021 10:26 pm The problem is not that the PV is wrong, but that it somehow doesn't see that Qxg8+ is (a full Queen) better. You would have to figure out what score that move had, and why. (This could have come from the TT.)

So just apply the standard method for debugging, of printing the move list with scores in every node from which the score propagated, or from the node that filled the TT entry where it came from.
Problem, I cannot reproduce it with fresh runs:

Code: Select all

position fen 6qQ/p7/1r4k1/5p2/4p2P/1P6/P4PP1/6K1 w - - 0 41
go movetime 20000
info depth 1 seldepth 2 tbhits 0 hashfull 0 nodes 60 nps 292974 score cp 715 time 0 pv h8g8 g6f6
info depth 2 seldepth 3 tbhits 0 hashfull 0 nodes 196 nps 357916 score cp 748 time 0 pv h8g8 g6f6 h4h5
info depth 3 seldepth 5 tbhits 30 hashfull 0 nodes 1064 nps 563503 score cp 768 time 1 pv h8g8 g6f6 g8h8 f6e6 h4h5
info depth 4 seldepth 5 tbhits 162 hashfull 1 nodes 4254 nps 561015 score cp 768 time 7 pv h8g8 g6f6 g8e8 b6a6 a2a4
info depth 5 seldepth 7 tbhits 823 hashfull 4 nodes 14966 nps 487753 score cp 779 time 30 pv h8g8 g6f6 g8d8 f6e6 d8c7 b6b4 c7a7
info depth 6 seldepth 8 tbhits 1872 hashfull 9 nodes 31517 nps 482572 score cp 832 time 65 pv h8g8 g6f6 h4h5 f6e5 g8g7 e5e6 g7a7 b6b5
info depth 7 seldepth 11 tbhits 3477 hashfull 17 nodes 51917 nps 468148 score cp 836 time 110 pv h8g8 g6f6 g8g5 f6e6 g5g6 e6e5 g6g7 b6f6 g7a7 f6g6 a7d7
info depth 8 seldepth 11 tbhits 8218 hashfull 34 nodes 103033 nps 456040 score cp 893 time 225 pv h8g8 g6f6 h4h5 f5f4 g8g6 f6e5 g6g5 e5d4 g5f4 b6a6 a2a4
info depth 9 seldepth 10 tbhits 18727 hashfull 72 nodes 210921 nps 453669 score cp 891 time 464 pv h8g8 g6f6 h4h5 b6b7 g1h2 e4e3 f2e3 b7b6 h5h6 b6b4
info depth 10 seldepth 14 tbhits 37153 hashfull 124 nodes 369964 nps 466364 score cp 912 time 793 pv h8g8 g6f6 h4h5 f6e5 g8g7 e5d5 g7a7 b6d6 a7f7 d5e5 f7g7 e5d5 h5h6 f5f4
info depth 11 seldepth 16 tbhits 130223 hashfull 387 nodes 1355215 nps 489540 score cp 1017 time 2768 pv h8g8 g6f6 h4h5 e4e3 f2e3 f6e7 g8g7 e7d8 g7g5 d8c7 g5f5 a7a6 f5c5 b6c6 c5e7 c7b6
info depth 12 seldepth 17 tbhits 259447 hashfull 539 nodes 2346519 nps 487749 score cp 1041 time 4810 pv h8g8 g6h6 g8g5 h6h7 g5f5 h7h6 f5e4 b6a6 e4e2 a6b6 e2e3 h6g6 e3e7 b6f6 h4h5 g6f5 e7a7
info depth 13 seldepth 23 tbhits 543950 hashfull 728 nodes 4470942 nps 483233 score cp 1045 time 9252 pv h8g8 g6h6 g8g5 h6h7 g5f5 h7h6 f5e4 b6a6 e4e3 h6g7 e3e7 g7g6 h4h5 g6h6 e7e3 h6g7 e3c3 g7f7 c3f3 a6f6 f3b7 f7e6 b7a7
info depth 13 seldepth 23 tbhits 543950 hashfull 728 nodes 4470942 nps 483231 score cp 1045 time 9252 pv h8g8 g6h6 g8g5 h6h7 g5f5 h7h6 f5e4 b6a6 e4e3 h6g7 e3e7 g7g6 h4h5 g6h6 e7e3 h6g7 e3c3 g7f7 c3f3 a6f6 f3b7 f7e6 b7a7
bestmove h8g8 ponder g6h6
quit
At the moment, I am suspecting this to be the issue:

Code: Select all

		if score > bestscore {
			// Potential PV move, lets copy it to the current pv-line
			e.innerLines[searchHeight].AddFirst(move)
			e.innerLines[searchHeight].ReplaceLine(e.innerLines[searchHeight+1])
			if score >= beta {
				e.TranspositionTable.Set(hash, move, score, depthLeft, LowerBound, e.Ply)
				// e.AddKillerMove(move, searchHeight)
				e.AddHistory(move, move.MovingPiece(), move.Destination(), depthLeft)
				return score
			}
			bestscore = score
			hashmove = move
		}
I update innerLines (which later becomes PV and bestmove, if the node is root) before checking for betacutoff. And this, in some cases may probably result in a bad move... at least my instinct says so. I am running a match between the current version, and a version where the update happens after beta-cutoff check.

But the fact that I cannot reproduce it, makes me not have so much faith in this hunch :'(
User avatar
hgm
Posts: 28318
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: TT bug? or something else

Post by hgm »

Blunders that are not reproducible are sort of hopeless. Sometimes it helps to start one or two moves earlier, and also have it think on those (roughly as long as in the original game), to get the same filling of the TT.
amanjpro
Posts: 883
Joined: Sat Mar 13, 2021 1:47 am
Full name: Amanj Sherwany

Re: TT bug? or something else

Post by amanjpro »

hgm wrote: Mon Jun 14, 2021 10:38 pm Blunders that are not reproducible are sort of hopeless. Sometimes it helps to start one or two moves earlier, and also have it think on those (roughly as long as in the original game), to get the same filling of the TT.
That is actually a very good idea! Thanks for the tip... will do that next
amanjpro
Posts: 883
Joined: Sat Mar 13, 2021 1:47 am
Full name: Amanj Sherwany

Re: TT bug? or something else

Post by amanjpro »

So, I tried going back a few moves and couldn't reproduce the issue.

What I thought was happening was:

- In root node, first move is hashmove, and it finds the bestmove
- Other nodes are tried, and one of them is causing beta-cutoff (in this case, Qe5). And since the code above updates PV when the score is higher than bestscore, and before betacutoff is tried, it updates PV to that move...

Not sure if this is the scenario, but I don't have a better explanation than that, gotta think about it harder :(
amanjpro
Posts: 883
Joined: Sat Mar 13, 2021 1:47 am
Full name: Amanj Sherwany

Re: TT bug? or something else

Post by amanjpro »

amanjpro wrote: Tue Jun 15, 2021 1:44 am So, I tried going back a few moves and couldn't reproduce the issue.

What I thought was happening was:

- In root node, first move is hashmove, and it finds the bestmove
- Other nodes are tried, and one of them is causing beta-cutoff (in this case, Qe5). And since the code above updates PV when the score is higher than bestscore, and before betacutoff is tried, it updates PV to that move...

Not sure if this is the scenario, but I don't have a better explanation than that, gotta think about it harder :(
Mmmmm... thinking abou ti, I believe it makes sense...
amanjpro
Posts: 883
Joined: Sat Mar 13, 2021 1:47 am
Full name: Amanj Sherwany

Re: TT bug? or something else

Post by amanjpro »

The only reason I doubt this, is the frequency where this bug happens, I have only seen it a few times... which makes me doubt the above hypothesis
User avatar
hgm
Posts: 28318
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: TT bug? or something else

Post by hgm »

amanjpro wrote: Tue Jun 15, 2021 1:44 am So, I tried going back a few moves and couldn't reproduce the issue.

What I thought was happening was:

- In root node, first move is hashmove, and it finds the bestmove
- Other nodes are tried, and one of them is causing beta-cutoff (in this case, Qe5). And since the code above updates PV when the score is higher than bestscore, and before betacutoff is tried, it updates PV to that move...

Not sure if this is the scenario, but I don't have a better explanation than that, gotta think about it harder :(
Well, this 'scenario' is how alpha-beta with iterative deepening works in general. You start with the move that was best in the previous iteration, and when you find a better one you switch to that. Of course in the root you cannot have a beta cutoff, as beta is +infinite there. Unless you would use aspiration, and then you would re-search after enlarging the window.

Qe5 had a poor score. It can only have superceded Qxg8 if that had an even lower score. And this is obviously wrong, as this line is about a Queen better.