SEE accuracy

[sandermvdb]

I was interested in the accuracy of my Static Exchange Evaluation so I compared this score against the Quiescence score (on the same square of course). I took a testset containing 1.5 million violent positions and calculated both scores for all possible attacks for these positions. Initially this gave me the following accuracy (correct/total):
0.9961

I was a bit surprised that the accuracy was already this good!
When using the pinned-pieces for certain types in the SEE evaluation I got the number up to
0.9966

I was already playing with discovered checks and because I am already calculating the pinned-pieces, calculating the discovered check pieces was easy. Adding this knowledge to the SEE improved the accuracy to
0.9973

Next I looked at positions where the SEE failed on. This was when a pinned-piece was not pinned anymore because the pinner attacked another piece so the pinned-piece was not pinned anymore and should now actually be involved in the SEE calculation. This however meant that somehow I needed to update the pinned-pieces during the SEE calculation. I was curious about its effect and thought that I could calculate this quite effectively so after I implemented this and the accuracy improved to
0,9997!

I can see that the nodecount has been decreased but the nps has also decreased, both by about 3%.

Positions where the SEE score is still wrong is for instance when a pinned pawn is being captured by an en-passant capture. But this is so rare (and cumbersome to implement) that I am not going to look into this.

I still need to investigate how important the accuracy of the SEE evaluation really is (elo wise). I use the SEE score for move ordering and for pruning bad captures in the Q-search but maybe everything above 0.99 is more than enough. Still, I like these kind of tweaks because these are easy to tune/fix and you don't need to play 1000 games to check the impact of a certain change in the code. (I know that eventually you do...)

Regarding the testset, see the following link by the author of Zurichess, again
http://www.talkchess.com/forum/viewtopic.php?t=61427

[hgm]

What do you mean, by "Quiescence score on the same square"? It sounds a bit like what the definition of SEE is...

[sandermvdb]

What do you mean, by "Quiescence score on the same square"? It sounds a bit like what the definition of SEE is...

Quiescence is 100% accurate. SEE is optimized for speed and if it does not take into account pinned pieces, checks, ..., it will not be 100% accurate

[hgm]

SEE is defined as the result of a search that only takes legal moves to a single given square into account. Not every implementation manages to calculate this perfectly, for efficiency reasons.

And that makes perfect sense, as the main reason SEE results differ greatly from QS is not the occasional pin or check, but overloaded protectors, soft-pins, capture of an essential protector and the like. I.e. going-ons on different squares. E.g. a 'bad capture' RxN can be winning if the N was the only piece protecting a B attacked by a R.

[sandermvdb]

SEE is defined as the result of a search that only takes legal moves to a single given square into account. Not every implementation manages to calculate this perfectly, for efficiency reasons.

And that makes perfect sense, as the main reason SEE results differ greatly from QS is not the occasional pin or check, but overloaded protectors, soft-pins, capture of an essential protector and the like. I.e. going-ons on different squares. E.g. a 'bad capture' RxN can be winning if the N was the only piece protecting a B attacked by a R.

I understand completely what you are saying. So the question remained, how accurate is my SEE implementation? It was 0.9961 and with some tweaks I was able to get it to 0,9997, not knowing (yet) if this results in a stronger engine. That is all I am saying!

If you don't think this is interesting, please ignore my post.

[hgm]

Well, I was not sure what you were actually testing, which is why I enquired.

And indeed, if the difference between the QS of a given move and its see would be wrong ~20% of the time, it is not very interesting to know whether this would be 20,00% or 20.39%. The most important conclusion of your research seems to be "don't bother about pins and checks in SEE, as there is virtually nothing to gain there".