Stockfish's Asymmetric Evaluations

[Stephen Ham]

Dear gents,

In the last few months, I've perceived numerous examples of asymmetric evaluations from Stockfish. For example, in the following position:
r2r2k1/pp2n1b1/2ppqnpp/4p3/1PP1P3/1QNR1PP1/P3BB2/R5K1 w - - 0 21,
Stockfish 270618 selects 21 Rad1 after 20 minutes. It evaluates White having a 0.64 center-pawn advantage.

However, after the hash is cleared and it then computes a 21st move reply for Black, it immediately shows equality and continues to show equality through a further 20 minutes, selecting 21...Kh7 with a 0.12 evaluation.

I label this common behavior "asymmetric" as it favors one side after selecting its move. But when that move is played, it then rejects its previous evaluation, sometimes immediately, and then evaluates contrarily. I've even seen cases where it prefers White when it's White's move, and then prefers Black when it's Black's turn to respond.

Can you replicate the above example? This is just the most recent example I've perceived and not the most glaring. Also, please explain why Stockfish's evaluations are so asymmetric. I don't see this behavior from other chess engines, otherwise I'd mentally write this off as being related to non-determinism.

Sorry for my ignorance as this behavior may be commonly known. But you gents are the experts, so I hope to learn from you.

Thanks in advance.

-Steve-

[Dann Corbit]

If you are analyzing rather than game playing, make sure that you set the contempt to zero.
There is a built in offset for the side to move, otherwise.

[tpoppins]

The default contempt is 21, the quoted difference is 52. I'm afraid the contempt is a red herring here.

[yanquis1972]

IF it worked that simply, that'd mean the true evals were +.44 & +.33, which is reasonable. i'm also pretty sure it doesn't work that simply.

but i thought 'analysis contempt' had to be manually ticked, otherwise it'd only kick in during gameplay.

[Stephen Ham]

Hi Dann,

Thanks for your reply.

You wrote, "If you are analyzing rather than game playing, make sure that you set the contempt to zero.
There is a built in offset for the side to move, otherwise."

Yes, I'm analyzing in Infinite Analysis, but I don't see how contempt is a factor since Black is given the same contempt setting as White when it's Black's turn to move. And as I stated, Stockfish calculated/evaluated for White, and then for Black's reply after hash was cleared. Both colors were given 20 minutes of calculation with the same engine settings.

I can run this position again with contempt set to 0 and see if it changes matters. But my hypothesis is that since contempt remained static for both colors, then it should not be a factor. Also, given non-determinism, I'd probably need to run multiple tests, but lack the time and patience. Nonetheless, this asymmetric evaluation seems common only to Stockfish, since Komodo also has a contempt factor.

All the best,
Steve

[Dann Corbit]

Seems to me that contempt active for both sides doubles the problem.

Suppose that a true eval is 0.00, but you have +21 for contempt (let's also say white to move). You report a score of +21.
You finish analyzing and switch to the new position and analyze again.
Suppose that the new evaluation is also 0.0 for black to move in the position, now one ply ahead.
Due to contempt, black will say it has a +21 advantage for its score.
So the window is 42 centipawns wide.

White claims to be +21 ahead and so does black. So the error is doubled. I never use contempt, since I only analyze so I turn it off at the UCI level with a default setting of 0.0

Depending on what GUI you are using, the engine might not go into analysis mode when analyzing. It might still be in game play mode.

[Stephen Ham]

Thanks again for your valued input, Dann.

I didn't realize that SF's contempt factor of 21 merely added 21 to the evaluation. Is that truly how it attempts to avoid draws? If so, then how does that make any sense?

After all, since increased contempt is an anti-draw algorithm, then how does increasing the evaluation avoid draws and change move selection (i.e. less drawish moves)? If the search finds the best move is X, then calling the evaluation 0.00 or 0.21 doesn't change the fact that the best move is still X, so X still gets selected. So whether a centipawn window is created, or not, the result is the same. Hence, I doubt that contempt of 21 simply adds 21 to the evaluation.

I'm analyzing a different position now. After 20 minutes with a 21 contempt factor, the move X was selected with an evaluation of 0.00. After reading your posts, Dann, I then reduced contempt to 0, cleared the hash, and ran a new search. After 15' 52", the same move of X is selected and the evaluation remains 0.00. I'll keep going for 20 minutes. But the point is, there's no change in evaluation due to removing the contempt factor.

I use the Fritz 11 GUI, Dann, since I believe that was the last one prior to switching to "ribbons", which I hate.

All the best,
-Steve-

[Stephen Ham]

Dann, as a follow up, SF analyzed two plies deeper after 21' 57" and still selected move X with an evaluation of 0.00.

Given non-determinism from using 4-cores, it's possible that if I were to test again for the same period, I'd get different results. Nonetheless, it's interesting that roughly 20 minutes of search resulted in move X being selected, whether the contempt factor was 21 or 0. And the valuation was unchanged at 0.00. Since we're 15 moves deep into an opening, we're not in a drawn endgame. So, the 0.00 valuation is compelling.

In the interest of full transparency, the position under review now is:
r2q1rk1/p3ppbp/1p3np1/2p5/2NP3B/2PQP3/P4PPP/1R3RK1 b - - 0 15

Move X = 15...Qd7.

All the best,
-Steve-

[Uri Blass]

Dann, as a follow up, SF analyzed two plies deeper after 21' 57" and still selected move X with an evaluation of 0.00.

Given non-determinism from using 4-cores, it's possible that if I were to test again for the same period, I'd get different results. Nonetheless, it's interesting that roughly 20 minutes of search resulted in move X being selected, whether the contempt factor was 21 or 0. And the valuation was unchanged at 0.00. Since we're 15 moves deep into an opening, we're not in a drawn endgame. So, the 0.00 valuation is compelling.

In the interest of full transparency, the position under review now is:
r2q1rk1/p3ppbp/1p3np1/2p5/2NP3B/2PQP3/P4PPP/1R3RK1 b - - 0 15

Move X = 15...Qd7.

All the best,
-Steve-

I guess that stockfish evaluates a draw by repetition as 0.00 regardless of contempt.
When stockfish say 0.00 with contempt then it means that it finds nothing better in the search than repetition.
It does not mean a forced draw because it is possible that there is a winning line that stockfish is wrong to evaluate it as 0.3 pawns for the opponent with no contempt and the contempt is not high enough to change the evaluation to be better for the computer.

In the relevant position you can increase contempt to 100 and get a clear advantage for black.

[Joerg Oster]

If you are analyzing rather than game playing, make sure that you set the contempt to zero.
There is a built in offset for the side to move, otherwise.

Please note, even if you do this, you only deactivate static contempt.
Dynamic contempt (based upon the current best score) will still get applied and evals distorted.