strange : evaluation differs with colors reversed

[tissatussa]

I encounter some strange behaviour of StockFish when evaluating the same position with colors reversed !?
can anybody explain this and reproduce the outcome ? do other chess engines act alike ?

I think SF does this in all (?) positions. I have no explanation, but it seems a basic question.
Maybe SF calculates differently for K and Q side .. strange ..

Example : screenshots of my Scid program when evaluating both positions with my latest StockFish (v7 in the Ubuntu repo) :


[/img]

[zenpawn]

Maybe SF calculates differently for K and Q side .. strange ..

You could test the hypothesis by flipping along the horizontal axis versus the diagonal.

[bob]

There are lots of things that can cause this. You are doing a search, which is a problematic way to look at this. If you flip the board, you alter move ordering since the moves are now generated in a different order thanks to bit board first one / last one operations. If you search moves in different orders, then you will reduce them differently. Giving different results.

First test is to set up the position and use the "score" (or whatever stockfish uses) to display the static score. Then flip the colors and type score again. It ought to be the same except with the sign reversed. Make sure that when you flip the position you also flip the side to move as well.

[Sven]

The SF command to display the static evaluation on console is "eval".

[mjlef]

There are lots of things that can cause this. You are doing a search, which is a problematic way to look at this. If you flip the board, you alter move ordering since the moves are now generated in a different order thanks to bit board first one / last one operations. If you search moves in different orders, then you will reduce them differently. Giving different results.

First test is to set up the position and use the "score" (or whatever stockfish uses) to display the static score. Then flip the colors and type score again. It ought to be the same except with the sign reversed. Make sure that when you flip the position you also flip the side to move as well.

In the past I know Stockfish (and is some cases Komodo) took shortcuts. I recently wrote code to flip the board around the line between ranks 4 and 5. It converted white to black pieces, adjusted castling rights, hash keys, etc. The I have Komodo run through the eval, first as whatever color was moving, then with the flipped board. It can find evaluation unsymmetries.

One I found was a term that was propagating orthogonal attacks through a pawn. In this case, a pawn on say e4, had a queen on e5, a bishop on e6 and a rook on e7. The code masked off the squares e4, e5, e6, e6 and e8, and confirmed at least one rook or queen was there. The function worked with the position flipped one way, but not the other. Bit scan functions fine the first set bit, but depending on the instruction you use, could find the rook or the queen first. If it finds the rook, then the program did not propagate the attack, because the rook blocked it. With the board flipped, it finds the queen first, and see the attack. Best practice would be to check every possible R/Q behind that pawn, but that takes more time. BTW, I fixed this. But such minor bug fixes rarely amount to many elo.

I do recall something similar in Stockfish at one time. It was a while ago and I do not remember the details. But it is sometimes done to save time, when scoring something more than once is not worth it.

[Ajedrecista]

Hello:

The SF command to display the static evaluation on console is "eval".

I have just tried the 'eval' command with the following results:

Code: Select all

Stockfish 8 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
position fen 3rr2k/p4pp1/5n1p/q1PNb3/p5P1/3PP2P/2P2QB1/1RK2R2 w - - 0 1
eval
      Eval term |    White    |    Black    |    Total
                |   MG    EG  |   MG    EG  |   MG    EG
----------------+-------------+-------------+-------------
       Material |   ---   --- |   ---   --- |  1.25  1.46
      Imbalance |   ---   --- |   ---   --- |  0.02  0.02
          Pawns |   ---   --- |   ---   --- |  0.27  0.29
        Knights |  0.17  0.04 |  0.00  0.00 |  0.17  0.04
         Bishop | -0.13 -0.19 | -0.06 -0.15 | -0.06 -0.05
          Rooks |  0.26  0.11 |  0.16  0.06 |  0.10  0.05
         Queens |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
       Mobility |  0.57  1.52 |  0.60  1.61 | -0.02 -0.09
    King safety | -0.34 -0.10 |  0.44 -0.06 | -0.78 -0.04
        Threats |  0.33  0.36 |  0.54  0.53 | -0.20 -0.17
   Passed pawns |  0.23  0.68 |  0.64  0.41 | -0.41  0.27
          Space |  0.29  0.00 |  0.16  0.00 |  0.13  0.00
----------------+-------------+-------------+-------------
          Total |   ---   --- |   ---   --- |  0.46  1.94

Total Evaluation: 0.70 (white side)

position fen 2r2kr1/1bq2p2/p2pp3/1p5P/3Bnp1Q/P1N5/1PP4P/K2RR3 b - - 0 1
eval
      Eval term |    White    |    Black    |    Total
                |   MG    EG  |   MG    EG  |   MG    EG
----------------+-------------+-------------+-------------
       Material |   ---   --- |   ---   --- | -1.25 -1.46
      Imbalance |   ---   --- |   ---   --- | -0.02 -0.02
          Pawns |   ---   --- |   ---   --- | -0.27 -0.29
        Knights |  0.00  0.00 |  0.17  0.04 | -0.17 -0.04
         Bishop | -0.06 -0.15 | -0.13 -0.19 |  0.06  0.05
          Rooks |  0.16  0.06 |  0.26  0.11 | -0.10 -0.05
         Queens |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
       Mobility |  0.60  1.61 |  0.57  1.52 |  0.02  0.09
    King safety |  0.44 -0.06 | -0.34 -0.10 |  0.78  0.04
        Threats |  0.54  0.53 |  0.33  0.36 |  0.20  0.17
   Passed pawns |  0.64  0.41 |  0.23  0.68 |  0.41 -0.27
          Space |  0.16  0.00 |  0.29  0.00 | -0.13  0.00
----------------+-------------+-------------+-------------
          Total |   ---   --- |   ---   --- | -0.46 -1.94

Total Evaluation: -0.70 (white side)

I do not see problems here: both positions are evaluated as +0.7 for the side to move. Am I missing something?

Regards from Spain.

Ajedrecista.

[Dann Corbit]

The four identical positions:
[d]k2rr3/1pp4p/p1n5/3bNP1q/1P5p/P2PP3/1BQ2P2/2R2KR1 w - -
[d]3rr2k/p4pp1/5n1p/q1PNb3/p5P1/3PP2P/2P2QB1/1RK2R2 w - -
[d]2r2kr1/1bq2p2/p2pp3/1p5P/3Bnp1Q/P1N5/1PP4P/K2RR3 b - -
[d]1rk2r2/2p2qb1/3pp2p/P5p1/Q1pnB3/5N1P/P4PP1/3RR2K b - -

[Dann Corbit]

I got this:

Code: Select all

G:\chess\Stockfish-master2\src>stockfish-x64-mingw.exe
Stockfish 241216 64 POPCNT by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
position fen k2rr3/1pp4p/p1n5/3bNP1q/1P5p/P2PP3/1BQ2P2/2R2KR1 w - -
eval
      Eval term |    White    |    Black    |    Total
                |   MG    EG  |   MG    EG  |   MG    EG
----------------+-------------+-------------+-------------
       Material |   ---   --- |   ---   --- |  1.21  1.46
      Imbalance |   ---   --- |   ---   --- |  0.02  0.02
          Pawns |   ---   --- |   ---   --- |  0.27  0.28
        Knights |  0.17  0.04 |  0.00  0.00 |  0.17  0.04
         Bishop | -0.13 -0.19 | -0.06 -0.15 | -0.06 -0.05
          Rooks |  0.26  0.11 |  0.16  0.06 |  0.10  0.05
         Queens |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
       Mobility |  0.57  1.52 |  0.60  1.61 | -0.02 -0.09
    King safety | -0.26 -0.10 |  0.50 -0.06 | -0.76 -0.04
        Threats |  0.46  0.39 |  1.05  0.63 | -0.59 -0.24
   Passed pawns |  0.23  0.68 |  0.64  0.41 | -0.41  0.27
          Space |  0.29  0.00 |  0.16  0.00 |  0.13  0.00
----------------+-------------+-------------+-------------
          Total |   ---   --- |   ---   --- |  0.06  1.85

Total Evaluation: 0.35 (white side)

position fen 3rr2k/p4pp1/5n1p/q1PNb3/p5P1/3PP2P/2P2QB1/1RK2R2 w - -
eval
      Eval term |    White    |    Black    |    Total
                |   MG    EG  |   MG    EG  |   MG    EG
----------------+-------------+-------------+-------------
       Material |   ---   --- |   ---   --- |  1.21  1.46
      Imbalance |   ---   --- |   ---   --- |  0.02  0.02
          Pawns |   ---   --- |   ---   --- |  0.27  0.28
        Knights |  0.17  0.04 |  0.00  0.00 |  0.17  0.04
         Bishop | -0.13 -0.19 | -0.06 -0.15 | -0.06 -0.05
          Rooks |  0.26  0.11 |  0.16  0.06 |  0.10  0.05
         Queens |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
       Mobility |  0.57  1.52 |  0.60  1.61 | -0.02 -0.09
    King safety | -0.26 -0.10 |  0.50 -0.06 | -0.76 -0.04
        Threats |  0.46  0.39 |  1.05  0.63 | -0.59 -0.24
   Passed pawns |  0.23  0.68 |  0.64  0.41 | -0.41  0.27
          Space |  0.29  0.00 |  0.16  0.00 |  0.13  0.00
----------------+-------------+-------------+-------------
          Total |   ---   --- |   ---   --- |  0.06  1.85

Total Evaluation: 0.35 (white side)

position fen 2r2kr1/1bq2p2/p2pp3/1p5P/3Bnp1Q/P1N5/1PP4P/K2RR3 b - -
eval
      Eval term |    White    |    Black    |    Total
                |   MG    EG  |   MG    EG  |   MG    EG
----------------+-------------+-------------+-------------
       Material |   ---   --- |   ---   --- | -1.21 -1.46
      Imbalance |   ---   --- |   ---   --- | -0.02 -0.02
          Pawns |   ---   --- |   ---   --- | -0.27 -0.28
        Knights |  0.00  0.00 |  0.17  0.04 | -0.17 -0.04
         Bishop | -0.06 -0.15 | -0.13 -0.19 |  0.06  0.05
          Rooks |  0.16  0.06 |  0.26  0.11 | -0.10 -0.05
         Queens |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
       Mobility |  0.60  1.61 |  0.57  1.52 |  0.02  0.09
    King safety |  0.50 -0.06 | -0.26 -0.10 |  0.76  0.04
        Threats |  1.05  0.63 |  0.46  0.39 |  0.59  0.24
   Passed pawns |  0.64  0.41 |  0.23  0.68 |  0.41 -0.27
          Space |  0.16  0.00 |  0.29  0.00 | -0.13  0.00
----------------+-------------+-------------+-------------
          Total |   ---   --- |   ---   --- | -0.06 -1.85

Total Evaluation: -0.35 (white side)

position fen 1rk2r2/2p2qb1/3pp2p/P5p1/Q1pnB3/5N1P/P4PP1/3RR2K b - -
eval
      Eval term |    White    |    Black    |    Total
                |   MG    EG  |   MG    EG  |   MG    EG
----------------+-------------+-------------+-------------
       Material |   ---   --- |   ---   --- | -1.21 -1.46
      Imbalance |   ---   --- |   ---   --- | -0.02 -0.02
          Pawns |   ---   --- |   ---   --- | -0.27 -0.28
        Knights |  0.00  0.00 |  0.17  0.04 | -0.17 -0.04
         Bishop | -0.06 -0.15 | -0.13 -0.19 |  0.06  0.05
          Rooks |  0.16  0.06 |  0.26  0.11 | -0.10 -0.05
         Queens |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
       Mobility |  0.60  1.61 |  0.57  1.52 |  0.02  0.09
    King safety |  0.50 -0.06 | -0.26 -0.10 |  0.76  0.04
        Threats |  1.05  0.63 |  0.46  0.39 |  0.59  0.24
   Passed pawns |  0.64  0.41 |  0.23  0.68 |  0.41 -0.27
          Space |  0.16  0.00 |  0.29  0.00 | -0.13  0.00
----------------+-------------+-------------+-------------
          Total |   ---   --- |   ---   --- | -0.06 -1.85

Total Evaluation: -0.35 (white side)

[Vinvin]

I got this:...

So, the difference come probably from the search, not from the eval.

[Dann Corbit]

I got this:...

So, the difference come probably from the search, not from the eval.

Which is unsurprising.
Move ordering difference, for instance.