SF270317 vs. SF121216, 4CPU, TC 600+10

[majortom]

OS: Windows Server 2012 R2
CPU: Intel® Xeon® Processor E3-1270 v5 @3.60GHz
GUI: Cutechess-cli 0.6.0
Openings: Noomen Testsuite 2016, based on the TCEC superfinal PGN
50 positions x2 (with different colours) = 100 games
Hash: 16384 MB
EGTB: Syzygy 5-men
TC: 600 sec + 10 sec (per move)

Stockfish 270317 BMI2: bench 6197938, the latest SF version.
Stockfish 121216 BMI2: bench 4684146, the strongest SF version in the Stefan Pohl's (SPCC) ratinglist.

Both version are compiled and optimized on Intel® Xeon® Processor E3-1270 v5 @3.60GHz.

[majortom]

Code: Select all

1 Stockfish 280317 64 BMI2  :    0   11 (+  1,=  9,-  1), 50.0 %
2 Stockfish 121216 64 BMI2  :    0   11 (+  1,=  9,-  1), 50.0 %

http://treu.ru/2017/sf270317_sf121216_600_10.pgn

[majortom]

BTW, my SF BMI2 compile is about 7% faster than Ultimaiq's on Intel Core i3-6100U @ 2.30GHz

Code: Select all

FishBench test 1:

Base - my version
Test - Ultimaiq's

Results for 20 tests for each version:

            Base      Test      Diff      
    Mean    1004826   933970    70856     
    StDev   77513     74132     8298      

p-value: 0
speedup: -0,071

FishBench test 2:

Results for 20 tests for each version:

            Base      Test      Diff      
    Mean    1055134   983258    71876     
    StDev   45519     45226     7524      

p-value: 0
speedup: -0,068

UCI bench:

Ultimaiq's
Total time (ms) : 4680
Nodes searched  : 6197938
Nodes/second    : 1324345

My version:
Total time (ms) : 4381
Nodes searched  : 6197938
Nodes/second    : 1414731

6.8249%

[majortom]

Code: Select all

  1 Stockfish 280317 64 BMI2  :   12   29 (+  3,= 24,-  2), 51.7 %
  2 Stockfish 121216 64 BMI2  :    0   29 (+  2,= 24,-  3), 48.3 %

[majortom]

Code: Select all

    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Stockfish 280317 64 BMI2       :    0   42  42    40    50.0 %      0   85.0 %
  2 Stockfish 121216 64 BMI2       :    0   42  42    40    50.0 %      0   85.0 %

  1 Stockfish 280317 64 BMI2       :    0   40 (+  3,= 34,-  3), 50.0 %
  2 Stockfish 121216 64 BMI2       :    0   40 (+  3,= 34,-  3), 50.0 %

[majortom]

1 Stockfish 280317 64 BMI2 : 0 49 (+ 6,= 37,- 6), 50.0 %
2 Stockfish 121216 64 BMI2 : 0 49 (+ 6,= 37,- 6), 50.0 %

[Milos]

1 Stockfish 280317 64 BMI2 : 0 49 (+ 6,= 37,- 6), 50.0 %
2 Stockfish 121216 64 BMI2 : 0 49 (+ 6,= 37,- 6), 50.0 %

Let me guess the final score: +12/=76/+12 .
In worst it will be +13/=75/+12 or +12/=75/+13 .
With this amount of games it is pretty much like testing an engine against identical copy of it.

[majortom]

Code: Select all

1 Stockfish 280317 64 BMI2  :    6   60 (+  8,= 45,-  7), 50.8 %
2 Stockfish 121216 64 BMI2  :    0   60 (+  7,= 45,-  8), 49.2 %

There are two missed games: 18 & 31.
They were lost when the net storage was unreachable via wi-fi.
That games will be replayed and added later.

[majortom]

Code: Select all

  1 Stockfish 280317 64 BMI2  :         8     78 (+ 12,= 56,- 10), 51.3 %
  2 Stockfish 121216 64 BMI2  :         0     78 (+ 10,= 56,- 12), 48.7 %

[majortom]

Code: Select all

    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Stockfish 280317 64 BMI2       :    0   33  33   100    50.0 %      0   76.0 %
  2 Stockfish 121216 64 BMI2       :    0   33  33   100    50.0 %      0   76.0 %

  1 Stockfish 280317 64 BMI2       :    0  100 (+ 12,= 76,- 12), 50.0 %
  2 Stockfish 121216 64 BMI2       :    0  100 (+ 12,= 76,- 12), 50.0 %