Testmatch SF 141130 (TCEC) - SF 141112

[fastgm]

Dual AMD Opteron 6376, 2.3 GHz

Cutechess, 1 Core, 128 MB Hash, no ponder, 8moves_v3.pgn, 120s+1.2s

Code: Select all

    Program                Elo    +   -   Games   Score   Av.Op.  Draws
-------------------------------------------------------------------------------------------
  1 Stockfish 141130 T1  : 3000   4   4   6002    50.0 %   3000   74.4 %  (+771,=4464,-767)
  2 Stockfish 141112 T1  : 3000   4   4   6002    50.0 %   3000   74.4 %  (+767,=4464,-771)

Total equal after 6000 games.

[Carlos777]

Dual AMD Opteron 6376, 2.3 GHz

Cutechess, 1 Core, 128 MB Hash, no ponder, 8moves_v3.pgn, 120s+1.2s

Code: Select all

    Program                Elo    +   -   Games   Score   Av.Op.  Draws
-------------------------------------------------------------------------------------------
  1 Stockfish 141130 T1  : 3000   4   4   6002    50.0 %   3000   74.4 %  (+771,=4464,-767)
  2 Stockfish 141112 T1  : 3000   4   4   6002    50.0 %   3000   74.4 %  (+767,=4464,-771)

Total equal after 6000 games.

I think it would be more interesting if both versions play against a set of strong engines like Komodo, Gull and Houdini.

IMO this kind of testing used by the Stockfish team is wrong, I mean testing Stockfish with a previous version of itself.

[JJJ]

Funny, I was running the almost same test. Ok, we prove they equal against each others. What about versus others engines ?

[fastgm]

Here the results against Komodo 8, same conditions:

Code: Select all

    Program                Elo    +   -   Games   Score   Av.Op.  Draws
 --------------------------------------------------------------------------------------------
  1 Stockfish 141112 T1  : 3027   6   6   6000    57.6 %   2973   51.2 %  (+1920,=3071,-1009) 3455.5/6000
  2 Komodo 8 T1          : 2973   6   6   6000    42.4 %   3027   51.2 %  (+1009,=3071,-1920)

    Program                Elo    +   -   Games   Score   Av.Op.  Draws
 --------------------------------------------------------------------------------------------
  1 Stockfish 141130 T1  : 3026   6   6   6000    57.4 %   2974   51.4 %  (+1904,=3082,-1014) 3445.0/6000
  2 Komodo 8 T1          : 2974   6   6   6000    42.6 %   3026   51.4 %  (+1014,=3082,-1904)

Andreas

[fenchel]

Here the results against Komodo 8, same conditions:

Code: Select all

    Program                Elo    +   -   Games   Score   Av.Op.  Draws
 --------------------------------------------------------------------------------------------
  1 Stockfish 141112 T1  : 3027   6   6   6000    57.6 %   2973   51.2 %  (+1920,=3071,-1009) 3455.5/6000
  2 Komodo 8 T1          : 2973   6   6   6000    42.4 %   3027   51.2 %  (+1009,=3071,-1920)

    Program                Elo    +   -   Games   Score   Av.Op.  Draws
 --------------------------------------------------------------------------------------------
  1 Stockfish 141130 T1  : 3026   6   6   6000    57.4 %   2974   51.4 %  (+1904,=3082,-1014) 3445.0/6000
  2 Komodo 8 T1          : 2974   6   6   6000    42.6 %   3026   51.4 %  (+1014,=3082,-1904)

Andreas

Andreas, can you confirm that you compiled these stockfish versions yourself, and moreover compiled them both in the same way (compiler version, compiler flags, stc.)?

(If you follow SF's mailing list, you'll know why I bring this up...)

[M ANSARI]

Where are you downloading your SF versions? Is it from the main site? Hard to keep up with all the SF versions and figure out which one is which and if it has TB support or not.

[fastgm]

No, that's the offical versions from http://abrok.eu/stockfish/ downloaded on 10th December.

[Jouni]

There is also test in SF forum, which shows definitely NO PROGRESS in last month! It was 178200 games for each version !! Was there bug in syzygy addition and/or compiler change maybe? Tomorrow is last day to make TCEC final compile .

[M ANSARI]

I think people have to be careful when testing different compilers as the improvement or regression could simply be because the compiler was tuned for certain hardware. It is very easy to lose track of what is going on or if there actually is an improvement if testing is done with different hardware and different compilers without specifying such differences.

[JJJ]

I think SF 141112 is still the best Stockfish so far.