Stockfish-1.6.2 Benchmarks for 1 to 8 Threads

[zullil]

Stockfish-1.6.2-mSD7 (8 threads) defeated Stockfish-1.6.2-mSD4 (8 threads) by a score of 291 to 120, with 589 draws.

Thanks Louis !

this for me is _the_ news of this sunday


Waiting for mSD7 vs mSD6.....

You're welcome, Marco. Glad to be of assistance. By the way, here's the shortest non-draw game from the match:

[Event "mSD4-7_TEST_MATCH"]
[Site "MacPro4,1-2.26GHz-8_Threads-512M_Hash"]
[Date "2010.01.03"]
[Round "744"]
[White "Stockfish 1.6.2-mSD7 64bit"]
[Black "Stockfish 1.6.2-mSD4 64bit"]
[Result "1-0"]
[PlyCount "61"]
[TimeControl "40"]

1.e4 {book} 1...c5 {book} 2.Nf3 {book} 2...d6 {book} 3.d4 {book} 3...cxd4 {book} 4.Nxd4 {book} 4...Nf6 {book} 5.Nc3 {book} 5...Nc6 {book} 6.Bg5 {book} 6...e6 {book} 7.Qd2 {book} 7...Be7 {book} 8.O-O-O {book} 8...O-O {book} 9.Ndb5 {book} 9...Qa5 {0.00/13 1.5s} 10.Kb1 {book} 10...Rd8 {+0.36/14 0.99s} 11.f3 {-0.36/13 1.1s} 11...a6 {+0.28/13 0.88s} 12.Nd4 {-0.36/14 0.83s} 12...Qc7 {+0.28/13 1.3s} 13.Nxc6 {-0.32/14 2.2s} 13...bxc6 {+0.32/14 1.2s} 14.Qe1 {-0.36/14 1.7s} 14...h6 {+0.40/13 1.0s} 15.Bh4 {-0.36/13 0.73s} 15...a5 {+0.56/13 1.9s} 16.g4 {-0.28/13 0.98s} 16...a4 {+0.40/12 0.74s} 17.Rg1 {-0.44/12 1.5s} 17...Qb7 {+0.36/12 0.98s} 18.g5 {+0.20/13 0.86s} 18...hxg5 {0.00/13 0.87s} 19.Bxg5 {0.00/13 0.74s} 19...Rb8 {0.00/13 1.2s} 20.Nxa4 {+1.01/15 3.0s} 20...Qa7 {-0.64/13 2.6s} 21.Bc4 {+1.09/12 0.88s} 21...Qxa4 {-0.60/8 0.054s} 22.Qh4 {+2.18/13 1.3s} 22...Qb4 {-1.89/11 0.83s} 23.Bb3 {+4.40/13 1.3s} 23...Ne8 {-4.28/12 1.4s} 24.Bxe7 {+5.05/9 0.11s} 24...Rd7 {-4.28/9 0.041s} 25.Rxg7+ {+M25/17 2.0s} 25...Nxg7 {-M12/13 0.85s} 26.Bf6 {+M11/10 0.15s} 26...Qa3 {-M10/9 0.027s} 27.Qh6 {+M9/6 0.006s} 27...Qxa2+ {-M8/6 0.004s} 28.Bxa2 {+M7/6 0.001s} 28...Rxb2+ {-M6/6 0.004s} 29.Kc1 {+M5/6 0.002s} 29...Rxc2+ {-M4/6 0.003s} 30.Kxc2 {+M3/6 0.001s} 30...c5 {-M2/6 0.001s} 31.Qxg7# {White mates} 1-0

[Joerg Oster]

Wow, a clear win for MSD=7. Very intersting.

On my Quad MSD=7 seems to be slightly worse than the default setting. After 90 games SF 1.6.2 default is leading by 3 wins (+10 =73 -7).
MSD=7 is too high in the middlegame, especially.

I'm just preparing a post on this topic.

[mcostalba]

By the way, here's the shortest non-draw game from the match:

The key move from white is from here:

[d] 1rbr2k1/q3bpp1/2pppn2/6B1/N3P3/5P2/PPP4P/1K1RQBR1 w - - 0 21


It is amazing your super 8 cores monster finds the winning move in less then one second (0.88s)

[mcostalba]

Wow, a clear win for MSD=7. Very intersting.

On my Quad MSD=7 seems to be slightly worse than the default setting. After 90 games SF 1.6.2 default is leading by 3 wins (+10 =73 -7).
MSD=7 is too high in the middlegame, especially.

I'm just preparing a post on this topic.

Probably a QUAD will require a different setup from an OCTAL machine, I don't know if the current default setup is the best one for a QUAD, I hope not so we have some margin to gain

[Joerg Oster]

Hi Tord,
Hi Marco,

I did some thinking on this...

First, let me say, I now understand, why creating 8 Threads or even more on a 8-core machine is probably a bad idea. ) The effort doing so is simply too much. At least the first tests done by Louis imply this.
So I guess reasonable numbers to test are 3, 4 and 5. 2 is probably too less for a N-core machine. (N >= 4)

And now to the second value. Min. Split Depth (MSD)
While watching my testmatch SF default against SF MSD=7 I got the impression, that MSD=7 is too high in the middlegame. The search of SF default seems more 'efficient', reaching certain plies a bit faster, though the nps looked often similar.
OTOH, SF MSD=7 got higher nps in the endgame and searched 1 ply deeper (sometimes), which seems to be a bit better therefore.

And this seems to make sense to me.
In the endgame we usually search some plies deeper than in the middlegame. Therefore we can raise the Min. Split Depth in the endgame to 7 or even 8 in the late endgame.

This leads me to the following idea:
why not adapting MSD to the game phase?

For example, MSD=4 for midgame, MSD=5 for late midgame, MSD=6 for (early) endgame and MSD=7 for late endgame. Or something similar.
Louis' test implies, that for an 8-Core machine these values may even be higher, something like 6, 7, 8 and 9.

To me, this looks like an interesting idea. But of course, I'm only a lay on this matter and may be totally wrong.
Did anybody else already consider this? Already implemented in an engine?
What do you think?

Kind regards,
Joerg.

[mcostalba]

Well, it seems interesting but I fail to understand the rationale behind this idea, perhaps Tord can give more hindsight about this approach.

Anyhow of one thing I am quite confident: Before to start a second order optimization we should first finish the first order one, aka we have to find the optimal _static_ parameters for the QUAD, then above this we can try the dynamic approach.

[Joerg Oster]

Agreed.
Of course, you are right.

[Joerg Oster]

Probably a QUAD will require a different setup from an OCTAL machine, I don't know if the current default setup is the best one for a QUAD, I hope not so we have some margin to gain

At least mSD=7, MNTpSP=5 is no improvement on a QUAD.
A bit disappointing, but after knowing Louis' results on his OCTAL, this is not a surprise to me.

Here is the final result.
Blitz 4m+2s Ponder off, 512MB Hash each, Q6600@3000MHz
50 Start Positions, Colors changed = 100 games

Code: Select all

1   Stockfish 1.6.2 JA 64bit       +11/=81/-8 51.50%   51.5/100
2   Stockfish 1.6.2 JA 64bit MSD7  +8/=81/-11 48.50%   48.5/100

[zullil]

By the way, here's the shortest non-draw game from the match:

The key move from white is from here:

[d] 1rbr2k1/q3bpp1/2pppn2/6B1/N3P3/5P2/PPP4P/1K1RQBR1 w - - 0 21


It is amazing your super 8 cores monster finds the winning move in less then one second (0.88s)

Actually, finding 20. Nxa4 in 3 seconds seems very impressive too:

[d] 1rbr2k1/1q2bpp1/2pppn2/6B1/p3P3/2N2P2/PPP4P/1K1RQBR1 w - - 1 19

[zullil]

Probably a QUAD will require a different setup from an OCTAL machine, I don't know if the current default setup is the best one for a QUAD, I hope not so we have some margin to gain

At least mSD=7, MNTpSP=5 is no improvement on a QUAD.
A bit disappointing, but after knowing Louis' results on his OCTAL, this is not a surprise to me.

Here is the final result.
Blitz 4m+2s Ponder off, 512MB Hash each, Q6600@3000MHz
50 Start Positions, Colors changed = 100 games

Code: Select all

1   Stockfish 1.6.2 JA 64bit       +11/=81/-8 51.50%   51.5/100
2   Stockfish 1.6.2 JA 64bit MSD7  +8/=81/-11 48.50%   48.5/100

Hi Jeorg, Marco, Tord, ...

Before starting the big mSD6--mSD7 (8 threads) match, I decided to spend some time benchmarking some 4 thread configurations. The data below seems to suggest that mSD=6 or 7 and MNTpSP=4 is good. Not sure how to reconcile this with Jeorg's results.

Code: Select all

Four threads, with MNTpSP=4. (Two runs each for mSD=3,4,5,6,7)

./stockfish-1.6.2-mSD3 bench 512 4 16 default depth

===============================
Total time (ms) : 124315
Nodes searched  : 354548302
Nodes/second    : 2852015

./stockfish-1.6.2-mSD4 bench 512 4 16 default depth

===============================
Total time (ms) : 120437
Nodes searched  : 362829355
Nodes/second    : 3012607

./stockfish-1.6.2-mSD5 bench 512 4 16 default depth

===============================
Total time (ms) : 120842
Nodes searched  : 383827333
Nodes/second    : 3176274

./stockfish-1.6.2-mSD6 bench 512 4 16 default depth

Total time (ms) : 131231
Nodes searched  : 414905010
Nodes/second    : 3161638

./stockfish-1.6.2-mSD7 bench 512 4 16 default depth

===============================
Total time (ms) : 124055
Nodes searched  : 398094830
Nodes/second    : 3209018

./stockfish-1.6.2-mSD3 bench 512 4 16 default depth

===============================
Total time (ms) : 119761
Nodes searched  : 344334260
Nodes/second    : 2875178

./stockfish-1.6.2-mSD4 bench 512 4 16 default depth

===============================
Total time (ms) : 128208
Nodes searched  : 386194937
Nodes/second    : 3012253

./stockfish-1.6.2-mSD5 bench 512 4 16 default depth

===============================
Total time (ms) : 126272
Nodes searched  : 401336018
Nodes/second    : 3178345

./stockfish-1.6.2-mSD6 bench 512 4 16 default depth
===============================
Total time (ms) : 133322
Nodes searched  : 423942061
Nodes/second    : 3179835

./stockfish-1.6.2-mSD7 bench 512 4 16 default depth

===============================
Total time (ms) : 133062
Nodes searched  : 421112311
Nodes/second    : 3164782

----------------------------------------------------------------

Code: Select all

Four threads, with MNTpSP=3. (Two runs each for mSD=3,4,5,6,7)


./stockfish-1.6.2-mSD3 bench 512 4 16 default depth

===============================
Total time (ms) : 113434
Nodes searched  : 320008371
Nodes/second    : 2821097

./stockfish-1.6.2-mSD4 bench 512 4 16 default depth

===============================
Total time (ms) : 126276
Nodes searched  : 382498108
Nodes/second    : 3029064

./stockfish-1.6.2-mSD5 bench 512 4 16 default depth

===============================
Total time (ms) : 124254
Nodes searched  : 390340214
Nodes/second    : 3141470

./stockfish-1.6.2-mSD6 bench 512 4 16 default depth

===============================
Total time (ms) : 139257
Nodes searched  : 444234962
Nodes/second    : 3190036

./stockfish-1.6.2-mSD7 bench 512 4 16 default depth

===============================
Total time (ms) : 121394
Nodes searched  : 386793892
Nodes/second    : 3186268

./stockfish-1.6.2-mSD3 bench 512 4 16 default depth

===============================
Total time (ms) : 142147
Nodes searched  : 407689137
Nodes/second    : 2868081

./stockfish-1.6.2-mSD4 bench 512 4 16 default depth

===============================
Total time (ms) : 124920
Nodes searched  : 380905179
Nodes/second    : 3049192

./stockfish-1.6.2-mSD5 bench 512 4 16 default depth

===============================
Total time (ms) : 119173
Nodes searched  : 380134016
Nodes/second    : 3189766

./stockfish-1.6.2-mSD6 bench 512 4 16 default depth

===============================
Total time (ms) : 121259
Nodes searched  : 387653555
Nodes/second    : 3196905

./stockfish-1.6.2-mSD7 bench 512 4 16 default depth

===============================
Total time (ms) : 117844
Nodes searched  : 372886392
Nodes/second    : 3164237

----------------------------------------------------------------

Code: Select all

Four threads, with MNTpSP=2. (One run each for mSD=3,4,5,6,7)


./stockfish-1.6.2-mSD3 bench 512 4 16 default depth

===============================
Total time (ms) : 148938
Nodes searched  : 370948205
Nodes/second    : 2490621

./stockfish-1.6.2-mSD4 bench 512 4 16 default depth

===============================
Total time (ms) : 115499
Nodes searched  : 310072673
Nodes/second    : 2684635

./stockfish-1.6.2-mSD5 bench 512 4 16 default depth
===============================
Total time (ms) : 115117
Nodes searched  : 342239837
Nodes/second    : 2972973

./stockfish-1.6.2-mSD6 bench 512 4 16 default depth
===============================
Total time (ms) : 106314
Nodes searched  : 329603492
Nodes/second    : 3100283

./stockfish-1.6.2-mSD7 bench 512 4 16 default depth
===============================
Total time (ms) : 113644
Nodes searched  : 346826237
Nodes/second    : 3051865