Stockfish "Use Sleeping Threads" Test

[zullil]

4) Set TC (time control) to 10"+0.1 because functionality is the same is only speed that changes.

Are you sure that 10 second games are appropriate for this test?

Before running 5000 games I would make some tests to verify that the average node speed of the 8-threads version is actually higher than for the 4-threads version.

The average node speed over 250 milliseconds will not be quite the same as in the 600 second benchmark.

Yes, I thought to do some preliminary testing too.

[zullil]

Well, this is also no good statistically, but Crafty-23.4 gets an 11% and a 15% increase in nps using hyperthreading. This is based on the crafty bench command, which searches positions to a fixed depth (as does the SF bench). As Robert Houdart suggests, perhaps this increase has no effect on game results. Note that the speed gain seems to increase with the depth of the fixed-depth searches.

Code: Select all

LZsMacPro-OSX6: ~/Documents/Chess/Crafty/Crafty-23.4] ./crafty-23.4 
unable to open book file [./Books/book.bin].
book is disabled
unable to open book file [./Books/books.bin].
Warning--  xboard 'cores' option disabled
max threads set to 16.
maximum thread group size set to 12.
minimum nodes before a split 4000.
EGTB access enabled
using tbpath=../TB
0 piece tablebase files found
EGTB cache memory = 256M bytes.
Warning--  xboard 'memory' option disabled
hash table memory = 1024M bytes (64M entries).
Warning--  xboard 'memory' option disabled
pawn hash table memory = 256M bytes (8M entries).
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering disabled.
Audio output disabled
 game/10 minutes primary time control


Crafty v23.4 (16 cpus)

White(1): bench+3
Running benchmark 3. . .
......
Total nodes: 2520046720
Raw nodes per second: 25540151
Total elapsed time: 98.67
White(1): quit



LZsMacPro-OSX6: ~/Documents/Chess/Crafty/Crafty-23.4] ./crafty-23.4 
unable to open book file [./Books/book.bin].
book is disabled
unable to open book file [./Books/books.bin].
Warning--  xboard 'cores' option disabled
max threads set to 16.
maximum thread group size set to 12.
minimum nodes before a split 4000.
EGTB access enabled
using tbpath=../TB
0 piece tablebase files found
EGTB cache memory = 256M bytes.
Warning--  xboard 'memory' option disabled
hash table memory = 1024M bytes (64M entries).
Warning--  xboard 'memory' option disabled
pawn hash table memory = 256M bytes (8M entries).
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering disabled.
Audio output disabled
 game/10 minutes primary time control


Crafty v23.4 (16 cpus)

White(1): mt=8
Warning--  xboard 'cores' option disabled
max threads set to 8.
White(1): bench+3
Running benchmark 3. . .
......
Total nodes: 2265323093
Raw nodes per second: 23084918
Total elapsed time: 98.13
White(1): quit



LZsMacPro-OSX6: ~/Documents/Chess/Crafty/Crafty-23.4] ./crafty-23.4 
unable to open book file [./Books/book.bin].
book is disabled
unable to open book file [./Books/books.bin].
Warning--  xboard 'cores' option disabled
max threads set to 16.
maximum thread group size set to 12.
minimum nodes before a split 4000.
EGTB access enabled
using tbpath=../TB
0 piece tablebase files found
EGTB cache memory = 256M bytes.
Warning--  xboard 'memory' option disabled
hash table memory = 1024M bytes (64M entries).
Warning--  xboard 'memory' option disabled
pawn hash table memory = 256M bytes (8M entries).
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering disabled.
Audio output disabled
 game/10 minutes primary time control


Crafty v23.4 (16 cpus)

White(1): bench+5
Running benchmark 5. . .
......
Total nodes: 8855963985
Raw nodes per second: 27566344
Total elapsed time: 321.26
White(1): quit



LZsMacPro-OSX6: ~/Documents/Chess/Crafty/Crafty-23.4] ./crafty-23.4 
unable to open book file [./Books/book.bin].
book is disabled
unable to open book file [./Books/books.bin].
Warning--  xboard 'cores' option disabled
max threads set to 16.
maximum thread group size set to 12.
minimum nodes before a split 4000.
EGTB access enabled
using tbpath=../TB
0 piece tablebase files found
EGTB cache memory = 256M bytes.
Warning--  xboard 'memory' option disabled
hash table memory = 1024M bytes (64M entries).
Warning--  xboard 'memory' option disabled
pawn hash table memory = 256M bytes (8M entries).
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering disabled.
Audio output disabled
 game/10 minutes primary time control


Crafty v23.4 (16 cpus)

White(1): mt=8
Warning--  xboard 'cores' option disabled
max threads set to 8.
White(1): bench+5
Running benchmark 5. . .
......
Total nodes: 6220117960
Raw nodes per second: 23970549
Total elapsed time: 259.49
White(1): quit

[Houdini]

Note that the speed gain seems to increase with the depth of the fixed-depth searches.

That's exactly as expected, multiple threads become more interesting with increasing TC.
It's the main reason why ultra-fast games might not very well reflect the true performance of an 8-threaded engine.

[PawnStormZ]

Hi Marco.

Am I missing something here? How could a match like this be run on 1 pc? The hyperthreading is a BIOS setting that needs to be either on or off. There is not a way to play one engine where the cpu is using it and one not on the same pc.

[Joerg Oster]

Your test is no good. You need to run _several_ different positions, multiple times each, and then average all the times together. SMP is highly non-deterministic and you need a significant number of samples to get a reasonable estimate.

The stockfish bench uses 16 positions for each run. Perhaps Marco can clarify this.

I understand that SMP is quite variable. When you say that the "test is no good" do you mean more than "the results are statistically insignificant?"

I wonder how many times I would need to run each test in order for the average values of the nps results to be significant.

Hello Louis,

I'm afraid Bob is right.
For example: I have run the Stockfish benchmark over a set of 25 opening position, up to depth 17 with 4 cores, 5 runs each. I got 105 sec, 97 sec, 107 sec, 89 sec and 82 sec.
This is a maximum difference of 25 seconds. A huge difference I would say.

I don't know how many runs are necessary, but one run is almost meaningless.

Joerg.

[zullil]

2) On first engine disable "Sleeping Threads" and set "Threads" to 4

3) On the second engine enable "Sleeping Threads" and set "Threads" to 8

I have 8 physical cores---16 logical with HT. It seems one engine would need to use 8 threads (say with Sleeping Threads disabled) while the other used 16 (with Sleeping Threads enabled). Pondering off of course. Right?

[zullil]

Hi Marco.

Am I missing something here? How could a match like this be run on 1 pc? The hyperthreading is a BIOS setting that needs to be either on or off. There is not a way to play one engine where the cpu is using it and one not on the same pc.

Hyperthreading would be enabled. One engine would use 8 threads (which would be equivalent to no hyperthreading, since I have 8 physical cores). The other would use 16 threads (with Sleeping Threads enabled). Pondering would be disabled for both engines.

[Tord Romstad]

I understand that SMP is quite variable. When you say that the "test is no good" do you mean more than "the results are statistically insignificant?"

He means that comparing N/s between 8 and 16 threads isn't very interesting. The strength of Stockfish (and any other program) doesn't derive from seeing lots of nodes, but from searching very deeply. The interesting question, therefore, isn't whether the average N/s with 16 threads is higher than with 8 threads, but whether Stockfish is on average able to complete deeper searches in a given amount of time with 16 threads. Searching a 10% higher number of nodes per second doesn't help if you need to search 20% more nodes to search to the same depth.

The only way to be sure is to play lots of games, but I'd be extremely surprised if Stockfish is stronger with 16 than with 8 threads on your machine. Using 16 threads is a huge handicap, and it would be very remarkable if a tiny 10% increase in N/s is enough to compensate.

[zullil]

I understand that SMP is quite variable. When you say that the "test is no good" do you mean more than "the results are statistically insignificant?"

The interesting question, therefore, isn't whether the average N/s with 16 threads is higher than with 8 threads, but whether Stockfish is on average able to complete deeper searches in a given amount of time with 16 threads.

Thanks. So my test should have been to give each version of the engine some fixed amount of time to search and record the depth that it reached. (And to do this with a collection of positions and repeat these a large number of times to ensure significance of the results.)

[Karlo Bala]

I understand that SMP is quite variable. When you say that the "test is no good" do you mean more than "the results are statistically insignificant?"

He means that comparing N/s between 8 and 16 threads isn't very interesting. The strength of Stockfish (and any other program) doesn't derive from seeing lots of nodes, but from searching very deeply. The interesting question, therefore, isn't whether the average N/s with 16 threads is higher than with 8 threads, but whether Stockfish is on average able to complete deeper searches in a given amount of time with 16 threads. Searching a 10% higher number of nodes per second doesn't help if you need to search 20% more nodes to search to the same depth.

The only way to be sure is to play lots of games, but I'd be extremely surprised if Stockfish is stronger with 16 than with 8 threads on your machine. Using 16 threads is a huge handicap, and it would be very remarkable if a tiny 10% increase in N/s is enough to compensate.

Why not try with 9 or 10 threads? On my i5 mobile (dual core with HT) stockfish works best with 3 cores.