Elo gain by core doubling - Komodo 14, Stockfish 11

[Laskos]

Do you have the complete table of the results (w/d/l)?

Komodo 14

Code: Select all

Threads    2 vs 1          4 vs 2          8 vs 4          16 vs 8        32 vs 16        64 vs 32       128 vs 64
w-d-l	815-2030-155	783-2064-153	657-2187-156	520-2326-154	392-2431-177	430-2427-143	251-2531-218

Stockfish 11

Code: Select all

Threads     2 vs 1         4 vs 2          8 vs 4         16 vs 8         32 vs 16        64 vs 32       128 vs 64
w-d-l	1004-1764-232	902-1879-219	762-2038-200	626-2171-203	592-2212-196	560-2280-160	363-2395-242

Here is the added Normalized Elo, which usually shows better the scaling. We can see that Komodo and Stockfish scale almost the same to 16 threads (SF just a bit better), but SF scales much better to 32, 64 and 128 threads.

Komodo 14

Code: Select all

Threads    2 vs 1          4 vs 2          8 vs 4          16 vs 8        32 vs 16        64 vs 32       128 vs 64
w-d-l	815-2030-155	783-2064-153	657-2187-156	520-2326-154	392-2431-177	430-2427-143	251-2531-218
Norm Elo   0.420           0.406           0.339            0.266          0.167           0.224           0.028
Standard Deviation: 0.018

Stockfish 11

Code: Select all

Threads     2 vs 1         4 vs 2          8 vs 4         16 vs 8         32 vs 16        64 vs 32       128 vs 64
w-d-l	1004-1764-232	902-1879-219	762-2038-200	626-2171-203	592-2212-196	560-2280-160	363-2395-242
Norm Elo    0.438          0.401           0.351           0.278           0.267           0.283           0.090
Standard Deviation: 0.018

[Alayan]

Comparing elo is also flawed because the starting elo isn't equal. The elo compression we see as TC increases stems from the mistakes being made being smaller and smaller.

While any engine at 10+0.1 is blundering left and right, Stockfish 11 is noticeably stronger than Komodo 14 at this TC.

To put these results in perspective, another test would be needed : time odds games. SF11 and K14 with 10.1s against themselves with 2x TC, and so on. This would allow to compute the treading efficiency compared to increasing time.

[MikeB]

Interesting information is also Mn/sec benchmark (startpos go depth 28) at different code counts.
To differ scaling problems (speed gain after doubling cores) and diminishing returns (elo gain after doubling nodes)
Even scaling problems is hard to differ from AMD Precision Boost technology

This is Freq vs Threads


Also, was it Linux or Windows?
If CPU has more than 64 logical cores, Windows split them in 2 Groups


Application need special support to be able to run in multiple groups (up to 64 thread each group)

It is not an accident I emphasized in every time testers would not use HT (SMT) and frequency turbo, only physical cores with fixed CPU clock. Another important viewpoint is the good cooling of the machine.

Excellent point and 100% correct, that is the only way to get the most reliable results. Also I recommend reserving at least one core for OS and perhaps at least two cores at 32 cores. Running with less one core when you have 32 cores only changes the testing time by~ 3% - so its not big expense to get a more consistent results. With these settings I get remarkably repeatable consistent runs with 1000 game sets. And believe it or not, fishest does not enforce those settings (well maybe do now, not sure, but at one time they did not).

[Vinvin]

AMD Precision Boost was disabled, of course. Speed was 3.7 GHz

Stockfish 11, 2048 MB Hash, depth 28

1 thread

info depth 28 seldepth 40 multipv 1 score cp 61 nodes 41087252 nps 1.724.398 hashfull 163 tbhits 0 time 23827 pv d2d4 d7d5 c2c4 g8f6 g1f3 c7c5 c4d5 c5d4 f3d4 d8d5 b1c3 d5a5 d4b3 a5h5 h2h4 c8d7 c1d2 e7e5 f2f3 h5g6 g2g4 f6g4 f3g4 g6g4 h1h3 b8c6 e2e4 f8e7 d2e3 e7h4 h3h4 g4h4 e3f2 h4h2
bestmove d2d4 ponder d7d5
-------------------------
...

I have a request for you : start 16 instances of SF with 1 thread for a couple of minutes and report the speed. Thanks !

[fastgm]

Stockfish get much better scaling results according to this data, but Stockfish 11 has contempt and Komodo 14 has 0 as default contempt. And this might make a noticeable difference, as contempt will expand the doubling gain in such tests.

Maybe I'll repeat the test with contempt 0 for both, Stockfish 11 and Komodo 14.

Just for your info:
Komodo 14 has 4 as default contempt.

[lkaufman]

Do you have the complete table of the results (w/d/l)?

Komodo 14

Code: Select all

Threads    2 vs 1          4 vs 2          8 vs 4          16 vs 8        32 vs 16        64 vs 32       128 vs 64
w-d-l	815-2030-155	783-2064-153	657-2187-156	520-2326-154	392-2431-177	430-2427-143	251-2531-218

Stockfish 11

Code: Select all

Threads     2 vs 1         4 vs 2          8 vs 4         16 vs 8         32 vs 16        64 vs 32       128 vs 64
w-d-l	1004-1764-232	902-1879-219	762-2038-200	626-2171-203	592-2212-196	560-2280-160	363-2395-242

Based on this, if we use WILO (i.e. ignoring draws) to avoid the effect of Contempt, Komodo scales better for each doubling thru 16 threads, then starting at 32 threads Stockfish scales better.

[fastgm]

AMD Precision Boost was disabled, of course. Speed was 3.7 GHz

Stockfish 11, 2048 MB Hash, depth 28

1 thread

info depth 28 seldepth 40 multipv 1 score cp 61 nodes 41087252 nps 1.724.398 hashfull 163 tbhits 0 time 23827 pv d2d4 d7d5 c2c4 g8f6 g1f3 c7c5 c4d5 c5d4 f3d4 d8d5 b1c3 d5a5 d4b3 a5h5 h2h4 c8d7 c1d2 e7e5 f2f3 h5g6 g2g4 f6g4 f3g4 g6g4 h1h3 b8c6 e2e4 f8e7 d2e3 e7h4 h3h4 g4h4 e3f2 h4h2
bestmove d2d4 ponder d7d5
-------------------------
...

I have a request for you : start 16 instances of SF with 1 thread for a couple of minutes and report the speed. Thanks !

Stockfish 11, 1 thread, 128 MB Hash, go depth 34

info depth 34 seldepth 51 multipv 1 score cp 57 nodes 349662654 nps 1.857.605 hashfull 1000 tbhits 0 time 188233 pv e2e4 e7e6 g1f3 d7d5 e4d5 e6d5 h2h3 c7c6 d2d4 g8f6 f1d3 f8d6 d1e2 c8e6 f3g5 d8e7 e1g1 e8g8 f1e1 f8e8 c2c3 h7h6 g5e6 e7e6 e2e6 e8e6 c1e3 b8d7 b1d2 a8e8 d3f5 e6e7 a2a4 a7a5 f5c2 d7f8 d2f3 f8e6 g2g3 f6e4

----------------------------------------

Stockfish 11, 16 x 1 thread, 128 MB Hash, go depth 34

info depth 34 seldepth 51 multipv 1 score cp 57 nodes 349662654 nps 1.858.454 hashfull 1000 tbhits 0 time 188147 pv e2e4 e7e6 g1f3 d7d5 e4d5 e6d5 h2h3 c7c6 d2d4 g8f6 f1d3 f8d6 d1e2 c8e6 f3g5 d8e7 e1g1 e8g8 f1e1 f8e8 c2c3 h7h6 g5e6 e7e6 e2e6 e8e6 c1e3 b8d7 b1d2 a8e8 d3f5 e6e7 a2a4 a7a5 f5c2 d7f8 d2f3 f8e6 g2g3 f6e4

----------------------------------------

Stockfish 11, 32 x 1 thread, 128 MB Hash, go depth 34

info depth 34 seldepth 51 multipv 1 score cp 57 nodes 349662654 nps 1.803.453 hashfull 1000 tbhits 0 time 193885 pv e2e4 e7e6 g1f3 d7d5 e4d5 e6d5 h2h3 c7c6 d2d4 g8f6 f1d3 f8d6 d1e2 c8e6 f3g5 d8e7 e1g1 e8g8 f1e1 f8e8 c2c3 h7h6 g5e6 e7e6 e2e6 e8e6 c1e3 b8d7 b1d2 a8e8 d3f5 e6e7 a2a4 a7a5 f5c2 d7f8 d2f3 f8e6 g2g3 f6e4

----------------------------------------

Stockfish 11, 48 x 1 thread, 128 MB Hash, go depth 34

info depth 34 seldepth 51 multipv 1 score cp 57 nodes 349662654 nps 1.637.587 hashfull 1000 tbhits 0 time 213523 pv e2e4 e7e6 g1f3 d7d5 e4d5 e6d5 h2h3 c7c6 d2d4 g8f6 f1d3 f8d6 d1e2 c8e6 f3g5 d8e7 e1g1 e8g8 f1e1 f8e8 c2c3 h7h6 g5e6 e7e6 e2e6 e8e6 c1e3 b8d7 b1d2 a8e8 d3f5 e6e7 a2a4 a7a5 f5c2 d7f8 d2f3 f8e6 g2g3 f6e4

----------------------------------------

Stockfish 11, 64 x 1 thread, 128 MB Hash, go depth 34

info depth 34 seldepth 51 multipv 1 score cp 57 nodes 349662654 nps 1.403.118 hashfull 1000 tbhits 0 time 249204 pv e2e4 e7e6 g1f3 d7d5 e4d5 e6d5 h2h3 c7c6 d2d4 g8f6 f1d3 f8d6 d1e2 c8e6 f3g5 d8e7 e1g1 e8g8 f1e1 f8e8 c2c3 h7h6 g5e6 e7e6 e2e6 e8e6 c1e3 b8d7 b1d2 a8e8 d3f5 e6e7 a2a4 a7a5 f5c2 d7f8 d2f3 f8e6 g2g3 f6e4

----------------------------------------

Stockfish 11, 128 x 1 thread, 128 MB Hash, go depth 34

info depth 34 seldepth 51 multipv 1 score cp 57 nodes 349662654 nps 1.059.522 hashfull 1000 tbhits 0 time 330019 pv e2e4 e7e6 g1f3 d7d5 e4d5 e6d5 h2h3 c7c6 d2d4 g8f6 f1d3 f8d6 d1e2 c8e6 f3g5 d8e7 e1g1 e8g8 f1e1 f8e8 c2c3 h7h6 g5e6 e7e6 e2e6 e8e6 c1e3 b8d7 b1d2 a8e8 d3f5 e6e7 a2a4 a7a5 f5c2 d7f8 d2f3 f8e6 g2g3 f6e4

[Vinvin]

Stockfish 11, 1 thread, 128 MB Hash, go depth 34
1.857.605
----------------------------------------
Stockfish 11, 16 x 1 thread, 128 MB Hash, go depth 34
1.858.454
----------------------------------------
Stockfish 11, 32 x 1 thread, 128 MB Hash, go depth 34
1.803.453
----------------------------------------
Stockfish 11, 48 x 1 thread, 128 MB Hash, go depth 34
1.637.587
----------------------------------------
Stockfish 11, 64 x 1 thread, 128 MB Hash, go depth 34
1.403.118
----------------------------------------
Stockfish 11, 128 x 1 thread, 128 MB Hash, go depth 34
1.059.522

Very impressive scaling until 32 threads.
Still very good (-12% compare to 1 th.) with 48 threads.
Drop (-24% compare to 1 th.) with 64 threads.

[Dann Corbit]

Stockfish 11, 1 thread, 128 MB Hash, go depth 34
1.857.605
----------------------------------------
Stockfish 11, 16 x 1 thread, 128 MB Hash, go depth 34
1.858.454
----------------------------------------
Stockfish 11, 32 x 1 thread, 128 MB Hash, go depth 34
1.803.453
----------------------------------------
Stockfish 11, 48 x 1 thread, 128 MB Hash, go depth 34
1.637.587
----------------------------------------
Stockfish 11, 64 x 1 thread, 128 MB Hash, go depth 34
1.403.118c
----------------------------------------
Stockfish 11, 128 x 1 thread, 128 MB Hash, go depth 34
1.059.522

Very impressive scaling until 32 threads.
Still very good (-12% compare to 1 th.) with 48 threads.
Drop (-24% compare to 1 th.) with 64 threads.

And yet there is something very impressive about the 128 core setting.
Multiply 128 by 1.06
Multiply 64 by 1.4
Pretty astonishing that the total throughput is that high with that many threads.
If you had a giant batch of positions you wanted to analyze to a billion nodes, you should have 128 instances running.
I think that is not unexpected. But the extent how much better is perhaps a little surprising.
Must be a 3990x

[corres]

Stockfish 11, 1 thread, 128 MB Hash, go depth 34
1.857.605
----------------------------------------
Stockfish 11, 16 x 1 thread, 128 MB Hash, go depth 34
1.858.454
----------------------------------------
Stockfish 11, 32 x 1 thread, 128 MB Hash, go depth 34
1.803.453
----------------------------------------
Stockfish 11, 48 x 1 thread, 128 MB Hash, go depth 34
1.637.587
----------------------------------------
Stockfish 11, 64 x 1 thread, 128 MB Hash, go depth 34
1.403.118c
----------------------------------------
Stockfish 11, 128 x 1 thread, 128 MB Hash, go depth 34
1.059.522

Very impressive scaling until 32 threads.
Still very good (-12% compare to 1 th.) with 48 threads.
Drop (-24% compare to 1 th.) with 64 threads.

And yet there is something very impressive about the 128 core setting.
Multiply 128 by 1.06
Multiply 64 by 1.4
Pretty astonishing that the total throughput is that high with that many threads.
If you had a giant batch of positions you wanted to analyze to a billion nodes, you should have 128 instances running.
I think that is not unexpected. But the extent how much better is perhaps a little surprising.
Must be a 3990x

And a cold storage for the machine together with a household nuclear power plant.