237 Mn/s for Stockfish on an 2xEPYC 7742

[Raphexon]

You'll be surprised just how strong SF is, even at low node counts.

This is especially true with the latest version of SF. I think SF has gained incredible strength at low nodes thanks to tuning against Lc0!

I've tested numerous SF versions on 1+0.01s on my i5-4460, i5-3470 and an i3-6100U and current SFDev definitely isn't stronger than SF7, 8, 9 and 10 at ultra low TC.

Current development version has received numerous patches that are either elo neutral or elo negative on very low TC but elo positive on longer TC.
That's why there's probably a regression.

These are the 20000 test games I did on my i3-6100U laptop:
TC= 1sec + 0.01sec.
Settings are all default.

Code: Select all

    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 stockfish_9_x64                :   38    7   7  8000    56.7 %     -9   27.6 %
  2 stockfish_10_x64               :   28    7   7  8000    55.1 %     -7   25.8 %
  3 stockfish_8_x64                :   23    6   6  8000    54.2 %     -6   28.2 %
  4 stockfish 7 x64                :  -30    7   7  8000    44.7 %      8   26.9 %
  5 stockfish_20011312_x64         :  -60    7   7  8000    39.4 %     15   21.6 %

And the PGN file.

http://www.mediafire.com/file/qx8v0mlub ... c.pgn/file

[Alayan]

Yes, SF hasn't become any stronger at 1s+0.01.

This is particularly related to this patch : https://github.com/official-stockfish/S ... /pull/2260

It lost 40-50 elo at 2s+0.02 ; lost ~4 elo at 10s+0.1 @ 1.6mnps ; gained ~1 elo at 20s+0.2 ; gained ~5 elo at 60s+0.6 and around ~12 elo at 180s+1.8.

[Raphexon]

Yes, SF hasn't become any stronger at 1s+0.01.

This is particularly related to this patch : https://github.com/official-stockfish/S ... /pull/2260

It lost 40-50 elo at 2s+0.02 ; lost ~4 elo at 10s+0.1 @ 1.6mnps ; gained ~1 elo at 20s+0.2 ; gained ~5 elo at 60s+0.6 and around ~12 elo at 180s+1.8.

Are there still plans to change STC from 10+0.1 to something higher?

[Alayan]

Are there still plans to change STC from 10+0.1 to something higher?

There has been a push for 15+0.15 (or even 20+0.2), but nothing has come out of it yet.

I expect it to happen and the STC bounds to get less wide than they are now.

[JohnWoe]

I think comparing move generator to search speed is much more ideal.

For exampke my engine generates 20 mnps and search 4 mnps. So the slowdown here is 5x. That is due to the fact Sapeli has so simple evaluation + search. It doesn't slow down.

However Stockfish:
They have removed NPS but to me it seems to take about 1 second. So 119 mnps. Compared to Sapeli's 20 mnps! That 6x faster.

Code: Select all

Stockfish 160120 64 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
go perft 6
a2a3: 4463267
b2b3: 5310358
c2c3: 5417640
d2d3: 8073082
e2e3: 9726018
f2f3: 4404141
g2g3: 5346260
h2h3: 4463070
a2a4: 5363555
b2b4: 5293555
c2c4: 5866666
d2d4: 8879566
e2e4: 9771632
f2f4: 4890429
g2g4: 5239875
h2h4: 5385554
b1a3: 4856835
b1c3: 5708064
g1f3: 5723523
g1h3: 4877234

Nodes searched: 119060324

Here SF searching only 900_000 NPS. So the mgen -> search slowdown is 100x. As there is a lot more stuff in SF's search and eval.

Code: Select all

Stockfish 160120 64 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
go movetime 1000
info depth 1 seldepth 1 multipv 1 score cp 114 nodes 20 nps 6666 tbhits 0 time 3 pv e2e3
info depth 2 seldepth 2 multipv 1 score cp 126 nodes 54 nps 18000 tbhits 0 time 3 pv e2e3 b7b6
info depth 3 seldepth 3 multipv 1 score cp 124 nodes 147 nps 36750 tbhits 0 time 4 pv e2e3 b7b6 f1c4
info depth 4 seldepth 4 multipv 1 score cp 75 nodes 574 nps 114800 tbhits 0 time 5 pv d2d4 e7e6 e2e3 b8c6
info depth 5 seldepth 5 multipv 1 score cp 86 nodes 782 nps 130333 tbhits 0 time 6 pv c2c3 b8c6 d2d4 d7d6
info depth 6 seldepth 6 multipv 1 score cp 56 nodes 2734 nps 210307 tbhits 0 time 13 pv d2d4 c7c6 g1f3 d7d5 c2c3 g8f6
info depth 7 seldepth 8 multipv 1 score cp 43 nodes 5780 nps 361250 tbhits 0 time 16 pv c2c3 b8c6 d2d4 d7d5 e2e3 g8f6 f1e2
info depth 8 seldepth 10 multipv 1 score cp 46 nodes 9568 nps 455619 tbhits 0 time 21 pv b1c3 d7d5 e2e4 d5e4 c3e4 c7c6 g1f3 g8f6
info depth 9 seldepth 13 multipv 1 score cp 48 nodes 14134 nps 543615 tbhits 0 time 26 pv e2e4 d7d5 e4d5 d8d5 d2d4 d5e6 c1e3 g8f6 g1e2 b8c6
info depth 10 seldepth 14 multipv 1 score cp 105 nodes 18613 nps 600419 tbhits 0 time 31 pv e2e4 e7e6 b1c3 b8c6 d2d4 d7d5 e4d5 e6d5 g1f3
info depth 11 seldepth 14 multipv 1 score cp 110 nodes 37332 nps 704377 tbhits 0 time 53 pv e2e4 e7e5 g1f3 d7d5 d2d4 b8c6 f3e5 c6e5 d4e5 d5e4 d1d8 e8d8
info depth 12 seldepth 21 multipv 1 score cp 110 nodes 99541 nps 809276 tbhits 0 time 123 pv e2e4 e7e5 g1f3 b8c6 d2d4 e5d4 f3d4 d8h4 d4b5 h4e4 d1e2 e4e2 f1e2 e8d8 c1f4
info depth 13 seldepth 21 multipv 1 score cp 90 nodes 161123 nps 826271 tbhits 0 time 195 pv e2e4 d7d5 e4d5 d8d5 b1c3 d5a5 f1e2 e7e5 g1f3 b8c6 e1g1 c8e6 d2d4 e8c8 c3b5 f7f6 c2c3 e6a2
info depth 14 seldepth 24 multipv 1 score cp 91 nodes 235857 nps 836372 tbhits 0 time 282 pv e2e4 e7e5 d2d4 e5d4 d1d4 b8c6 d4e3 b7b6 g1f3 f8c5 e3f4 d7d6 b1c3 g8f6
info depth 15 seldepth 23 multipv 1 score cp 65 nodes 525778 nps 861931 tbhits 0 time 610 pv e2e4 e7e6 d2d4 d7d5 b1c3 f8b4 e4d5 e6d5 g1f3 b8c6 f1d3 d8e7 d1e2 g8f6 e1g1 b4c3 b2c3 e8g8 c1g5 h7h6
info depth 16 seldepth 25 multipv 1 score cp 100 nodes 654750 nps 863786 tbhits 0 time 758 pv e2e4 e7e6 d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 d1e2 c8e6 f3g5 b8c6 c2c3 d8e7 c1f4 e8c8 g5e6 e7e6 e2e6 f7e6 b1d2
info depth 17 seldepth 29 multipv 1 score cp 80 nodes 864196 nps 863332 hashfull 382 tbhits 0 time 1001 pv e2e4 e7e6
bestmove e2e4 ponder e7e6
quit

[Uri Blass]

How far it is above Crafty really doesn't affect my comments. 2K nodes per second is not going to beat a GM. Might not beat a master, since most of the speculative pruning stuff really needs exceptional search depth to bypass the holes it causes.

1 second per move vs 180 seconds per move? Maybe. But certainly not 360K nodes total vs 180 seconds for a GM.

As far as the other numbers you quoted, I don't think any of 'em are even close to reality. Stockfish at 1K nodes per second is going to get roasted by DB. Completely roasted. That is only a factor of 100,000 in terms of nodes per second. Would be pretty easy to test. Take the online Cray Blitz source, and let it run for 180 * 200M nodes total and limit stockfish to 360K total. Cray Blitz was fairly similar to DB and others of that time frame, primarily using recursive null-move but nothing else regarding forward pruning, except maybe in the last few plies before q-search. Would be an interesting thing to try, but it would certainly take forever for a single game. I'm pretty sure nobody would want to scale that down to 2M nodes per second vs 10 nodes per second to keep the time ratio the same. I can certainly predict how badly that will end.

I believe 360K nodes per move is enough for stockfish to beat GM's at 180 seconds per move.

Stockfish is basically optimized for bullet games of 60+0.6 with one core.
It get more than 360K nodes per move in these conditions but still part of the testing is done with 10+0.1 time control that is not a lot faster than 360K nodes per move.

You should test this yourself. You might be surprised. It will be tactically very weak with such shallow searches.

Another type of test would be stockfish on an old pentium pro 200. Might hit 50K nodes per second. 6 secs per move for stockfish, 180 seconds per move for human? I know how computers performed back then. They were becoming pretty problematic against GM players at very fast games. But give the GMs 3 minutes to defeat a 300K node (total search tree) program would not offer him much of a challenge. Note that the program could NOT "ponder" which would skew the total nodes searched unacceptably. Would be interesting to see how deeply stockfish could search given a computational budget of 300K nodes total.

For your 10 + .1 time control, assume first time control = 40 moves, so that is 14 seconds used over 40 moves, or a third of a second per move. Crafty, on my MacBook hits around 10M nodes per second. That would equate to about 3M nodes per move, if pondering is not allowed. That is still 10x the 300K. I could always be wrong here, but my experience says 300K total nodes per search is not enough to play at GM level when a GM has three full minutes to use per move. Larry K might dive in here with his thoughts. Just doesn't seem like "enough tree" to me based on what we saw back when those sized trees were quite common.

10M nodes per second is not correct for stockfish when it does not use more than one thread.

I can see the number of nodes per second in the page of the testing

http://tests.stockfishchess.org/tests

I see 1.41M nodes per second(it is not constant and can be slightly more than it or slightly less than it).

with 1/3 second per move you get initially 470K nodes per move that is more than 360K nodes per move but not by a big difference and not in the first 40 moves you have to use only 1/10 second per move not to lose on time so you get less than 360K nodes per move.

[Uri Blass]

Yes, SF hasn't become any stronger at 1s+0.01.

This is particularly related to this patch : https://github.com/official-stockfish/S ... /pull/2260

It lost 40-50 elo at 2s+0.02 ; lost ~4 elo at 10s+0.1 @ 1.6mnps ; gained ~1 elo at 20s+0.2 ; gained ~5 elo at 60s+0.6 and around ~12 elo at 180s+1.8.

I think that it is better if stockfish comes with some versions that are optimized for different time control including 2s+0.02 so maclane does not complain that stockfish play weak at low number of nodes per move because he will be able to use also a version that does not use the counterproductive changes at 2+0.02

[Laskos]

How far it is above Crafty really doesn't affect my comments. 2K nodes per second is not going to beat a GM. Might not beat a master, since most of the speculative pruning stuff really needs exceptional search depth to bypass the holes it causes.

1 second per move vs 180 seconds per move? Maybe. But certainly not 360K nodes total vs 180 seconds for a GM.

As far as the other numbers you quoted, I don't think any of 'em are even close to reality. Stockfish at 1K nodes per second is going to get roasted by DB. Completely roasted. That is only a factor of 100,000 in terms of nodes per second. Would be pretty easy to test. Take the online Cray Blitz source, and let it run for 180 * 200M nodes total and limit stockfish to 360K total. Cray Blitz was fairly similar to DB and others of that time frame, primarily using recursive null-move but nothing else regarding forward pruning, except maybe in the last few plies before q-search. Would be an interesting thing to try, but it would certainly take forever for a single game. I'm pretty sure nobody would want to scale that down to 2M nodes per second vs 10 nodes per second to keep the time ratio the same. I can certainly predict how badly that will end.

I would be willing to play stockfish at 100 nodes per second myself, and I have not actively played chess in 30 years (for tournaments anyway).

18000 nodes total is NOT going to play like a GM, in any shape for form.

With SF_dev at 5 kn/s I have no doubts SF_dev is no less than 2700 FIDE Elo level at tournament time control. Bob, software has advanced much. In some maybe 600-700 computer Elo-ish points since 1990s. Meaning some 400 FIDE Elo-ish points against humans.

There are many ways to reach this conclusion. One of the many is that Rebel 9 in 1997 was already some 2600 FIDE Elo level TPR on P200 in a good number of games against humans, having some 60 kn/s IIRC.

Do the math.

[Dann Corbit]

There was an analysis in the programming forum a while back that showed software advance in strength is exponential, just like the hardware advance.

[Ovyron]

I think chess engines have become so much stronger than humans that ELO breaks down when trying to measure by how much.