Chess.com 2018 computer chess championship

[AndrewGrant]

I've taken some time to run some tests and gather NPS for Ethereal (as you requested) and Stockfish.
I ran them each with 46 and 92 threads for 10 seconds:
(I'm presenting only the last PV output before 'bestmove')

Code: Select all

Ethereal 10.86 (bmi2)

setoption name Threads value 46
info string set Threads to 46
go movetime 10000
info depth 25 seldepth 32 score cp 24 time 3125 nodes 206966264 nps 66208000 tbhits 0 hashfull 996 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8b4 c4d5 e6d5 c1g5 e8g8 e2e3 b8d7 f1d3 c7c5 e1g1 b4c3 b2c3 c5c4 d3c2 d8a5 g5f4 f6e4 c2e4
bestmove d2d4 ponder g8f6

setoption name Threads value 92
info string set Threads to 92
go movetime 10000
info depth 22 seldepth 31 score cp 26 time 7984 nodes 533074064 nps 66759000 tbhits 0 hashfull 1000 pv d2d4 d7d5 g1f3 e7e6 c2c4 g8f6 b1c3 c7c5 c1g5 c5d4 f3d4 d5c4 e2e3 b8d7 f1c4 f8e7 e1g1 e8g8 c4e2 a7a6 d4f3 h7h6 g5f4 g7g5
bestmove d2d4 ponder d7d5

nps +1%

Code: Select all

Stockfish 130818 (bmi2)

setoption name Threads value 46
go movetime 10000
info depth 27 seldepth 26 multipv 1 score cp 46 upperbound nodes 720732280 nps 72066021 hashfull 999 tbhits 0 time 10001 pv d2d4 g8f6
bestmove d2d4 ponder g8f6

setoption name Threads value 92
go movetime 10000
info depth 26 seldepth 32 multipv 1 score cp 45 nodes 916696131 nps 91660447 hashfull 999 tbhits 0 time 10001 pv d2d4 g8f6
bestmove d2d4 ponder g8f6

nps +27%

As you can see there's a big difference in how effectively they scale above 40-50 threads.

Ethereal is scaling quite poorly and Stockfish doing much better...perhaps due to Lazy SMP as some are suggesting, but even with that 27% increase, we're talking about an relatively insignificant net Elo gain of approx +25 Elo. Not really an efficient use of the resources if you ask me, especially since some engine may not benefit at all.

Norman, I feel like you have done this test knowing the results beforehand ...

I'm sure you know that on a Windows machine, without special Numa code to handle 64+ threads, the threads are left on the same processor group.

Furthermore, if I were to send the latest Ethereal, that Numa support does exist, and Ethereal would exhibit the same scaling as Stockfish.

[kranium]

Norman, I feel like you have done this test knowing the results beforehand ...
I'm sure you know that on a Windows machine, without special Numa code to handle 64+ threads, the threads are left on the same processor group.
Furthermore, if I were to send the latest Ethereal, that Numa support does exist, and Ethereal would exhibit the same scaling as Stockfish.

Andrew I have no doubt about your ability to support NUMA in Ethereal, it appears there's very little that escapes you. I ran that comparison simply in an attempt to provide info, and respond to Carlos and satisfy Dr. Cliche, etc., and to illustrate my point posted earlier: "Not all engines have good scaling...and it probably can vary from system to system."
It was never my intention to embarass you, if that's what happened. Are you? You shouldn't be IMO.

I think using ponder is a joke, they should give 90+ threads to each engine with no ponder, and it would be way better.

What was Ethereal's average NPS in your tests? One thing I appreciate about Andrew Grant is that when he tests, he puts numbers to paper rather than making vague claims like "should perform well". Are you outperforming Andrew Grant's $720 processor? Are you outperforming TCEC or the now defunct YLCET? One would expect some pretty gaudy NPS numbers from the CPU engines if the claim that the CCCC will "generate the best possible chess" is to be taken seriously.

I do hope that rather than classifying any and all criticism as merely the "egotistical" ramblings of "haters", you make a real effort to address the substantive issues being raised calmly and politely by a large community of reasonable and knowledgeable viewers.

[jp]

(And lest you assume I'm not a fan of Leela, I'm actually a huge fan of the project. That's why I want Leela to achieve dominance in an environment that reasonably approximates fairness, without much room for griping and grumbling, and not because of a 60:1 effective hardware cost advantage.)


From what I can tell, most reasonable people deem the GPU hardware to be downright pornographic in comparison to the CPU hardware. It's impossible to draw a line in the sand saying exactly what would be reasonable, or fair, or interesting, or whatever any claimed goal would be. But people know sheer ridiculousness when they see it. $40,000 of graphics cards versus Ethereal pulling less NPS than on Andrew Grant's $720 CPU is simply ridiculous.

[kranium]

Andrew,

Feel free to send the NUMA version anytime, I'd be happy to make some time and run it on the server at a high thread count for you, maybe a couple games...

But it can"t be until later this week, or this weekend, because we're planning to test Lc0 RC2 next.

Norm

[AndrewGrant]

Andrew,

Feel free to send the NUMA version anytime, I'd be happy to make some time and run it on the server at a high thread count for you, maybe a couple games...

But it can"t be until later this week, or this weekend, because we're planning to test Lc0 RC2 next.

Norm

Norm, the latest version is here ( https://www.dropbox.com/s/sbe0e01ty9l0x ... 8.zip?dl=0 ) grab the PEXT.

This should make no difference for your 46 thread tests of course, but would resolve things for the 64+ test you did.

Thanks, Andrew Grant

[kranium]

(And lest you assume I'm not a fan of Leela, I'm actually a huge fan of the project. That's why I want Leela to achieve dominance in an environment that reasonably approximates fairness, without much room for griping and grumbling, and not because of a 60:1 effective hardware cost advantage.)


From what I can tell, most reasonable people deem the GPU hardware to be downright pornographic in comparison to the CPU hardware. It's impossible to draw a line in the sand saying exactly what would be reasonable, or fair, or interesting, or whatever any claimed goal would be. But people know sheer ridiculousness when they see it. $40,000 of graphics cards versus Ethereal pulling less NPS than on Andrew Grant's $720 CPU is simply ridiculous.

Yes JP, we're all aware that running an engine on a high-end GPU system is much more expensive than on a high-end CPU system.
But does that mean we shouldn't do it?

I haven't heard one engine author complain yet about the cost ratio...on the contrary, they're very excited to participate.

PS - copying another users post, and bolding text in it, is just plain spamming...is that all you have to offer?

You know, there are are many types of people...ones that jump in, work hard, and make something happen...
ones that graciously accept what's been given for free, and make the best of it and enjoy it...
and ones that stand on the sidelines, waving their arms, complaining about everything.

[jhellis3]

Well, the 2080Ti should be fairly close to a V100 for LC0 purposes, and will go on sale in the 1 - 1.2k price range in the not too distant future. So that should bring the cost down by roughly 1 order of magnitude. Though when it comes to perf/dollar I suspect the 2070 will be the one to beat.

[Guenther]

I never saw anything besides the download symbol which is a link to 'download pgn'
and this always resulted in plain moves.

I'll look into it and try to take care of it asap.

Well, nothing has changed despite the tournament already has started.
Still no eval/depth/time not even a round number.

'Download full tournament' which I hoped would give all yet played games
just gives a 'network error'.

Code: Select all

tournament.pgn = Error: Network Error

Code: Select all

[Event "CCCC 1: Rapid Rumble (15|5) Stage 1"]
[Site "Chess.com"]
[Date "2018.09.01"]
[Round "?"]
[White "Fritz 16.10"]
[Black "Nemorino 5.00"]
[Result "*"]
[WhiteElo "3200"]
[BlackElo "3099"]
[WhiteType "program"]
[BlackType "program"]
[ECO "D02"]
[Opening "Queen's pawn game"]
[TimeControl "900+5"]
[Time "05:19:04"]
[Termination "unterminated"]
[PlyCount "56"]

1. Nf3 d5 2. d4 Bf5 3. c4 e6 4. Nc3 Bb4 5. cxd5 exd5 6. Bg5 Ne7 7. e3 c6 8. Bf4
Ng6 9. Bg3 Nd7 10. a3 Bxc3+ 11. bxc3 Nf6 12. Nd2 h5 13. h3 O-O 14. Be2 h4 15.
Bh2 Ne4 16. Nxe4 Bxe4 17. O-O Qg5 18. Bf3 Bxf3 19. Qxf3 f5 20. Rab1 b6 21. Rfe1
Rae8 22. Bd6 Rf7 23. c4 Re6 24. Bf4 Nxf4 25. exf4 Qe7 26. Qc3 dxc4 27. Rxe6 Qxe6
28. Re1 Qd5 29. Re5 Qd7 30. Qxc4 Kh7 31. Qd3 Rf6 *

[AdminX]

I never saw anything besides the download symbol which is a link to 'download pgn'
and this always resulted in plain moves.

I'll look into it and try to take care of it asap.

Well, nothing has changed despite the tournament already has started.
Still no eval/depth/time not even a round number.

'Download full tournament' which I hoped would give all yet played games
just gives a 'network error'.

Code: Select all

tournament.pgn = Error: Network Error

Code: Select all

[Event "CCCC 1: Rapid Rumble (15|5) Stage 1"]
[Site "Chess.com"]
[Date "2018.09.01"]
[Round "?"]
[White "Fritz 16.10"]
[Black "Nemorino 5.00"]
[Result "*"]
[WhiteElo "3200"]
[BlackElo "3099"]
[WhiteType "program"]
[BlackType "program"]
[ECO "D02"]
[Opening "Queen's pawn game"]
[TimeControl "900+5"]
[Time "05:19:04"]
[Termination "unterminated"]
[PlyCount "56"]

1. Nf3 d5 2. d4 Bf5 3. c4 e6 4. Nc3 Bb4 5. cxd5 exd5 6. Bg5 Ne7 7. e3 c6 8. Bf4
Ng6 9. Bg3 Nd7 10. a3 Bxc3+ 11. bxc3 Nf6 12. Nd2 h5 13. h3 O-O 14. Be2 h4 15.
Bh2 Ne4 16. Nxe4 Bxe4 17. O-O Qg5 18. Bf3 Bxf3 19. Qxf3 f5 20. Rab1 b6 21. Rfe1
Rae8 22. Bd6 Rf7 23. c4 Re6 24. Bf4 Nxf4 25. exf4 Qe7 26. Qc3 dxc4 27. Rxe6 Qxe6
28. Re1 Qd5 29. Re5 Qd7 30. Qxc4 Kh7 31. Qd3 Rf6 *

You have to click on the Full Tournament Download button to get the PGN with eval data. It works over here, maybe a browser issue. I am using Chrome browser.

Correction: I thought it had downloaded and opened in Chessbase but it had not. You sir are right, I click but no download. I just get the Network error you posted about.

[AdminX]

After a little bit ( 5 Min) of probing their HTML code, I found a temp solution to the problem.

Grab it from here: https://newman.chess.com/games.pgn

PS:

Pro Tip: If you use the non secure version of the link http://newman.chess.com/games.pgn it will also work in Hiarcs Chess Explorer and you can then use your favorite engine to analyze the current game while it is in progress.