Chess.com 2018 computer chess championship

[arunsoorya1309]

There's no indication that the CPU engines will be given full access to the anemic CPU machine during games against Leela. The official documentation simply states the CPU engines will be limited to 46 threads and 8192 MB of hash without mention of any exceptions.

The CPU engines will be the only engine running on the system, and will have full access to the processors (w/ 46 threads addressing 48 physical CPUs) while playing vs. Lc0
I recently ran Ethereal in this manner and the NPS was impressive, so it's clear that that the CPU engines will be running at full or near full strength, and should perform well in games against Lc0.

With facts like these it becomes hard not see an agenda, and the claim that "the CCCC will run on cutting-edge technology to generate the best possible chess" becomes laughable.

I can assure you there's no 'agenda'. We are simply trying to provide a fun and entertaining event that includes high-level and competitive chess. All of which I can guarantee. We are of course very interested in the Lc0 project, and are fascinated by it's unusual yet strong style of play (as are many), and have tried to provide it with a suitable platform in an effort to help it play to it's full and current potential. If funding allowed we would have gone with 8x Tesla v100s. We're not overly concerned if the CPU vs GPU hardware platforms equate perfectly or not. This is not an official (software) world championship event.

These events will be run 24/7/365, a new one each month, with a enormous variety of formats and configurations.
Time Control: blitz, rapid, and classical
Ponder: on or off
Opening book: no book, 2 moves, 4 moves, 6 move, and 8 moves, and on occasion: thematic openings
etc.

We've specifically chosen ponder 'on' for this 1st inaugural event for 1 reason only...the chess.com team feels very strongly about letting viewers see both engine's thinking simultaneously, in an effort to highlight the new web UI capabilities (which we've written from the ground up with the newest modern web technologies). This includes a groundbreaking "live PVs" web broadcast feature.

I do hope you can enjoy it instead of picking it apart, leveling accusations, and being so critical of the effort. I certainly realize there's a lot of 'haters', and egotistical people who believe they could do much better, but please don't forget: these events are being produced, funded, and broadcast for free by chess.com for the benefit of community...as the amount of new subscribers necessary to pay for it all is astronomical.

With all due respect, live pv was already showcased very well in YLCET and now is part of TCEC, not sure what is groundbreaking in that. Also for blitz and rapids the way it is done, is more problematic than useful for viewers, you cannot follow anything in the pv

[DrCliche]

The CPU engines will be the only engine running on the system, and will have full access to the processors (w/ 46 threads addressing 48 physical CPUs) while playing vs. Lc0

The machine in question supports 96 logical cores, while the CPU engines are being limited to 46 in all games. We all understand it's standard practice to reserve one physical core for the OS, so full access for the CPU engines would entail 94 logical cores. 46 is less than 94, therefore the CPU engines will not have full access to the processors.

I recently ran Ethereal in this manner and the NPS was impressive, so it's clear that that the CPU engines will be running at full or near full strength, and should perform well in games against Lc0.

What was Ethereal's average NPS in your tests? One thing I appreciate about Andrew Grant is that when he tests, he puts numbers to paper rather than making vague claims like "should perform well". Are you outperforming Andrew Grant's $720 processor? Are you outperforming TCEC or the now defunct YLCET? One would expect some pretty gaudy NPS numbers from the CPU engines if the claim that the CCCC will "generate the best possible chess" is to be taken seriously. From what I can tell, most reasonable and knowledgeable people believe that claim to be laughable.

(And of course we all understand any given entity will, in practice, be limited by resources. "Best possible chess" is simply shorthand. But does Chess.com appear to have made a reasonable effort to ensure all engines will "generate the best possible chess" within the context of the apparent limit of the resources allocated to the project? I know of no reasonably independent party that believes that to be the case.)

I can assure you there's no 'agenda'. We are simply trying to provide a fun and entertaining event that includes high-level and competitive chess. All of which I can guarantee. We are of course very interested in the Lc0 project, and are fascinated by it's unusual yet strong style of play (as are many), and have tried to provide it with a suitable platform in an effort to help it play to it's full and current potential. If funding allowed we would have gone with 8x Tesla v100s.

Assurances or no, providing one and only one engine "with a suitable platform in an effort to help it play to its full and current potential" reasonably appears to many to plainly be your agenda.

(And lest you assume I'm not a fan of Leela, I'm actually a huge fan of the project. That's why I want Leela to achieve dominance in an environment that reasonably approximates fairness, without much room for griping and grumbling, and not because of a 60:1 effective hardware cost advantage.)

We're not overly concerned if the CPU vs GPU hardware platforms equate perfectly or not. This is not an official (software) world championship event.

Of course it's impossible to ensure that both hardware platforms equate perfectly, as no one can agree what that even means. I'm reminded of Justice Potter Stewart's most famous quote:

I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description ["hard-core pornography"], and perhaps I could never succeed in intelligibly doing so. But I know it when I see it, and the motion picture involved in this case is not that.

From what I can tell, most reasonable people deem the GPU hardware to be downright pornographic in comparison to the CPU hardware. It's impossible to draw a line in the sand saying exactly what would be reasonable, or fair, or interesting, or whatever any claimed goal would be. But people know sheer ridiculousness when they see it. $40,000 of graphics cards versus Ethereal pulling less NPS than on Andrew Grant's $720 CPU is simply ridiculous.

We've specifically chosen ponder 'on' for this 1st inaugural event for 1 reason only...the chess.com team feels very strongly about letting viewers see both engine's thinking simultaneously, in an effort to highlight the new web UI capabilities (which we've written from the ground up with the newest modern web technologies). This includes a groundbreaking "live PVs" web broadcast feature.

While watching some of your testing I found the constantly shifting PVs to be exceedingly distracting. Of course, someone, somewhere, may derive some value from watching two PV windows flash wildly many times a second, but I personally detest it, not to put too fine a point on it. May I request that you allow viewers to disable this feature if they desire?

(It's also disappointing to me because most reasonable and knowledgeable people seem to agree that ponder will result in lower quality chess, at least in games between top CPU engines. It seems a shame to produce lower quality chess merely for the sake of two simultaneously updating live PV displays that many people couldn't care less about, or actively dislike. Furthermore, it's simply bizarre and without precedent in computer chess that the Houdini facing Leela won't be the same as the Houdini facing Komodo.)

I do hope you can enjoy it instead of picking it apart, leveling accusations, and being so critical of the effort. I certainly realize there's a lot of 'haters', and egotistical people who believe they could do much better, but please don't forget: these events are being produced, funded, and broadcast for free by chess.com for the benefit of community...as the amount of new subscribers necessary to pay for it all is astronomical.

I do hope that rather than classifying any and all criticism as merely the "egotistical" ramblings of "haters", you make a real effort to address the substantive issues being raised calmly and politely by a large community of reasonable and knowledgeable viewers.

[Nay Lin Tun]

AFAIK, they have 96 logical cores, so dividing half resources (46 logical cores for both AB engines ) with ponder ON make sense. Ponder OFF and give full 90+ cores to each engine in their individual turn will give higher quality of chess.(because you dont need waste time guessing/calculating what opponent will be doing) However Ponder On makes more realistic human tournament condition. (A player/engine who can make more accurate prediction for opponent's move on opponent's time will get higher scores)

[hgm]

If you want the highest quality chess for this hardware, you should play 92 single-core ponder-off games in parallel, with a 92 times slower PC.

[Laskos]

If you want the highest quality chess for this hardware, you should play 92 single-core ponder-off games in parallel, with a 92 times slower PC.

TC, right?
Haha, that's true.

[whereagles]

This lack of proof frustrates me tremendously. Hard to accept that lazy smp works.

Sometimes you just have to rely on.. huh.. let's call it "experimental physics"

[kranium]

With all due respect, live pv was already showcased very well in YLCET and now is part of TCEC, not sure what is groundbreaking in that. Also for blitz and rapids the way it is done, is more problematic than useful for viewers, you cannot follow anything in the pv

Hi Arun,
I viewed YLCET once early on when it was 1st announced...I don't remember seeing anything that resembled 'live PVs'. Perhaps this was a later development?

I've been viewing TCEC for years...it shows 1 PV per turn, the last one the engine has output before making a move. Has something changed?

Yes I agree with your last statement...the PV boards change rapidly, especially in the beginning of search when the engine is outputting many depths and corresponding PVs. After some time though, this should stabilize and slow down considerably. With long Time Controls, the engine will output new PVs at a much slower rate. Perhaps this is something we can tweak in the coming weeks.

[kranium]

It's fairly well known Carlos that most engines scale very poorly above 30-40 threads, with Elo gains almost flatlining, so allocating 90+ threads would be the real joke...a huge waste of resources.

I believe this is outdated information. There's been a fair amount of testing over the past couple of years (as well as improvements to Lazy SMP implementations) that show CPU engines have good scaling as far as anyone has been able to test:

Not all engines have good scaling...and it probably can vary from system to system.

What was Ethereal's average NPS in your tests? One thing I appreciate about Andrew Grant is that when he tests, he puts numbers to paper rather than making vague claims like "should perform well". Are you outperforming Andrew Grant's $720 processor? Are you outperforming TCEC or the now defunct YLCET? One would expect some pretty gaudy NPS numbers from the CPU engines if the claim that the CCCC will "generate the best possible chess" is to be taken seriously. From what I can tell, most reasonable and knowledgeable people believe that claim to be laughable.

The quote "generate the best possible chess" which is causing you so much dismay, is simply a catch phrase utilized in the announcement by the marketing team. Believe me, if they knew that it was going to cause you so much grief, the would have likely omitted it.
I have no idea if we outperform TCEC or YLCET...that was not high in our list of goals for the events, and has not been measured.

I've taken some time to run some tests and gather NPS for Ethereal (as you requested) and Stockfish.
I ran them each with 46 and 92 threads for 10 seconds:
(I'm presenting only the last PV output before 'bestmove')

Code: Select all

Ethereal 10.86 (bmi2)

setoption name Threads value 46
info string set Threads to 46
go movetime 10000
info depth 25 seldepth 32 score cp 24 time 3125 nodes 206966264 nps 66208000 tbhits 0 hashfull 996 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8b4 c4d5 e6d5 c1g5 e8g8 e2e3 b8d7 f1d3 c7c5 e1g1 b4c3 b2c3 c5c4 d3c2 d8a5 g5f4 f6e4 c2e4
bestmove d2d4 ponder g8f6

setoption name Threads value 92
info string set Threads to 92
go movetime 10000
info depth 22 seldepth 31 score cp 26 time 7984 nodes 533074064 nps 66759000 tbhits 0 hashfull 1000 pv d2d4 d7d5 g1f3 e7e6 c2c4 g8f6 b1c3 c7c5 c1g5 c5d4 f3d4 d5c4 e2e3 b8d7 f1c4 f8e7 e1g1 e8g8 c4e2 a7a6 d4f3 h7h6 g5f4 g7g5
bestmove d2d4 ponder d7d5

nps +1%

Code: Select all

Stockfish 130818 (bmi2)

setoption name Threads value 46
go movetime 10000
info depth 27 seldepth 26 multipv 1 score cp 46 upperbound nodes 720732280 nps 72066021 hashfull 999 tbhits 0 time 10001 pv d2d4 g8f6
bestmove d2d4 ponder g8f6

setoption name Threads value 92
go movetime 10000
info depth 26 seldepth 32 multipv 1 score cp 45 nodes 916696131 nps 91660447 hashfull 999 tbhits 0 time 10001 pv d2d4 g8f6
bestmove d2d4 ponder g8f6

nps +27%

As you can see there's a big difference in how effectively they scale above 40-50 threads.

Ethereal is scaling quite poorly and Stockfish doing much better...perhaps due to Lazy SMP as some are suggesting, but even with that 27% increase, we're talking about an relatively insignificant net Elo gain of approx +25 Elo. Not really an efficient use of the resources if you ask me, especially since some engine may not benefit at all.

[Guenther]

...
Perhaps this is something we can tweak in the coming weeks.

I hope pgn download will be 'improved' too later for the real tournament.
Currently it contains just the plain moves. No eval/depth/time, the games are quite pointless this way.

[kranium]

...
Perhaps this is something we can tweak in the coming weeks.

I hope pgn download will be 'improved' too later for the real tournament.
Currently it contains just the plain moves. No eval/depth/time, the games are quite pointless this way.

Hi Guenther,
The Pgn does include those things:

1. e4 { (e2e4 c7c5 Ng1f3 e7e6 Nb1c3 Nb8c6 Bf1e2 Ng8f6 d2d4 c5xd4 Nf3xd4
Bf8b4 Qd1d3 OO OO e6e5 Nd4xc6 b7xc6 Rf1d1 Qd8e7 Bc1e3 Bc8b7 h2h3 Bb4c5
Nc3a4 Bc5xe3 Qd3xe3 d7d6 Na4c3) +0.15/25 94 }

How did you download? (there are couple methods available)
If that data has been stripped by the web front end somehow, I'll definitely look into it.
Thx for the heads up!