Chess.com 2018 computer chess championship

[kranium]

Also, seldepth=3. Max tree size is <= 256^3. Its not a ponder bug, its a hardware issue.

Andrew,

May I quote you from CCCC chat?

A handful of engines (5) are randomly locked out of thread access, causing their trivial depth 1 searches to take well over a second, when usually they are < 1 micro second.
The working theory is that the 5 engines impacted are ones which realloc threads on each search. I know for example that Stockfish only does that at startup, so its less likely to be hurt by OS related tasking.

Its an OS problem. Getting locked out of CPU time is the fault of the OS
This occurs because of the hyperthreading and pondering. Windows is known to lock up contexts heavily
Stockfish does a 1 time thread alloc / binding. This reduces the overhead of swapping contexts, so the issue occurs less.

And now you're insisting it's a hardware problem...

You were well aware of the hardware platform and intended configuration as early as Aug. 8th...and responded that you were happy to participate. At that time you said you had no interest in supporting pondering in Ethereal...yet you implemented it a short time later in version 10.88, and agreed to have it used in CCCC.

Our moderator Greg had a inciteful perspective:
NJ_Greg: @AndrewGrantEthereal FWIW, I see both sides... Authors _could_ take steps to address it (to deal with crappy OS), but it _is_ also a level playing field. Some engines are (at least sort of) OK with it.

That all being said, I told you in chat I would discuss with the chess.com team to see what we could possibly do in this regard but can't promise anything.

We'd certainly also not be happy if Ethereal missed the cut because of the way it handles thread allocation.

[AndrewGrant]

Also, seldepth=3. Max tree size is <= 256^3. Its not a ponder bug, its a hardware issue.

Andrew,

May I quote you from CCCC chat?

A handful of engines (5) are randomly locked out of thread access, causing their trivial depth 1 searches to take well over a second, when usually they are < 1 micro second.
The working theory is that the 5 engines impacted are ones which realloc threads on each search. I know for example that Stockfish only does that at startup, so its less likely to be hurt by OS related tasking.

Its an OS problem. Getting locked out of CPU time is the fault of the OS
This occurs because of the hyperthreading and pondering. Windows is known to lock up contexts heavily
Stockfish does a 1 time thread alloc / binding. This reduces the overhead of swapping contexts, so the issue occurs less.

And now you're insisting it's a hardware problem...

You were well aware of the hardware platform and intended configuration as early as Aug. 8th...and responded that you were happy to participate. At that time you said you had no interest in supporting pondering in Ethereal...yet you implemented it a short time later in version 10.88, and agreed to have it used in CCCC.

Our moderator Greg had a inciteful perspective:
NJ_Greg: @AndrewGrantEthereal FWIW, I see both sides... Authors _could_ take steps to address it (to deal with crappy OS), but it _is_ also a level playing field. Some engines are (at least sort of) OK with it.

That all being said, I told you in chat I would discuss with the chess.com team to see what we could possibly do in this regard but can't promise anything.

We'd certainly also not be happy if Ethereal missed the cut because of the way it handles thread allocation.

Alright. Time to clear the air about all of this crap.

Hardware problem, thread switching problem, OS context problem. Whatever. Its both at once, or one, no one can know. Windows is in charge of giving each engine fair CPU time. The hardware is in charge of splitting physical cores into virtual ones. Somewhere there is a disconnect, and one engine's threads are allowed to throttle the others.

I was not asked, INITIALLY, to be apart of CCCC. The press release / news article was posted on chesscom's website before I was emailed by Erik. That release listed Ethereal as an active participant. I, in no way shape or form, agreed to that. After that was posted, I received an email from Erik. At which point, I agreed to participate, and said I was "Happy to" or whatever. I was trying to be nice and respectful, despite not being shown any respect initially, by not asking me.

And then again I decided to be nice and respectful, and implement the features which CCCC planned to use. Erik was hellbent on pondering, and how fun it would be to see the engine pvs live (total garbage). Adding the phrase "in a short time" is your not-so-subtle way of telling the rest of this thread that really my pondering is broken, which its not. This move here had nothing to do with pondering, as it did not come after a ponder hit.

As for NJ_Greg ... sorry ... but he has (admittedly) no idea what he is talking about. So far over 21% of the active AB field has shown issues. I'm guessing that if you share the rest of the log file with me, I fill find additional engines afflicted.

That all being said, I told you in chat I would discuss with the chess.com team to see what we could possibly do in this regard but can't promise anything.

Really? You can't promise to fix the trivial problems, which were known well in advance, by virtually everyone in this thread?

We'd certainly also not be happy if Ethereal missed the cut because of the way it handles thread allocation.

There is no need to patronize me Norman. Another subtle misdirection from the faults of chesscom. I would have hoped that you would use your position as the author of the 4th best engine to inform those around you about the technical aspects of this operation.

[Modern Times]

In my opinion, the only proper way to do ponder on matches is with two machines, so each engine has an entire machine and its resources to itself. Otherwise use ponder off.

[AndrewGrant]

Here is a sample for the impacted engines, one for each.

2018-09-10 08:30:45.016<--2:id name Nirvanachess 2.4 (64-Bit Popcount)
2018-09-10 08:33:55.385<--2:info depth 1 seldepth 19 multipv 1 score cp 64 nodes 136158436 nps 8094069 time 16822 pv b1a3 e7e5

2018-09-10 09:00:10.936<--1:id name Nemorino
2018-09-10 09:01:12.569<--1:info depth 1 seldepth 2 multipv 1 score cp -20 nodes 1334 nps 81 hashfull 190 tbhits 0 time 16407 pv e7e5

2018-09-10 10:18:35.627<--1:id name Laser 1.6
2018-09-10 10:21:44.105<--1:info depth 1 seldepth 48 score cp -5 time 8736 nodes 36956201 nps 4230326 tbhits 0 hashfull 25 pv g8f6

2018-09-10 11:04:00.354<--1:id name Ethereal 10.88 (PEXT)
2018-09-10 11:07:33.288<--1:info depth 1 seldepth 1 score cp -19 time 8531 nodes 174556732 nps 20459000 tbhits 0 hashfull 0 pv f8c5

2018-09-10 13:45:44.557<--1:id name Deep Shredder 13 x64
2018-09-10 13:46:28.001<--1:info depth 1 seldepth 6 time 1203 score cp -2 nodes 50 nps 41 hashfull 0 pv d7d5

2018-09-10 22:50:34.734<--2:id name Senpai 2.0
2018-09-10 22:51:28.710<--2:info depth 1 seldepth 1 score cp 43 nodes 49 time 4607 nps 11 pv d1h5


Nemorino and Shredder both often exhibit the problem, but when playing together only one suffers:

2018-09-11 08:42:28.604<--1:id name Deep Shredder 13 x64
2018-09-11 08:42:40.387<--2:id name Nemorino
2018-09-11 08:43:42.473<--2:info depth 1 seldepth 2 multipv 1 score cp 36 nodes 1932 nps 125 hashfull 815 tbhits 0 time 15374 pv b8c6 f1b5
2018-09-11 08:43:42.473<--2:info depth 1 seldepth 2 multipv 1 score cp 36 nodes 1932 nps 125 hashfull 815 tbhits 0 time 15374 pv b8c6 f1b5
2018-09-11 08:44:17.727<--2:info depth 1 seldepth 2 multipv 1 score cp 48 nodes 1610 nps 106 hashfull 945 tbhits 0 time 15179 pv f1c4
2018-09-11 08:45:05.499<--2:info depth 1 seldepth 2 multipv 1 score cp 39 nodes 1978 nps 125 hashfull 994 tbhits 0 time 15761 pv f8e7 d2d4
2018-09-11 08:45:39.629<--2:info depth 1 seldepth 3 multipv 1 score cp 35 nodes 2714 nps 170 hashfull 999 tbhits 0 time 15959 pv e5d4 f3d4
.... < many more lines >

Turning pondering off for an individual engine won't make a difference.
Turning hyper threading off for an individual engine won't make a difference.

Six of the Twenty-Three engines are suffering -- OFTEN -- and this is only from partial log files.

From Norman ....

Debug file split into 5 parts w/GSplit...this is the last one.
Here is a link:

https://www.dropbox.com/s/13ejha9buslcd ... 5.gsd?dl=0

Cheers, Andrew Grant

[chessdev]

Hey all! First off, thank you to everyone who has taken an interest in the CCCC. We're doing our best to create a new kind of chess event and make it awesome for fans of computer chess. This is obviously our first event and there are a lot of moving pieces with software, hardware, OS's, formats, time controls, settings, etc. It's also been an EXHAUSTING endeavor so far and everyone on the team has worked crazy hours to make it happen. The reception has been great and we again appreciate the fans and engine developers for making this worthwhile!

I do want to publicly address a few points. First off, this event was kind of "leaked" out before we were ready. We were in the midst of planning and getting things ready when the main article/URL was publicly leaked out by an engine developer before we had gotten all of our invitations out. That is on me and I'm sorry. I wanted to have that sequenced differently, but it was pushed out before we were ready to announce. My apologies.

Also, while we wanted to have the cooperation from all engine developers, we didn't get responses from everyone - and understandably so! Nobody knew what this was really, or if it was worth paying attention to. And that's fine! Again, it's the first season and we're all still learning from it. We are definitely open to feedback from everyone - fans, engine developers, etc - so we can make it better and better.

Finally, I've read through this thread and received some personal messages from Andrew from Ethereal with regards to the depth issues he presented here. I appreciate what he has shared and the passion he (and the rest of you!) bring to the game. We will absolutely take this seriously and look at what is going on. Everybody wants (and deserves!) a fair game, and we want to make sure that happens.

Please give us a little time to look at this and then share our responses. We can all agree that each engine should have access to the resources it deserves to play the best chess possible. I'd love it if we could also all agree that we're trying to do something new and interesting and that we all want to do great things for fans - and we need a little bit of patience and the benefit of the doubt to get there!

Thank you for listening (and watching!),

Erik
CEO, Chess.com

[DrCliche]

Finally, I've read through this thread and received some personal messages from Andrew from Ethereal with regards to the depth issues he presented here. I appreciate what he has shared and the passion he (and the rest of you!) bring to the game. We will absolutely take this seriously and look at what is going on. Everybody wants (and deserves!) a fair game, and we want to make sure that happens.

Andrew Grant has already looked at "what is going on", correctly diagnosed the problem, and proposed a quick and uncomplicated solution. If you want to "take this seriously" then cut the Trumpian deflection and lying, admit your mistake (which you were warned about countless times before the event started), and fix it. Immediately.

Please give us a little time to look at this and then share our responses. We can all agree that each engine should have access to the resources it deserves to play the best chess possible. I'd love it if we could also all agree that we're trying to do something new and interesting and that we all want to do great things for fans - and we need a little bit of patience and the benefit of the doubt to get there!

You have already been given "a little time". In that short amount of time, a competent, honest, and highly credible individual -- with limited access to a small subset of your logs -- correctly diagnosed the problem and proposed a quick and uncomplicated solution.

Thank you for listening (and watching!),

Erik
CEO, Chess.com

Norman's disgraceful and pathetic attempts to deflect blame onto the affected engines is absolutely breathtaking in its sheer treachery and dishonesty. You and your team appear to be as mendacious as you are incompetent. Your responses to this disastrous and never-ending cavalcade of blunders have been appalling and shameful.

[whereagles]

what's up with all this negativity? CCCC is a free event, high graphical quality, great hardware, and lots of fun to watch.

show some gratitude

[mjlef]

Here is a sample for the impacted engines, one for each.

2018-09-10 08:30:45.016<--2:id name Nirvanachess 2.4 (64-Bit Popcount)
2018-09-10 08:33:55.385<--2:info depth 1 seldepth 19 multipv 1 score cp 64 nodes 136158436 nps 8094069 time 16822 pv b1a3 e7e5

2018-09-10 09:00:10.936<--1:id name Nemorino
2018-09-10 09:01:12.569<--1:info depth 1 seldepth 2 multipv 1 score cp -20 nodes 1334 nps 81 hashfull 190 tbhits 0 time 16407 pv e7e5

2018-09-10 10:18:35.627<--1:id name Laser 1.6
2018-09-10 10:21:44.105<--1:info depth 1 seldepth 48 score cp -5 time 8736 nodes 36956201 nps 4230326 tbhits 0 hashfull 25 pv g8f6

2018-09-10 11:04:00.354<--1:id name Ethereal 10.88 (PEXT)
2018-09-10 11:07:33.288<--1:info depth 1 seldepth 1 score cp -19 time 8531 nodes 174556732 nps 20459000 tbhits 0 hashfull 0 pv f8c5

2018-09-10 13:45:44.557<--1:id name Deep Shredder 13 x64
2018-09-10 13:46:28.001<--1:info depth 1 seldepth 6 time 1203 score cp -2 nodes 50 nps 41 hashfull 0 pv d7d5

2018-09-10 22:50:34.734<--2:id name Senpai 2.0
2018-09-10 22:51:28.710<--2:info depth 1 seldepth 1 score cp 43 nodes 49 time 4607 nps 11 pv d1h5


Nemorino and Shredder both often exhibit the problem, but when playing together only one suffers:

2018-09-11 08:42:28.604<--1:id name Deep Shredder 13 x64
2018-09-11 08:42:40.387<--2:id name Nemorino
2018-09-11 08:43:42.473<--2:info depth 1 seldepth 2 multipv 1 score cp 36 nodes 1932 nps 125 hashfull 815 tbhits 0 time 15374 pv b8c6 f1b5
2018-09-11 08:43:42.473<--2:info depth 1 seldepth 2 multipv 1 score cp 36 nodes 1932 nps 125 hashfull 815 tbhits 0 time 15374 pv b8c6 f1b5
2018-09-11 08:44:17.727<--2:info depth 1 seldepth 2 multipv 1 score cp 48 nodes 1610 nps 106 hashfull 945 tbhits 0 time 15179 pv f1c4
2018-09-11 08:45:05.499<--2:info depth 1 seldepth 2 multipv 1 score cp 39 nodes 1978 nps 125 hashfull 994 tbhits 0 time 15761 pv f8e7 d2d4
2018-09-11 08:45:39.629<--2:info depth 1 seldepth 3 multipv 1 score cp 35 nodes 2714 nps 170 hashfull 999 tbhits 0 time 15959 pv e5d4 f3d4
.... < many more lines >

Turning pondering off for an individual engine won't make a difference.
Turning hyper threading off for an individual engine won't make a difference.

Six of the Twenty-Three engines are suffering -- OFTEN -- and this is only from partial log files.

From Norman ....

Debug file split into 5 parts w/GSplit...this is the last one.
Here is a link:

https://www.dropbox.com/s/13ejha9buslcd ... 5.gsd?dl=0

Cheers, Andrew Grant

Andrew, do you know if someone has tried setting program affinity for each engine to run on a different NUMA node? This should keep either program from affecting whatever the other program does with its threads. I think the format is:

start /AFFINITY [n] program.exe

where the [n] part is a hexadecimal mask with a bit for each processor.

Is the best guess of what is happening that the Windows scheduler is either putting a thread in an already busy hyperthread, or perhaps moving it around a lot?

If someone here could run some tests on a suitable machine, let me know. I only have a 4 core laptop until I get home Friday, but I could try some experiments when I get home to a 20 core machine I can boot into Windows.

Mark

[megamau]

I find DrCliche attitude absolutely appalling.

Yes, CCCC did multiple mistakes in setting up the event.
Yes, for many of these they were informed in advance on this forum.
Yes, before Erik stepped in the replies were kind of "aloof".

But let's not forget that

* Everything is completely free !
* They set-up a very interesting sporting event for the community
* The hardware and user interface is top notch
* They gave an alternative to TCEC (which had its own hardware and setup problems)
* They plan to keep doing it and improving it.

I think we should all be grateful to CCCC for the work done (I'm personally thinking to subscribe to chess.com or to donate).
The amount of resources and effort to actually DO something is order of magnitudes greater than the one for complaining.

We should keep the suggestions and requests for improvement coming, but in a respectful and attitude free way.

[DrCliche]

I find DrCliche attitude absolutely appalling.

Yes, CCCC did multiple mistakes in setting up the event.
Yes, for many of these they were informed in advance on this forum.
Yes, before Erik stepped in the replies were kind of "aloof".

But let's not forget that

* Everything is completely free !
* They set-up a very interesting sporting event for the community
* The hardware and user interface is top notch
* They gave an alternative to TCEC (which had its own hardware and setup problems)
* They plan to keep doing it and improving it.

I think we should all be grateful to CCCC for the work done (I'm personally thinking to subscribe to chess.com or to donate).
The amount of resources and effort to actually DO something is order of magnitudes greater than the one for complaining.

We should keep the suggestions and requests for improvement coming, but in a respectful and attitude free way.

CCCC has shown and continues to show utter disregard for and gross disrespect to the engines, authors, fans, and casual viewers alike. Since you place such importance on everything being "completely free!", please note that the majority of the participating engines are free, and were exploitatively entered into this fiasco without regard to the desires of the authors, as evidenced by Andrew Grant's anecdote, and as confirmed by Erik.

Granted, permission isn't necessary for the open source engines, and probably isn't legally necessary for any of the commercial engines, either, but then why did Norman (kranium) mention it at all, and in such a way that was so misrepresentative of the actual facts that it can only be interpreted as intentionally deceptive?

Because that is the consistent pattern of Chess.com and its agents with regard to this tournament. They make bad decisions, are politely cautioned by a large consensus of knowledgeable people that their decisions are nonsensical and likely to lead to unmitigated disaster, then when the very things they were warned about transpire, they lie, deflect, and attempt to cover up their mistakes without making any effort to actually address the problems. But at least they do it all for free! (As if anything in this world is free.)

You draw a comparison to snafus that occurred in the early divisions of TCEC season 13, when in reality, the comparison of how TCEC and CCCC handled their respective problems couldn't be more stark:

• TCEC wasn't forewarned about any of the problems they experienced. They went out of their way to follow every last shred of advice from the Leela developers.
• TCEC brought on an experienced tournament director and Leela contributor to help manage the tournament.
• It took more than a division and a half for even the Leela developers to definitively notice there were any issues, at which point TCEC were fully cooperative, immediately investigated, and were completely open about everything they discovered.
• There were multiple NightBot commands in chat clearly explaining the problems and what steps were being taken to mitigate them.
• TCEC immediately made changes to address the overheating by throttling the GPUs.
• TCEC were completely forthright and admitted that they didn't have the resources to rent more and better GPUs in time for the rest of the tournament, but promised to have new and improved hardware in time for season 14.

No, TCEC didn't magically make everyone happy with the outcome of division 3. They didn't replay games, and they didn't rent new hardware. They said, "We're sorry, but we don't have the resources to fix these problems right this second, and the show must go on. We promise to do better next time." It is what it is. But they admitted their mistakes, and were at all times fully cooperative, eager to listen and learn, and completely open and honest.