Chess.com 2018 computer chess championship

[jp]

They should use only 5-men Syzygy, otherwise it's a serious issue with ponder on.

If you think TB access may be an issue, I'll seriously consider limiting to 5 man...
but I must say, we've been running this exact config (46 threads/ponder-on, and 6-man TBs) all summer with no issues.

How do you know that if there are issues you'd be aware of them? It's not easy to see issues depending what the issues are.

[mar]

I'm quite familiar with hyperthreading. My concerns remain.

Indeed, this shows the incompetence of the organizers by not understanding the difference between a logical and a physical core.
I guess nobody even bothered to measure NPS ponder off vs ponder on (assuming running on the same machine),
not to mention that the two processes will compete not only for CPU resources but also for memory.
It's unclear what ponder on would do to two Leelas; most likely this would depend on how well Leela can utilize the GPU(s).
48 threads per engine, ponder off, problem solved. HT should easily guarantee overall system responsiveness. They may even save some electricity.

Ponder on is how humans think, sure, ponder on is awesome. But humans don't share brains.
So yes, on different machines that's something else and actually very nice.

[AndrewGrant]

I think using ponder is a joke, they should give 90+ threads to each engine with no ponder, and it would be way better.

.... STOP PONDERING ON CCLS !!!!! .....

[AndrewGrant]

1. 4x V 100 for Leela vs 48 cores ( GPU/CPU ratio will be 3.6 times in favor for Leela than TCEC, approximately + 90 elo)

You can double that ratio to 7.2, since Leela won't be sharing CPU/GPU time as she will have her own machine, where as the other engines will be sharing the same RAM/CPU/SSD.

[jhellis3]

Well, all the other engines won't be sharing resources while playing LC0, so she will face engines with 40-something "real" threads with no memory or disk contention. On the other hand, while the conventional engines will face weaker competition than Leela, they will also be weaker when doing so...

The upshot is that the games vs LC0 should be of higher overall quality.

[jkiliani]

Well, all the other engines won't be sharing resources while playing LC0, so she will face engines with 40-something "real" threads with no memory or disk contention. On the other hand, while the conventional engines will face weaker competition than Leela, they will also be weaker when doing so...

The upshot is that the games vs LC0 should be of higher overall quality.

Makes sense, but I'm wondering why they didn't just specify no pondering in matches between engines playing on the regular server (or between multiple engines on the GPU server), but pondering allowed between engines playing on different servers.

[Jhoravi]

Well, all the other engines won't be sharing resources while playing LC0, so she will face engines with 40-something "real" threads with no memory or disk contention. On the other hand, while the conventional engines will face weaker competition than Leela, they will also be weaker when doing so...

The upshot is that the games vs LC0 should be of higher overall quality.

With ponder ON, AB engines will become much stronger when facing against Leela because this time no other engine is utilizing half of the resource. It means AB engine strength is not constant throughout the tournament. Not a good way to measure relative strength IMO. The problem is easily fixed by turning the ponder OFF though.

[AndrewGrant]

Well, all the other engines won't be sharing resources while playing LC0, so she will face engines with 40-something "real" threads with no memory or disk contention. On the other hand, while the conventional engines will face weaker competition than Leela, they will also be weaker when doing so...

The upshot is that the games vs LC0 should be of higher overall quality.

With ponder ON, AB engines will become much stronger when facing against Leela because this time no other engine is utilizing half of the resource. It means AB engine strength is not constant throughout the tournament. Not a good way to measure relative strength IMO. The problem is easily fixed by turning the ponder OFF though.

I just watched an Ethereal vs Shredder game. Ethereal was pulling 30mnps throughout.

On my single Ryzen 1950x I get 60mnps.

[DrCliche]

It's fairly well known Carlos that most engines scale very poorly above 30-40 threads, with Elo gains almost flatlining, so allocating 90+ threads would be the real joke...a huge waste of resources.

I believe this is outdated information. There's been a fair amount of testing over the past couple of years (as well as improvements to Lazy SMP implementations) that show CPU engines have good scaling as far as anyone has been able to test:

• 384-thread Stockfish is 94±12 Elo stronger than 64-thread Stockfish: http://tests.stockfishchess.org/tests/v ... 02b2e5fec5
• 384-thread Stockfish is 22±10 Elo stronger than 192-thread Stockfish: http://tests.stockfishchess.org/tests/v ... 02b2e60162

Note that those tests were run on a single 192-core machine, so the second test merely pitted Stockfish against itself with hyperthreading enabled in one instance, and disabled in the other. At the level of the top engines, I don't think anyone would consider that to be poor scaling at all. I'm actually astonished that Stockfish on 192 cores gets 22±10 Elo stronger simply by turning on hyperthreading.

The perennial top dogs in various tournaments like TCEC and the now defunct YLCET all use Lazy SMP, all support large numbers of threads, and all continue to scale well along with them. (Stockfish supports 512, Houdini 128, Komodo 64, Ethereal 2048, Fire 64, etc.) While some of the less prominent engines might not be able to make use of large numbers of threads, it's a peculiar decision to punish the best engines for the shortcomings of the worst.

It also seems peculiar to use a setup and settings that result in CPU engine performance significantly weaker than TCEC, and even weaker than mid-range consumer hardware like Andrew Grant's. With the amount of firepower being thrown at Leela, the only way the competition could be interesting is if her opponents were similarly armed. It's not promising that the CPU engine hardware fails to match the performance a CPU you can get for $720 on Amazon, while Leela will be crunching away on four graphics cards that each retail for over $10,000.

With facts like these it becomes hard not see an agenda, and the claim that "the CCCC will run on cutting-edge technology to generate the best possible chess" becomes laughable. Furthermore, even if you claim it's not true -- and even if it isn't true -- many observers will reasonably believe that the CPU engines are being intentionally handicapped because of the drama and attention that would be generated by Leela's dominance. Often times it's important to adequately manage perceptions, which I believe is being done poorly here. (And even more poorly than TCEC did with regards to their own "experimental" Season 13, which is saying something.)

Well, all the other engines won't be sharing resources while playing LC0, so she will face engines with 40-something "real" threads with no memory or disk contention. On the other hand, while the conventional engines will face weaker competition than Leela, they will also be weaker when doing so...

The upshot is that the games vs LC0 should be of higher overall quality.

There's no indication that the CPU engines will be given full access to the anemic CPU machine during games against Leela. The official documentation simply states the CPU engines will be limited to 46 threads and 8192 MB of hash without mention of any exceptions.

[jhellis3]

There's no indication that the CPU engines will be given full access to the anemic CPU machine during games against Leela.

Fortunately, that wasn't the point I was making ....