Concurrent testing, some cores are more equal than others

[chrisw]

Bonjour, mostly to testers and programmers,

I'm getting a laptop next week with Intel Core Ultra 9, this comes with 24 cores, of which 8 are P and 16 E.

Core performance, and therefore nps one guesses, are not the same between P and E. How they differ is yet to be discovered here.

This will have implications (or will it?) for cutechess high concurrency engine-engine matches since we don't know which engine in any one pair will get a P core or an E core. This would seem to induce a random bias into the games scores and noisy results. True, false? How to get round it?

[Modern Times]

I wouldn't say that CPU is a good option for ratings list testing. Double the amount of E cores than P cores.

My earlier i7 12700 desktop has a different mix - 8 P cores and 4 E cores. On that, you don't lose a lot by disabling the E cores and running concurrency 8 on the remaining P cores.

[Aleks Peshkov]

I have only AMD and Linux, but I prefer to use `fastchess restart=on` option (per each engine config). More chances that lucky and unlucky performance startups compensate each other.

[chrisw]

I wouldn't say that CPU is a good option for ratings list testing. Double the amount of E cores than P cores.

My earlier i7 12700 desktop has a different mix - 8 P cores and 4 E cores. On that, you don't lose a lot by disabling the E cores and running concurrency 8 on the remaining P cores.

Yes, you have a different problem than I. You’re forced for rating list work to use movetime, so mixing P and E cores will introduce noise. My case is development, so I can measure progress and perform comparative testing using nodes as the limit, not time. Constant nodes each move. Then it matters not if one engine is running slower than the other via cores or other factors.

Actually, thinking more, I tell a lie here. I can make comparative tests between different versions of my own slightly changed engine against original itself as long as those changes haven’t affected NPS. Minor search tweaks would be ok, an entirely new and slower net would not.

[cpeters]

You should be able to enumerate/get the precise number for each core (and its attributes) from the system and make use of that value. Under Linux/*BSD - which you don't operate on - you would have to use

Code: Select all

taskset

to play with.

Windows might provide you with something similarily useful (perhaps wait a little; progress is underway:
https://github.com/microsoft/coreutils)

Maybe disable things/cores before boot (effzweitaste) in your personal computer.

[adnoh]

https://bitsum.com/apps/coredirector/

[towforce]

For several reasons, the best approach is probably:

1. run some code on startup that does a simple task using multiple threads

2. try it with varying numbers of cores

3. pick the number of cores that does the task quickest

4. get the task entirely cleared from memory

5. start the chess task

[cpeters]

For several reasons, the best approach is probably:
1. run some code on startup that does a simple task using multiple threads
2. try it with varying numbers of cores
3. pick the number of cores that does the task quickest
4. get the task entirely cleared from memory
5. start the chess task

Why the guessing? The cores get their attributes (P/E) not at runtime/dynamically - it's fixed. You may even disable them (either P/E) in UEFI. And they get a number (which will likely never change). So with the right tool, you can easily (under Linux and *BSD - the solution for windows should be googable) direct a task to a specific core.

[towforce]

For several reasons, the best approach is probably:
1. run some code on startup that does a simple task using multiple threads
2. try it with varying numbers of cores
3. pick the number of cores that does the task quickest
4. get the task entirely cleared from memory
5. start the chess task

Why the guessing? The cores get their attributes (P/E) not at runtime/dynamically - it's fixed. You may even disable them (either P/E) in UEFI. And they get a number (which will likely never change). So with the right tool, you can easily (under Linux and *BSD - the solution for windows should be googable) direct a task to a specific core.

"So with the right tool..." - if a tool can find out, why can't the engine? Can the engine pick its own cores, or must this decision be made before the engine starts? If so, could a "starter" program do the testing and then start the main engine with a good configuration before ending itself?

Is it fair to say that when the engine starts, it won't know what cores are available? Other processes might be using some cores, the device configuration may have changed, or the engine may be running on a different device.

Each task an engine has to do could be rated for suitability for either a P (performance) core or an E (efficiency) core, and if their aren't enough P cores for all tasks, relatively light tasks can be directed to available E cores.

My quoted post above suggesting trying different combinations of cores was an idea I had to actually find the answer out at runtime rather than setting a fixed combination which may not always be the best. Also, just using all the available cores won't always be the best way to go.

[cpeters]

"So with the right tool..." - if a tool can find out, why can't the engine? Can the engine pick its own cores, or must this decision be made before the engine starts? If so, could a "starter" program do the testing and then start the main engine with a good configuration before ending itself?

https://man7.org/linux/man-pages/man1/taskset.1.html

The rest of your post I'm too stupid/annoyed to process/evaluate.