how to measure elo from a hardwarecoupled system

[flok]

Hello,

I've got an ESP microcontroller running a chess program. I would like to measure the elo-rating of the combination ESP+program.
Now measuring that of the software at itself would be easy: just put it on the laptop together with a program with a known rating.
But in this case I would like to know it for the specific ESP32 + software combination. As the cpu is something special (xtensa with 32 kB cache for example).
How can I do so? I tried compiling tscp on the esp but that's problematic (fastchess (tournamentmanager) doesn't do xboard).

[Ras]

The easiest way: Measure your kNPS on laptop and ESP32, then benchmark your engine on laptop with accordingly asymmetric thinking time.

The most accurate way: use a wrapper console program on the laptop that acts as if it were an engine, but actually relays the GUI commands to the ESP32 over some communication interface such as UART or Wifi.

[flok]

The easiest way: Measure your kNPS on laptop and ESP32, then benchmark your engine on laptop with accordingly asymmetric thinking time.

But they're totally different architectures. Also: then I know how much faster my laptop is than the esp, but I don't know to what elo difference that amounts to. Because the CCRL ratings are on a different computer than mine. Or am I making a thinking error in my reasoning?

The most accurate way: use a wrapper console program on the laptop that acts as if it were an engine, but actually relays the GUI commands to the ESP32 over some communication interface such as UART or Wifi.

that would require a known reference system. Hmmm, I saw that stockfish 10 does this on my laptop (I checked the CCRL):

Code: Select all

Total time (ms) : 1687
Nodes searched  : 3854608
Nodes/second    : 2284889

so if I'm right, I now should know how fast sf10 is on the ccrl systems, right?

[Ras]

But they're totally different architectures.

By and large, that doesn't matter much. Only how fast it is, in the end. Sure, the hashtable size would make some difference, especially in the endgame.

Also: then I know how much faster my laptop is than the esp, but I don't know to what elo difference that amounts to. Because the CCRL ratings are on a different computer than mine. Or am I making a thinking error in my reasoning?

You know the Elo on CCRL. And Elo is measuring relative winning scores. So, if, on your laptop, your engine with appropriately reduced thinking times scores about 50% against a selection of engines where you know the CCRL Elo, then the average of that Elo is what your engine would be on CCRL with that reduced thinking time. Same of course if you use the wrapper interface and let your engine play on the controller.