First MEA rating list

[Rebel]

Announcement only.

With NICE you can easily and by fast computing create reasonable valid rating lists based on a large volume of EPD positions which are analyzed by Stockfish 17 at long time control with MultiPV=4 and the 4 best moves of each position are calculated a bonus, for the best move 10 points and for the remaining 3 moves 0-9 points based on the score difference with the best move.

In this way we selected 41.539 EPD positions and made a first rating list of it with MEA at 1000ms (one second) per position. With NICE we split the 41.539 EPD positions over 16 cores, running time only 43 minutes per engine. The whole first rating list (of 93 engines) was created in only 10 days. The first list of January 2025 contained engines released in 2024 with the exception of golden oldies like Fruit, Senpai, Shredder, Crafty and the very first versions of Komodo and Stockfish to further check the reliability of the system.

List posted in the Tournament and Matches section.

viewtopic.php?t=84764

[fsanders]

Thank you for your good work!
Would it be possible that you show here the bat file you used for NICE to do the splitting and than computing??

[Rebel]

Thank you for your good work!
Would it be possible that you show here the bat file you used for NICE to do the splitting and than computing??

Well, it's not very user friendly, but here goes -

First step, download the new 41.539 EPD set and store it in the EPD folder.



As an example, let's use the Rebel-16.3 engine (already installed in the ENGINES folder) and split the 41.539 positions over (say) 8 threads.

Create a batch file (*.bat) with the following content -

Code: Select all

del debug*.*

set MT=100
set HASH=128
set THREADS=1
set PROTOCOL=uci
set EPD=epd\mea-stripped.epd
set OPTIONS=MultiPV=1
set MRL=MEA-STRIPPED

set EXE=engines\Rebel-16.3.exe
set NAME=Rebel-16.3

mea.exe --engine %EXE% --name %NAME% --hash %HASH% --threads %THREADS% --protocol %PROTOCOL% --epd %EPD% --movetime %MT% --log --eoption %OPTIONS%
csv %MRL% %EPD% %OPTIONS% %NAME% %MT% %CCRL%
mrl %MRL% %MRL% %EPD% %MT%

exit

Then double click NICE.EXE and select Split Run and type 8 for the number of threads, or any other suitable number. NICE will split the 41.539 positions into 8 parts and as the screen states creates a new batch file @start.bat

Then double click @start.bat and the analysis will start and as defined in set MT=100 it will use 100ms per position.

When finished start NICE.EXE again and this time select Combine Results because that's what is needed, make one result file from the 8 threads, steps -

1. Scroll with the arrow down key to x1.csv and press enter.
2. Press "A" if there are no errors found, which is the case of 95% of the engines I tested.
3. Scroll with the arrow down key to MEA-STRIPPED.csv and press enter. It's where the final result will be added as defined by the set MRL=MEA-STRIPPED setting.

And your browser will show you the result.

That's it, a mouth full.

[fsanders]

Thank you very kind, on a first test with a smaller epd it worked like a charm !

[fsanders]

Another question I would like to ask is, on the REBEL homepage is written

With NICE you can easily and by fast computing create reasonable valid rating lists based on a large volume of EPD positions which are analyzed by Stockfish 17 at long time control with MultiPV=4 and the 4 best moves of each position are calculated a bonus, for the best move 10 points and for the remaining 3 moves 0-9 points based on the score difference with the best move.""

How long was the long time control for Stockfish 17 to get the basis for the calculation of the points?

[Rebel]

Another question I would like to ask is, on the REBEL homepage is written

With NICE you can easily and by fast computing create reasonable valid rating lists based on a large volume of EPD positions which are analyzed by Stockfish 17 at long time control with MultiPV=4 and the 4 best moves of each position are calculated a bonus, for the best move 10 points and for the remaining 3 moves 0-9 points based on the score difference with the best move.""

How long was the long time control for Stockfish 17 to get the basis for the calculation of the points?

10 seconds per move, MT=10000

Best time controls to use : from 100ms to 2500ms.

[fsanders]

Thank you for that info. I forgott to ask was the 10 sec calculation with 1 thread or 16, 20,.....threads?

[Rebel]

Thank you for that info. I forgott to ask was the 10 sec calculation with 1 thread or 16, 20,.....threads?

One thread.

[fsanders]

Thank you, no more questions

[peter]

Well, it's not very user friendly, but here goes -

Split Run and Combine Results works well as long as I don't try to add more eoptions.
Tried the way, Ferdy once in a while way back then told me as for MEA with
--eoption "..."
after
--movetime %MT% --log
in batch- file, and it doesn't work with adding more than the one at
set OPTIONS=MultiPV=1
separated with comma and space just after MultiPV=1,
".." doesn't help neither
together with
--eoption %OPTIONS%
after
--movetime %MT% --log

Could you give syntax to define more eoptions than the one of MultiPV only, please?
Thanks in advance,