NICE 1.1

[Rebel]

NICE 1.1 is a considerable update over version 1.0 including better documentation, hence a new thread.

Some snippets.....

• While 1.0 had it's focus on fast time controls between 100-500ms for evaluation tuning mainly we added 2 formula for longer time controls (1000ms and 2000ms) in order to tune search settings more reliable.

• Speeding up the MEA analysis process by dividing the work over all the available cores (threads) on your PC in a more convenient way than in version 1.0. Our tests ran 2½ times faster, if we had used the full capacity of the PC it would had been 5 times faster. Maximum cores (threads) is 64.

• Improved formula for calculation the points.

• Time Bonus - use csvx.exe to exclude the (much discussed) Time Bonus from the calculation formula.

• MEA produces an excellent EPD with scores, depth of each position in the epd_out folder.

• Our experience (so far) has been we had more success with the sfx EPD's than with the lcx EPD's.

• The disadvantage of the MEA fixed movetime [MT] is that it abrupts aborts the search when the movetime limit is reached. An alternative is including depth, for example.......

• For easy sharing results a text file is produced, sf5.txt in this case.

.....

http://rebel13.nl/misc/nice.html#one

[Rebel]

I tried if NICE is able to measure the progress of Stockfish by month.

Stockfish 11 (as far as I can tell) was released half of januari.

I downloaded 5 development versions from the abrok.eu site and put them into NICE using Lcx.epd (81.xxx positions), time control 250ms. The result:

Code: Select all

    EPD  : epd\lcx.epd
    Time : 250ms
                                                        Solving       Max    Total   Time   Hash          
    Engine            Score    Used Time  Found   Pos     Time       Score    Rate    ms     Mb  Cpu  CCRL
 1  2020-05-09       1772683  06:40:22.2  45781  83542  00:36:21.7  2506260  70.7%    250   128    1  3200
 2  2020-04-05       1764524  06:39:39.1  45599  83542  00:35:55.1  2506260  70.4%    250   128    1  3200
 3  2020-03-07       1760136  06:40:24.0  45462  83542  00:35:40.2  2506260  70.2%    250   128    1  3200
 4  2020-01-27       1759177  06:41:28.7  45585  83542  00:35:50.4  2506260  70.2%    250   128    1  3200
 5  2020-01-07       1754829  06:41:57.5  45363  83542  00:36:06.7  2506260  70.0%    250   128    1  3200

Or in html - http://rebel13.nl/mea/sf-lcx-by-month.html

[Rebel]

Forgot to add that NICE 1.1 predicts the Stockfish May 9 version is 0.7% * 40 = 28 elo points stronger than the released version 11. More on this (riddle) in the next post.

[Rebel]

Code: Select all

    EPD  : epd\lcx.epd
    Time : 250ms
                                                        Solving       Max    Total   Time   Hash          
    Engine            Score    Used Time  Found   Pos     Time       Score    Rate    ms     Mb  Cpu  CCRL
 1  2020-05-09       1772683  06:40:22.2  45781  83542  00:36:21.7  2506260  70.7%    250   128    1  3200
 2  2020-04-05       1764524  06:39:39.1  45599  83542  00:35:55.1  2506260  70.4%    250   128    1  3200
 3  2020-03-07       1760136  06:40:24.0  45462  83542  00:35:40.2  2506260  70.2%    250   128    1  3200
 4  2020-01-27       1759177  06:41:28.7  45585  83542  00:35:50.4  2506260  70.2%    250   128    1  3200
 5  2020-01-07       1754829  06:41:57.5  45363  83542  00:36:06.7  2506260  70.0%    250   128    1  3200

In NICE 1.1 one percent means 40 elo. And so 70.7% - 70.0% = 0.7% * 40 is an elo increase of 28 elo points. It also means that when comparing versions 0.1% is 4 elo.

In the next version this is further refined. First by displaying the score with 2 decimals and increasing one percent to 45 elo. I tested 5 versions (at 250ms) if this assumption of one percent = 45 elo reasonable matches the CCRL 40/2 results. The idea is that with 2 decimals NICE can report an elo gain or regression with a precision of 1 elo.

Measuring the elo progress of the last version and the forelast version -

Arasan version 22 gains 29 elo points on CCRL 40/2, NICE reports 11 elo.

Code: Select all

    EPD  : epd\sfx.epd
    Time : 250ms
                                                              Max            Time  Hash          
    Engine           Points  Used Time   Found   Pos    Elo   Score   Score   ms    Mb  Cpu  CCRL
 1  Arasan 22        575681  06:51:03.4  38485  84621  3061  846210  68.03%  250   128    1  3179
 2  Arasan 21.4      573555  06:57:43.7  38012  84621  3050  846210  67.78%  250   128    1  3148

Defendchess version 2.2 gains 47 elo points on CCRL 40/2, NICE reports 23 elo.

Code: Select all

                                                               Max            Time   Hash          
    Engine           Points  Used Time   Found   Pos    Elo   Score   Score    ms     Mb  Cpu  CCRL
 1  Defenchess 2.2   611008  06:13:58.0  41315  84621  3249  846210  72.21%    250   128    1  3270
 2  Defenchess 2.1   606812  06:15:45.9  40948  84621  3226  846210  71.71%    250   128    1  3223

Ethereal version 12 gains 28 elo points on CCRL 40/2, NICE reports 24 elo.

Code: Select all

                                                               Max            Time   Hash          
    Engine           Points  Used Time   Found   Pos    Elo   Score   Score    ms     Mb  Cpu  CCRL
 1  Ethereal 12      618514  07:06:36.2  41917  84621  3289  846210  73.09%    250   128    1  3375
 2  Ethereal 11.75   614059  07:03:36.6  41708  84621  3265  846210  72.57%    250   128    1  3347

Komodo the free version 11 from the Komodo site gains 76 elo points on CCRL 40/2, NICE reports 33 elo.

Code: Select all

                                                               Max            Time   Hash          
    Engine           Points  Used Time   Found   Pos    Elo   Score   Score    ms     Mb  Cpu  CCRL
 1  Komodo 11        627082  06:14:13.1  42739  84621  3334  846210  74.10%    250   128    1  3413
 2  Komodo 10        620798  06:14:18.2  42316  84621  3301  846210  73.36%    250   128    1  3337

Laser version 1.7 gains 86 elo points on CCRL 40/2, NICE reports 81 elo.

Code: Select all

                                                               Max            Time   Hash          
    Engine           Points  Used Time   Found   Pos    Elo   Score   Score    ms     Mb  Cpu  CCRL
 1  Laser 1.7        609810  06:50:28.8  41174  84621  3242  846210  72.06%    250   128    1  3274
 2  Laser 1.6        594462  07:06:09.9  39922  84621  3161  846210  70.25%    250   128    1  3188

Conclusions
1. An abrupt time control 250ms isn't comparable with a) a regular time control and b) and the CCRL 3000ms due to scaling.
2. Running the 84,621 positions at 250ms at 20 cores takes about 20 minutes. The idea to invest 20 minutes before you put an engine change into cute-chess looks attractive to me when NICE (hopefully) can report an elo gain or regression with a precision of 1 elo.

[jpqy]

Thanks for this nice tool..

About:
• Speeding up the MEA analysis process by dividing the work over all the available cores (threads) on your PC in a more convenient way than in version 1.0. Our tests ran 2½ times faster, if we had used the full capacity of the PC it would had been 5 times faster. Maximum cores (threads) is 64.

Threads are used for engine,how many cores/threads it will use..
But dividing the work over all availabe cores ,how to set this? ,i don't see a concurrency setting for example.

Can you help me out here what i'm missing.

Thanks in advance,
JP.

[Rebel]

Thanks for this nice tool..

About:
• Speeding up the MEA analysis process by dividing the work over all the available cores (threads) on your PC in a more convenient way than in version 1.0. Our tests ran 2½ times faster, if we had used the full capacity of the PC it would had been 5 times faster. Maximum cores (threads) is 64.

Threads are used for engine,how many cores/threads it will use..
But dividing the work over all availabe cores ,how to set this? ,i don't see a concurrency setting for example.

Can you help me out here what i'm missing.

Thanks in advance,
JP.

Hi JP,

1. Start NICE and choose Split Run and select sf5-default.bat from the menu.
2. Then you will be prompted how many threads you want to use, type for instance 8.
3. NICE will split the q1.epd of 40,000 positions into 8 parts of 5,000 and create the needed 8 batch files.
4. The program will exit with the message to double click the created @start.bat and the MEA analysis will begin.

The more threads you type the faster the 40,000 positions are ready.

Hope this helps.

[jpqy]

Thanks Ed ,running it with 18cores ,working great!

-Then i did Combine results
-After Quit i get my html table ,40.000positions

Into sf5-default.bat i changed engine exe & name to Stockfish 11 (with putting SF11.exe into engines folder)
save it before i click on @start.bat
When all finished i see in html table still as engine SF5-default?

JP.

[Rebel]

Thanks Ed ,running it with 18cores ,working great!

-Then i did Combine results
-After Quit i get my html table ,40.000positions

Into sf5-default.bat i changed engine exe & name to Stockfish 11 (with putting SF11.exe into engines folder)
save it

So far so good. And now you will have to create new batch files. So NICE -> Split Run, then you have the new @start.bat

BTW, new version tomorrow.

before i click on @start.bat
When all finished i see in html table still as engine SF5-default?

JP.

[Dann Corbit]

For these two rows, the e.p. flag is spuriouis:

Code: Select all

#Endgame.epd (    337): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; c0 "Bf4+=10, Re2=9, b6=3, Re4=0";
   lcx-8.epd (   9429): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; bm Re2; c0 "Re2=10, b6=5, Re4=2, Bf4+=0";
     lcx.epd (   1489): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; bm Re2; c0 "Re2=10, b6=5, Re4=2, Bf4+=0";
      Q1.epd (  31148): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; c0 "Bf4+=10, Re2=9, b6=3, Re4=0";
   sfx-4.epd (   1150): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; c0 "Bf4+=10, Re2=9, b6=3, Re4=0";
     sfx.epd (   1538): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; c0 "Bf4+=10, Re2=9, b6=3, Re4=0";
   lcx-7.epd (   6793): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; bm Rd2; c0 "h5=10, Rd2=8, Re7=0, Re8=0";
     lcx.epd (  27267): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; bm Rd2; c0 "h5=10, Rd2=8, Re7=0, Re8=0";
      Q2.epd (  39964): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; c0 "h5=10, Rd2=9, Kh8=0, Kg8=0";
   sfx-8.epd (   9968): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; c0 "h5=10, Rd2=9, Kh8=0, Kg8=0";
     sfx.epd (  27621): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; c0 "h5=10, Rd2=9, Kh8=0, Kg8=0";

[Rebel]

For these two rows, the e.p. flag is spuriouis:

Code: Select all

#Endgame.epd (    337): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; c0 "Bf4+=10, Re2=9, b6=3, Re4=0";
   lcx-8.epd (   9429): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; bm Re2; c0 "Re2=10, b6=5, Re4=2, Bf4+=0";
     lcx.epd (   1489): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; bm Re2; c0 "Re2=10, b6=5, Re4=2, Bf4+=0";
      Q1.epd (  31148): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; c0 "Bf4+=10, Re2=9, b6=3, Re4=0";
   sfx-4.epd (   1150): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; c0 "Bf4+=10, Re2=9, b6=3, Re4=0";
     sfx.epd (   1538): 1n2r3/ppb2kp1/5p1p/2p2N1P/6P1/2B5/PPP2P2/2KR4 b - g3 bm Re2; c0 "Bf4+=10, Re2=9, b6=3, Re4=0";
   lcx-7.epd (   6793): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; bm Rd2; c0 "h5=10, Rd2=8, Re7=0, Re8=0";
     lcx.epd (  27267): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; bm Rd2; c0 "h5=10, Rd2=8, Re7=0, Re8=0";
      Q2.epd (  39964): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; c0 "h5=10, Rd2=9, Kh8=0, Kg8=0";
   sfx-8.epd (   9968): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; c0 "h5=10, Rd2=9, Kh8=0, Kg8=0";
     sfx.epd (  27621): 3r4/3r2bk/1p4pp/pN2n3/P6P/4Q1P1/1P2B1K1/8 b - h3 bm Rd2; c0 "h5=10, Rd2=9, Kh8=0, Kg8=0";

I don't understand, in the first position white's last move is 1.g4 and in the second position white's last move has been 1.h4

What am I missing?