Update Speedy Rating List with the latest NNUE engines

[Rebel]

Update Speedy Rating List with the latest NNUE engines.

Or... the NNUE festival.

Code: Select all

1000ms
Stockfish 12        +174
Rubichess 1.8 NNUE  +223
Minic 2.48 NNUE     +238
Igel 2.70 NNUE      +199

4000ms
Stockfish 12        +151
Rubichess 1.8 NNUE  +224
Minic 2.48 NNUE     +260
Igel 2.70 NNUE      +167

[Rebel]

Forgot the link.

http://rebel13.nl/download/speedy-rating-list.html

[David Carteau]

Hi Ed,

I saw your post and decided to provide the result of my NNUE experiments with Orion. If you could add new version (v0.7) and its NNUE declinaison (v0.7.nnue) to your Speedy Rating List, it would be great !

Direct link to Orion v0.7.nnue : here.
To make it work, you'll have to download and put in the same directory this net : nn-82215d0fd0df.nnue.

Please note that the NNUE version should not be considered as the "official" version of Orion, as stated here, it is released only for entertainment purposes !

[Rebel]

Hi Ed,

I saw your post and decided to provide the result of my NNUE experiments with Orion. If you could add new version (v0.7) and its NNUE declinaison (v0.7.nnue) to your Speedy Rating List, it would be great !

Direct link to Orion v0.7.nnue : here.
To make it work, you'll have to download and put in the same directory this net : nn-82215d0fd0df.nnue.

Please note that the NNUE version should not be considered as the "official" version of Orion, as stated here, it is released only for entertainment purposes !

1000ms result, this is not normal.

Code: Select all

    Engine           Points  Used Time   Found   Pos    Elo   Score   Score    ms     Mb  Cpu  Errors
 1  Stockfish 12     341638  11:34:09.7  25383  40000  3416  400000  85.41%   1000   128    1     0
 2  Rubi 1.8 NNUE    328244  11:30:21.7  23651  40000  3282  400000  82.06%   1000   128    1     0
 3  Orion 0.7 NNUE   318886  11:43:55.1  22654  40000  3188  400000  79.72%   1000   128    1     0
 4  Komodo 14        317110  11:09:21.4  22398  40000  3171  400000  79.28%   1000   128    1     0
 5  Houdini 6.03     314596  11:27:54.8  22218  40000  3146  400000  78.65%   1000   128    1     0
 ....
60  Orion 0.6        268120  11:31:45.3  17895  40000  2681  400000  67.03%   1000   128    1    41

+507 (!!)

Running 4000ms now...

[Rebel]

4000ms

Code: Select all

    Engine           Points  Used Time   Found   Pos    Elo   Score   Score    ms     Mb  Cpu  Errors
 1  Stockfish 12     348460  44:54:23.9  26480  40000  3484  400000  87.12%   4000   128    1     0
 2  Rubi 1.8 NNUE    337312  44:48:11.8  24972  40000  3373  400000  84.33%   4000   128    1     0
 3  Orion 0.7 NNUE   329921  44:58:38.8  23892  40000  3299  400000  82.48%   4000   128    1     0
 4  Minic 2.48 NNUE  329645  42:13:25.3  23854  40000  3296  400000  82.41%   4000   128    1     0
 5  Komodo 14        326627  44:27:14.3  23558  40000  3266  400000  81.66%   4000   128    1     0
 6  Houdini 6.03     323896  44:47:34.1  23189  40000  3238  400000  80.97%   4000   128    1     0
 7  Ethereal 12.50   322740  44:54:28.2  23201  40000  3227  400000  80.69%   4000   128    1     0
 8  igel 2.70 NNUE   322472  44:52:01.2  23334  40000  3224  400000  80.62%   4000   128    1     0

NICE 1.2 has been generous to Orion, let's see if CUTE is also.

[David Carteau]

(...)
1000ms result, this is not normal.
(...)
+507 (!!)
(...)
4000ms
(...)
NICE 1.2 has been generous to Orion, let's see if CUTE is also.

Thanks a lot for testing ! The result seems impressive, in fact. But :

1) according to CCRL and CEGT lists, v0.7 is around +130 elo against v0.6 (I think this will more or less reflect in your own list when you will test "normal" v0.7). So the difference is lower, maybe around +370 elo (which remains considerable) ;

2) Orion's search is maybe a lot less agressive in reductions and prunings : perhaps (?) it can explain why the result is so good at "short" time controls.

Anyway, I'm happy that an independent test confirms my own results : Orion's current evaluation is really WEAK compared to state of the art

Thanks again, Ed, for the test !

[Rebel]

You are correct, Orion's search depth prevents a top rating, but the result is still fantastic, quick and dirty CUTE results, TC=40/20:

Code: Select all

Fruit 2.1 CCRL=2695
Orion 0.7 CCRL=2760
Score of Orion-0.7-NNUE vs Fruit-2.1: 158 - 17 - 27  [0.849] 202
Elo difference: 300.0 +/- 58.8, LOS: 100.0 %, DrawRatio: 13.4 %

Code: Select all

Senpai 1.0 CCRL=2961
Orion 0.7 CCRL=2760
Score of Orion-0.7-NNUE vs Senpai_1.0: 114 - 51 - 45  [0.650] 210
Elo difference: 107.5 +/- 43.3, LOS: 100.0 %, DrawRatio: 21.4 %

Code: Select all

Texel 1.07 CCRL=3075
Orion 0.7 CCRL=2760
Score of Orion-0.7-NNUE vs Texel_1.7: 431 - 318 - 251  [0.556] 1000
Elo difference: 39.4 +/- 18.7, LOS: 100.0 %, DrawRatio: 25.1 %

Code: Select all

Laser 1.7  CCRL=3207
Orion 0.7 CCRL=2760
Score of Orion-0.7-NNUE vs Laser_1.7: 210 - 537 - 253  [0.337] 1000
Elo difference: -117.9 +/- 19.3, LOS: 0.0 %, DrawRatio: 25.3 %

Seems that Laser 1.7 is one bridge too far but based on the result against Texel 1.07 Orion's rating is ~3100, which would be an elo gain of 340, a number you estimated yourself

Depth statistics

Code: Select all

Engine                 Depth       Time   Games     Moves  Average Forfeit  Book Depth     MIDG   EARLY    ENDG    LATE
Laser_1.7              17.95    8:54:33    1000     63719    0.50     0     7943  7.94    15.43 | 15.40 | 17.51 | 26.37
Orion-0.7-NNUE         12.35    7:58:38    1000     63794    0.45     0     8496  8.50    10.31 | 11.10 | 12.85 | 17.44

Also that Orion's time control can be improved, 0.45 is an indication that Orion plays too fast.

[David Carteau]

You are correct, Orion's search depth prevents a top rating, but the result is still fantastic, quick and dirty CUTE results, TC=40/20:
(...)
Also that Orion's time control can be improved, 0.45 is an indication that Orion plays too fast.

Wow, Orion at ~3100 elo, that's unbelievable

*** Stop dreaming ***

Thanks a lot for this detailed feedback !

Yes, time management has room for improvement... For the moment, it is really basic : I take the remaining time, divide it by the number of remaining moves, and at each iteration (iterative deepening), I check whether there is or not at least half of the resulting value. If not, search is stopped. Basic... like a lot of features in the engine

At least, it seems I managed to fix the '41 errors' encountered by the v0.6 in your Speedy Rating List : that's a good start !