Commercial Release of Ethereal 13.00 (NNUE) for AVX2 Systems

[Rebel]

If you see any elo gain at all, then the Network is working as intended. Not having a working Network would be - hundreds of elo, since the NNUE would be spitting out random evals essentially.

OK great news, thanks for confirming.
I am indeed running the Ethereal-13.00-pext-avx2 executable.

Then I'de expect your test to eventually end in the 50-90 range. But YMMV, and the opponent pool plays a role in whether it hits the upper end or the lower end. Ethereal tends to lose to Stockfish and its derivatives (NNUE wise, or Fire / Houdini) more so than it loses to the rest of the AB field. That can be seen on CCRLs elo diffs on individual breakdowns quite broadly over the last few releases. So a pool with stockfishes will deflate the rating. A pool with only Komodo, Xiphos, Laser, Igel (original net version), will inflate the rating.

:shrug:

Not something to shrug about. It's simply true. And not right. A pool of engines should be balanced. At the beginning of the GRL I had put Arasan 22.2 in a too strong engine pool (I had to start somewhere) and it got a too low rating because of that. The new 22.3 was put in a balanced pool and made a mighthy elo jump while Jon said -- This version is modestly stronger than 22.2 in my testing --

Another example, ProDeo 3.1, initially it was matched in a +/- 100 elo stronger pool at CCRL resulting in a lower rating than 3.0 (impossible), later it played its games against ~equal opponents and eventually came out stronger.

Regarding the use of derivatives, it's simply unfair competition. Not on my list.

Shall see. Not too concerned either way, as selfplay testing has proved itself reliable for years now

For sanity's sake, here is the regression test with the 8movesv3 book and LTC (60s here, but effectively 100s due to OpenBench worker speeds, scaled to Fishtest). http://chess.grantnet.us/test/11256/ [ELO | 76.31 +- 6.21 (95%)] This method has been the predictor for Ethereal at CCRL for the last few years, and is generally in the ball park. Where as most other lists see variations based on their book preferences.

[Modern Times]

Will be interesting to see where it sits under all the various test conditions. This is what I have at FRC:

http://ccrl.chessdom.com/ccrl/404FRC/ra ... _pure.html

Lower than I expected, but I've double-checked everything.

[Damir]

did you try and use FRC.nnue ( Fischerandom Chess) when testing Ethereal ?

[AndrewGrant]

Will be interesting to see where it sits under all the various test conditions. This is what I have at FRC:

http://ccrl.chessdom.com/ccrl/404FRC/ra ... _pure.html

Lower than I expected, but I've double-checked everything.

Oh! Have we been talking about FRC this whole time? If so, those results are good and what I was looking for. SF > K > E, thats the idea, and that has Ethereal at #3 edging out Houdini finally. FRC gains are smaller than Standard gains, due to the nature of Networks essentially learning opening theory. I managed to push more elo than usual, by going with the FRC only net, but rough figures were something like (At 10s+.01s, unbalanced book): +40 Standard NNUE in FRC, +70 Fischer NNUE in FRC, +120 Standard NNUE in Standard, +50 Fischer NNUE in Standard.

And just to confirm off what Damir said: This is the e13.fischer.nnue Network? Looks like it based off of the results to me.

[Modern Times]

And just to confirm off what Damir said: This is the e13.fischer.nnue Network? Looks like it based off of the results to me.

Correct, yes.

[Modern Times]

40/15 1CPU - these results have it at about +38 Elo to 12.75 and the #4 engine. It jumps several places on the list as it is quite tight below the Top 3. I need to play further opponents below Pedone 3.1. The rating could change slightly and of course the error bars will reduce with more games.

Code: Select all

Ethereal 13.00 64-bit   - SlowChess 2.54 64-bit       26.0 - 26.0    +7/=38/-7    50.00%
Ethereal 13.00 64-bit   - Houdini 6 64-bit            27.5 - 24.5    +6/=43/-3    52.88%
Ethereal 13.00 64-bit   - Pedone 3.1 64-bit           27.5 - 24.5    +6/=43/-3    52.88%
Ethereal 13.00 64-bit   - Stockfish 110121 64-bit     13.0 - 39.0    +0/=26/-26    25.00%
Ethereal 13.00 64-bit   - Fat Fritz 2 64-bit          13.5 - 40.5    +0/=27/-27    25.00%
Ethereal 13.00 64-bit   - Dragon By Komodo 64-bit     19.5 - 34.5    +0/=39/-15    36.11%
Ethereal 13.00 64-bit   - Igel 3.0.5 64-bit           30.0 - 24.0    +6/=48/-0    55.56%
Ethereal 13.00 64-bit   - Nemorino 6.00 64-bit        29.5 - 24.5    +7/=45/-2    54.63%
Ethereal 13.00 64-bit   - RubiChess 2.1 64-bit        27.0 - 25.0    +5/=44/-3    51.92%

Let's see how our standard chess blitz results come out, and also CEGT results.

(and yes this is using the standard net)

[AlexChess]

40/15 1CPU - these results have it at about +38 Elo to 12.75 and the #4 engine. It jumps several places on the list as it is quite tight below the Top 3. I need to play further opponents below Pedone 3.1. The rating could change slightly and of course the error bars will reduce with more games.

(and yes this is using the standard net)

There must be something wrong on NNUE... It is 232 Elo under Stockfish and only 36 Elo points better than 12.75 without the net...
And on CCRL stronger engines like Cfish, Corchess, Shashchess, SugaR, Honey, Bluefish, Oki Maguro, Harmon, Black Diamond are missing while Houdini is reported as 5th... Maybe with a custom NNUE would be better.





https://ccrl.chessdom.com/ccrl/404FRC/r ... _pure.html



Regards, AlexChess

[connor_mcmonigle]

40/15 1CPU - these results have it at about +38 Elo to 12.75 and the #4 engine. It jumps several places on the list as it is quite tight below the Top 3. I need to play further opponents below Pedone 3.1. The rating could change slightly and of course the error bars will reduce with more games.

(and yes this is using the standard net)

There must be something wrong on NNUE... It is 232 Elo under Stockfish and only 36 Elo points better than 12.75 without the net...
And on CCRL stronger engines like Cfish, Corchess, Shashchess, SugaR, Honey, Bluefish, Oki Maguro, Harmon, Black Diamond are missing while Houdini is reported as 5th... Maybe with a custom NNUE would be better.





https://ccrl.chessdom.com/ccrl/404FRC/r ... _pure.html



Regards, AlexChess

They aren't listed on CCRL because they're not unique engines. Rather, they're just a bunch of Stockfish derivatives which differ only slightly and will almost all play effectively identical to Stockfish with nearly identical evaluations.

[Modern Times]

There must be something wrong on NNUE... It is 232 Elo under Stockfish and only 36 Elo points better than 12.75 without the net...

I don't think so, Andrew said if the net wasn't being used then it would perform several hundred Elo worse. And he is only expecting +50 to +60 Elo anyway. So it is only slightly worse than he is expecting.

We will see what other test results come out at, like CEGT, Ed's tests and the CCRL standard chess blitz results.

[pohl4711]

I don't think so, Andrew said if the net wasn't being used then it would perform several hundred Elo worse.

That is correct. If I start the commercial Ethereal 13 avx2 without net access in console-mode and start it with "go infinite", it wants to play 1.b4 in the starting position and the whole pv-line is complete nonsense. So, if the net is not found, the Elo performance of Ethereal 13 avx2 would be a complete mess.
Only problem, I can imagine, is, to use the wrong (FRC) nnue-net for classical chess instead of the standard net.

Before my test-machine crashed (it is now for repair at XMG), more than 1000 of the 7000 games testrun of Ethereal 13 avx2 were played and I got a performance around +75 Elo compared to Ethereal 12.75.