Engine strength improvements by tweaking NNUE files?

[royb]

I realize that the transition from for a non-NNUE engine to becoming coded to support NNUE technology has a very
significant effect on that engine's playing strength - 100+ ELO perhaps as a rough guess?

However I'm wondering how much strength improvement is typical for a given NNUE-based engine as newer NNUE data files are created for that engine. Are we looking at 10-15 ELO on average per tweak to the NNUE files? More than that? Of course it will vary for each NNUE-based engine and even from one set of NNUE files to another (some may well bring larger strength increases than others), but I'm curious what this highly experienced group thinks about this.

Thanks for your thoughts!

[Jouni]

Typical example from SF:
Update default main net to nn-1ceb1ade0001.nnue

Code: Select all

Passed STC:
https://tests.stockfishchess.org/tests/view/65e3ddd1f2ef6c733362ae5c
LLR: 2.92 (-2.94,2.94) <0.00,2.00>
Total: 149792 W: 39153 L: 38661 D: 71978 Elo +1.14

Passed LTC:
https://tests.stockfishchess.org/tests/view/65e4d91c416ecd92c162a69b
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 64416 W: 16517 L: 16135 D: 31764 Elo +2.06

[Antihelion]

Example from Ethereal:

Code: Select all

Elo   | 3.91 +- 3.29 (95%)
SPRT  | 10.0+0.10s Threads=1 Hash=8MB
LLR   | 2.95 (-2.94, 2.94) [0.00, 3.00]
Games | N: 20536 W: 5059 L: 4828 D: 10649
Penta | [103, 2399, 5070, 2556, 140]
http://chess.grantnet.us/test/36465/

Elo   | 5.30 +- 4.04 (95%)
SPRT  | 60.0+0.60s Threads=1 Hash=64MB
LLR   | 3.00 (-2.94, 2.94) [0.00, 3.00]
Games | N: 13034 W: 3099 L: 2900 D: 7035
Penta | [10, 1439, 3428, 1622, 18]
http://chess.grantnet.us/test/36466/

[royb]

Thanks for the info folks! So the gain is typically small single digit increases in strength from one NNUE file to the next based on the replies.

So while NNUE itself resulted in a huge jump on strength, it's slower going (by a significant amount) as NNUE files are fine tuned.

Thanks again!

[Antihelion]

Thanks for the info folks! So the gain is typically small single digit increases in strength from one NNUE file to the next based on the replies.

So while NNUE itself resulted in a huge jump on strength, it's slower going (by a significant amount) as NNUE files are fine tuned.

Thanks again!

I wouldn't say the single elo conclusion applies to all engines. For example from Alexandria:

Code: Select all

Elo | 10.94 +- 6.04 (95%)
SPRT | 8.0+0.08s Threads=1 Hash=16MB
LLR | 2.96 (-2.94, 2.94) [0.00, 3.00]
Games | N: 6512 W: 1771 L: 1566 D: 3175
Penta | [37, 717, 1573, 862, 67]
https://github.com/PGG106/Alexandria/commit/77657e382f04dd1022e35f303af2351bb53b0d98

From Obsidian:

Code: Select all

Elo   | 9.79 +- 5.74 (95%)
SPRT  | 8.0+0.08s Threads=1 Hash=16MB
LLR   | 2.90 (-2.25, 2.89) [0.00, 3.00]
Games | N: 6994 W: 1842 L: 1645 D: 3507
Penta | [33, 764, 1727, 919, 54]

Smallbrain:

Code: Select all

ELO   | 13.93 +- 7.72 (95%)
SPRT  | 8.0+0.08s Threads=1 Hash=16MB
LLR   | 2.95 (-2.94, 2.94) [0.00, 5.00]
GAMES | N: 3968 W: 1092 L: 933 D: 1943
https://github.com/Disservin/Smallbrain/commit/ea0dbaccf3ffc6a19a3d83255f6f4275077fac88

[Viz]

The stronger net is the weaker gains usuall are, obviously.
Also can be pretty different with net arch changes.

[AndrewGrant]

Also Ethereal: https://github.com/AndyGrant/Ethereal/c ... 7680f11df2

Code: Select all

Elo   | 42.67 +- 11.49 (95%)
SPRT  | 10.0+0.10s Threads=1 Hash=8MB
LLR   | 2.95 (-2.94, 2.94) [0.00, 3.00]
Games | N: 1604 W: 463 L: 267 D: 874
Penta | [4, 118, 375, 288, 17]
http://chess.grantnet.us/test/36386/

Elo   | 35.63 +- 8.10 (95%)
Conf  | 60.0+0.60s Threads=1 Hash=64MB
Games | N: 3356 W: 968 L: 625 D: 1763
Penta | [4, 247, 837, 582, 8]
http://chess.grantnet.us/test/36388/