Engine strength improvements by tweaking NNUE files?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

royb
Posts: 546
Joined: Thu Mar 09, 2006 12:53 am

Engine strength improvements by tweaking NNUE files?

Post by royb »

I realize that the transition from for a non-NNUE engine to becoming coded to support NNUE technology has a very
significant effect on that engine's playing strength - 100+ ELO perhaps as a rough guess?

However I'm wondering how much strength improvement is typical for a given NNUE-based engine as newer NNUE data files are created for that engine. Are we looking at 10-15 ELO on average per tweak to the NNUE files? More than that? Of course it will vary for each NNUE-based engine and even from one set of NNUE files to another (some may well bring larger strength increases than others), but I'm curious what this highly experienced group thinks about this.

Thanks for your thoughts!
Jouni
Posts: 3324
Joined: Wed Mar 08, 2006 8:15 pm

Re: Engine strength improvements by tweaking NNUE files?

Post by Jouni »

Typical example from SF:
Update default main net to nn-1ceb1ade0001.nnue

Code: Select all

Passed STC:
https://tests.stockfishchess.org/tests/view/65e3ddd1f2ef6c733362ae5c
LLR: 2.92 (-2.94,2.94) <0.00,2.00>
Total: 149792 W: 39153 L: 38661 D: 71978 Elo +1.14

Passed LTC:
https://tests.stockfishchess.org/tests/view/65e4d91c416ecd92c162a69b
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 64416 W: 16517 L: 16135 D: 31764 Elo +2.06
Jouni
User avatar
Antihelion
Posts: 30
Joined: Tue Mar 26, 2024 8:21 pm
Full name: Lyndon S. Sears

Re: Engine strength improvements by tweaking NNUE files?

Post by Antihelion »

Example from Ethereal:

Code: Select all

Elo   | 3.91 +- 3.29 (95%)
SPRT  | 10.0+0.10s Threads=1 Hash=8MB
LLR   | 2.95 (-2.94, 2.94) [0.00, 3.00]
Games | N: 20536 W: 5059 L: 4828 D: 10649
Penta | [103, 2399, 5070, 2556, 140]
http://chess.grantnet.us/test/36465/

Elo   | 5.30 +- 4.04 (95%)
SPRT  | 60.0+0.60s Threads=1 Hash=64MB
LLR   | 3.00 (-2.94, 2.94) [0.00, 3.00]
Games | N: 13034 W: 3099 L: 2900 D: 7035
Penta | [10, 1439, 3428, 1622, 18]
http://chess.grantnet.us/test/36466/
royb
Posts: 546
Joined: Thu Mar 09, 2006 12:53 am

Re: Engine strength improvements by tweaking NNUE files?

Post by royb »

Thanks for the info folks! So the gain is typically small single digit increases in strength from one NNUE file to the next based on the replies.

So while NNUE itself resulted in a huge jump on strength, it's slower going (by a significant amount) as NNUE files are fine tuned.

Thanks again!
User avatar
Antihelion
Posts: 30
Joined: Tue Mar 26, 2024 8:21 pm
Full name: Lyndon S. Sears

Re: Engine strength improvements by tweaking NNUE files?

Post by Antihelion »

royb wrote: Sun May 05, 2024 4:50 am Thanks for the info folks! So the gain is typically small single digit increases in strength from one NNUE file to the next based on the replies.

So while NNUE itself resulted in a huge jump on strength, it's slower going (by a significant amount) as NNUE files are fine tuned.

Thanks again!
I wouldn't say the single elo conclusion applies to all engines. For example from Alexandria:

Code: Select all

Elo | 10.94 +- 6.04 (95%)
SPRT | 8.0+0.08s Threads=1 Hash=16MB
LLR | 2.96 (-2.94, 2.94) [0.00, 3.00]
Games | N: 6512 W: 1771 L: 1566 D: 3175
Penta | [37, 717, 1573, 862, 67]
https://github.com/PGG106/Alexandria/commit/77657e382f04dd1022e35f303af2351bb53b0d98
From Obsidian:

Code: Select all

Elo   | 9.79 +- 5.74 (95%)
SPRT  | 8.0+0.08s Threads=1 Hash=16MB
LLR   | 2.90 (-2.25, 2.89) [0.00, 3.00]
Games | N: 6994 W: 1842 L: 1645 D: 3507
Penta | [33, 764, 1727, 919, 54]
Smallbrain:

Code: Select all

ELO   | 13.93 +- 7.72 (95%)
SPRT  | 8.0+0.08s Threads=1 Hash=16MB
LLR   | 2.95 (-2.94, 2.94) [0.00, 5.00]
GAMES | N: 3968 W: 1092 L: 933 D: 1943
https://github.com/Disservin/Smallbrain/commit/ea0dbaccf3ffc6a19a3d83255f6f4275077fac88
Viz
Posts: 109
Joined: Tue Apr 09, 2024 6:24 am
Full name: Michael Chaly

Re: Engine strength improvements by tweaking NNUE files?

Post by Viz »

The stronger net is the weaker gains usuall are, obviously.
Also can be pretty different with net arch changes.
AndrewGrant
Posts: 1819
Joined: Tue Apr 19, 2016 6:08 am
Location: U.S.A
Full name: Andrew Grant

Re: Engine strength improvements by tweaking NNUE files?

Post by AndrewGrant »

Also Ethereal: https://github.com/AndyGrant/Ethereal/c ... 7680f11df2

Code: Select all

Elo   | 42.67 +- 11.49 (95%)
SPRT  | 10.0+0.10s Threads=1 Hash=8MB
LLR   | 2.95 (-2.94, 2.94) [0.00, 3.00]
Games | N: 1604 W: 463 L: 267 D: 874
Penta | [4, 118, 375, 288, 17]
http://chess.grantnet.us/test/36386/

Elo   | 35.63 +- 8.10 (95%)
Conf  | 60.0+0.60s Threads=1 Hash=64MB
Games | N: 3356 W: 968 L: 625 D: 1763
Penta | [4, 247, 837, 582, 8]
http://chess.grantnet.us/test/36388/
Friendly reminder that stealing is a crime, is wrong, and makes you a thief.
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )