I realize that the transition from for a non-NNUE engine to becoming coded to support NNUE technology has a very
significant effect on that engine's playing strength - 100+ ELO perhaps as a rough guess?
However I'm wondering how much strength improvement is typical for a given NNUE-based engine as newer NNUE data files are created for that engine. Are we looking at 10-15 ELO on average per tweak to the NNUE files? More than that? Of course it will vary for each NNUE-based engine and even from one set of NNUE files to another (some may well bring larger strength increases than others), but I'm curious what this highly experienced group thinks about this.
Thanks for your thoughts!
Engine strength improvements by tweaking NNUE files?
Moderator: Ras
-
- Posts: 553
- Joined: Thu Mar 09, 2006 12:53 am
-
- Posts: 3537
- Joined: Wed Mar 08, 2006 8:15 pm
- Full name: Jouni Uski
Re: Engine strength improvements by tweaking NNUE files?
Typical example from SF:
Update default main net to nn-1ceb1ade0001.nnue
Update default main net to nn-1ceb1ade0001.nnue
Code: Select all
Passed STC:
https://tests.stockfishchess.org/tests/view/65e3ddd1f2ef6c733362ae5c
LLR: 2.92 (-2.94,2.94) <0.00,2.00>
Total: 149792 W: 39153 L: 38661 D: 71978 Elo +1.14
Passed LTC:
https://tests.stockfishchess.org/tests/view/65e4d91c416ecd92c162a69b
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 64416 W: 16517 L: 16135 D: 31764 Elo +2.06
Jouni
-
- Posts: 30
- Joined: Tue Mar 26, 2024 8:21 pm
- Full name: Lyndon S. Sears
Re: Engine strength improvements by tweaking NNUE files?
Example from Ethereal:
Code: Select all
Elo | 3.91 +- 3.29 (95%)
SPRT | 10.0+0.10s Threads=1 Hash=8MB
LLR | 2.95 (-2.94, 2.94) [0.00, 3.00]
Games | N: 20536 W: 5059 L: 4828 D: 10649
Penta | [103, 2399, 5070, 2556, 140]
http://chess.grantnet.us/test/36465/
Elo | 5.30 +- 4.04 (95%)
SPRT | 60.0+0.60s Threads=1 Hash=64MB
LLR | 3.00 (-2.94, 2.94) [0.00, 3.00]
Games | N: 13034 W: 3099 L: 2900 D: 7035
Penta | [10, 1439, 3428, 1622, 18]
http://chess.grantnet.us/test/36466/
-
- Posts: 553
- Joined: Thu Mar 09, 2006 12:53 am
Re: Engine strength improvements by tweaking NNUE files?
Thanks for the info folks! So the gain is typically small single digit increases in strength from one NNUE file to the next based on the replies.
So while NNUE itself resulted in a huge jump on strength, it's slower going (by a significant amount) as NNUE files are fine tuned.
Thanks again!
So while NNUE itself resulted in a huge jump on strength, it's slower going (by a significant amount) as NNUE files are fine tuned.
Thanks again!
-
- Posts: 30
- Joined: Tue Mar 26, 2024 8:21 pm
- Full name: Lyndon S. Sears
Re: Engine strength improvements by tweaking NNUE files?
I wouldn't say the single elo conclusion applies to all engines. For example from Alexandria:royb wrote: ↑Sun May 05, 2024 4:50 am Thanks for the info folks! So the gain is typically small single digit increases in strength from one NNUE file to the next based on the replies.
So while NNUE itself resulted in a huge jump on strength, it's slower going (by a significant amount) as NNUE files are fine tuned.
Thanks again!
Code: Select all
Elo | 10.94 +- 6.04 (95%)
SPRT | 8.0+0.08s Threads=1 Hash=16MB
LLR | 2.96 (-2.94, 2.94) [0.00, 3.00]
Games | N: 6512 W: 1771 L: 1566 D: 3175
Penta | [37, 717, 1573, 862, 67]
https://github.com/PGG106/Alexandria/commit/77657e382f04dd1022e35f303af2351bb53b0d98
Code: Select all
Elo | 9.79 +- 5.74 (95%)
SPRT | 8.0+0.08s Threads=1 Hash=16MB
LLR | 2.90 (-2.25, 2.89) [0.00, 3.00]
Games | N: 6994 W: 1842 L: 1645 D: 3507
Penta | [33, 764, 1727, 919, 54]
Code: Select all
ELO | 13.93 +- 7.72 (95%)
SPRT | 8.0+0.08s Threads=1 Hash=16MB
LLR | 2.95 (-2.94, 2.94) [0.00, 5.00]
GAMES | N: 3968 W: 1092 L: 933 D: 1943
https://github.com/Disservin/Smallbrain/commit/ea0dbaccf3ffc6a19a3d83255f6f4275077fac88
-
- Posts: 223
- Joined: Tue Apr 09, 2024 6:24 am
- Full name: Michael Chaly
Re: Engine strength improvements by tweaking NNUE files?
The stronger net is the weaker gains usuall are, obviously.
Also can be pretty different with net arch changes.
Also can be pretty different with net arch changes.
-
- Posts: 1936
- Joined: Tue Apr 19, 2016 6:08 am
- Location: U.S.A
- Full name: Andrew Grant
Re: Engine strength improvements by tweaking NNUE files?
Also Ethereal: https://github.com/AndyGrant/Ethereal/c ... 7680f11df2
Code: Select all
Elo | 42.67 +- 11.49 (95%)
SPRT | 10.0+0.10s Threads=1 Hash=8MB
LLR | 2.95 (-2.94, 2.94) [0.00, 3.00]
Games | N: 1604 W: 463 L: 267 D: 874
Penta | [4, 118, 375, 288, 17]
http://chess.grantnet.us/test/36386/
Elo | 35.63 +- 8.10 (95%)
Conf | 60.0+0.60s Threads=1 Hash=64MB
Games | N: 3356 W: 968 L: 625 D: 1763
Penta | [4, 247, 837, 582, 8]
http://chess.grantnet.us/test/36388/