Has Stockfish NNUE reached it's limit now?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Jouni
Posts: 2391
Joined: Wed Mar 08, 2006 7:15 pm

Has Stockfish NNUE reached it's limit now?

Post by Jouni » Wed Dec 09, 2020 1:52 pm

No progress in any list for some weeks. May be next step is a bigger net!?
Jouni

Madeleine Birchfield
Posts: 296
Joined: Tue Sep 29, 2020 2:29 pm
Location: Dublin, Ireland
Full name: Madeleine Birchfield

Re: Has Stockfish NNUE reached it's limit now?

Post by Madeleine Birchfield » Wed Dec 09, 2020 4:19 pm

Jouni wrote:
Wed Dec 09, 2020 1:52 pm
No progress in any list for some weeks. May be next step is a bigger net!?
Sergio Vieri was the only person on the Stockfish team who really knew how to train nets, and when he suddenly stopped training nobody else could really step in and replicate his efforts. So they've had to start from scratch.

Pi4Chess
Posts: 237
Joined: Mon Nov 16, 2020 11:13 am
Full name: Manuel Rivera

Re: Has Stockfish NNUE reached it's limit now?

Post by Pi4Chess » Thu Dec 10, 2020 12:42 am

It seems there is no elo progress against Stockfish 7 since 22 October dev release. But may be SF 12 is too far from Stockfish 7 to see any improvement ?

https://nextchessmove.com/dev-builds

RogerC
Posts: 35
Joined: Tue Oct 29, 2019 7:33 pm
Location: French Polynesia
Full name: Roger C.

Re: Has Stockfish NNUE reached it's limit now?

Post by RogerC » Thu Dec 10, 2020 1:43 am

To mesure a progression in engines ELO, first you have to play a lot of games as the differences between versions are near 1 ELO or less.

50000 games would be the minimum to have a good evaluation of progression (or regression). NCM chess dev-builds evaluations plays only 20000 games, and vs a very weak engine (SF7) so ELO is now biased by the big 78% of wins.

The best way to be sure that SF is progressing (or not) is the 60000 games vs SF12 at LTC that is played quite often. The last run was november 29 : SFdev was +30,61 ELO vs SF12 (https://github.com/glinscott/fishtest/w ... sion-Tests).

But sure, we can see that SF NNUE has stopped his rate of progression since November 01 (just +2 ELO in 1 month).

Pi4Chess
Posts: 237
Joined: Mon Nov 16, 2020 11:13 am
Full name: Manuel Rivera

Re: Has Stockfish NNUE reached it's limit now?

Post by Pi4Chess » Thu Dec 10, 2020 1:52 am

@RogerC Thx for the link and infos. There is some little plateau since late october. My bet is that evolution of NNUE is needed. May be changing the size/structure of the net ? Using multiple nets (by opening, by mid/late game) ? I really don't know but this is very interesting :-)

carldaman
Posts: 2081
Joined: Sat Jun 02, 2012 12:13 am

Re: Has Stockfish NNUE reached it's limit now?

Post by carldaman » Thu Dec 10, 2020 2:59 am

Or maybe go back to classical eval?

:twisted:

Pi4Chess
Posts: 237
Joined: Mon Nov 16, 2020 11:13 am
Full name: Manuel Rivera

Re: Has Stockfish NNUE reached it's limit now?

Post by Pi4Chess » Thu Dec 10, 2020 1:48 pm

Anyone knowing the process of stockfish development can answer this question please :

Are the dev versions published before a new net is found retro tested in fishtest with the New net ? Would it be relevant ?

Jouni
Posts: 2391
Joined: Wed Mar 08, 2006 7:15 pm

Re: Has Stockfish NNUE reached it's limit now?

Post by Jouni » Mon Dec 14, 2020 8:00 am

Finally 14.12. version shows nice gain again! But not from NNUE.
Jouni

Nay Lin Tun
Posts: 697
Joined: Mon Jan 16, 2012 5:34 am

Re: Has Stockfish NNUE reached it's limit now?

Post by Nay Lin Tun » Mon Dec 14, 2020 3:59 pm

Meanwhile SPRT 1000 nodes per move test showed Leela improved +40 elo from last season.

https://imgur.com/FzXNL4b

Jouni
Posts: 2391
Joined: Wed Mar 08, 2006 7:15 pm

Re: Has Stockfish NNUE reached it's limit now?

Post by Jouni » Mon Dec 14, 2020 4:12 pm

I am really sceptical about 1000 nodes test. And FGRL shows only regression :( .
Jouni

Post Reply