Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: Harvey Williamson, bob, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Chessqueen
Posts: 1078
Joined: Wed Sep 05, 2018 12:16 am
Full name: Nancy M Pichardo

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by Chessqueen » Mon Sep 07, 2020 3:17 am

Laskos wrote:
Sat Sep 05, 2020 7:04 pm
Werewolf wrote:
Sat Sep 05, 2020 5:57 pm
Because the Neural Net in Lc0 is bigger than the one in SF NNUE, do we have any examples of where Lc0 evaluates a position better?
I'm thinking of positions where a deep AB search won't help much, such as blocked positions.

Openings. Positionally in the openings, it seems Lc0 > SF NNUE > SF AB > other AB.
Let see How Stockfish NNUE handle a 1 move Opening that are balanced like 1.h3 or 1.g3 which does NOT give an edge to White since the first move only occupy the 3rd rank. Let see what happens NEXT game? https://tcec-chess.com/live.html

https://www.chess.com/forum/view/chess- ... o-play-1h3

Please do NOT think that 1.h3 automatically give Black the edge ==>

This game was played between Smallfish = Stockfish for iphone and Komodo 13.3 using My Dell inspiron 22 All in one. I fed the first move 1.h3 the Clemenz Opening knowing that none of the two had the best variations programmed to their opening book, I just wanted to see how both engines and iphone Xs Max versus my Dell handle this not too well known opening.


Werewolf
Posts: 1314
Joined: Thu Sep 18, 2008 8:24 pm

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by Werewolf » Tue Sep 08, 2020 12:31 pm

mwyoung wrote:
Sat Sep 05, 2020 3:49 am


Lc0 is clearly improving faster then Stockfish at this point in time. Even at 3m+2s time controls vs past matches at the same time controls.

Is this definitely true? The T60 graph has been flat for ages. I know there are other improvements outside of the nets, but this is quite a big thing

Milos
Posts: 3923
Joined: Wed Nov 25, 2009 12:47 am

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by Milos » Tue Sep 08, 2020 1:10 pm

mwyoung wrote:
Sat Sep 05, 2020 3:49 am
Chessqueen wrote:
Sat Sep 05, 2020 3:21 am
Dann Corbit wrote:
Wed Sep 02, 2020 11:55 pm
Stockfish nnue has a secret weapon. The Kamehameha blast. Of course, he has to go to level 5 before he can use it. You don't just go Kamehameha blasting stuff willy-nilly.
At the very end it will be LCZero Vs Stockfish NNUE, but I predict a very close encounter of the 3rd kind, LCZero from Planet 1140b Vs StockFish NNUE from Planet Earth, Now I am more convinced than ever :roll:
https://tcec-chess.com/live.html
I agree. I just played 200 games with Stockfish 12 Vs Lc0 26.2. Stockfish 12 won by only 24 Elo in 200 games at 3m+2s. And in testing. We can see how badly Stockfish NNUE has scaled in past testing. At longer time controls.

Both are the best chess engines, and the winner may only be decided by hardware and time controls.

The sprinter Stockfish 12 vs. the marathon runner Lc0. Who wins the race. May depend on the distance of the race!


Lc0 is clearly improving faster then Stockfish at this point in time. Even at 3m+2s time controls vs past matches at the same time controls.

Code: Select all

Result:
--------------------------------------------------------------------------
  #  name          games    wins   draws  losses   score    los%  elo+/-
  1. Stockfish 12    200      16     182       2   107.0   100.0    24.4
  2. Lc0 v0.26.2     200       2     182      16    93.0     0.0   -24.4

Cross table:
--------------------------------------------------------------------------
  #  name             score   games                                                                                                                                                                                                        1                                                                                                                                                                                                        2
  1. Stockfish 12     107.0     200                                                                                                                                                                                                        x =====1==1===1====================1======1========11========================1==========================1============================================1====1===1=========1==0================1===1=1==0====
  2. Lc0 v0.26.2       93.0     200 =====0==0===0====================0======0========00========================0==========================0============================================0====0===0=========0==1================0===0=0==1====                                                                                                                                                                                                        x

Tech:
--------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name            nodes/m         NPS  depth/m   time/m    moves     time
  1. Stockfish 12    125173K    26565996     42.5      4.7     54.1    255.1
  2. Lc0 v0.26.2        101K       20342     10.0      4.9     54.1    267.2
     all ---          61216K    12984844     26.3      4.8     54.1    261.2
For years ppl come up with the BS theory that A/B engines tuned in micro-bullet would be weak in LTC and for years they are so bluntly proven wrong. Impact of eval on horizon effects is minimal and it doesn't change whether you search to depth 20 or depth 100. SF-NN search is SF and SF is proven to scale better than Lc0 (and as a matter of fact any MCTS engine) in LTC. Ergo SF-NN scales better than Lc0 in LTC.
Your claims are simply BS reflecting your cluelessness in the matter. You effectively draw conclusions from STC (just because it's not micro-bullet but blitz instead) with a sample size that is a joke.
The result in the superfinal will be much worse sweep than last year. And then ppl like you would be astonished and would come up with all kind of ridiculous excuses to justify what is basically their cluelessness.

mwyoung
Posts: 2433
Joined: Wed May 12, 2010 8:00 pm

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by mwyoung » Tue Sep 08, 2020 3:29 pm

Milos wrote:
Tue Sep 08, 2020 1:10 pm
mwyoung wrote:
Sat Sep 05, 2020 3:49 am
Chessqueen wrote:
Sat Sep 05, 2020 3:21 am
Dann Corbit wrote:
Wed Sep 02, 2020 11:55 pm
Stockfish nnue has a secret weapon. The Kamehameha blast. Of course, he has to go to level 5 before he can use it. You don't just go Kamehameha blasting stuff willy-nilly.
At the very end it will be LCZero Vs Stockfish NNUE, but I predict a very close encounter of the 3rd kind, LCZero from Planet 1140b Vs StockFish NNUE from Planet Earth, Now I am more convinced than ever :roll:
https://tcec-chess.com/live.html
I agree. I just played 200 games with Stockfish 12 Vs Lc0 26.2. Stockfish 12 won by only 24 Elo in 200 games at 3m+2s. And in testing. We can see how badly Stockfish NNUE has scaled in past testing. At longer time controls.

Both are the best chess engines, and the winner may only be decided by hardware and time controls.

The sprinter Stockfish 12 vs. the marathon runner Lc0. Who wins the race. May depend on the distance of the race!


Lc0 is clearly improving faster then Stockfish at this point in time. Even at 3m+2s time controls vs past matches at the same time controls.

Code: Select all

Result:
--------------------------------------------------------------------------
  #  name          games    wins   draws  losses   score    los%  elo+/-
  1. Stockfish 12    200      16     182       2   107.0   100.0    24.4
  2. Lc0 v0.26.2     200       2     182      16    93.0     0.0   -24.4

Cross table:
--------------------------------------------------------------------------
  #  name             score   games                                                                                                                                                                                                        1                                                                                                                                                                                                        2
  1. Stockfish 12     107.0     200                                                                                                                                                                                                        x =====1==1===1====================1======1========11========================1==========================1============================================1====1===1=========1==0================1===1=1==0====
  2. Lc0 v0.26.2       93.0     200 =====0==0===0====================0======0========00========================0==========================0============================================0====0===0=========0==1================0===0=0==1====                                                                                                                                                                                                        x

Tech:
--------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name            nodes/m         NPS  depth/m   time/m    moves     time
  1. Stockfish 12    125173K    26565996     42.5      4.7     54.1    255.1
  2. Lc0 v0.26.2        101K       20342     10.0      4.9     54.1    267.2
     all ---          61216K    12984844     26.3      4.8     54.1    261.2
For years ppl come up with the BS theory that A/B engines tuned in micro-bullet would be weak in LTC and for years they are so bluntly proven wrong. Impact of eval on horizon effects is minimal and it doesn't change whether you search to depth 20 or depth 100. SF-NN search is SF and SF is proven to scale better than Lc0 (and as a matter of fact any MCTS engine) in LTC. Ergo SF-NN scales better than Lc0 in LTC.
Your claims are simply BS reflecting your cluelessness in the matter. You effectively draw conclusions from STC (just because it's not micro-bullet but blitz instead) with a sample size that is a joke.
The result in the superfinal will be much worse sweep than last year. And then ppl like you would be astonished and would come up with all kind of ridiculous excuses to justify what is basically their cluelessness.
The only one that is clueless here is you. As I test at the longer time controls, as well as short time controls. Along with 1 core testing, and up to 32 threads.

And I am not talking about A/B engine only testing at micro-bullet. And I never have. I am talking about NNUE! And my sample size is huge. This is not my only test. I test non stop.

My conclusion is what the data is showing us, and if it changes all will see that also. I test openly, and to video.


"SF-NN search is SF and SF is proven to scale better than Lc0" :lol:
Professing themselves to be wise, they became fools,
take on me. Foes 0.

mwyoung
Posts: 2433
Joined: Wed May 12, 2010 8:00 pm

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by mwyoung » Tue Sep 08, 2020 4:21 pm

mwyoung wrote:
Tue Sep 08, 2020 3:29 pm
Milos wrote:
Tue Sep 08, 2020 1:10 pm
mwyoung wrote:
Sat Sep 05, 2020 3:49 am
Chessqueen wrote:
Sat Sep 05, 2020 3:21 am
Dann Corbit wrote:
Wed Sep 02, 2020 11:55 pm
Stockfish nnue has a secret weapon. The Kamehameha blast. Of course, he has to go to level 5 before he can use it. You don't just go Kamehameha blasting stuff willy-nilly.
At the very end it will be LCZero Vs Stockfish NNUE, but I predict a very close encounter of the 3rd kind, LCZero from Planet 1140b Vs StockFish NNUE from Planet Earth, Now I am more convinced than ever :roll:
https://tcec-chess.com/live.html
I agree. I just played 200 games with Stockfish 12 Vs Lc0 26.2. Stockfish 12 won by only 24 Elo in 200 games at 3m+2s. And in testing. We can see how badly Stockfish NNUE has scaled in past testing. At longer time controls.

Both are the best chess engines, and the winner may only be decided by hardware and time controls.



The sprinter Stockfish 12 vs. the marathon runner Lc0. Who wins the race. May depend on the distance of the race!


Lc0 is clearly improving faster then Stockfish at this point in time. Even at 3m+2s time controls vs past matches at the same time controls.

Code: Select all

Result:
--------------------------------------------------------------------------
  #  name          games    wins   draws  losses   score    los%  elo+/-
  1. Stockfish 12    200      16     182       2   107.0   100.0    24.4
  2. Lc0 v0.26.2     200       2     182      16    93.0     0.0   -24.4

Cross table:
--------------------------------------------------------------------------
  #  name             score   games                                                                                                                                                                                                        1                                                                                                                                                                                                        2
  1. Stockfish 12     107.0     200                                                                                                                                                                                                        x =====1==1===1====================1======1========11========================1==========================1============================================1====1===1=========1==0================1===1=1==0====
  2. Lc0 v0.26.2       93.0     200 =====0==0===0====================0======0========00========================0==========================0============================================0====0===0=========0==1================0===0=0==1====                                                                                                                                                                                                        x

Tech:
--------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name            nodes/m         NPS  depth/m   time/m    moves     time
  1. Stockfish 12    125173K    26565996     42.5      4.7     54.1    255.1
  2. Lc0 v0.26.2        101K       20342     10.0      4.9     54.1    267.2
     all ---          61216K    12984844     26.3      4.8     54.1    261.2
For years ppl come up with the BS theory that A/B engines tuned in micro-bullet would be weak in LTC and for years they are so bluntly proven wrong. Impact of eval on horizon effects is minimal and it doesn't change whether you search to depth 20 or depth 100. SF-NN search is SF and SF is proven to scale better than Lc0 (and as a matter of fact any MCTS engine) in LTC. Ergo SF-NN scales better than Lc0 in LTC.
Your claims are simply BS reflecting your cluelessness in the matter. You effectively draw conclusions from STC (just because it's not micro-bullet but blitz instead) with a sample size that is a joke.
The result in the superfinal will be much worse sweep than last year. And then ppl like you would be astonished and would come up with all kind of ridiculous excuses to justify what is basically their cluelessness.
The only one that is clueless here is you. As I test at the longer time controls, as well as short time controls. Along with 1 core testing, and up to 32 threads.

And I am not talking about A/B engine only testing at micro-bullet. And I never have. I am talking about NNUE! And my sample size is huge. This is not my only test. I test non stop.

My conclusion is what the data is showing us, and if it changes all will see that also. I test openly, and to video.


"SF-NN search is SF and SF is proven to scale better than Lc0" :lol:
"The result in the superfinal will be much worse sweep than last year. And then ppl like you would be astonished and would come up with all kind of ridiculous excuses to justify what is basically their cluelessness" :lol:

For reference here are the results of last season's superfinal...

TCEC Season 184 May 2020 – Jul 2020 Stockfish 202006170741 LCZero v0.25.1-svjio-t60-3972-mlh + 23 = 61 - 16

Code: Select all

Wins	Losses	Draws
23
16
61
Points		Games
53.5
/	
100
Winning percentage
53.5

Elo difference : 
+24
Professing themselves to be wise, they became fools,
take on me. Foes 0.

Jouni
Posts: 2227
Joined: Wed Mar 08, 2006 7:15 pm

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by Jouni » Wed Sep 09, 2020 11:15 am

In CCC current score is same for Classic and NNUE :o !
Jouni

Cornfed
Posts: 99
Joined: Sun Apr 26, 2020 9:40 pm
Full name: Brian D. Smith

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by Cornfed » Wed Sep 09, 2020 6:13 pm

Quick TCEC question as you guys would know:
Are all 8 remaining engines NN's with the exception of Komodo? Or is Komodo getting lapped by A/B engines as well?

Leo
Posts: 969
Joined: Fri Sep 16, 2016 4:55 pm
Location: USA/Minnesota
Full name: Leo Anger

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by Leo » Wed Sep 09, 2020 6:22 pm

In my own testing SF NNUE is doing great.
Advanced Micro Devices fan.

mwyoung
Posts: 2433
Joined: Wed May 12, 2010 8:00 pm

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by mwyoung » Wed Sep 09, 2020 9:41 pm

Leo wrote:
Wed Sep 09, 2020 6:22 pm
In my own testing SF NNUE is doing great.
SF NNUE is the best program under most conditions. You are just not going to see crazy Elo performance with more threads, and/or a longer time control. Like you do when testing at bullet or fast blitz. On my system for example it crushes Lc0 under those conditions, and all other engines. But at longer time controls, it can perform under 20 Elo or lower at 15m+15s. For a huge drop in performance.

Some feel maybe this is due to reaching the end of gains for chess engine development. As the other best engines can reach this level with more time, and faster computers. Time will tell in short order.
Professing themselves to be wise, they became fools,
take on me. Foes 0.

rcmaddox
Posts: 180
Joined: Mon Oct 14, 2013 6:59 pm
Location: Winder, GA
Full name: Robert C. Maddox

Re: Are we sure that Stockfish NNUE is better than the Normal Stockfish ?

Post by rcmaddox » Wed Sep 09, 2020 10:20 pm

Dann Corbit wrote:
Wed Sep 02, 2020 11:55 pm
Stockfish nnue has a secret weapon. The Kamehameha blast. Of course, he has to go to level 5 before he can use it. You don't just go Kamehameha blasting stuff willy-nilly.
This discussion has left me in the dust.

Post Reply