Finally something closer to my results. I don't quite understand recent +40 - +60 Elo results at fast TC, either the net is old or compile is old or the openings are flattening the difference.lkaufman wrote: ↑Tue Jul 28, 2020 10:49 pmWith a later net (0109), 312.5 out of 500 (57% draws) for same test, +89 elo.lkaufman wrote: ↑Tue Jul 28, 2020 7:25 pmTest finished with 308.5 out of 500 games (56% draws), which is +83 elo! Quite remarkable given the not too short tc and fast (5 GHz) comp.lkaufman wrote: ↑Tue Jul 28, 2020 5:18 pmWith the latest net (1817) vs. latest SF (july 17) at same 0.5' + 0.5", one thread, Hert lowdraw book, I got 60.5 out of 100, +74 elo, so far.kranium wrote: ↑Tue Jul 28, 2020 2:10 pm Just to clarify any confusion in this thread...
We're seeing different Elo results presented
Those are of course the results of 'nnue' vs sf-dev test, and are very dependent on exactly which nn.bin is being tested
In the case of Larry and Mark, this detail is not indicated
This has much less to do with the quality of the compile (except for Ed's test of course), in which it's perfectly clear what's being tested.
My recommendation for a simple method of testing which compile is fastest on a particular system is:
1. make sure both binaries being compared load the same NNUE eval file (normally nn.bin)
2. type 'ucinewgame' and verify the nn.bin is found and loaded
(not needed for my recent compiles in which the nn.bin is loaded at startup)
3. run 'bench' at least twice for each compile, and average the result
(I know most here already know this, please don't be offended...I'm including it for clarity, and in case it helps someone).
Also, as Henk pointed out...to compare to a nodchip compile, make sure to select a 'nnue' version and make sure to type 'ucinewgame' to load the nn.bin before running 'bench'.
I believe Laskos 'nnue' vs sf-dev results with the newest SV net are significant...
I've been using ultra-fast for many years and have great confidence that ultra-fast results scale down in a meaningful way as TC increases.
At this point, I believe it's safe to say that nnue is pushing +60 elo or more? (Ed's results have it at +65)
which is fantastic considering it was at -50 to sf-dev in the very beginning, and just +30 just a short time ago
New sf+nnue play-only compiles
Moderators: hgm, Rebel, chrisw
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: New sf+nnue play-only compiles
-
- Posts: 6995
- Joined: Thu Aug 18, 2011 12:04 pm
Re: New sf+nnue play-only compiles
Final result 59.7%.Rebel wrote: ↑Tue Jul 28, 2020 8:24 pmDecided to play 2000 games instead of 1000.
First 1000 finished, so far more than +70 elo.Scales well.Code: Select all
Finished game 1018 (sf11 vs Norman): 1/2-1/2 {Draw by 3-fold repetition} Score of Norman vs sf11: 354 - 146 - 509 [0.603] 1009
Games: http://rebel13.nl/sergio-2344.7z
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 2872
- Joined: Wed Mar 08, 2006 10:09 pm
- Location: Germany
- Full name: Werner Schüle
Re: New sf+nnue play-only compiles
I repeated here a game with the play-only.exe and got a better result:
1 SFNNUE 270720 x64 SV2141 +70 +22/=76/-2 60.00% 60.0/100
2 Stockfish 11.0 x64 1CPU -70 +2/=76/-22 40.00% 40.0/100
1 SFNNUE 200719 SV2141 +45 +16/=81/-3 56.50% 56.5/100
2 Stockfish 11.0 x64 1CPU -45 +3/=81/-16 43.50% 43.5/100
1 SFNNUE 270720 x64 SV2141 +70 +22/=76/-2 60.00% 60.0/100
2 Stockfish 11.0 x64 1CPU -70 +2/=76/-22 40.00% 40.0/100
1 SFNNUE 200719 SV2141 +45 +16/=81/-3 56.50% 56.5/100
2 Stockfish 11.0 x64 1CPU -45 +3/=81/-16 43.50% 43.5/100
Werner
-
- Posts: 6995
- Joined: Thu Aug 18, 2011 12:04 pm
Re: New sf+nnue play-only compiles
Sergio-0109, 5000 games, tc 40m/20s, 96 elo stronger than Stockfish 11.
Incredible.
What net next?
Code: Select all
Score of Norman vs sf11: 2076 - 720 - 2204 [0.636] 5000
... Norman playing White: 1225 - 262 - 1013 [0.693] 2500
... Norman playing Black: 851 - 458 - 1191 [0.579] 2500
... White vs Black: 1683 - 1113 - 2204 [0.557] 5000
Elo difference: 96.6 +/- 7.2, LOS: 100.0 %, DrawRatio: 44.1 %
Finished match
What net next?
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: New sf+nnue play-only compiles
At the much slower tc of 5' + 3" (one thread, 5 GHz i7, net 1218, 244 games, 75% draws) I got +60 elo over SF July 17, quite good for that TC.Rebel wrote: ↑Wed Jul 29, 2020 3:35 pm Sergio-0109, 5000 games, tc 40m/20s, 96 elo stronger than Stockfish 11.
Incredible.Code: Select all
Score of Norman vs sf11: 2076 - 720 - 2204 [0.636] 5000 ... Norman playing White: 1225 - 262 - 1013 [0.693] 2500 ... Norman playing Black: 851 - 458 - 1191 [0.579] 2500 ... White vs Black: 1683 - 1113 - 2204 [0.557] 5000 Elo difference: 96.6 +/- 7.2, LOS: 100.0 %, DrawRatio: 44.1 % Finished match
What net next?
Komodo rules!
-
- Posts: 1535
- Joined: Sun Oct 25, 2009 2:30 am
Re: New sf+nnue play-only compiles
For some reason, this compile loses 100% of the games under cutechess-cli.
This is the log file.
This is the PGN.
This is the engines.json config for this engine:
This is the cli command:
No tablebases.
This is the log file.
This is the PGN.
This is the engines.json config for this engine:
Code: Select all
{
"command" : "sf+nnue-po.270720.halfkp_256x2-32-32.x64.bmi2.exe",
"name" : "nnue",
"options" : [
{
"name" : "Debug Log File",
"value" : "log.txt"
}
]
},
Code: Select all
cutechess-cli -engine conf=S11 tc=inf depth=13 -engine conf=nnue tc=inf depth=13 -openings file=SALC_V5_hc_10m.pgn start=1 -site test -event 10 -games 2 -rounds 5 -repeat -concurrency 1 -pgnout run.pgn -recover -resign movecount=1 score=300 -draw movenumber=25 movecount=5 score=5 -each proto=uci
-
- Posts: 215
- Joined: Sun Feb 24, 2008 2:08 am
Re: New sf+nnue play-only compiles
Probably not loading net.Ozymandias wrote: ↑Thu Jul 30, 2020 8:09 am For some reason, this compile loses 100% of the games under cutechess-cli.
This is the log file.
This is the PGN.
This is the engines.json config for this engine:This is the cli command:Code: Select all
{ "command" : "sf+nnue-po.270720.halfkp_256x2-32-32.x64.bmi2.exe", "name" : "nnue", "options" : [ { "name" : "Debug Log File", "value" : "log.txt" } ] },
No tablebases.Code: Select all
cutechess-cli -engine conf=S11 tc=inf depth=13 -engine conf=nnue tc=inf depth=13 -openings file=SALC_V5_hc_10m.pgn start=1 -site test -event 10 -games 2 -rounds 5 -repeat -concurrency 1 -pgnout run.pgn -recover -resign movecount=1 score=300 -draw movenumber=25 movecount=5 score=5 -each proto=uci
-
- Posts: 1535
- Joined: Sun Oct 25, 2009 2:30 am
-
- Posts: 215
- Joined: Sun Feb 24, 2008 2:08 am
Re: New sf+nnue play-only compiles
Why are you using depth instead of normal time control. Still not clear net is being loaded looking at he log file. Maybe specify net on cutechess command.
-
- Posts: 1535
- Joined: Sun Oct 25, 2009 2:30 am
Re: New sf+nnue play-only compiles
What would be the syntax for specifying the net on cutechess command?