Discussion of anything and everything relating to chess playing software and machines.
Moderators: hgm , Dann Corbit , Harvey Williamson
MikeB
Posts: 4889Joined: Thu Mar 09, 2006 6:34 amLocation: Pen Argyl, Pennsylvania
Post by MikeB Sat Jul 25, 2020 7:02 pm
MikeB wrote: ↑ Sat Jul 25, 2020 2:44 pm JohnS wrote: ↑ Sat Jul 25, 2020 10:06 am
Another quick test of 2344 against H6.03 using Nunn1 openings, G10s+0.2s - result +11 =8 -1.
Here is a crazy sacrifice by SFnnue on move 13. Of course - usual disclaimers and as Mike would say yomv and ymmv.
[pgn][Event "SF-NNUE - Houdini 6.03, Nunn1, G10s + 0.2s"]
[Site "Home"]
[Date "2020.07.25"]
[Round "1"]
[White "Stockfish+NNUE"]
[Black "Houdini 6.03"]
[Result "1-0"]
[TimeControl "10+0.2"]
[Time "17:49:43"]
[Board "9"]
[Termination "adjudication by engines' scores"]
[ECO "D11"]
[Opening "QGD Slav"]
1. d4 d5 2. c4 c6
3. Nf3 {D11: QGD Slav, 3.Nf3} e6 4. cxd5 exd5
5. Nc3 Nf6 6. Bg5 Be7
7. Qc2 Nbd7 8. e3 O-O
9. Bd3 Re8 10. O-O Nf8 {End of opening}
11. h3 {+0.55/21 1.7 1825739} Ne4 {-0.29/16 0.8 1659811} 12. Bf4 {+0.57/15 0.3 340609} f5 {-0.29/15 0.5 1077857}
13. Nxd5 {+2.78/17 0.3 365358} cxd5 {+2.19/14 0.2 541228} 14. Bc7 {-2.63/19 0.6 586658} Qd7 {+2.19/15 0.4 804820}
15. Rfc1 {-2.40/18 0.3 305302} a6 {+2.04/17 1.1 2499173} 16. Qb3 {-2.44/19 0.4 428054} Qe6 {+1.94/17 0.5 1132772}
17. Rc2 {-2.32/17 0.3 291583} b5 {+1.87/18 1.0 2388891} 18. Ne5 {-1.83/17 0.6 596654} Bb7 {+1.99/15 0.3 587827}
19. a4 {-1.68/21 1.1 1189223} b4 {+1.31/16 1.0 2281160} 20. f3 {-1.84/19 0.4 414811} Ng3 {+1.69/15 0.2 526878}
21. Rac1 {-0.68/17 0.4 413539} Qh6 {+0.89/17 1.9 4376992} 22. f4 {+0.00/20 0.2 258037} Ne6 {+0.78/17 0.8 1960946}
23. Rc6 {+0.00/20 0.4 479539} Rec8 {+0.49/17 0.9 2314155} 24. Rb6 {+0.00/20 0.3 363630} Ra7 {+1.47/12 0.2 418161}
25. Kh2 {+0.00/21 0.3 296852} Ne4 {-0.22/17 0.7 1695180} 26. Bxe4 {+1.25/19 0.3 307773} fxe4 {-0.22/16 0.0 637}
27. Qd1 {+1.91/23 1.9 2128183} Re8 {-0.73/19 1.1 2793150} 28. a5 {+3.28/17 0.2 208229} Qf6 {-1.09/18 0.9 2123492}
29. Qa4 {+3.83/21 0.3 350003} Rea8 {-1.21/19 0.4 1173762} 30. f5 {+3.02/23 0.8 895388} Qxf5 {-1.21/18 0.0 2235}
31. Qd7 {+4.07/19 0.2 282314} Bf8 {-1.31/19 0.4 974990} 32. Qxe6+ {+3.94/21 0.5 615758} Qxe6 {-1.20/18 0.3 819661}
33. Rxe6 {+4.06/21 0.3 422568} Rc8 {-1.35/19 0.3 1033236} 34. Kg3 {+4.81/19 0.3 344820} g6 {-1.55/17 0.6 1706853}
35. Rf1 {+4.70/17 0.3 399405} Ba8 {-2.14/17 0.4 952201} 36. Bd6 {+5.59/18 0.3 380173} Bg7 {-2.34/15 0.2 546724}
37. Kh4 {+6.52/17 0.2 270354} Kh8 {-2.27/14 0.2 544657} 38. Kg5 {+7.02/18 0.3 316968} Kg8 {-2.03/14 0.1 320824}
39. g4 {+7.70/18 0.3 413143} b3 {-2.38/15 0.3 742604} 40. h4 {+8.11/19 0.4 558349} Bh8 {-2.84/15 0.2 553155}
41. h5 {+8.56/18 0.3 419325} gxh5 {-2.91/17 0.2 474664} 42. gxh5 {+8.93/18 0.3 455113} Rd8 {-3.47/13 0.2 646262}
43. Ng4 {+9.46/18 0.4 506316} Bg7 {-6.58/15 0.2 586254} 44. Nh6+ {+9.53/16 0.1 155076} Bxh6+ {-6.46/18 0.2 615138}
45. Kxh6 {+11.84/19 0.2 245093} Rad7 {-5.65/16 0.1 312964} 46. Bc5 {+12.79/22 0.2 318832} Rc8 {-8.61/17 0.3 950241}
47. Ref6 {+18.81/29 0.2 309342} Rdd8 {-17.45/21 0.2 743347} 48. Be7 {M+7/41 0.2 353040} Bb7 {M-6/21 0.0 107765}
49. Rg1+ {M+6/51 0.2 326491} Kh8 {M-5/11 0.0 724} 1-0[/pgn]
Nice game
2344 will soon be "my" KOM at least, after 1134 's "lucky" run to stay on top of the heap. yomv and ymmv
two threads here - - not that it matters but these nets seem to test at very stable scaling once you get near 30+.03 or with SMP I will do that tc next with the same two nets to to see how close it it.
Code: Select all
pgn file: c:/cluster.mfb/pgn/2007250307-23441134.pgn
tc/base+inc: 60+0.60
games planned: 4000
Threads: 2
Hash: 256
Current date : time (EDST)
Date: 07/25/20 : 08:42:00
Projected-> Time: 8h:32m:0s
Running -> Time: 5h:34m:47s
2808 game(s) loaded
Rank Name Rating Δ + - # Σ Σ% W L D W% =% OppR
---------------------------------------------------------------------------------------------------------
1 2344 3502 0.0 8 8 2808 1418.5 50.5 644 615 1549 22.9 55.2 3498
2 1134 3498 3.8 8 8 2808 1389.5 49.5 615 644 1549 21.9 55.2 3502
---------------------------------------------------------------------------------------------------------
Δ = delta from the next higher rated opponent
# = number of games played
Σ = total score, 1 point for win, 1/2 point for draw
LOS:
23 11
2344 81
1134 18
2808 game(s) loaded
loops/scheduled: 149/228
waiting: 128
...seconds remaining: 128
As expected after 2800 games. 2344 finsined on top with a nice indicated gain of 5 Elo perhaps.
Final Results:
Code: Select all
pgn file: c:/cluster.mfb/pgn/2007250307-23441134.pgn
tc/base+inc: 60+0.60
games planned: 4000
Threads: 2
Hash: 256
Current date : time (EDST)
Date: 07/25/20 : 11:37:34
Projected-> Time: 8h:32m:0s
Running -> Time: 8h:30m:21s
4000 game(s) loaded
Rank Name Rating Δ + - # Σ Σ% W L D W% =% OppR
---------------------------------------------------------------------------------------------------------
1 2344 3503 0.0 7 7 4000 2028.0 50.7 928 872 2200 23.2 55.0 3497
2 1134 3497 5.0 7 7 4000 1972.0 49.3 872 928 2200 21.8 55.0 3503
---------------------------------------------------------------------------------------------------------
Δ = delta from the next higher rated opponent
# = number of games played
Σ = total score, 1 point for win, 1/2 point for draw
LOS:
23 11
2344 91
1134 8
4000 game(s) loaded
loops/scheduled: 228/228
waiting: 128
...seconds remaining: 0
done
4000 games and 8+ hours running of 50 concurrent games at once and the expected completion time falls within two minutes of the projected finish time. If this completion time indicator is off substantially, that usually means something went awry and the games should be checked for anomalies.
Next up is a 30+.3 set with the same nets, 4000 games with single thread to see how well of a proxy this TC is for longer games with multiple threads.
Below is the cutechess-cli output for the match above:
Code: Select all
Score of 2344 vs 1134: 928 - 872 - 2200 [0.507] 4000
... 2344 playing White: 911 - 29 - 1060 [0.721] 2000
... 2344 playing Black: 17 - 843 - 1140 [0.293] 2000
... White vs Black: 1754 - 46 - 2200 [0.714] 4000
Elo difference: 4.9 +/- 7.2, LOS: 90.7 %, DrawRatio: 55.0 %
Finished match
carldaman
Posts: 2277Joined: Sat Jun 02, 2012 2:13 am
Post by carldaman Sat Jul 25, 2020 7:43 pm
cdani wrote: ↑ Sat Jul 25, 2020 3:56 pm
Someone tested experimental nets? I don't see many comments about them.
viewtopic.php?f=6&t=74518&start=10#p852962
MikeB
Posts: 4889Joined: Thu Mar 09, 2006 6:34 amLocation: Pen Argyl, Pennsylvania
Post by MikeB Sat Jul 25, 2020 8:03 pm
I have tested a few - everyone was weaker.
chrisw
Posts: 4292Joined: Tue Apr 03, 2012 4:28 pm
Post by chrisw Sat Jul 25, 2020 8:12 pm
MikeB wrote: ↑ Sat Jul 25, 2020 8:03 pm
I have tested a few - everyone was weaker.
Yup, that happens when you test lots of things that are more or less the same and put them in some sort of ranked list. There’s one at the top and one at the bottom and the rest in the middle.
Rebel
Posts: 6946Joined: Thu Aug 18, 2011 12:04 pm
Post by Rebel Sat Jul 25, 2020 8:56 pm
I am giving up for the moment testing Sergio nets, although (just) 2000 games is far from accurate the magic seems to have gone quite quickly after the first releases. I have tested 22 versions. The wait is for the SF team, they have more man and computer power. Can't wait....
Code: Select all
SF-NNUE (popcount) vs Stockfish 11, 2000 games, tc 40m/20s, input 8moves.pgn
henk-2706 54.7% 2020-07-19
sergio-1432 56.1% 2020-07-21 15:56
sergio-1907 58.9% 2020-07-21 19:14
sergio-2323 58.4% 2020-07-21 23:23
sergio-0359 58.8% 2020-07-22 03:59
sergio-0944 58.9% 2020-07-22 09:44
sergio-1153 58.3% 2020-07-22 11:53
sergio-1807 58.0% 2020-07-22 18:07
sergio-2210 59.0% 2020-07-22 22:10
sergio-0511 58.2% 2020-07-23 05:11
sergio-1134 56.5% 2020-07-23 11:34 with 4moves.pgn 58.7%
sergio-1844 57.8% 2020-07-23 18:44
sergio-1843 58.7% 2020-07-23 18:43
sergio-0123 57.2% 2020-07-24 01:23
sergio-0640 56.7% 2020-07-24 06:40
sergio-1240 57.6% 2020-07-24 12:40
sergio-1224 58.1% 2020-07-24 12:24
sergio-1732 58.7% 2020-07-24 17:32
sergio-2344 57.2% 2020-07-24 23:54
ribbit-0.1 58.7% = 1134 (!!)
sergio-1313 58.5% 2020-07-25 13:13
sergio-2242 57.5% 2020-07-25 22:42
Note that the experimental versions are labeled with the time of day on the webpage.
90% of coding is debugging, the other 10% is writing bugs.
Sylwy
Posts: 4437Joined: Fri Apr 21, 2006 4:19 pmLocation: IASI - the historical capital of MOLDOVAFull name: SilvianR
Post by Sylwy Sat Jul 25, 2020 8:59 pm
cdani
Posts: 2204Joined: Sat Jan 18, 2014 10:24 amLocation: Andorra
Post by cdani Sat Jul 25, 2020 10:09 pm
Laskos wrote: ↑ Sat Jul 25, 2020 5:19 pm cdani wrote: ↑ Sat Jul 25, 2020 5:14 pm
20200725-2051.bin seems better than 20200723-0511 after a few hundred games. I publish the test later with more games.
About equal here after 500 games (to 2141).
Is possible that is a tiny bit stronger. 20+0.01
Code: Select all
# PLAYER : RATING ERROR POINTS PLAYED (%)
1 stnnuesergio20200725-2051 : 2856.6 2.7 2301.0 4586 50.2%
2 stnnuesergio20200723-0511 : 2855.4 2.7 2285.0 4586 49.8%
JJJ
Posts: 1346Joined: Sat Apr 19, 2014 1:47 pm
Post by JJJ Sat Jul 25, 2020 10:27 pm
Hello, I have trouble understanding how and where to dl Stockfish NNE and his best nets. Also, I d like to run it on arena. Can someone explain this to me ? Or is there a tutorial for this already ?
adnoh
Posts: 72Joined: Tue Jun 26, 2007 6:31 amFull name: Charles Wong
Post by adnoh Sat Jul 25, 2020 10:54 pm
JJJ wrote: ↑ Sat Jul 25, 2020 10:27 pm
Hello, I have trouble understanding how and where to dl Stockfish NNE and his best nets. Also, I d like to run it on arena. Can someone explain this to me ? Or is there a tutorial for this already ?
To get the SFNNUE executable, go here.
https://github.com/nodchip/Stockfish/releases
The current latest is 2020-07-19. There are a lot of exe in the archive so get the one that is suitable for your CPU but note the naming convention and make sure you get the one that actually uses the network file.
For my fairly old CPU, I am using this.
stockfish.sse42.halfkp_256x2-32-32.profile-nnue.2020-07-19.exe
For the nets, go here and place them in a subdirectory called "eval" (convention only)
https://www.comp.nus.edu.sg/~sergio-v/nnue/
There are many people testing which one is best. I find this one quite good for my conditions.
20200722-2141.bin
In Arena, install it like any other UCI engine and the important part is use the configure engine dialog to point the EvalFile option to where you placed the net.
JohnS
Posts: 215Joined: Sun Feb 24, 2008 2:08 am
Post by JohnS Sun Jul 26, 2020 3:52 am
Now 2344 demolishes Shredder 12 +20 =0 -0. This is getting scary.
