Hard-Talkchess-2020 set, final release

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
perejaslav
Posts: 233
Joined: Sat Mar 18, 2006 3:01 am
Location: Cold

Re: Hard-Talkchess-2020 set, final release

Post by perejaslav » Tue Dec 29, 2020 7:09 am



Engine: SugaR AI ICCF 1.20 (8192 MB)
by Stockfish Team, Marco Zerbinati, Khalid

1/01 0:00 +1.25 1.Bc2 (23.748) 5937

2/02 0:00 +1.25 1.Bc2 bxc3 (36.365) 7273

3/03 0:00 +1.25 1.Bc2 bxc3 2.bxc3 (94.748) 15791

...


19/33 0:15 +0.57 1.Nh4 bxc3 2.bxc3 Nxh4 3.Qh3 Nf5
4.Qxh5 Rb2 5.Bg4 Bd7 6.Kh1 Rxf2
7.Rg1 Bxe5 8.Bxe5 Qe7 9.Bxf5 exf5
10.g6 fxg6 11.Qxg6 Rf7 12.Qg3 Re2 (373.026.292) 23759
...

best move: Nf3-h4 time: 0:17.906 min n/s: 23.652.532 nodes: 423.758.766

Threads=24
Engine Options:
multiPV Search=4

Jouni
Posts: 2384
Joined: Wed Mar 08, 2006 7:15 pm

Re: Hard-Talkchess-2020 set, final release

Post by Jouni » Tue Dec 29, 2020 8:28 am

Lc0 don't consider other moves:

Analysis by Lc0 v0.26.3:

1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/23 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 78kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Qf7 10.h4 Ba5 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.63) Depth: 15/26 00:00:02 81kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.63) Depth: 15/27 00:00:02 81kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.64) Depth: 15/27 00:00:07 92kN
Jouni

Ferdy
Posts: 4527
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Hard-Talkchess-2020 set, final release

Post by Ferdy » Tue Dec 29, 2020 1:14 pm

Vinvin wrote:
Mon Dec 28, 2020 3:51 pm
Ferdy wrote:
Mon Dec 28, 2020 9:46 am
Vinvin wrote:
Mon Dec 28, 2020 7:01 am
Each line is a position of the test. The number is the second needed to find the solution. If there's no number, the solution is not found.
When you go down after line #213, you will see the average time and the number of solutions found.
What is your setting in arena?

Image
At left, it's not very important, it's just the lines displayed after n-depth. I set to "10" but you can leave at "2".
I activate "abort", minimal depth= "24", if correct move is held "3" additional plies.
I settled for minimal analysis depth = 10 and Abort if the correct move is held = 3 with activate abort analysis set to ON.

Results, Lc0 is at rank 26.

Code: Select all

+----+----------------------+------------+---------------+-------------+------------------------------+----------------+-----------+
|    | name                 |   numsolve |   meantimesec |   pospermin | cpu                          | gpu            |   threads |
|----+----------------------+------------+---------------+-------------+------------------------------+----------------+-----------|
|  0 | Bluefish-FD-XI-DC    |        191 |            25 |           5 | AMD Ryzen Threadripper 3970X | na             |         2 |
|  1 | Bluefish-FD-XI-TD    |        181 |           153 |          30 | i7-4930K                     | na             |         1 |
|  2 | Eman_5.40_fad_12     |        167 |           193 |          30 | i7-4930K                     | na             |         1 |
|  3 | SugaR.AI.ICCF.1.00   |        166 |           113 |          30 | i7-4930K                     | na             |         1 |
|  4 | ShashChess15.1       |        166 |           170 |          30 | i7-4930K                     | na             |         1 |
|  5 | Bluefish-12-W        |        162 |           187 |          30 | i7-4930K                     | na             |         1 |
|  6 | Crystal-2020-06-29   |        159 |           199 |          30 | i7-4930K                     | na             |         1 |
|  7 | Bluefish-XI          |        159 |           222 |          30 | i7-4930K                     | na             |         1 |
|  8 | CorChess v6          |        156 |           327 |          30 | i7-4930K                     | na             |         1 |
|  9 | Houdidi_6.03_tac_1   |        155 |           196 |          30 | i7-4930K                     | na             |         1 |
| 10 | Stockfish_2020-07-19 |        155 |           239 |          30 | i7-4930K                     | na             |         1 |
| 11 | ShashChess 12.1      |        154 |           248 |          30 | i7-4930K                     | na             |         1 |
| 12 | Stockfish_20121407   |        153 |           229 |          30 | i7-4930K                     | na             |         1 |
| 13 | Stockfish_20062710   |        153 |           331 |          30 | i7-4930K                     | na             |         1 |
| 14 | Eman 6.61            |        153 |           258 |          30 | i7-4930K                     | na             |         1 |
| 15 | SugaR.AI.1.00        |        149 |           280 |          30 | i7-4930K                     | na             |         1 |
| 16 | Stockfish 11         |        141 |           366 |          30 | i7-4930K                     | na             |         1 |
| 17 | Honey-XI             |        139 |           290 |          30 | i7-4930K                     | na             |         1 |
| 18 | Stockfish 10         |        137 |           450 |          30 | i7-4930K                     | na             |         1 |
| 19 | Black-Diamond-XI     |        136 |           267 |          30 | i7-4930K                     | na             |         1 |
| 20 | Black-Diamond-XR7    |        136 |           231 |          30 | i7-4930K                     | na             |         1 |
| 21 | Crystal 2020-05-02   |        136 |           187 |          30 | i7-4930K                     | na             |         1 |
| 22 | Crystal 1.1          |        135 |           270 |          30 | i7-4930K                     | na             |         1 |
| 23 | Black-Diamond-XI-r2  |        135 |           158 |          30 | i7-4930K                     | na             |         1 |
| 24 | McCain-v2            |        130 |           389 |          30 | i7-4930K                     | na             |         1 |
| 25 | Stockfish 9          |        130 |           473 |          30 | i7-4930K                     | na             |         1 |
| 26 | Houdidi_6.03_tac_def |        125 |           349 |          30 | i7-4930K                     | na             |         1 |
| 27 | Lc0 v0.26.3 s1       |        123 |            10 |           2 | i7-2600K                     | GTX 1650 super |         2 |
| 28 | Sting-SF-16          |        122 |           234 |          30 | i7-4930K                     | na             |         1 |
| 29 | Sting-SF-20          |        119 |           240 |          30 | i7-4930K                     | na             |         1 |
| 30 | Ethereal 12.25       |         98 |           396 |          30 | i7-4930K                     | na             |         1 |
| 31 | Houdini 1.5 x64      |         97 |           283 |          30 | i7-4930K                     | na             |         1 |
| 32 | Critter 1.6a 64bit   |         96 |           240 |          30 | i7-4930K                     | na             |         1 |
| 33 | Stockfish 8          |         94 |           507 |          30 | i7-4930K                     | na             |         1 |
| 34 | Xiphos 0.6           |         81 |           330 |          30 | i7-4930K                     | na             |         1 |
| 35 | Ethereal 11.75       |         77 |           613 |          30 | i7-4930K                     | na             |         1 |
| 36 | Stockfish 7          |         76 |           487 |          30 | i7-4930K                     | na             |         1 |
+----+----------------------+------------+---------------+-------------+------------------------------+----------------+-----------+
I break down the header due to long character.

Header Legend:

Code: Select all

Stockfish_2020-07-19: Stockfish.sse42.halfkp_256x2-32-32.Profile-nnue.2020-07-19
Bluefish-FD-XI-DC: Bluefish-FD-XI-T=2;D=N;DC 3970X 5 minutes
Bluefish-FD-XI-TD: Bluefish-FD-XI-(Tact=2,defens=no)
Houdidi_6.03_tac_1: Houdidit603-tact=1
Houdidi_6.03_tac_def: Houdidit603(default)
Eman_5.40_fad_12: Eman 5.40, FAD=12
Lc0 v0.26.3 s1: Lc0 v0.26.3 cudnn-fp16, 66680, GTX 1650 super, i7-2600K, 2 minutes/pos
I created a repo for this, saving the epd, csv files for table, ranking etc. which can be found in docs/folder. If you are not ok with this, I will delete this repo.

Jouni
Posts: 2384
Joined: Wed Mar 08, 2006 7:15 pm

Re: Hard-Talkchess-2020 set, final release

Post by Jouni » Tue Dec 29, 2020 1:38 pm

NN engines should score 100% soon! Either with learning data or by NN learning, if someone shows position in learning? No?
Jouni

Werewolf
Posts: 1348
Joined: Thu Sep 18, 2008 8:24 pm

Re: Hard-Talkchess-2020 set, final release

Post by Werewolf » Tue Dec 29, 2020 7:13 pm

Jouni wrote:
Tue Dec 29, 2020 8:28 am
Lc0 don't consider other moves:

Analysis by Lc0 v0.26.3:

1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/23 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 78kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Qf7 10.h4 Ba5 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.63) Depth: 15/26 00:00:02 81kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.63) Depth: 15/27 00:00:02 81kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.64) Depth: 15/27 00:00:07 92kN
Yes I've just discovered this myself. Incredible.

Is position 23 (this position) regarded as the hardest position in the testsuite?

Vinvin
Posts: 4909
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: Hard-Talkchess-2020 set, final release

Post by Vinvin » Tue Dec 29, 2020 9:18 pm

Werewolf wrote:
Tue Dec 29, 2020 7:13 pm
Jouni wrote:
Tue Dec 29, 2020 8:28 am
Lc0 don't consider other moves:

Analysis by Lc0 v0.26.3:

1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/23 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 77kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5
+/= (0.62) Depth: 15/24 00:00:00 78kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Qf7 10.h4 Ba5 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.63) Depth: 15/26 00:00:02 81kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.63) Depth: 15/27 00:00:02 81kN
1.Nh4 Kh7 2.Bxh5 Rh8 3.Qg3 a5 4.Ng2 Kg8 5.Bxg6 fxg6 6.Nf4 a4 7.a3 bxc3 8.bxc3 Rb3 9.Re3 Ba5 10.h4 Qf7 11.h5 Rxh5 12.Nxh5 gxh5
+/= (0.64) Depth: 15/27 00:00:07 92kN
Yes I've just discovered this myself. Incredible.

Is position 23 (this position) regarded as the hardest position in the testsuite?
For others engines than LC0, yes !

Vinvin
Posts: 4909
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: Hard-Talkchess-2020 set, final release

Post by Vinvin » Wed Dec 30, 2020 12:13 am

perejaslav wrote:
Tue Dec 29, 2020 7:09 am


Engine: SugaR AI ICCF 1.20 (8192 MB)
by Stockfish Team, Marco Zerbinati, Khalid
...
19/33 0:15 +0.57 1.Nh4 bxc3 2.bxc3 Nxh4 3.Qh3 Nf5
4.Qxh5 Rb2 5.Bg4 Bd7 6.Kh1 Rxf2
7.Rg1 Bxe5 8.Bxe5 Qe7 9.Bxf5 exf5
10.g6 fxg6 11.Qxg6 Rf7 12.Qg3 Re2 (373.026.292) 23759
...

best move: Nf3-h4 time: 0:17.906 min n/s: 23.652.532 nodes: 423.758.766

Threads=24
Engine Options:
multiPV Search=4
Behavior with 1 thread and several threads can be very different.
If you test with several threads, please run it 4 or 5 times and make the average time.

I know my tests become not very realistic because almost everybody uses multi-thread.
I plan to buy a new computer with an AMD 5950x in late January. Likewise, I will be able to run the set faster and with multi-threading.

abgursu
Posts: 46
Joined: Thu May 14, 2020 1:34 pm
Full name: Ahmet Burak Gursu

Re: Hard-Talkchess-2020 set, final release

Post by abgursu » Wed Dec 30, 2020 2:07 pm


I was expecting to see this position in Hard-TalkChess. A lot harder than lots of positions in this set.
Taken from:

abgursu
Posts: 46
Joined: Thu May 14, 2020 1:34 pm
Full name: Ahmet Burak Gursu

Re: Hard-Talkchess-2020 set, final release

Post by abgursu » Wed Dec 30, 2020 2:13 pm

abgursu wrote:
Wed Dec 30, 2020 2:07 pm

I was expecting to see this position in Hard-TalkChess. A lot harder than lots of positions in this set.
Taken from:
Oh, there is. I just see it. Great position by the way.

Vinvin
Posts: 4909
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: Hard-Talkchess-2020 set, final release

Post by Vinvin » Thu Dec 31, 2020 2:10 am

Werewolf wrote:
Tue Dec 29, 2020 7:13 pm
...
Here a list
Yes I've just discovered this myself. Incredible.

Is position 23 (this position) regarded as the hardest position in the testsuite?
Here is a list of the 25 hardest positions for Houdini, Stockfish and their family ...

Code: Select all

 23) Nf3-h4      
 33) Nf7-d6      
 38) Ne4-c3      
  7) Rh4xh1      
 74) Kb7-c8      
 57) Nc2-d4      
 67) f3-f4       
204) Bc1-g5      
184) .. Bf5-g4   
 68) Qc7-d8      
 77) Nc4-a5      
 91) Be3xc5      
 93) Nc6xe7      
131) Na8-c7      
 70) a7-a8B      
153) Ne2-d4      
 69) Re8xe5      
150) Bd4-f6      
 26) Ne2-f4      
 99) .. Qd3xg3   
144) Bg5-f6      
 78) Qd6xe5      
130) f2-f4       
 89) .. Re2xd2   
 18) Be1-d2

Post Reply