Hard-Talkchess-2020 set, final release

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Vinvin
Posts: 4909
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: Hard-Talkchess-2020 set, final release

Post by Vinvin » Mon Dec 28, 2020 7:01 am

Ferdy wrote:
Mon Dec 28, 2020 6:27 am
Vinvin wrote:
Sun Dec 27, 2020 10:09 pm
Results for 6 new engines (NN file used : nn-62ef826d1a6d.nnue) :

Code: Select all

SugaR.AI.ICCF.1.00 : 166 solved
ShashChess15.1     : 166 solved
Bluefish-12-W      : 162 solved
Stockfish_20121407 : 153 solved
Eman 6.61          : 153 solved
SugaR.AI.1.00      : 149 solved
The 3 first ones are new records if we omit the 2 tricky version of Eman and BlueFish who are very weak in play.
These 3 engines combined, find 193 (!!!) solutions !
If we add Crystal-2020-06-29, they find 199 solutions !
To solved 11 remaining positions, you need to add Sting-SF (more specialized in fortress detection and positions without progress). A winning pack of 5 to not overlook tricky tactics and singular shots.

It's still 3 positions not found with this very clever pack :

Code: Select all

1rb1qrk1/2b2pp1/p3pBn1/3pP1Pp/1ppP4/2P1QN2/PP3P1P/R2BR1K1 w - - bm Nh4; id "Hard-Talkchess-2020.023" ;
2bqrr1k/p5b1/1p1p2pp/nPpBp3/P1P1N2P/3PN1P1/R4P1K/3Q1R2 w - - bm Nc3; id "Hard-Talkchess-2020.038" ;
r2r2k1/pqp1bppp/1pn1pn2/8/2NP1B2/2N1P1P1/PPQ2P1P/R4RK1 b - - bm Rxd4; id "Hard-Talkchess-2020.179" ;
Some engines that will be tested in the following days : Crystal_x64_110920, Cfish NUMA 261020 and Sting-sf-25.

Link to the sheet of results : https://www.dropbox.com/scl/fi/rmt10d1y ... pnp1c6kvmq
How to interpret the table? I understand there are 213 positions in this test.

Image
Each line is a position of the test. The number is the second needed to find the solution. If there's no number, the solution is not found.
When you go down after line #213, you will see the average time and the number of solutions found.

Ferdy
Posts: 4527
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Hard-Talkchess-2020 set, final release

Post by Ferdy » Mon Dec 28, 2020 7:30 am

Vinvin wrote:
Mon Dec 28, 2020 7:01 am
Ferdy wrote:
Mon Dec 28, 2020 6:27 am
Vinvin wrote:
Sun Dec 27, 2020 10:09 pm
Results for 6 new engines (NN file used : nn-62ef826d1a6d.nnue) :

Code: Select all

SugaR.AI.ICCF.1.00 : 166 solved
ShashChess15.1     : 166 solved
Bluefish-12-W      : 162 solved
Stockfish_20121407 : 153 solved
Eman 6.61          : 153 solved
SugaR.AI.1.00      : 149 solved
The 3 first ones are new records if we omit the 2 tricky version of Eman and BlueFish who are very weak in play.
These 3 engines combined, find 193 (!!!) solutions !
If we add Crystal-2020-06-29, they find 199 solutions !
To solved 11 remaining positions, you need to add Sting-SF (more specialized in fortress detection and positions without progress). A winning pack of 5 to not overlook tricky tactics and singular shots.

It's still 3 positions not found with this very clever pack :

Code: Select all

1rb1qrk1/2b2pp1/p3pBn1/3pP1Pp/1ppP4/2P1QN2/PP3P1P/R2BR1K1 w - - bm Nh4; id "Hard-Talkchess-2020.023" ;
2bqrr1k/p5b1/1p1p2pp/nPpBp3/P1P1N2P/3PN1P1/R4P1K/3Q1R2 w - - bm Nc3; id "Hard-Talkchess-2020.038" ;
r2r2k1/pqp1bppp/1pn1pn2/8/2NP1B2/2N1P1P1/PPQ2P1P/R4RK1 b - - bm Rxd4; id "Hard-Talkchess-2020.179" ;
Some engines that will be tested in the following days : Crystal_x64_110920, Cfish NUMA 261020 and Sting-sf-25.

Link to the sheet of results : https://www.dropbox.com/scl/fi/rmt10d1y ... pnp1c6kvmq
How to interpret the table? I understand there are 213 positions in this test.

Image
Each line is a position of the test. The number is the second needed to find the solution. If there's no number, the solution is not found.
When you go down after line #213, you will see the average time and the number of solutions found.
Thanks, nice table it is detailed. I plan two things, one run this test with Lc0 (on my current gtx gpu) at lower movetime not 30 minutes per position just to see how it performs and second find the top 100 easier positions. Criteria for easy could be the high number of engines where the position is solved and the lower time where the position is solved. If you have a rating list for those engines, another criteria to include would be, the positions solved by lower rated engine would also contribute to determine the easier positions. Any idea on how to find the top 100 easier positions?

Jouni
Posts: 2384
Joined: Wed Mar 08, 2006 7:15 pm

Re: Hard-Talkchess-2020 set, final release

Post by Jouni » Mon Dec 28, 2020 8:45 am

Bluefish tactical is in own class, but is running in 3 best line mode cheating?
Jouni

Ferdy
Posts: 4527
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Hard-Talkchess-2020 set, final release

Post by Ferdy » Mon Dec 28, 2020 9:46 am

Vinvin wrote:
Mon Dec 28, 2020 7:01 am
Each line is a position of the test. The number is the second needed to find the solution. If there's no number, the solution is not found.
When you go down after line #213, you will see the average time and the number of solutions found.
What is your setting in arena?

Image

Ferdy
Posts: 4527
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Hard-Talkchess-2020 set, final release

Post by Ferdy » Mon Dec 28, 2020 2:27 pm

This is the top 100 easier positions based on larger count of engines that solved the positions followed by a smaller average time the engines solved the positions.

Code: Select all

+----+----------------+---------------------+------------------+
|    | Solution       |   NumEngSolvedCount |   AverageTimeSec |
|----+----------------+---------------------+------------------|
|  0 | 9) Bd3xh7      |                  36 |               13 |
|  1 | 128) Bb2-a3    |                  36 |               21 |
|  2 | 86) c2-c4      |                  36 |               43 |
|  3 | 143) Re1xe7    |                  36 |               58 |
|  4 | 145) Rd1-d8    |                  36 |               96 |
|  5 | 161) Bd3xh7    |                  36 |               98 |
|  6 | 188) Rd3-h3    |                  36 |              168 |
|  7 | 193) .. c3xb2  |                  36 |              210 |
|  8 | 205) Rd1-g1    |                  36 |              295 |
|  9 | 162) Rh1xh7    |                  35 |               34 |
| 10 | 40) Kh3-g2     |                  35 |               98 |
| 11 | 92) .. Bb7-d5  |                  35 |              118 |
| 12 | 137) .. Nf6-h5 |                  35 |              165 |
| 13 | 71) .. h7-h5   |                  35 |              167 |
| 14 | 168) Nd5-f6    |                  35 |              212 |
| 15 | 199) d5-d6     |                  35 |              254 |
| 16 | 139) .. Ne8-c7 |                  35 |              274 |
| 17 | 210) Qd1-h5    |                  35 |              321 |
| 18 | 107) f4-f5     |                  34 |               86 |
| 19 | 63) h4-h5      |                  34 |              110 |
| 20 | 207) .. Ne4-d6 |                  34 |              197 |
| 21 | 142) Bf1-e2    |                  34 |              198 |
| 22 | 173) Qg5-h6    |                  34 |              201 |
| 23 | 175) Nd4-c6    |                  34 |              202 |
| 24 | 201) Bd3xh7    |                  34 |              261 |
| 25 | 132) .. Qe4xc4 |                  34 |              297 |
| 26 | 166) Nf8-g6    |                  34 |              388 |
| 27 | 213) Ng5xf7    |                  34 |              533 |
| 28 | 82) Ng7-f5     |                  33 |               87 |
| 29 | 115) Rf4-g4    |                  33 |              101 |
| 30 | 32) Ng3-h5     |                  33 |              117 |
| 31 | 17) Nf3-g5     |                  33 |              155 |
| 32 | 106) .. Rd7xd4 |                  33 |              161 |
| 33 | 4) e4-e5       |                  33 |              202 |
| 34 | 45) Qc6-f3     |                  33 |              282 |
| 35 | 191) Bd3xh7    |                  33 |              311 |
| 36 | 112) Rf1-c1    |                  32 |               56 |
| 37 | 151) Bd3xh7    |                  32 |              122 |
| 38 | 84) Rd3xb3     |                  32 |              125 |
| 39 | 174) g2-g4     |                  32 |              134 |
| 40 | 134) Rh1-h6    |                  32 |              161 |
| 41 | 96) Nc3-d5     |                  32 |              174 |
| 42 | 186) Rf3-f6    |                  32 |              190 |
| 43 | 180) .. O-O-O  |                  32 |              195 |
| 44 | 124) Ke2-d3    |                  32 |              196 |
| 45 | 50) Rf1-f5     |                  32 |              258 |
| 46 | 183) Ne3-g4    |                  32 |              273 |
| 47 | 206) Rf1-f6    |                  32 |              313 |
| 48 | 62) .. Qg5xg3  |                  32 |              343 |
| 49 | 171) Qf3-f6    |                  32 |              504 |
| 50 | 30) Rf1xf6     |                  31 |               59 |
| 51 | 22) g2-g4      |                  31 |               83 |
| 52 | 55) Qd8-a8     |                  31 |              138 |
| 53 | 202) e5-e6     |                  31 |              151 |
| 54 | 72) g5-g6      |                  31 |              173 |
| 55 | 212) Qe3-d3    |                  31 |              198 |
| 56 | 167) Nd5-f6    |                  31 |              270 |
| 57 | 146) Rd1-d8    |                  31 |              304 |
| 58 | 31) Rc8xc7     |                  31 |              335 |
| 59 | 147) Rd1-d8    |                  31 |              517 |
| 60 | 149) Bd3xh7    |                  30 |               71 |
| 61 | 5) Qc1-h6      |                  30 |              164 |
| 62 | 94) b3-b4      |                  30 |              180 |
| 63 | 198) Ne7-d5    |                  30 |              273 |
| 64 | 152) Rd1xd7    |                  30 |              292 |
| 65 | 100) a5-a6     |                  30 |              357 |
| 66 | 197) .. Qb2-c2 |                  30 |              432 |
| 67 | 127) .. Nb6xc4 |                  30 |              491 |
| 68 | 120) c3-c4     |                  29 |               72 |
| 69 | 41) c4-c5      |                  29 |              105 |
| 70 | 25) e4-e5      |                  29 |              143 |
| 71 | 121) Na5-b3    |                  29 |              162 |
| 72 | 52) Bd5xf7     |                  29 |              167 |
| 73 | 203) g2-g3     |                  29 |              182 |
| 74 | 85) h5-h6      |                  29 |              255 |
| 75 | 157) Kg1-f2    |                  29 |              306 |
| 76 | 148) Bd3xh7    |                  29 |              311 |
| 77 | 81) Rg2xg7     |                  29 |              497 |
| 78 | 165) b6-b7     |                  29 |              576 |
| 79 | 88) Qf5-e6     |                  28 |               54 |
| 80 | 111) .. e4-e3  |                  28 |               81 |
| 81 | 104) b7-b8R    |                  28 |              106 |
| 82 | 60) Nh3-g5     |                  28 |              107 |
| 83 | 122) .. Bf5-h3 |                  28 |              184 |
| 84 | 75) Rd7-d2     |                  28 |              204 |
| 85 | 16) Bd3xh7     |                  28 |              215 |
| 86 | 29) e4-e5      |                  28 |              266 |
| 87 | 12) Rg1xg7     |                  28 |              355 |
| 88 | 119) h4-h5     |                  28 |              406 |
| 89 | 189) Qf3xf4    |                  28 |              410 |
| 90 | 110) Kf2-e1    |                  28 |              412 |
| 91 | 35) Qe3xg5     |                  28 |              444 |
| 92 | 42) b2-b4      |                  27 |               56 |
| 93 | 172) Rd1-d3    |                  27 |               75 |
| 94 | 129) Bb4-a5    |                  27 |              205 |
| 95 | 49) Kh2-g3     |                  27 |              231 |
| 96 | 37) Re1-e6     |                  27 |              281 |
| 97 | 159) Bd3xh7    |                  27 |              301 |
| 98 | 56) Ne5xg6     |                  27 |              468 |
| 99 | 10) Rd6xf6     |                  26 |              177 |
+----+----------------+---------------------+------------------+


Ferdy
Posts: 4527
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Hard-Talkchess-2020 set, final release

Post by Ferdy » Mon Dec 28, 2020 2:41 pm

Result with 3 nn. Bigger nn performed better.

Code: Select all

CPU: i7-2600K, 3.4Ghz 4cores/8threads
GPU: GTX 1650 super
Cuda: V11.2.67
Name: Lc0 v0.26.3, backend: cudnn-fp16
Command: ./lc0 benchmark -w nn_name
Test Position: hard_talkchess_2020.epd
+-----------+-----------+-------+-----------+----------+---------+
|   nn_name | nn_size   |   nps |   TimeSec |   Solved |   Total |
|-----------+-----------+-------+-----------+----------+---------|
|     66680 | 30x384    |  2520 |        10 |       88 |     213 |
|     42850 | 20x256    |  8502 |        10 |       73 |     213 |
|    703810 | 10x128    | 44998 |        10 |       49 |     213 |
+-----------+-----------+-------+-----------+----------+---------+
I will run the nn 66680 later for 2 minutes per position.

Vinvin
Posts: 4909
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: Hard-Talkchess-2020 set, final release

Post by Vinvin » Mon Dec 28, 2020 3:51 pm

Ferdy wrote:
Mon Dec 28, 2020 9:46 am
Vinvin wrote:
Mon Dec 28, 2020 7:01 am
Each line is a position of the test. The number is the second needed to find the solution. If there's no number, the solution is not found.
When you go down after line #213, you will see the average time and the number of solutions found.
What is your setting in arena?

Image
At left, it's not very important, it's just the lines displayed after n-depth. I set to "10" but you can leave at "2".
I activate "abort", minimal depth= "24", if correct move is held "3" additional plies.

Jouni
Posts: 2384
Joined: Wed Mar 08, 2006 7:15 pm

Re: Hard-Talkchess-2020 set, final release

Post by Jouni » Mon Dec 28, 2020 3:59 pm

According to CSS forum SugaR AI ICCF 1.0 mit Night Nurse 0.3 now beats Bluefish tactical clearly!
Jouni

Vinvin
Posts: 4909
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: Hard-Talkchess-2020 set, final release

Post by Vinvin » Mon Dec 28, 2020 8:13 pm

Jouni wrote:
Mon Dec 28, 2020 3:59 pm
According to CSS forum SugaR AI ICCF 1.0 mit Night Nurse 0.3 now beats Bluefish tactical clearly!
Thanks for the information !
I will test this NN.

perejaslav
Posts: 233
Joined: Sat Mar 18, 2006 3:01 am
Location: Cold

Re: Hard-Talkchess-2020 set, final release

Post by perejaslav » Tue Dec 29, 2020 6:58 am



Engine: CF EXT 281220 x64 BMI2 N (8192 MB)
by The Stockfish developers

1/01 0:00 +1.49 1.Bc2 bxc3 (18.240) 9120

2/02 0:00 +1.49 1.Bc2 bxc3 (30.865) 15432

3/03 0:00 +1.49 1.Bc2 bxc3 2.bxc3 (47.426) 15808

...

26/41 0:55 +1.78 1.Nh4 bxc3 2.bxc3 Nxh4 3.Qh3 Nf5
4.Qxh5 Rb2 5.Kh1 Bd8 6.Bg4 Rxf2
7.Rg1 Qc6 8.Rab1 Bd7 9.Bxf5 exf5
10.Rb8 Rxa2 11.Rxd8 Be8 12.Qh3 Qe6
13.Qf3 Rb2 14.Qh5 (1.595.094.896) 28851

best move: Nf3-h4 time: 1:11.031 min n/s: 28.834.622 nodes: 2.048.152.049

Threads=24
Engine Options:
Use NNUE=Pure
EXT_ICCF_Analyses=5

Post Reply