ShashChess

[Jouni]

In my test with Arasan suite neither Tal nor MCTS improved result. Default was best.

[Frank Quisinsky]

Hi Jouni,

do you believe I should test this engine for my new KI-Ratings with default?
Conditions are "No learning".

I am very interest how many short win games Shash produced if I compare the results with Stockfish.
Very interesting on the move average for win games and on other statistics.

Honestly, I can't believe that any derivate Stockfish engine will produced a better result as the original Stockfish because the stats vs. TOP-40 from original Stockfish in mid-games are gigantic. But maybe ...

Best
Frank

[Vernon Crawford]

Hi Jouni,

do you believe I should test this engine for my new KI-Ratings with default?
Conditions are "No learning".

I am very interest how many short win games Shash produced if I compare the results with Stockfish.
Very interesting on the move average for win games and on other statistics.

Honestly, I can't believe that any derivate Stockfish engine will produced a better result as the original Stockfish because the stats vs. TOP-40 from original Stockfish in mid-games are gigantic. But maybe ...

Best
Frank

Hi Frank,
You could try Stockfish+ in full montecarlo mode
I estimate it to be 50 weaker than AB Stockfish but playing different & interesting lines.
I've used with good result in a few of my correspondence games.

I doubt it will do better on tests or stats though

[peter]

In my test with Arasan suite neither Tal nor MCTS improved result. Default was best.

MCTS single thread was probably a misunderstanding of mine of new UCI params of MCTS, up to latest but one version "Single" meant main thread, now it's one second thread besides main line, which stays A-B anyways, if I get new Readme at github right.
Already asked about that here

forum3/viewtopic.php?p=918678#p918678

You're right about that as for tactical single best moves probably, at least at short TC in tactical position-test it didn't help at my counter- check neither, even if it did (almost) no harm, about the same result.
Probably MCTS in the way its meant to set up now should have more MCTS- threads for tactical positions, don't know how much time to depth it costs and maybe it helps more in quiet positions, as for time to solution, more tests are necessary.

[Frank Quisinsky]

Hi Vernon,

what I can do is ...

Replace Shash with Stockfish+ in ... run-3 !!
So we have the result for both.

But two in one run is to much.



Best
Frank

[Jouni]

Frank one question. Is this meaning, that 1 CPU Stockfish is ahead of Lc0 in RTX-3070 Ti? Homepage has too much of everything

01. Stockfish 311221 NN dev : 3506.85 564 83.2 375 188 1 469.0 33.3 25.59 3197.16 14.45 40.6
02. Lc0 0.28.2 611062 GPU : 3484.95 160 80.9 99 61 0 129.5 38.1 45.85 3201.38 14.44 40.0 ... RTX-3070 Ti

[peter]

In my test with Arasan suite neither Tal nor MCTS improved result. Default was best.

Arasan -suite I didn't try, think it's too easy with TCs<(=)15"/pos for SF and its branches at modern hardware multicore to get selective results. How many positions were solved at your trials? Did you test single core? With what a TC?

At HTC108 with Tal option checked there are 10 positions more solved than unchecked (default) at 15"/pos., 30 threads of 16x3.5GHz CPU, bmi2 compile, 2G hash.
Default
78/108:

Code: Select all

Löse: C:\Users\Peter\Documents\ChessBase\Testsets\EL.cbh
Maximale Lösungszeit = 15s.

1. Hard-Talkchess-2020.001,  HTC108   Gelöst in 6.63s/44; Gelöst: 1
2. Hard-Talkchess-2020.002,  HTC108   Gelöst in 2.42s/24; Gelöst: 2
3. Hard-Talkchess-2020.003,  HTC108   Gelöst in 10.25s/27; Gelöst: 3
4. Hard-Talkchess-2020.007,  HTC108   > 15s.
5. Hard-Talkchess-2020.008,  HTC108   > 15s.
6. Hard-Talkchess-2020.010,  HTC108   Gelöst in 0.09s/15; Gelöst: 4
7. Hard-Talkchess-2020.011,  HTC108   > 15s.
8. Hard-Talkchess-2020.012,  HTC108   Gelöst in 8.86s/31; Gelöst: 5
9. Hard-Talkchess-2020.013,  HTC108   Gelöst in 2.95s/29; Gelöst: 6
10. Hard-Talkchess-2020.014,  HTC108   Gelöst in 3.27s/21; Gelöst: 7
11. Hard-Talkchess-2020.016,  HTC108   Gelöst in 1.53s/21; Gelöst: 8
12. Hard-Talkchess-2020.019,  HTC108   > 15s.
13. Hard-Talkchess-2020.020,  HTC108   Gelöst in 1.91s/25; Gelöst: 9
14. Hard-Talkchess-2020.021,  HTC108   Gelöst in 0.24s/22; Gelöst: 10
15. Hard-Talkchess-2020.023,  HTC108   > 15s.
16. Hard-Talkchess-2020.028,  HTC108   Gelöst in 3.34s/30; Gelöst: 11
17. Hard-Talkchess-2020.029,  HTC108   Gelöst in 2.17s/23; Gelöst: 12
18. Hard-Talkchess-2020.031,  HTC108   Gelöst in 0.39s/17; Gelöst: 13
19. Hard-Talkchess-2020.034,  HTC108   Gelöst in 3.69s/43; Gelöst: 14
20. Hard-Talkchess-2020.035,  HTC108   > 15s.
21. Hard-Talkchess-2020.036,  HTC108   Gelöst in 0.23s/18; Gelöst: 15
22. Hard-Talkchess-2020.039,  HTC108   > 15s.
23. Hard-Talkchess-2020.043,  HTC108   > 15s.
24. Hard-Talkchess-2020.046,  HTC108   Gelöst in 0.05s/8; Gelöst: 16
25. Hard-Talkchess-2020.047,  HTC108   > 15s.
26. Hard-Talkchess-2020.049,  HTC108   Gelöst in 0.06s/11; Gelöst: 17
27. Hard-Talkchess-2020.050,  HTC108   Gelöst in 3.56s/24; Gelöst: 18
28. Hard-Talkchess-2020.052,  HTC108   > 15s.
29. Hard-Talkchess-2020.053,  HTC108   Gelöst in 0.06s/11; Gelöst: 19
30. Hard-Talkchess-2020.054,  HTC108   > 15s.
31. Hard-Talkchess-2020.056,  HTC108   Gelöst in 13.30s/33; Gelöst: 20
32. Hard-Talkchess-2020.058,  HTC108   Gelöst in 0.14s/14; Gelöst: 21
33. Hard-Talkchess-2020.059,  HTC108   > 15s.
34. Hard-Talkchess-2020.061,  HTC108   Gelöst in 4.78s/30; Gelöst: 22
35. Hard-Talkchess-2020.065,  HTC108   Gelöst in 0.08s/14; Gelöst: 23
36. Hard-Talkchess-2020.066,  HTC108   Gelöst in 0.05s/10; Gelöst: 24
37. Hard-Talkchess-2020.067,  HTC108   > 15s.
38. Hard-Talkchess-2020.068,  HTC108   Gelöst in 4.66s/40; Gelöst: 25
39. Hard-Talkchess-2020.070,  HTC108   Gelöst in 5.66s/26; Gelöst: 26
40. Hard-Talkchess-2020.072,  HTC108   Gelöst in 0.22s/26; Gelöst: 27
41. Hard-Talkchess-2020.074,  HTC108   Gelöst in 6.28s/31; Gelöst: 28
42. Hard-Talkchess-2020.078,  HTC108   > 15s.
43. Hard-Talkchess-2020.081,  HTC108   > 15s.
44. Hard-Talkchess-2020.083,  HTC108   > 15s.
45. Hard-Talkchess-2020.087,  HTC108   Gelöst in 0.66s/42; Gelöst: 29
46. Hard-Talkchess-2020.089,  HTC108   Gelöst in 0.08s/13; Gelöst: 30
47. Hard-Talkchess-2020.090,  HTC108   Gelöst in 1.47s/44; Gelöst: 31
48. Hard-Talkchess-2020.091,  HTC108   > 15s.
49. Hard-Talkchess-2020.093,  HTC108   > 15s.
50. Hard-Talkchess-2020.094,  HTC108   > 15s.
51. Hard-Talkchess-2020.095,  HTC108   > 15s.
52. Hard-Talkchess-2020.096,  HTC108   Gelöst in 3.23s/36; Gelöst: 32
53. Hard-Talkchess-2020.097,  HTC108   Gelöst in 0.44s/18; Gelöst: 33
54. Hard-Talkchess-2020.099,  HTC108   > 15s.
55. Hard-Talkchess-2020.101,  HTC108   > 15s.
56. Hard-Talkchess-2020.103,  HTC108   Gelöst in 3.97s/30; Gelöst: 34
57. Hard-Talkchess-2020.105,  HTC108   Gelöst in 3.53s/46; Gelöst: 35
58. Hard-Talkchess-2020.108,  HTC108   Gelöst in 1.41s/29; Gelöst: 36
59. Hard-Talkchess-2020.109,  HTC108   Gelöst in 0.86s/32; Gelöst: 37
60. Hard-Talkchess-2020.110,  HTC108   Gelöst in 0.16s/22; Gelöst: 38
61. Hard-Talkchess-2020.113,  HTC108   Gelöst in 2.44s/26; Gelöst: 39
62. Hard-Talkchess-2020.114,  HTC108   Gelöst in 2.58s/48; Gelöst: 40
63. Hard-Talkchess-2020.116,  HTC108   Gelöst in 13.27s/49; Gelöst: 41
64. Hard-Talkchess-2020.117,  HTC108   Gelöst in 16.03s/47; Gelöst: 42
65. Hard-Talkchess-2020.119,  HTC108   Gelöst in 3.11s/40; Gelöst: 43
66. Hard-Talkchess-2020.121,  HTC108   Gelöst in 0.17s/23; Gelöst: 44
67. Hard-Talkchess-2020.122,  HTC108   Gelöst in 0.84s/29; Gelöst: 45
68. Hard-Talkchess-2020.125,  HTC108   Gelöst in 0.13s/19; Gelöst: 46
69. Hard-Talkchess-2020.126,  HTC108   > 15s.
70. Hard-Talkchess-2020.127,  HTC108   Gelöst in 0.34s/20; Gelöst: 47
71. Hard-Talkchess-2020.129,  HTC108   Gelöst in 0.23s/18; Gelöst: 48
72. Hard-Talkchess-2020.130,  HTC108   Gelöst in 0.22s/19; Gelöst: 49
73. Hard-Talkchess-2020.131,  HTC108   Gelöst in 3.11s/35; Gelöst: 50
74. Hard-Talkchess-2020.132,  HTC108   Gelöst in 0.81s/25; Gelöst: 51
75. Hard-Talkchess-2020.133,  HTC108   Gelöst in 1.28s/29; Gelöst: 52
76. Hard-Talkchess-2020.135,  HTC108   Gelöst in 6.55s/32; Gelöst: 53
77. Hard-Talkchess-2020.140,  HTC108   Gelöst in 0.08s/14; Gelöst: 54
78. Hard-Talkchess-2020.146,  HTC108   Gelöst in 1.02s/18; Gelöst: 55
79. Hard-Talkchess-2020.147,  HTC108   Gelöst in 1.34s/22; Gelöst: 56
80. Hard-Talkchess-2020.153,  HTC108   > 15s.
81. Hard-Talkchess-2020.155,  HTC108   Gelöst in 0.72s/21; Gelöst: 57
82. Hard-Talkchess-2020.156,  HTC108   > 15s.
83. Hard-Talkchess-2020.158,  HTC108   > 15s.
84. Hard-Talkchess-2020.159,  HTC108   Gelöst in 0.09s/13; Gelöst: 58
85. Hard-Talkchess-2020.160,  HTC108   Gelöst in 0.08s/12; Gelöst: 59
86. Hard-Talkchess-2020.164,  HTC108   Gelöst in 2.98s/26; Gelöst: 60
87. Hard-Talkchess-2020.166,  HTC108   Gelöst in 8.31s/26; Gelöst: 61
88. Hard-Talkchess-2020.170,  HTC108   Gelöst in 3.03s/32; Gelöst: 62
89. Hard-Talkchess-2020.171,  HTC108   Gelöst in 2.30s/27; Gelöst: 63
90. Hard-Talkchess-2020.179,  HTC108   > 15s.
91. Hard-Talkchess-2020.181,  HTC108   Gelöst in 0.55s/21; Gelöst: 64
92. Hard-Talkchess-2020.182,  HTC108   Gelöst in 3.08s/26; Gelöst: 65
93. Hard-Talkchess-2020.183,  HTC108   Gelöst in 0.66s/18; Gelöst: 66
94. Hard-Talkchess-2020.184,  HTC108   Gelöst in 0.11s/14; Gelöst: 67
95. Hard-Talkchess-2020.186,  HTC108   Gelöst in 0.11s/15; Gelöst: 68
96. Hard-Talkchess-2020.190,  HTC108   Gelöst in 13.20s/32; Gelöst: 69
97. Hard-Talkchess-2020.191,  HTC108   Gelöst in 1.08s/20; Gelöst: 70
98. Hard-Talkchess-2020.194,  HTC108   Gelöst in 0.06s/12; Gelöst: 71
99. Hard-Talkchess-2020.195,  HTC108   Gelöst in 1.50s/20; Gelöst: 72
100. Hard-Talkchess-2020.196,  HTC108   > 15s.
101. Hard-Talkchess-2020.198,  HTC108   Gelöst in 3.27s/29; Gelöst: 73
102. Hard-Talkchess-2020.200,  HTC108   Gelöst in 2.92s/25; Gelöst: 74
103. Hard-Talkchess-2020.203,  HTC108   Gelöst in 3.13s/28; Gelöst: 75
104. Hard-Talkchess-2020.208,  HTC108   Gelöst in 0.06s/10; Gelöst: 76
105. Hard-Talkchess-2020.209,  HTC108   Gelöst in 1.63s/26; Gelöst: 77
106. Hard-Talkchess-2020.210,  HTC108   Gelöst in 1.78s/26; Gelöst: 78
107. Hard-Talkchess-2020.211,  HTC108   > 15s.
108. Hard-Talkchess-2020.213,  HTC108   > 15s.

With option Tal checked 88/108 were solved with same conditions and hardware, see here:

forum3/viewtopic.php?p=918584#p918584

Statistically more significant is a suite, I made out of HTC108+Eret+20 positions added out of my problem- database, because 19 positions are doublets between HTC108 and Eret. So 222 positions all together, "new" as for (to me) known suites are only these 20:

Code: Select all

6k1/p1p1p1p1/P5Pq/7P/8/5R2/1P1P4/5K1B w - - bm Rf5; id "Stadlhofer Anton, 1";
3q3N/1p5k/5B2/8/1p2p1Rp/1P5K/p3PP2/8 w - - bm Rg7+; id "M. Garcia - P. Krug , Studie 2015, 2";
n2Bqk2/5p1p/Q4KP1/p7/8/8/8/8 w - - bm Qc8; id "Matous, Mario, 3";
k7/7p/2p5/2rp1PP1/pp6/7p/5K1P/4R3 w - - bm g6; id "Zepler, E.E., 4";
7k/pp1pBp1N/1p3Pp1/b3K1PR/P3P2p/R1n4P/2P1N1P1/3q4 w - - bm Rxc3; id "Josten, Gerhard, 5";
k7/P4pp1/1P6/2p1P3/2P1n3/p5pB/P4n2/K5b1 w - - bm e6; am Bg2; id "Hörning, Gerd, 6";
5N1n/3P1p1p/5P1k/1Pp1P2p/4KP1P/2p5/p4b2/5N2 w - - bm Ne3; id "Formela, Michal , 7";
r1b2rk1/3n1pb1/2p1p1p1/p5P1/Pp2B1NQ/1q4P1/2R2P2/3R2K1 w - - bm Rxd7; id "TCEC 2019, 8";
8/1pp2k2/p2p3p/3n1KP1/5P1P/8/1P1B4/8 w - - bm Ke4; id "Nebenvariante 41.Ke4!, 9";
1r1q1r1k/5pp1/3p1b2/pp1BpPB1/1n2P1QP/1PN5/1PPK4/R7 w - - bm Ne2; id "corr LIPEAD40 (PER) , 10";
r3nrk1/1bqp1ppp/p1n1p3/1p4PQ/3NP3/2N2R2/PPP1B2P/3RK3 w - - bm Nd5; id "Anton Stadlhofer, 11";
r1b2rk1/pp3p2/2p3pp/2n5/8/q3RN2/P4PPP/2RQ1BK1 b - - bm Qa4; id "Chessbase online, 12";
n6r/b7/1R4R1/2k5/2pppppp/8/1PPPPPP1/r4NK1 w - - bm b4+ Rgc6+; id "Korolkov W.A., 13";
r4r2/b2q2pk/3p2np/N2pp3/PP1P4/2P3PP/1B3P2/R3RQK1 b - - bm Rf5; id "Chessbase online , 14";
2r2krb/1qp2p2/2NPpN2/1P5B/p2Bp2p/1p5b/7p/R3K2Q w Q - bm Nh7+; id "Krug P., Garcia M.., 15";
8/7P/1P5B/2B1Q1n1/3nn2P/1PRnk1nR/3nnnK1/2B1nQBn w - - bm Qxe1; id "Goldsteen H., Horse Connection, 16";
r2n1r1k/1pp5/3ppn2/P2Pp2p/2P1P1pq/1Q2R2P/R2N1PP1/4N1K1 w - - bm g3; id "WC30/final, 17";
2b5/1pr4p/3bp1pk/1p6/1PpN2PP/K1P1n3/P3N1R1/3R4 w - - bm Nxb5; id "Mihai Neghina, 18";
3r2k1/p4pP1/1ppr4/5Bp1/P2qPnQP/4R3/5P2/5RK1 w - - bm e5; id "Tauber-Sikorsky ICCF 2018, 19";
4q1kr/p6p/1prQPppB/4n3/4P3/2P5/PP2B2P/R5K1 w - - bm Qxe5; id "Gusev-Auerbach, Molniya 1946, 20";

Whilst about 90% of Eret- positions are solved by ShashChess with this hardware- time, (Arasan about all of them for sure, if I look at SF dev in this one suite) out of the added 20 ShashChess default solves 15, so they fit with degree of difficulty to HTC108 and Eret to me as for hardware- time like the one I use.

Some runs with different engines and settings and nets I collect since quite a while in a chessbase- file of solutions of this suite, .cbh- files can be evaluated counting solution- times per position too by Frank Schubert's program EloStatTS (Test Suite).

https://glarean-magazin.ch/wp-content/u ... bert-1.pdf

From here

https://glarean-magazin.ch/2017/03/05/c ... test-eret/

the direct download- link:

https://glarean-magazin.ch/wp-content/u ... S-104-1.7z

So evaluated some older and newer results, the latest ones are the upper three ShashChess 20.2- trials, one with MCTS1, one with MCTS20(threads), both with Tal checked, and one default.

Code: Select all

    Program                                    Elo   +/-  Matches  Score   Av.Op.   S.Pos.   MST1    MST2   RIndex

  1 ShashChess20.2-MCTS1-Tal                 : 3566    4   4001    60.4 %   3493   197/222    2.2s    3.7s   0.85
  2 ShashChess20.2-Tal-MCTS20                : 3561    4   3923    59.6 %   3494   191/222    2.1s    3.9s   0.84
  3 ShashChess20.2                           : 3551    4   3825    58.0 %   3495   183/222    2.1s    4.4s   0.80
  4 Crystal251221                            : 3551    5   3767    57.8 %   3496   175/220    1.7s    4.4s   0.80
  5 ShashChess20.1-nn4401e826ebcc            : 3550    4   3849    57.8 %   3495   183/220    2.5s    4.6s   0.80
  6 CorChess3271221                          : 3533    4   3669    55.3 %   3496   169/220    2.4s    5.3s   0.77
  7 Stockfish+030122-MCTS                    : 3525    4   3597    53.9 %   3497   166/220    2.8s    5.8s   0.73
  8 Stockfish+030122                         : 3521    4   3589    53.4 %   3498   163/222    2.8s    6.0s   0.74
  9 Stockfish221221                          : 3515    4   3519    52.3 %   3499   157/220    2.7s    6.3s   0.71
 10 Stockfish171221-nn4401e826ebcc           : 3509    5   3512    51.4 %   3499   155/220    2.9s    6.5s   0.70
 11 Lc0v0.28.2-780486                        : 3508    5   3515    51.1 %   3500   150/222    2.3s    6.4s   0.67
 12 Dragon2.6byKomodoChess64-bit             : 3508    5   3496    51.1 %   3500   152/220    2.9s    6.6s   0.69
 13 Lc0v0.28.2-NN780084                      : 3506    5   3469    50.7 %   3501   144/220    2.0s    6.5s   0.67
 14 Lc0v0.28.2-NN610902                      : 3500    5   3456    49.8 %   3501   140/220    2.0s    6.7s   0.65
 15 Ceres0.94d-610902                        : 3469    5   3309    44.9 %   3505   122/220    2.2s    7.9s   0.55
 16 RubiChess2021NN-fb50f(avx2)              : 3469    5   3272    44.9 %   3504   124/220    2.6s    8.0s   0.58
 17 Ceres0.94d-780084                        : 3466    5   3283    44.4 %   3505   119/220    2.1s    8.0s   0.56
 18 Wasp5.20                                 : 3428    6   3171    38.6 %   3508   101/220    2.7s    9.4s   0.41
 19 DeepHIARCS15.0                           : 3412    6   3178    36.4 %   3509    97/222    3.5s   10.0s   0.34
 20 Fritz15                                  : 3398    6   3166    34.4 %   3510    87/222    3.4s   10.5s   0.33
 21 DarkToga1.1                              : 3377    6   3104    31.6 %   3511    73/222    2.8s   11.0s   0.30



MST1  : Mean solution time (solved positions only)
MST2  : Mean solution time (solved and unsolved positions)
RIndex: Score according to solution time ranking for each position

I've stored the whole chessbase- text- file of solutions of course too, but I guess boundary of signs is already soon to be reached for one posting.
One further advantage of EloStatTS is, the more engine- runs added, the lower the error bar gets. Every added run gets ratings of its own, as well as for the new ones as for the already before rated ones, because whole .pgn of positions and times of solutions is evaluated all over again with each new entry.

[peter]

...because 19 positions are doublets between HTC108 and Eret.

Typo found after edit- time over. 17 doublets, not 19.

[Frank Quisinsky]

Hi Jouni,

yes, at the moment Klaus and myself are working on an easy system for the rating idea in combination with a still-running tourney.
In reality it's not esay to find a way to display all the information for visitors with a simply way.

But we are working on it.
Today we open a new "Excel" selection!
Excel is in the future very important for the new idea around Ki-Ratings on my site.

But if we are ready with the work around the webpage and all is still running a longer time the people will like that in old html.
Hope so ...

To your question:
Yes, the complete CPU ratings are 1-core with 40 moves in 20 minutes (repeatedly) on 4.4Ghz machines.
One game need around ~ 1hour and 20 minutes (move average without resign mode is around 89-90).

And yes ... Lc0 0.28.2 GPU results are produced with a RTX-3070 Ti graphic card with full power vs. 1-Core engines!

Can be see on the detail site:
https://www.amateurschach.de/main/_fcp-tourney-ki.htm (the complete conditions).

If the Lc0 round-robin is ready other GPU engines will be follow ... Scorpio, others!
But the round-robin need a while ...

Best
Frank

I will have this one ...
https://2700chess.com/
For KI-Engines in combination with a still running tourney, Excel and so one and most important ...
In combination with information about playing styles of engines!!

So you have to know ... a lot of work.

[Jouni]

Peter: yes arasan suite is too easy to detect parameter change. And some positions seems to be incorrect I quess. With HTC and other suites MCTS/Tal is better.