Testing LazySMP

[supernova]

Thank you for sharing, Arthur! This is a minor bug.

I am curious to know how you classify a bug as minor when your engine is losing by time. I believe it is a major issue, and what you refer implies as minor, is the fix.

Time loss makes an engine untestable because the results become invalid.

I think you should also fix the smp issue since your engine is using only one thread. You mentioned it was smp what it is not currently.

To be honest, I may not release another version of the engine. Currently, there is no motivation to develop engine in a state that CCRL members don't want to test it and no effect if adding new feature. Please see my post on CCRL forum: https://kirill-kryukov.com/chess/discus ... hp?t=17166

I wish to point out a few inconsistencies I noticed in your previous message.

Chess computer rating lists are communities willing to test hundreds of engines and resources are not infinite. They have their own policies and cadence and we should thank them that they do this work for the computer chess community. While there may be imperfections, the rating serves as a helpful reference.

I should also mention that you've gotten great feedback from forum members (Brunetti, Tibono, hgm, chesskobra, etc.) regarding bug reports and suggestions. They're not necessarily members of the computer rating lists, but they'll help you anyway. While your engine is waiting for chess rating list testing, consider this a very positive approach.

Unfortunately, your previous message also invited me strongly to discontinue my testing and CPU usage time with your engine, as I find your attitude extremely unreasonable. I encourage you to develop a testing framework for your engine and address known bugs promptly, without relying solely on users to identify and report these issues.

Best regards.

[LazySMP]

I am curious to know how you classify a bug as minor when your engine is losing by time. I believe it is a major issue, and what you refer implies as minor, is the fix. Time loss makes an engine untestable because the results become invalid.

I noticed you shared several games while only one game is visible to others. I said that it is a minor bug related to your first game, which has more than 128 moves, because it can be fixed by changing the array value.

But for other games where time loss occurs, there is a time delay of milliseconds due to network and GUI overheads, which causes to losses on time in these cases. For example, I checked one of your other games:

Code: Select all

LazySMP Version 3.0
by Daniel Pierce
position startpos moves d2d4 g8f6 c2c4 d7d6 b1c3 c8f5 f2f3 e7e5 e2e4 e5d4 d1d4 b8c6 d4d1 f5e6 g1h3 e6h3 g2h3 f6d7 c1e3 f8e7 f1e2 e7h4 e1f1 f7f5 e4f5 e8g8 d1d5 g8h8 h1g1 c6e7 d5b7 e7f5 c3d1 g7g6 e3a7 c7c5 e2d3 d7e5 d3f5 f8f5 f1e2 f5f7 a7b6 f7b7 b6d8 a8d8 b2b3 e5c6 e2f1 h4f6 a1c1 f6d4 g1g4 c6e5 g4e4 e5f3 f1g2 f3g5 e4g4 g5f7 g4e4 f7e5 d1e3 d8f8 c1c2 d4e3 e4e3 e5c6 e3e4 c6d4 c2b2 d4f5 b2d2 h8g7 d2d5 b7a7 d5d2 f8a8 e4e2 f5d4 e2f2 d4f5 f2e2 f5d4 e2f2 a8e8 g2g1 e8e1 g1g2 e1e4 d2d1 a7e7 g2f1 e4e3 f1g2 e3e2 d1f1 e2f2 f1f2 e7a7
go wtime 1000 btime 10000 winc 2000 binc 2000
info depth 1 seldepth 1 score cp -269 nodes 20 nps 20000 time 1 pv f2f7
info depth 2 seldepth 2 score cp -357 nodes 127 nps 63500 time 2 pv a2a4 a7a4
info depth 3 seldepth 3 score cp -266 nodes 214 nps 53500 time 4 pv h3h4 h7h5 f2f7
info depth 4 seldepth 4 score cp -353 nodes 375 nps 75000 time 5 pv h3h4 h7h5 a2a4 a7a4
info depth 5 seldepth 6 score cp -263 nodes 1649 nps 235571 time 7 pv f2d2 d4e6 d2f2 a7e7 f2f7
info depth 6 seldepth 9 score cp -181 nodes 4355 nps 483888 time 9 pv f2d2 d4c6 d2d6 a7a2 g2g3 c6e5 d6g6
info depth 7 seldepth 10 score cp -278 nodes 15778 nps 1213692 time 13 pv f2b2 d4f5 g2g1 a7e7 b2d2 e7e1 g1g2 f5e3 g2g3
info depth 8 seldepth 11 score cp -272 nodes 21021 nps 1313812 time 16 pv f2b2 a7f7 b3b4 c5b4 b2b4 f7e7 b4b2 e7e3
info depth 9 seldepth 11 score cp -267 nodes 36678 nps 1746571 time 21 pv f2b2 d4f5 b2e2 a7d7 g2f2 d7f7 f2g2 f5h4 g2g3 h4f5 g3g2
info depth 10 seldepth 13 score cp -274 nodes 64463 nps 2148766 time 30 pv f2d2 d4f5 g2f3 a7e7 d2d3 h7h5 f3f2 e7f7 f2g2 f5e3
....
info depth 18 seldepth 23 score cp -276 nodes 4698830 nps 5041663 time 932 pv f2b2 g6g5 b2d2 a7e7 g2f1 h7h5 b3b4 d4f5 b4c5 d6c5 d2e2 e7e2 f1e2 g5g4 h3g4 h5g4 e2d2
bestmove f2b2 ponder g6g5

In this case, If the time delay of the network and GUI overhead is more than 70ms, it will cause time loss. If you use a better GUI, this will never happen.

I should also mention that you've gotten great feedback from forum members (Brunetti, Tibono, hgm, chesskobra, etc.) regarding bug reports and suggestions. They're not necessarily members of the computer rating lists, but they'll help you anyway. While your engine is waiting for chess rating list testing, consider this a very positive approach.

Unfortunately, there are many people (lucametehau, noobpwnftw, RubiChess, Guenther, etc.) who do not want my engine to be tested for chess rating list. As Mr. Gabor mentioned, he does not want to test my engine. This is a negative approach.

[supernova]

I noticed you shared several games while only one game is visible to others. I said tat it is a minor bug related to your first game, which has more than 128 moves, because it can be fixed by changing the array value.

But for other games where time loss occurs, there is a time delay of milliseconds due to network and GUI overheads, which causes to losses on time in these cases. For example, I checked one of your other games:

I have not experienced losses by time with other engines. The data I provided is specifically a sample of time losses from your engine. Additionally, I noticed that you do not have an UCI option for time overhead, which some other engines do offer.

I would share the complete PGN once the tests are finished, but at this point, I do not see a compelling reason to do so.

Unfortunately, there are many people (lucametehau, noobpwnftw, RubiChess, Guenther, etc.) who do not want my engine to be tested for chess rating list. As Mr. Gabor mentioned, he does not want to test my engine. This is a negative approach.

I’ve noticed that you tend to point out individuals, which I believe is not the most constructive approach. It seems that there tends to be a tendency to deflect responsibility onto other factors or individuals.

As for the rating list, I have already shared my thoughts with you. Wishing you the best in your endeavors.

PD: I was still running a long test...

Code: Select all

    Engine                                              Score               La
01: LazySMP 3.0 6t                                      125.0/506 ············ 
02: Aramis 1.40                                         12.0/12   111111111111 
02: EveAnn 1.73                                         12.0/12   111111111111 
02: Heimdall 1.1.1                                      12.0/12   111111111111 
02: Hermann 2.8                                         12.0/12   111111111111 
02: Lc0.cuda 0.31.1 maia-1300 6t1024h                   12.0/12   111111111111 
02: Lc0.cuda 0.31.1 maia-1400 6t1024h                   12.0/12   111111111111 
02: Lc0.cuda 0.31.1 maia-1500 6t1024h                   12.0/12   111111111111 
02: Lc0.cuda 0.31.1 maia-1600 6t1024h                   12.0/12   111111111111 
02: Lc0.cuda 0.31.1 maia-1700 6t1024h                   12.0/12   111111111111 
02: Lc0.cuda 0.31.1 maia-1800 6t1024h                   12.0/12   111111111111 
02: Lc0.cuda 0.31.1 maia-1900 6t1024h                   12.0/12   111111111111 
02: Lc0.cuda 0.31.1 Mean-girl.7 6t1024h                 12.0/12   111111111111 
02: Lc0.cuda 0.31.1 Mean-girl.8 6t1024h                 12.0/12   111111111111 
02: Nalwald 19                                          12.0/12   111111111111 
02: ProDeo 1.1                                          12.0/12   111111111111 
02: ProDeo 2.1 Agressive                                12.0/12   111111111111 
02: ProDeo 2.1 Strong-Club-Player                       12.0/12   111111111111 
19: Lc0.cuda 0.31.1 maia-1200 6t1024h                   11.5/12   111111=11111 
19: ProDeo 1.1 AntiGM.3-Playingstyle.3                  11.5/12   1111111111=1 
21: ProDeo 2.1 Average-Club-Player                      11.0/12   111101111111 
21: EveAnn 2.1nn                                        11.0/11   11111111111  
21: Rebel Mephisto-Gideon Style.Aggressive-AntiGM.Smart 11.0/11   11111111111  
21: Tcheran 4.0                                         11.0/11   11111111111  
25: AnMon 5.75                                          10.5/12   =11011111111 
25: ProDeo 1.1 Strong-Club-Player                       10.5/12   01111111=111 
25: Nejmet 3.07                                         10.5/12   1=111=11111= 
28: Rebel Mephisto-Gideon                               10.0/11   11101111111? 
28: Lc0.cuda 0.31.1 maia-1100 6t1024h                   10.0/12   111101110111 
28: ProDeo 1.1 Down                                     10.0/12   011011111111 
31: Dragon 4.6                                          9.5/12    111=11001111 
32: Joker 1.1.14                                        9.0/12    010111110111 
33: MangoPaola 1.0                                      8.0/12    001111==1110 
34: Rival 0094-ja                                       5.5/11    1=100101100  
35: Gullydeckel-2.16.pl2                                4.5/12    =00100110100 
35: Toad 1.0                                            4.5/11    0010=01=01=  
37: Colchess 8.0-ja                                     4.0/12    100000100011 
38: micro-Max 4.8                                       3.0/12    001=0010000= 
39: Skaki 1.23-ja                                       0.5/11    00=00000000  
40: Belofte 2.1.8.1                                     0.0/12    000000000000 
40: RoboKewlper 0.047a                                  0.0/11    00000000000  
40: ProDeo 1.1 Absolute-Novice-Player                   0.0/12    000000000000 
40: Tscp 181e-32-ja                                     0.0/11    00000000000  
40: StAndersen 1.31c-rev2-ja                            0.0/11    00000000000  

[hgm]

To be honest, I may not release another version of the engine. Currently, there is no motivation to develop engine in a state that CCRL members don't want to test it and no effect if adding new feature. Please see my post on CCRL forum: https://kirill-kryukov.com/chess/discus ... hp?t=17166

Well, remarks like this are certainly very effective for discouraging CCRL, as well as others to test your engine. You should realize that having your engine being tested by CCRL is a favor, and not a right that anyone could demand. One cannot extort favors, and the more you push people, the less likely it will become that they ever grant you one.

I don't understand your obsession with CCRL anyway. Many testers contribute their test results to CCRL, but there are many more engine versions than testers (perhaps hundreds of times as many), so even if they would test your engine, it would be extremely slow compared to what you can do yourself. The rating list they produce might be nice for getting an occasional calibration point for how you are doing compared to other engines, but is is completely useless for testing whether individual patches on your engine work well and are improvements. Turnaround is just much to slow for that; you would have to wait for months to get a result that you could yourself have produced overnight.

The main importance of the CCRL is that by playing your engine against some others that are in the list, the relative ratings you get from that can be translated to the CCRL scale because you have the ratings of the opponents as calibration.

[smatovic]

[....]
The main importance of the CCRL is that by playing your engine against some others that are in the list, the relative ratings you get from that can be translated to the CCRL scale because you have the ratings of the opponents as calibration.

+1

It is nice to know where you are ranked in the CCRL pool with others, but as you said, useless for own engine progress/development.

The STS v3 tool has an CCRL Elo rating function, but meanwhile off by ~100 to ~150 Elo points:

STS test suite and engine analysis interface
forum/viewtopic.php?t=56653

Re: Any Testsuites in EPD format you can recommend?
forum/viewtopic.php?p=967976#p967976

--
Srdja

[Graham Banks]

Unfortunately, there are many people (lucametehau, noobpwnftw, RubiChess, Guenther, etc.) who do not want my engine to be tested for chess rating list. As Mr. Gabor mentioned, he does not want to test my engine. This is a negative approach.

If you can reach a rough rating of 2450 Elo, plus make your engine open source giving credit where it's due. I'll consider testing it.

[Rebel]

Unfortunately, there are many people (lucametehau, noobpwnftw, RubiChess, Guenther, etc.) who do not want my engine to be tested for chess rating list. As Mr. Gabor mentioned, he does not want to test my engine. This is a negative approach.

If you can reach a rough rating of 2450 Elo, plus make your engine open source giving credit where it's due. I'll consider testing it.

It's between 1800-2000.

[hgm]

Unfortunately, there are many people (lucametehau, noobpwnftw, RubiChess, Guenther, etc.) who do not want my engine to be tested for chess rating list. As Mr. Gabor mentioned, he does not want to test my engine. This is a negative approach.

If you can reach a rough rating of 2450 Elo, plus make your engine open source giving credit where it's due. I'll consider testing it.

Open source? Is that a new CCRL requirement or just your own interest? I remember you tested Joker, the source of which I never published.

[Graham Banks]

Unfortunately, there are many people (lucametehau, noobpwnftw, RubiChess, Guenther, etc.) who do not want my engine to be tested for chess rating list. As Mr. Gabor mentioned, he does not want to test my engine. This is a negative approach.

If you can reach a rough rating of 2450 Elo, plus make your engine open source giving credit where it's due. I'll consider testing it.

Open source? Is that a new CCRL requirement or just your own interest? I remember you tested Joker, the source of which I never published.

I think that it is a reasonable request, considering all the flak that he has been taking.

[hgm]

I am not sure I understand that. When you are the target of a lynching mob, you must publish your source code???