SF not that good at short TC, Komodo very strong

xr_a_y · Post by **xr_a_y** » Sun Feb 23, 2020 10:17 am

Yesterday I did a Minic vs SF test at suddendeath 10s+0.25 and was surprised by MInic performance, only -150Elo.
In fact this is probably a SF counter performance because of TC management. SF is consuming many time for the middle game and only live on increment in the end-game. So there was two type of games, one were SF find an attacking line and crushed Minic soon, other were SF had to struggle to hold end-game.

I decided to run another tourney with more stronger engines at 10s+0.25 and here is the results

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   1 komodo-10-linux               246      12    3372   80.5%   23.5%
   2 Defenchess_2.2                101      10    3371   64.2%   33.1%
   3 stockfish                      -3      10    3371   49.6%   27.5%
   4 igel-last                    -130      10    3372   32.2%   30.2%
   5 minic_1.44                   -204      11    3372   23.6%   26.8%

As Komodo succeed to keep its rank and Elo gap at this TC, I wonder if this is not an error in SF short TC management ?

Raphexon · Post by **Raphexon** » Sun Feb 23, 2020 10:23 am

SF10-11 has had a few patches that are ELO negative on short TC but ELO positive at longer TC.

Although SF shouldn't be weaker than defencechess.

xr_a_y · Post by **xr_a_y** » Sun Feb 23, 2020 10:28 am

Look at time management, here for example

Code: Select all

[Event "My Tournament"]
[Site "?"]
[Date "2020.02.22"]
[Round "10"]
[White "minic_1.44"]
[Black "stockfish"]
[Result "1/2-1/2"]
[ECO "A01"]
[GameDuration "00:01:05"]
[GameEndTime "2020-02-22T09:43:57.396 CET"]
[GameStartTime "2020-02-22T09:42:52.276 CET"]
[Opening "Nimzovich-Larsen attack"]
[PlyCount "152"]
[TimeControl "20+0.25"]
[Variation "Classical Variation"]

1. b3 {book} d5 {book} 2. Nf3 {book} Nf6 {book} 3. Bb2 {book} e6 {book}
4. c4 {book} Be7 {book} 5. g3 {book} O-O {book} 6. Bg2 {book} c5 {book}
7. Ne5 {book} Nc6 {book} 8. f4 {book} Nd7 {book} 9. Nxc6 {book} bxc6 {book}
10. Qc2 {book} Rb8 {book} 11. O-O {book} Ba6 {book} 12. d3 {book} Bf6 {book}
13. Bxf6 {+0.41/17 1.2s} Nxf6 {-0.38/16 1.2s} 14. cxd5 {+0.04/17 0.93s}
c4 {-0.66/20 4.6s} 15. bxc4 {+0.10/16 0.76s} exd5 {-0.55/18 1.6s}
16. Nd2 {+0.17/17 1.0s} Ng4 {-0.69/19 3.2s} 17. Rfe1 {-0.19/16 0.91s}
Qf6 {-0.23/18 2.1s} 18. Nf3 {+0.06/15 0.59s} dxc4 {-0.17/18 2.0s}
19. dxc4 {+0.49/14 0.66s} Rb4 {+0.14/17 1.00s} 20. Ng5 {+0.68/15 0.63s}
g6 {-0.57/17 1.1s} 21. e3 {+0.62/15 0.89s} Rb2 {-0.35/17 1.5s}
22. Qd1 {+0.55/16 0.85s} h5 {0.00/19 1.3s} 23. Rb1 {+0.82/15 0.52s}
Rd8 {0.00/16 0.21s} 24. Ne4 {+0.47/18 0.80s} Qe7 {+0.33/20 0.87s}
25. Qxd8+ {-0.01/15 0.43s} Qxd8 {+0.24/17 0.73s} 26. Rxb2 {+0.01/19 0.76s}
Qa5 {+0.34/19 1.2s} 27. Rbb1 {+0.01/19 0.55s} Bxc4 {+0.60/17 0.50s}
28. Rbc1 {-0.01/18 0.71s} Qxa2 {+0.42/16 0.52s} 29. h3 {-0.21/18 0.69s}
Nh6 {+0.60/16 0.30s} 30. Ra1 {-0.46/14 0.41s} Qc2 {+0.76/16 0.26s}
31. Rac1 {-0.33/14 0.46s} Qa2 {+0.79/15 0.25s} 32. Ra1 {-0.37/14 0.36s}
Qb2 {+0.77/15 0.25s} 33. Reb1 {-0.40/15 0.63s} Qc2 {+0.76/16 0.25s}
34. Rc1 {-0.46/16 0.53s} Qd3 {+0.83/17 0.25s} 35. Nf2 {-0.39/14 0.37s}
Qe2 {+0.59/16 0.25s} 36. Re1 {-0.39/14 0.58s} Qb2 {+0.64/14 0.19s}
37. Rab1 {-0.61/13 0.34s} Qf6 {+0.83/15 0.31s} 38. Ne4 {-0.59/13 0.35s}
Qe6 {+1.11/14 0.24s} 39. Rb7 {-0.36/15 0.55s} a5 {+1.02/14 0.26s}
40. Ra7 {-0.63/13 0.38s} Bd5 {+1.04/16 0.25s} 41. Ng5 {-0.74/14 0.30s}
Qd6 {+1.09/15 0.25s} 42. Bxd5 {-0.64/14 0.52s} Qxd5 {+0.80/15 0.25s}
43. Rb1 {-0.79/14 0.51s} Qd2 {+0.91/14 0.25s} 44. Rb8+ {-0.89/16 0.34s}
Kg7 {+2.18/17 0.20s} 45. Re8 {-0.44/16 0.49s} Qc1+ {+0.49/16 0.30s}
46. Kf2 {-0.19/17 0.39s} Qd2+ {+0.31/18 0.25s} 47. Kf1 {-0.12/17 0.29s}
Qd3+ {0.00/18 0.25s} 48. Kf2 {-0.12/18 0.46s} a4 {0.00/19 0.080s}
49. Ne6+ {-0.01/20 0.26s} Kh7 {0.00/21 0.21s} 50. Ng5+ {-0.01/21 0.28s}
Kg7 {+0.58/1 0s} 51. Ne6+ {-0.01/21 0.28s} Kf6 {+0.66/21 0.34s}
52. Ng5 {-0.01/21 0.29s} Qd2+ {+0.32/21 0.55s} 53. Kf1 {-0.01/20 0.28s}
Qd3+ {+0.85/18 0.15s} 54. Kf2 {-0.01/20 0.25s} Qd2+ {+0.55/21 0.42s}
55. Kf1 {-0.01/20 0.26s} Qc1+ {+0.68/19 0.25s} 56. Ke2 {-0.01/20 0.42s}
Qb2+ {+0.27/19 0.25s} 57. Kf3 {-0.01/18 0.43s} Qb1 {0.00/19 0.12s}
58. Kf2 {-0.09/19 0.42s} Qb2+ {0.00/21 0.14s} 59. Kf3 {-0.01/19 0.24s}
Qb6 {0.00/21 0.11s} 60. Ree7 {-0.01/16 0.41s} Qb3 {0.00/17 0.35s}
61. Re8 {-0.01/19 0.24s} Kg7 {0.00/23 0.36s} 62. Ne6+ {-0.01/22 0.33s}
Kf6 {0.00/18 0.37s} 63. Ng5 {-0.01/18 0.36s} Qd5+ {0.00/22 0.098s}
64. Kf2 {-0.01/19 0.26s} Qd3 {0.00/22 0.12s} 65. Rh8 {-0.01/19 0.23s}
h4 {0.00/22 0.26s} 66. Rxh6 {-0.01/20 0.32s} hxg3+ {0.00/22 0.52s}
67. Kf3 {+0.01/25 0.27s} Qd1+ {0.00/16 0.28s} 68. Kxg3 {+0.01/27 0.26s}
Qg1+ {0.00/25 0.27s} 69. Kf3 {-0.01/30 0.26s} Qf1+ {0.00/25 0.25s}
70. Kg3 {+0.01/29 0.27s} Qe1+ {0.00/36 0.25s} 71. Kh2 {-0.01/26 0.22s}
Qf2+ {0.00/34 0.067s} 72. Kh1 {-0.01/31 0.21s} Qf1+ {0.00/21 0.34s}
73. Kh2 {-0.01/32 0.26s} Qe2+ {0.00/28 0.34s} 74. Kh1 {+0.01/36 0.30s}
Qf1+ {0.00/20 0.25s} 75. Kh2 {+0.01/34 0.22s} Qe2+ {0.00/27 0.25s}
76. Kh1 {+0.01/34 0.39s} Qf1+ {0.00/19 0.25s, Draw by 3-fold repetition} 1/2-1/2

4 sec at move 14 ! 3sec at move 16

Alayan · Post by **Alayan** » Sun Feb 23, 2020 12:47 pm

I'm skeptical. Bullet TC is typical of fishtest conditions, such a massive regression would never have went unnoticed.

I ran games with asymmetric TC between SF and Ethereal, and SF with 20s+0.2s was demolishing Ethereal with 50+0.5s.

SF with 60s+0.6s tops the FastGM bullet list.

For me the most likely explanation is that something is wrong in how you compiled or set up SF.

xr_a_y · Post by **xr_a_y** » Sun Feb 23, 2020 12:54 pm

I didn't touch default parameters, maybe something are not well suited to this TC.
What Move overhead shall I use, what "slow mover"
And why the hell my UCI_LimitStrenght was activated by default ...

Ok error is on my side probably. Let me try again !

Alayan · Post by **Alayan** » Sun Feb 23, 2020 12:56 pm

What is the version you used ?

xr_a_y · Post by **xr_a_y** » Sun Feb 23, 2020 12:57 pm

git master

But maybe I played with my cutechess params some time ago and didn't remember, I reset UCI config.

xr_a_y · Post by **xr_a_y** » Sun Feb 23, 2020 2:20 pm

Ok my bad: the strenght limit was activated. and in fact SF is crazy strong at short TC !!!

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   1 stockfish                     295      41     285   84.6%   24.6%
   2 komodo-10-linux                91      36     284   62.9%   25.0%
   3 Defenchess_2.2                -24      35     285   46.5%   24.9%
   4 minic_1.44                   -469      69     286    6.3%    8.4%

570 of 6000 games finished.

Alayan · Post by **Alayan** » Sun Feb 23, 2020 2:28 pm

That looks better. When getting strange results, most of the time it's because of an unnoticed mistake.

SF not that good at short TC, Komodo very strong

SF not that good at short TC, Komodo very strong

Re: SF not that good at short TC, Komodo very strong

Re: SF not that good at short TC, Komodo very strong

Re: SF not that good at short TC, Komodo very strong

Re: SF not that good at short TC, Komodo very strong

Re: SF not that good at short TC, Komodo very strong

Re: SF not that good at short TC, Komodo very strong

False alarm, my bad (SF very strong at short TC)

Re: SF not that good at short TC, Komodo very strong