Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

[voffka]

Hello everyone,

I am trying to hunt down some issues in Igel (CCRL 40/40 ~2800 elo) when running games at a very quick time control, and one of my ideas was to measure the "time to depth" in Igel as well as see if I should improve evaluation or search.

So I decided to run a few games between Stockfish 10 and Igel 1.8.3 at a fixed depth=10 and the results are very surprising to me:

Code: Select all

tc=inf
depth=10
Score of Igel 1.8.3 64 POPCNT vs Stockfish 10 64 POPCNT: 105 - 79 - 45  [0.557] 229
Elo difference: 39.62 +/- 40.67

How come the performance of Stockfish is ~39 elo worse than Igel at a fixed depth=10? Here is an example of one game:

Code: Select all

[Event "?"]
[Site "?"]
[Date "2019.07.31"]
[Round "1"]
[White "Igel 1.8.3 64 POPCNT"]
[Black "Stockfish 10 64 POPCNT"]
[Result "1-0"]
[ECO "B04"]
[Opening "Alekhine's defense"]
[PlyCount "110"]
[Termination "adjudication"]
[TimeControl "inf"]
[Variation "Modern, Fianchetto Variation"]

1. e4 {book} Nf6 {book} 2. e5 {book} Nd5 {book} 3. d4 {book} d6 {book}
4. Nf3 {book} g6 {book} 5. Bc4 {book} Nb6 {book} 6. Bb3 {book} Bg7 {book}
7. Ng5 {book} e6 {book} 8. Qf3 {book} Qe7 {book} 9. Ne4 {+0.15/10 1.1s}
O-O {-0.52/10 0.021s} 10. Nf6+ {+0.70/10 0.34s} Kh8 {-0.05/10 0.004s}
11. Bg5 {+0.66/10 0.73s} h6 {-1.17/10 0.015s} 12. Qh3 {+1.96/10 0.87s}
N8d7 {-0.52/10 0.020s} 13. Ne4 {+1.85/10 0.58s} Qe8 {-3.75/10 0.008s}
14. Bxh6 {+2.67/10 0.66s} Kg8 {-3.15/10 0.004s} 15. exd6 {+2.70/10 0.44s}
Bxd4 {-4.12/10 0.011s} 16. Nbc3 {+3.10/10 0.43s} Nd5 {-4.16/10 0.029s}
17. dxc7 {+3.57/10 1.9s} Nxc3 {-4.98/10 0.015s} 18. Nxc3 {+3.69/10 0.26s}
Bg7 {-5.06/10 0.016s} 19. Bxg7 {+3.96/10 0.98s} Kxg7 {-4.31/10 0.012s}
20. Ne4 {+3.92/10 0.50s} Qe7 {-4.68/10 0.004s} 21. O-O-O {+3.79/10 0.57s}
Nf6 {-4.31/10 0.002s} 22. Qc3 {+3.95/10 0.48s} e5 {-4.55/10 0.006s}
23. Nxf6 {+4.00/10 0.54s} Qxf6 {-5.03/10 0.017s} 24. Rhe1 {+4.00/10 0.003s}
Re8 {-5.00/10 0.022s} 25. Rd5 {+3.88/10 0.69s} e4 {-5.20/10 0.009s}
26. Rd4 {+4.15/10 1.3s} Bf5 {-4.12/10 0.009s} 27. h3 {+3.43/10 1.0s}
Rac8 {-4.27/10 0.011s} 28. g4 {+3.53/10 0.15s} Be6 {-4.23/10 0.007s}
29. Rexe4 {+3.73/10 0.58s} Re7 {-3.43/10 0.007s} 30. Rf4 {+3.41/10 0.89s}
Rexc7 {-3.65/10 0.004s} 31. Rxf6 {+3.33/10 0.23s} Rxc3 {-3.37/10 0.001s}
32. Rxe6 {+3.33/10 0.012s} Rxb3 {-3.36/10 0.010s} 33. axb3 {+3.35/10 0.080s}
fxe6 {-3.67/10 0.004s} 34. Rd7+ {+3.35/10 0.001s} Kf6 {-3.48/10 0.001s}
35. Rxb7 {+3.35/10 0s} Rh8 {-3.21/10 0.002s} 36. Rxa7 {+3.81/10 0.054s}
Rxh3 {-3.20/10 0.004s} 37. b4 {+3.81/10 0s} Rf3 {-3.85/10 0.004s}
38. b5 {+4.56/10 0.038s} Ke5 {-4.31/10 0.007s} 39. b6 {+4.00/10 0.050s}
Kd6 {-3.25/10 0.003s} 40. Rg7 {+4.41/10 0.093s} Rxf2 {-3.16/10 0.003s}
41. b7 {+4.92/10 0.041s} Rf8 {-4.21/10 0.005s} 42. Rxg6 {+5.16/10 0.10s}
Kc6 {-4.27/10 0.004s} 43. Rg7 {+5.96/10 0.11s} Kb6 {-4.99/10 0.008s}
44. g5 {+6.02/10 0.11s} Ka7 {-5.38/10 0.004s} 45. g6 {+6.18/10 0.038s}
Rf6 {-5.61/10 0.008s} 46. Kd2 {+6.23/10 0.058s} e5 {-5.78/10 0.005s}
47. Kd3 {+6.79/10 0.064s} e4+ {-5.39/10 0.005s} 48. Kxe4 {+7.56/10 0.093s}
Rb6 {-6.53/10 0.012s} 49. c4 {+8.25/10 0.051s} Rb4 {-8.14/10 0.008s}
50. Rf7 {+10.17/10 0.046s} Rxc4+ {-6.69/10 0.003s} 51. Kd5 {+14.07/10 0.050s}
Rb4 {-13.06/10 0.005s} 52. g7 {+14.07/10 0.008s} Rb5+ {-55.80/10 0.005s}
53. Kc6 {+21.22/10 0.078s} Rb6+ {-13.49/10 0.002s} 54. Kc7 {+15.66/10 0.11s}
Rxb7+ {-7.22/10 0.005s} 55. Kd8 {+14.59/10 0.17s}
Rxf7 {-6.06/10 0.009s, White wins by adjudication: SyzygyTB} 1-0

As you can see, around move 11 the evaluation of stockfish drops a lot:

Code: Select all

11. Bg5 {+0.66/10 0.73s} h6 {-1.17/10 0.015s}

... yet the time difference spent on a move between two moves is not so big: 0.73s versus 0.015s (a factor of x48), then the score for black suddenly drops and the game is pretty much over.

I can't rationally explain this difference in performance. Perhaps a bug in SF at shallow depths?

[xr_a_y]

I don't know about stockfish, but at fixed depth 10, Igel is killing Minic. At TC 40/20sec, Minic is killing Igel.

I guess those fixed depth tests will not reveal what you are looking for.

At fixed depth 10, I see that Igel is spending nearly 15 secondes to make the first 40 moves while Minic needs only 2sec...

Maybe you are looking for a short TC pruning tuning.

Regards

[PK]

There is a circular dependency that goes as follows: Stockfish prunes and reduces insane amounts of moves. It needs depth to compensate for that. Insane prunings and reductions are needed to reach that depth.

[Look]

You may try games with short time not fixed depth. As lots of factors affect the depth of an engine. You may try 1min+1sec games to begin with.

[Ferdy]

Hello everyone,

I am trying to hunt down some issues in Igel (CCRL 40/40 ~2800 elo) when running games at a very quick time control, and one of my ideas was to measure the "time to depth" in Igel as well as see if I should improve evaluation or search.

So I decided to run a few games between Stockfish 10 and Igel 1.8.3 at a fixed depth=10 and the results are very surprising to me:

Code: Select all

tc=inf
depth=10
Score of Igel 1.8.3 64 POPCNT vs Stockfish 10 64 POPCNT: 105 - 79 - 45  [0.557] 229
Elo difference: 39.62 +/- 40.67

How come the performance of Stockfish is ~39 elo worse than Igel at a fixed depth=10? Here is an example of one game:

Code: Select all

[Event "?"]
[Site "?"]
[Date "2019.07.31"]
[Round "1"]
[White "Igel 1.8.3 64 POPCNT"]
[Black "Stockfish 10 64 POPCNT"]
[Result "1-0"]
[ECO "B04"]
[Opening "Alekhine's defense"]
[PlyCount "110"]
[Termination "adjudication"]
[TimeControl "inf"]
[Variation "Modern, Fianchetto Variation"]

1. e4 {book} Nf6 {book} 2. e5 {book} Nd5 {book} 3. d4 {book} d6 {book}
4. Nf3 {book} g6 {book} 5. Bc4 {book} Nb6 {book} 6. Bb3 {book} Bg7 {book}
7. Ng5 {book} e6 {book} 8. Qf3 {book} Qe7 {book} 9. Ne4 {+0.15/10 1.1s}
O-O {-0.52/10 0.021s} 10. Nf6+ {+0.70/10 0.34s} Kh8 {-0.05/10 0.004s}
11. Bg5 {+0.66/10 0.73s} h6 {-1.17/10 0.015s} 12. Qh3 {+1.96/10 0.87s}
N8d7 {-0.52/10 0.020s} 13. Ne4 {+1.85/10 0.58s} Qe8 {-3.75/10 0.008s}
14. Bxh6 {+2.67/10 0.66s} Kg8 {-3.15/10 0.004s} 15. exd6 {+2.70/10 0.44s}
Bxd4 {-4.12/10 0.011s} 16. Nbc3 {+3.10/10 0.43s} Nd5 {-4.16/10 0.029s}
17. dxc7 {+3.57/10 1.9s} Nxc3 {-4.98/10 0.015s} 18. Nxc3 {+3.69/10 0.26s}
Bg7 {-5.06/10 0.016s} 19. Bxg7 {+3.96/10 0.98s} Kxg7 {-4.31/10 0.012s}
20. Ne4 {+3.92/10 0.50s} Qe7 {-4.68/10 0.004s} 21. O-O-O {+3.79/10 0.57s}
Nf6 {-4.31/10 0.002s} 22. Qc3 {+3.95/10 0.48s} e5 {-4.55/10 0.006s}
23. Nxf6 {+4.00/10 0.54s} Qxf6 {-5.03/10 0.017s} 24. Rhe1 {+4.00/10 0.003s}
Re8 {-5.00/10 0.022s} 25. Rd5 {+3.88/10 0.69s} e4 {-5.20/10 0.009s}
26. Rd4 {+4.15/10 1.3s} Bf5 {-4.12/10 0.009s} 27. h3 {+3.43/10 1.0s}
Rac8 {-4.27/10 0.011s} 28. g4 {+3.53/10 0.15s} Be6 {-4.23/10 0.007s}
29. Rexe4 {+3.73/10 0.58s} Re7 {-3.43/10 0.007s} 30. Rf4 {+3.41/10 0.89s}
Rexc7 {-3.65/10 0.004s} 31. Rxf6 {+3.33/10 0.23s} Rxc3 {-3.37/10 0.001s}
32. Rxe6 {+3.33/10 0.012s} Rxb3 {-3.36/10 0.010s} 33. axb3 {+3.35/10 0.080s}
fxe6 {-3.67/10 0.004s} 34. Rd7+ {+3.35/10 0.001s} Kf6 {-3.48/10 0.001s}
35. Rxb7 {+3.35/10 0s} Rh8 {-3.21/10 0.002s} 36. Rxa7 {+3.81/10 0.054s}
Rxh3 {-3.20/10 0.004s} 37. b4 {+3.81/10 0s} Rf3 {-3.85/10 0.004s}
38. b5 {+4.56/10 0.038s} Ke5 {-4.31/10 0.007s} 39. b6 {+4.00/10 0.050s}
Kd6 {-3.25/10 0.003s} 40. Rg7 {+4.41/10 0.093s} Rxf2 {-3.16/10 0.003s}
41. b7 {+4.92/10 0.041s} Rf8 {-4.21/10 0.005s} 42. Rxg6 {+5.16/10 0.10s}
Kc6 {-4.27/10 0.004s} 43. Rg7 {+5.96/10 0.11s} Kb6 {-4.99/10 0.008s}
44. g5 {+6.02/10 0.11s} Ka7 {-5.38/10 0.004s} 45. g6 {+6.18/10 0.038s}
Rf6 {-5.61/10 0.008s} 46. Kd2 {+6.23/10 0.058s} e5 {-5.78/10 0.005s}
47. Kd3 {+6.79/10 0.064s} e4+ {-5.39/10 0.005s} 48. Kxe4 {+7.56/10 0.093s}
Rb6 {-6.53/10 0.012s} 49. c4 {+8.25/10 0.051s} Rb4 {-8.14/10 0.008s}
50. Rf7 {+10.17/10 0.046s} Rxc4+ {-6.69/10 0.003s} 51. Kd5 {+14.07/10 0.050s}
Rb4 {-13.06/10 0.005s} 52. g7 {+14.07/10 0.008s} Rb5+ {-55.80/10 0.005s}
53. Kc6 {+21.22/10 0.078s} Rb6+ {-13.49/10 0.002s} 54. Kc7 {+15.66/10 0.11s}
Rxb7+ {-7.22/10 0.005s} 55. Kd8 {+14.59/10 0.17s}
Rxf7 {-6.06/10 0.009s, White wins by adjudication: SyzygyTB} 1-0

As you can see, around move 11 the evaluation of stockfish drops a lot:

Code: Select all

11. Bg5 {+0.66/10 0.73s} h6 {-1.17/10 0.015s}

... yet the time difference spent on a move between two moves is not so big: 0.73s versus 0.015s (a factor of x48), then the score for black suddenly drops and the game is pretty much over.

Sf is optimized to play strong moves at minimum time and not on minimum depth.
What you did is not running at a very quick time control but depth control.

I can't rationally explain this difference in performance. Perhaps a bug in SF at shallow depths?

One hint is to compare the average nodes searched at depth 10 of both engines, although nodes between different engines have different meanings. I suspect Igel is searching more nodes than Sf.

Generally time to depth measurements is not too important if you are also looking for performance. Strong moves with less time/depth ratio (trying to fix the depth) is what you are looking for or just strong moves with less time, forget the depth.

Engine performance is affected by evaluation and search. To find out if you need to improve evaluation or search is vague. Sometimes you try to make changes to the evaluation that you think would improve its play, but you were not successful after so many tries. Later you found out that you have a bug in the search. The same is true when you reverse the situation, you try to improve the search and there is bug in the evaluation, engine performance could still not improve. What is needed is to fix that bug.

Sometimes it is not really a bug but just something that is there that worked in the past but now it is blocking performance if you attempt to improve other areas in the program. One of it is pruning in the search. No matter how much you improve the evaluation the engine performance will not improve because that pruning is triggered first. So it is recommended that every time you make some changes that affects strength, it is better to enable/disable/modify conditions of those pruning methods. And of course you still have search reduction to handle too. Examine the aggressive reduction first.

Could you tell us what issues that you have been hunting?

[Raphexon]

Igel prunes less than Stockfish so it shouldn't be a surprise it's stronger at a fixed depth.
Stockfish reaches the same depth in (much) less time.

And from what I've tested with Igel in the past, its insanely slow at reaching even modest (displayed) depths so it being a monster at fixed depth isn't really a surprise to me either. It seems to barely prune.


On a sidenote Herman 2.8 (2529 ELO on CCRL 40/40) beats up Stockfish as long as you limit "nodes per move" to something abnormally small like 1500 nodes per moves.
Aggressive pruning is nice and better in most cases, but not great for fixed depth or abnormally small node counts.

[bob]

Simple answer: All plies are not created equal. For an aggressive pruner like stockfish, 10 plies is an instant search. For example, I just played 20 moves with Crafty and the longest search was 0.06 seconds at sd=10. Less aggressive pruners will move slower and see more at that same depth.

[Rebel]

Stockfish strength at lower depths is meaningless, it will even lose against ProDeo, see first subject on this page:

http://rebel13.nl/rebel13/experiments.html

[voffka]

Thanks to all of you for your comments. Yes, indeed, taking into account the fact that Igel does not prune much, testing it with fixed depth against SF makes no sense after your explanations

The original issue came after I was analyzing some games between Igel and Cheng when Igel just could not see a good move in a short time and it required much more time than Cheng. I will improve the pruning in next versions of Igel