Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
voffka
Posts: 62
Joined: Sat Jun 30, 2018 8:58 pm
Location: Ukraine
Full name: Volodymyr Shcherbyna

Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by voffka » Wed Jul 31, 2019 11:33 pm

Hello everyone,

I am trying to hunt down some issues in Igel (CCRL 40/40 ~2800 elo) when running games at a very quick time control, and one of my ideas was to measure the "time to depth" in Igel as well as see if I should improve evaluation or search.

So I decided to run a few games between Stockfish 10 and Igel 1.8.3 at a fixed depth=10 and the results are very surprising to me:

Code: Select all

tc=inf
depth=10
Score of Igel 1.8.3 64 POPCNT vs Stockfish 10 64 POPCNT: 105 - 79 - 45  [0.557] 229
Elo difference: 39.62 +/- 40.67
How come the performance of Stockfish is ~39 elo worse than Igel at a fixed depth=10? Here is an example of one game:

Code: Select all

[Event "?"]
[Site "?"]
[Date "2019.07.31"]
[Round "1"]
[White "Igel 1.8.3 64 POPCNT"]
[Black "Stockfish 10 64 POPCNT"]
[Result "1-0"]
[ECO "B04"]
[Opening "Alekhine's defense"]
[PlyCount "110"]
[Termination "adjudication"]
[TimeControl "inf"]
[Variation "Modern, Fianchetto Variation"]

1. e4 {book} Nf6 {book} 2. e5 {book} Nd5 {book} 3. d4 {book} d6 {book}
4. Nf3 {book} g6 {book} 5. Bc4 {book} Nb6 {book} 6. Bb3 {book} Bg7 {book}
7. Ng5 {book} e6 {book} 8. Qf3 {book} Qe7 {book} 9. Ne4 {+0.15/10 1.1s}
O-O {-0.52/10 0.021s} 10. Nf6+ {+0.70/10 0.34s} Kh8 {-0.05/10 0.004s}
11. Bg5 {+0.66/10 0.73s} h6 {-1.17/10 0.015s} 12. Qh3 {+1.96/10 0.87s}
N8d7 {-0.52/10 0.020s} 13. Ne4 {+1.85/10 0.58s} Qe8 {-3.75/10 0.008s}
14. Bxh6 {+2.67/10 0.66s} Kg8 {-3.15/10 0.004s} 15. exd6 {+2.70/10 0.44s}
Bxd4 {-4.12/10 0.011s} 16. Nbc3 {+3.10/10 0.43s} Nd5 {-4.16/10 0.029s}
17. dxc7 {+3.57/10 1.9s} Nxc3 {-4.98/10 0.015s} 18. Nxc3 {+3.69/10 0.26s}
Bg7 {-5.06/10 0.016s} 19. Bxg7 {+3.96/10 0.98s} Kxg7 {-4.31/10 0.012s}
20. Ne4 {+3.92/10 0.50s} Qe7 {-4.68/10 0.004s} 21. O-O-O {+3.79/10 0.57s}
Nf6 {-4.31/10 0.002s} 22. Qc3 {+3.95/10 0.48s} e5 {-4.55/10 0.006s}
23. Nxf6 {+4.00/10 0.54s} Qxf6 {-5.03/10 0.017s} 24. Rhe1 {+4.00/10 0.003s}
Re8 {-5.00/10 0.022s} 25. Rd5 {+3.88/10 0.69s} e4 {-5.20/10 0.009s}
26. Rd4 {+4.15/10 1.3s} Bf5 {-4.12/10 0.009s} 27. h3 {+3.43/10 1.0s}
Rac8 {-4.27/10 0.011s} 28. g4 {+3.53/10 0.15s} Be6 {-4.23/10 0.007s}
29. Rexe4 {+3.73/10 0.58s} Re7 {-3.43/10 0.007s} 30. Rf4 {+3.41/10 0.89s}
Rexc7 {-3.65/10 0.004s} 31. Rxf6 {+3.33/10 0.23s} Rxc3 {-3.37/10 0.001s}
32. Rxe6 {+3.33/10 0.012s} Rxb3 {-3.36/10 0.010s} 33. axb3 {+3.35/10 0.080s}
fxe6 {-3.67/10 0.004s} 34. Rd7+ {+3.35/10 0.001s} Kf6 {-3.48/10 0.001s}
35. Rxb7 {+3.35/10 0s} Rh8 {-3.21/10 0.002s} 36. Rxa7 {+3.81/10 0.054s}
Rxh3 {-3.20/10 0.004s} 37. b4 {+3.81/10 0s} Rf3 {-3.85/10 0.004s}
38. b5 {+4.56/10 0.038s} Ke5 {-4.31/10 0.007s} 39. b6 {+4.00/10 0.050s}
Kd6 {-3.25/10 0.003s} 40. Rg7 {+4.41/10 0.093s} Rxf2 {-3.16/10 0.003s}
41. b7 {+4.92/10 0.041s} Rf8 {-4.21/10 0.005s} 42. Rxg6 {+5.16/10 0.10s}
Kc6 {-4.27/10 0.004s} 43. Rg7 {+5.96/10 0.11s} Kb6 {-4.99/10 0.008s}
44. g5 {+6.02/10 0.11s} Ka7 {-5.38/10 0.004s} 45. g6 {+6.18/10 0.038s}
Rf6 {-5.61/10 0.008s} 46. Kd2 {+6.23/10 0.058s} e5 {-5.78/10 0.005s}
47. Kd3 {+6.79/10 0.064s} e4+ {-5.39/10 0.005s} 48. Kxe4 {+7.56/10 0.093s}
Rb6 {-6.53/10 0.012s} 49. c4 {+8.25/10 0.051s} Rb4 {-8.14/10 0.008s}
50. Rf7 {+10.17/10 0.046s} Rxc4+ {-6.69/10 0.003s} 51. Kd5 {+14.07/10 0.050s}
Rb4 {-13.06/10 0.005s} 52. g7 {+14.07/10 0.008s} Rb5+ {-55.80/10 0.005s}
53. Kc6 {+21.22/10 0.078s} Rb6+ {-13.49/10 0.002s} 54. Kc7 {+15.66/10 0.11s}
Rxb7+ {-7.22/10 0.005s} 55. Kd8 {+14.59/10 0.17s}
Rxf7 {-6.06/10 0.009s, White wins by adjudication: SyzygyTB} 1-0
As you can see, around move 11 the evaluation of stockfish drops a lot:

Code: Select all

11. Bg5 {+0.66/10 0.73s} h6 {-1.17/10 0.015s}
... yet the time difference spent on a move between two moves is not so big: 0.73s versus 0.015s (a factor of x48), then the score for black suddenly drops and the game is pretty much over.

I can't rationally explain this difference in performance. Perhaps a bug in SF at shallow depths?

User avatar
xr_a_y
Posts: 729
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by xr_a_y » Thu Aug 01, 2019 5:08 am

I don't know about stockfish, but at fixed depth 10, Igel is killing Minic. At TC 40/20sec, Minic is killing Igel.

I guess those fixed depth tests will not reveal what you are looking for.

At fixed depth 10, I see that Igel is spending nearly 15 secondes to make the first 40 moves while Minic needs only 2sec...

Maybe you are looking for a short TC pruning tuning.

Regards

PK
Posts: 819
Joined: Mon Jan 15, 2007 10:23 am
Location: Warsza
Contact:

Re: Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by PK » Thu Aug 01, 2019 6:14 am

There is a circular dependency that goes as follows: Stockfish prunes and reduces insane amounts of moves. It needs depth to compensate for that. Insane prunings and reductions are needed to reach that depth.

Look
Posts: 218
Joined: Thu Jun 05, 2014 12:14 pm
Location: Iran
Full name: Mehdi Amini
Contact:

Re: Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by Look » Thu Aug 01, 2019 10:24 am

You may try games with short time not fixed depth. As lots of factors affect the depth of an engine. You may try 1min+1sec games to begin with.

Ferdy
Posts: 4071
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by Ferdy » Fri Aug 02, 2019 2:29 am

voffka wrote:
Wed Jul 31, 2019 11:33 pm
Hello everyone,

I am trying to hunt down some issues in Igel (CCRL 40/40 ~2800 elo) when running games at a very quick time control, and one of my ideas was to measure the "time to depth" in Igel as well as see if I should improve evaluation or search.

So I decided to run a few games between Stockfish 10 and Igel 1.8.3 at a fixed depth=10 and the results are very surprising to me:

Code: Select all

tc=inf
depth=10
Score of Igel 1.8.3 64 POPCNT vs Stockfish 10 64 POPCNT: 105 - 79 - 45  [0.557] 229
Elo difference: 39.62 +/- 40.67
How come the performance of Stockfish is ~39 elo worse than Igel at a fixed depth=10? Here is an example of one game:

Code: Select all

[Event "?"]
[Site "?"]
[Date "2019.07.31"]
[Round "1"]
[White "Igel 1.8.3 64 POPCNT"]
[Black "Stockfish 10 64 POPCNT"]
[Result "1-0"]
[ECO "B04"]
[Opening "Alekhine's defense"]
[PlyCount "110"]
[Termination "adjudication"]
[TimeControl "inf"]
[Variation "Modern, Fianchetto Variation"]

1. e4 {book} Nf6 {book} 2. e5 {book} Nd5 {book} 3. d4 {book} d6 {book}
4. Nf3 {book} g6 {book} 5. Bc4 {book} Nb6 {book} 6. Bb3 {book} Bg7 {book}
7. Ng5 {book} e6 {book} 8. Qf3 {book} Qe7 {book} 9. Ne4 {+0.15/10 1.1s}
O-O {-0.52/10 0.021s} 10. Nf6+ {+0.70/10 0.34s} Kh8 {-0.05/10 0.004s}
11. Bg5 {+0.66/10 0.73s} h6 {-1.17/10 0.015s} 12. Qh3 {+1.96/10 0.87s}
N8d7 {-0.52/10 0.020s} 13. Ne4 {+1.85/10 0.58s} Qe8 {-3.75/10 0.008s}
14. Bxh6 {+2.67/10 0.66s} Kg8 {-3.15/10 0.004s} 15. exd6 {+2.70/10 0.44s}
Bxd4 {-4.12/10 0.011s} 16. Nbc3 {+3.10/10 0.43s} Nd5 {-4.16/10 0.029s}
17. dxc7 {+3.57/10 1.9s} Nxc3 {-4.98/10 0.015s} 18. Nxc3 {+3.69/10 0.26s}
Bg7 {-5.06/10 0.016s} 19. Bxg7 {+3.96/10 0.98s} Kxg7 {-4.31/10 0.012s}
20. Ne4 {+3.92/10 0.50s} Qe7 {-4.68/10 0.004s} 21. O-O-O {+3.79/10 0.57s}
Nf6 {-4.31/10 0.002s} 22. Qc3 {+3.95/10 0.48s} e5 {-4.55/10 0.006s}
23. Nxf6 {+4.00/10 0.54s} Qxf6 {-5.03/10 0.017s} 24. Rhe1 {+4.00/10 0.003s}
Re8 {-5.00/10 0.022s} 25. Rd5 {+3.88/10 0.69s} e4 {-5.20/10 0.009s}
26. Rd4 {+4.15/10 1.3s} Bf5 {-4.12/10 0.009s} 27. h3 {+3.43/10 1.0s}
Rac8 {-4.27/10 0.011s} 28. g4 {+3.53/10 0.15s} Be6 {-4.23/10 0.007s}
29. Rexe4 {+3.73/10 0.58s} Re7 {-3.43/10 0.007s} 30. Rf4 {+3.41/10 0.89s}
Rexc7 {-3.65/10 0.004s} 31. Rxf6 {+3.33/10 0.23s} Rxc3 {-3.37/10 0.001s}
32. Rxe6 {+3.33/10 0.012s} Rxb3 {-3.36/10 0.010s} 33. axb3 {+3.35/10 0.080s}
fxe6 {-3.67/10 0.004s} 34. Rd7+ {+3.35/10 0.001s} Kf6 {-3.48/10 0.001s}
35. Rxb7 {+3.35/10 0s} Rh8 {-3.21/10 0.002s} 36. Rxa7 {+3.81/10 0.054s}
Rxh3 {-3.20/10 0.004s} 37. b4 {+3.81/10 0s} Rf3 {-3.85/10 0.004s}
38. b5 {+4.56/10 0.038s} Ke5 {-4.31/10 0.007s} 39. b6 {+4.00/10 0.050s}
Kd6 {-3.25/10 0.003s} 40. Rg7 {+4.41/10 0.093s} Rxf2 {-3.16/10 0.003s}
41. b7 {+4.92/10 0.041s} Rf8 {-4.21/10 0.005s} 42. Rxg6 {+5.16/10 0.10s}
Kc6 {-4.27/10 0.004s} 43. Rg7 {+5.96/10 0.11s} Kb6 {-4.99/10 0.008s}
44. g5 {+6.02/10 0.11s} Ka7 {-5.38/10 0.004s} 45. g6 {+6.18/10 0.038s}
Rf6 {-5.61/10 0.008s} 46. Kd2 {+6.23/10 0.058s} e5 {-5.78/10 0.005s}
47. Kd3 {+6.79/10 0.064s} e4+ {-5.39/10 0.005s} 48. Kxe4 {+7.56/10 0.093s}
Rb6 {-6.53/10 0.012s} 49. c4 {+8.25/10 0.051s} Rb4 {-8.14/10 0.008s}
50. Rf7 {+10.17/10 0.046s} Rxc4+ {-6.69/10 0.003s} 51. Kd5 {+14.07/10 0.050s}
Rb4 {-13.06/10 0.005s} 52. g7 {+14.07/10 0.008s} Rb5+ {-55.80/10 0.005s}
53. Kc6 {+21.22/10 0.078s} Rb6+ {-13.49/10 0.002s} 54. Kc7 {+15.66/10 0.11s}
Rxb7+ {-7.22/10 0.005s} 55. Kd8 {+14.59/10 0.17s}
Rxf7 {-6.06/10 0.009s, White wins by adjudication: SyzygyTB} 1-0
As you can see, around move 11 the evaluation of stockfish drops a lot:

Code: Select all

11. Bg5 {+0.66/10 0.73s} h6 {-1.17/10 0.015s}
... yet the time difference spent on a move between two moves is not so big: 0.73s versus 0.015s (a factor of x48), then the score for black suddenly drops and the game is pretty much over.
Sf is optimized to play strong moves at minimum time and not on minimum depth.
What you did is not running at a very quick time control but depth control.
voffka wrote:
Wed Jul 31, 2019 11:33 pm
I can't rationally explain this difference in performance. Perhaps a bug in SF at shallow depths?
One hint is to compare the average nodes searched at depth 10 of both engines, although nodes between different engines have different meanings. I suspect Igel is searching more nodes than Sf.

Generally time to depth measurements is not too important if you are also looking for performance. Strong moves with less time/depth ratio (trying to fix the depth) is what you are looking for or just strong moves with less time, forget the depth.

Engine performance is affected by evaluation and search. To find out if you need to improve evaluation or search is vague. Sometimes you try to make changes to the evaluation that you think would improve its play, but you were not successful after so many tries. Later you found out that you have a bug in the search. The same is true when you reverse the situation, you try to improve the search and there is bug in the evaluation, engine performance could still not improve. What is needed is to fix that bug.

Sometimes it is not really a bug but just something that is there that worked in the past but now it is blocking performance if you attempt to improve other areas in the program. One of it is pruning in the search. No matter how much you improve the evaluation the engine performance will not improve because that pruning is triggered first. So it is recommended that every time you make some changes that affects strength, it is better to enable/disable/modify conditions of those pruning methods. And of course you still have search reduction to handle too. Examine the aggressive reduction first.

Could you tell us what issues that you have been hunting?

Raphexon
Posts: 83
Joined: Sun Mar 17, 2019 11:00 am
Full name: Henk Drost

Re: Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by Raphexon » Wed Aug 14, 2019 5:32 am

Igel prunes less than Stockfish so it shouldn't be a surprise it's stronger at a fixed depth.
Stockfish reaches the same depth in (much) less time.

And from what I've tested with Igel in the past, its insanely slow at reaching even modest (displayed) depths so it being a monster at fixed depth isn't really a surprise to me either. It seems to barely prune.


On a sidenote Herman 2.8 (2529 ELO on CCRL 40/40) beats up Stockfish as long as you limit "nodes per move" to something abnormally small like 1500 nodes per moves.
Aggressive pruning is nice and better in most cases, but not great for fixed depth or abnormally small node counts.

bob
Posts: 20471
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by bob » Wed Aug 14, 2019 6:41 pm

Simple answer: All plies are not created equal. For an aggressive pruner like stockfish, 10 plies is an instant search. For example, I just played 20 moves with Crafty and the longest search was 0.06 seconds at sd=10. Less aggressive pruners will move slower and see more at that same depth.

User avatar
Rebel
Posts: 4658
Joined: Thu Aug 18, 2011 10:04 am

Re: Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by Rebel » Thu Aug 15, 2019 2:41 pm

Stockfish strength at lower depths is meaningless, it will even lose against ProDeo, see first subject on this page:

http://rebel13.nl/rebel13/experiments.html
Everybody is unique, except me.

voffka
Posts: 62
Joined: Sat Jun 30, 2018 8:58 pm
Location: Ukraine
Full name: Volodymyr Shcherbyna

Re: Poor performance of Stockfish 10 against Igel 1.8.3 at (shallow) fixed depth=10

Post by voffka » Wed Sep 04, 2019 6:46 pm

Thanks to all of you for your comments. Yes, indeed, taking into account the fact that Igel does not prune much, testing it with fixed depth against SF makes no sense after your explanations :)

The original issue came after I was analyzing some games between Igel and Cheng when Igel just could not see a good move in a short time and it required much more time than Cheng. I will improve the pruning in next versions of Igel :)

Post Reply