AlphaZero Chess test - SF8 16 GB vs. 512 MB

[Milos]

I did some single core test 1 MB vs. 64 MB. HERT500.PGN book. With fixed time=1 sec 1 MB was better than bigger? With 60s + 0,6s it changed totally:

Code: Select all

                   
1   SF 64MB   +42  +33/=158/-9 56.00%  112.0/200
2   SF 1MB    -42  +9/=158/-33 44.00%   88.0/200

Obviously fixed time per move DESTROYS SF SEARCH BADLY!

How does fixed time destroy SF's search? Does the search operate in a different manner based on the time control?

If you cannot research on a fail high/low you knowingly play a bad move. This could be interpreted as destroying search and it costs a hell of a lot of Elo.

[syzygy]

I did some single core test 1 MB vs. 64 MB. HERT500.PGN book. With fixed time=1 sec 1 MB was better than bigger? With 60s + 0,6s it changed totally:

Code: Select all

                   
1   SF 64MB   +42  +33/=158/-9 56.00%  112.0/200
2   SF 1MB    -42  +9/=158/-33 44.00%   88.0/200

Obviously fixed time per move DESTROYS SF SEARCH BADLY!

How does fixed time destroy SF's search? Does the search operate in a different manner based on the time control?

It does not destroy SF's search at all.

The "bad" thing that can happen is that SF realises that its move from iteration N isn't as good as it had hoped at iteration N+1 and time runs out before it can finish iteration N+1. But if this happens, SF will still play a move that - at iteration N+1 - is at least as good as the best move from iteration N (either it won't switch or it will switch to a better move, just possibly not the move that is best at iteration N+1 because it didn't have the time to figure out what that best move is).

[Ovyron]

It does not destroy SF's search at all.

The "bad" thing that can happen is that SF realises that its move from iteration N isn't as good as it had hoped at iteration N+1 and time runs out before it can finish iteration N+1. But if this happens, SF will still play a move that - at iteration N+1 - is at least as good as the best move from iteration N (either it won't switch or it will switch to a better move, just possibly not the move that is best at iteration N+1 because it didn't have the time to figure out what that best move is).

This can be tested.

Just like 16 GB vs. 512 MB was tested, you can test how crippled is Stockfish at 1 min/move by matching it against, say, a Stockfish with 40 moves in 40 minutes repeating time control. The second one will be able to manage its time as it wishes, while the first one must move in one minute.

[Adam Hair]

I did some single core test 1 MB vs. 64 MB. HERT500.PGN book. With fixed time=1 sec 1 MB was better than bigger? With 60s + 0,6s it changed totally:

Code: Select all

                   
1   SF 64MB   +42  +33/=158/-9 56.00%  112.0/200
2   SF 1MB    -42  +9/=158/-33 44.00%   88.0/200

Obviously fixed time per move DESTROYS SF SEARCH BADLY!

I can't replicate your results. In two sets of tests using a 3 move epd from Kai and the HERT500 pgn, SF8 with 64MB hash is better than 1MB hash at 1sec/move and 60"+0.6":

Code: Select all

3moves_2200Elo.epd @ 1"/move

   # PLAYER    :  RATING  ERROR  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 64MB      :    25.4   15.1     500    54     100   85  366   49    73
   2 1MB       :     0.0   ----     500    46     ---   49  366   85    73

3moves_2200Elo.epd @ 60"+0.6"

   # PLAYER    :  RATING  ERROR  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 64MB      :    29.1   13.5     500    54     100   71  399   30    80
   2 1MB       :     0.0   ----     500    46     ---   30  399   71    80

HERT500.pgn @ 1"/move

   # PLAYER    :  RATING  ERROR  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 64MB      :    20.6   11.9    1000    53     100  181  696  123    70
   2 1MB       :     0.0   ----    1000    47     ---  123  696  181    70

HERT500.pgn @ 60"+0.6"

   # PLAYER    :  RATING  ERROR  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 64MB      :    27.7    9.9    1000    54     100  154  770   76    77
   2 1MB       :     0.0   ----    1000    46     ---   76  770  154    77

Being able to allocate more time early in the game and when needed does amplify the effect of having more hash, but not so much as to support the claim that fixed time destroys the search.

Did you use Stockfish 8 in your tests?

[Adam Hair]

I did some single core test 1 MB vs. 64 MB. HERT500.PGN book. With fixed time=1 sec 1 MB was better than bigger? With 60s + 0,6s it changed totally:

Code: Select all

                   
1   SF 64MB   +42  +33/=158/-9 56.00%  112.0/200
2   SF 1MB    -42  +9/=158/-33 44.00%   88.0/200

Obviously fixed time per move DESTROYS SF SEARCH BADLY!

How does fixed time destroy SF's search? Does the search operate in a different manner based on the time control?

If you cannot research on a fail high/low you knowingly play a bad move. This could be interpreted as destroying search and it costs a hell of a lot of Elo.

There is a limit to how much time Stockfish can use to resolve a fail high/low no matter the time control. So it is not an issue that arises solely with fixed time. It is just a question of how much more it occurs at fixed time rather than a comparable tc.

The Elo cost does not seem to be extreme at 1"/move and I believe it would be less at 1'/move (even in normalized Elo).

[Adam Hair]

I did some single core test 1 MB vs. 64 MB. HERT500.PGN book. With fixed time=1 sec 1 MB was better than bigger? With 60s + 0,6s it changed totally:

Code: Select all

                   
1   SF 64MB   +42  +33/=158/-9 56.00%  112.0/200
2   SF 1MB    -42  +9/=158/-33 44.00%   88.0/200

Obviously fixed time per move DESTROYS SF SEARCH BADLY!

How does fixed time destroy SF's search? Does the search operate in a different manner based on the time control?

It does not destroy SF's search at all.

The "bad" thing that can happen is that SF realises that its move from iteration N isn't as good as it had hoped at iteration N+1 and time runs out before it can finish iteration N+1. But if this happens, SF will still play a move that - at iteration N+1 - is at least as good as the best move from iteration N (either it won't switch or it will switch to a better move, just possibly not the move that is best at iteration N+1 because it didn't have the time to figure out what that best move is).

Thanks! This how I thought it worked.