That's what we do. We will run a few hundred games at various fixed depth levels and have the instrumentation that gives us the average time per move up to some arbitrary move number. I think we currently use move 70. We think this is superior to just running 100 positions and timing them. We only do this when we want data on something that is supposed to be a speedup - normally we just run time games to measure general improvements and things that involve trade-offs.mcostalba wrote:Yes, we have.Rebel wrote:I am sure the SF team have done considerable testing here.
At the end of endless trials with differnt formulas and values we end up in starting with a small window value:
That is increased after a fail low/high with the following formula:Code: Select all
delta = Value(16); alpha = RootMoves[PVIdx].prevScore - delta; beta = RootMoves[PVIdx].prevScore + delta;
Code: Select all
if (bestValue >= beta) { beta += delta; delta += delta / 2; } else if (bestValue <= alpha) { alpha -= delta; delta += delta / 2; }
I want to add that this is the Ippo* formula and I think that very probably it is what is used in _all_ the top engines from Rybka 3 to Houdini. Although we knew the Ippo formula since when sources were published we moved to that only one year later, after having tried all the possible different combinations: some are weaker, some are equivalent ELO wise, but more complex, so this is the simplest formula (we know) that guarantees top performance.
Answering to Uri: I am not interested in tweaking the engine on a sample position. I only use real games to validate a change.
optimal aspiration window for stockfish question
Moderators: hgm, Rebel, chrisw
-
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: optimal aspiration window for stockfish question
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
- Posts: 1471
- Joined: Tue Mar 16, 2010 12:00 am
Re: optimal aspiration window for stockfish question
Houdini doesn't use these formulae.mcostalba wrote:I want to add that this is the Ippo* formula and I think that very probably it is what is used in _all_ the top engines from Rybka 3 to Houdini.
Please don't spread incorrect assumptions about Houdini.
Regards,
Robert
-
- Posts: 1471
- Joined: Tue Mar 16, 2010 12:00 am
Re: optimal aspiration window for stockfish question
If you're really interested, no version of Houdini has used the formulae that Marco quoted above.Uri Blass wrote:The more interesting question is which houdini does not use that formula because there are more than one houdini version.
Houdini uses an aspiration window that widens much faster.
Robert
-
- Posts: 10282
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: optimal aspiration window for stockfish question
Thanks Robert.Houdini wrote:If you're really interested, no version of Houdini has used the formulae that Marco quoted above.Uri Blass wrote:The more interesting question is which houdini does not use that formula because there are more than one houdini version.
Houdini uses an aspiration window that widens much faster.
Robert
My intuition also tells me that it is better to have less researches(increasing delta faster by a different formula may be a way to do it)
Maybe the stockfish team may try something like
if (bestValue >= beta)
{
beta += delta;
delta += delta*2;
}
else if (bestValue <= alpha)
{
alpha -= delta;
delta += delta *2;
}
-
- Posts: 4565
- Joined: Sun Mar 12, 2006 2:40 am
- Full name:
Re: optimal aspiration window for stockfish question
@Uri: It is probably not such a bad idea to let it depend not only on the depth but also the amount of hash. If there are more holes in the hash table because of saturation, it is likely that you will see more failed searches for good PV nodes, and that leads to more fail lows or fail highs. I don't think that a wider window by itself will be more efficient if there are a lot of Fail Lows or Fail Highs, but the fail low or fail high searches will take longer if they find no stored positions in hash. And then it is possibly better to just use a wider window. I don't think Uri's correspondence chess example is really a case where this already applies, because the fail lows all seem to go pretty fast and the endresult is actually lower so the fail lows did their work on the right side as it were. But if Marco wants to test your theory and he needs a timecontrol of at least 40 minutes per move it is going to be rather timeconsuming. But you should be able to simulate my theory of the hash effect on short timecontrols a bit.
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
-
- Posts: 10282
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: optimal aspiration window for stockfish question
The fail low is not justified in my exampleEelco de Groot wrote:@Uri: It is probably not such a bad idea to let it depend not only on the depth but also the amount of hash. If there are more holes in the hash table because of saturation, it is likely that you will see more failed searches for good PV nodes, and that leads to more fail lows or fail highs. I don't think that a wider window by itself will be more efficient if there are a lot of Fail Lows or Fail Highs, but the fail low or fail high searches will take longer if they find no stored positions in hash. And then it is possibly better to just use a wider window. I don't think Uri's correspondence chess example is really a case where this already applies, because the fail lows all seem to go pretty fast and the endresult is actually lower so the fail lows did their work on the right side as it were. But if Marco wants to test your theory and he needs a timecontrol of at least 40 minutes per move it is going to be rather timeconsuming. But you should be able to simulate my theory of the hash effect on short timecontrols a bit.
stockfish started with 1.97 and the first fail low said lower than 1.89 but the final score was 1.93
35/79 40:24 10,618,711,296 4,378,972 +1.97
36/79- 42:46 11,265,468,653 4,388,811 +1.89
36/79- 43:36 11,486,371,530 4,389,792 +1.81
36/79- 44:35 11,747,022,497 4,390,999 +1.69
36/79+ 46:55 12,371,982,241 4,393,810 +2.06
36/79 55:58 14,833,537,002 4,417,125 +1.93
second example from the same search
The score did not change so fail high and fail low are not justified.
23/56 00:36 130,367,750 3,606,399 +1.37
24/56+ 00:43 158,498,424 3,604,776 +1.45
24/56- 00:46 169,841,034 3,623,584 +1.29
24/56- 00:47 172,222,913 3,627,349 +1.17
24/56- 00:48 175,065,644 3,613,624 +0.98
24/56+ 00:50 183,805,380 3,619,356 +1.53
24/64 01:03 231,112,085 3,634,123 +1.37
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: optimal aspiration window for stockfish question
your initial delta is only 1/16-th of a pawn ?mcostalba wrote:Code: Select all
delta = Value(16);
-
- Posts: 4565
- Joined: Sun Mar 12, 2006 2:40 am
- Full name:
Re: optimal aspiration window for stockfish question
Well no, because oflucasart wrote:your initial delta is only 1/16-th of a pawn ?mcostalba wrote:Code: Select all
delta = Value(16);
Code: Select all
const Value PawnValueMidgame = Value(0x0C6);
const Value PawnValueEndgame = Value(0x102);
Eelco
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
-
- Posts: 373
- Joined: Wed Mar 22, 2006 10:17 am
- Location: Novi Sad, Serbia
- Full name: Karlo Balla
Re: optimal aspiration window for stockfish question
My intuition is somewhat different than yours. If your PV search is very similar to non-PV search, it should be a small percentage of false PV nodes, so that you can use a wider window. If your PV search is very differen from non-PV search (which is common today because of the LMR, NM, futility, delta, etc.) then it should be more false PV nodes. In that case you do not want to search a lot of false PV nodes with a wide window.Uri Blass wrote:Thanks Robert.Houdini wrote:If you're really interested, no version of Houdini has used the formulae that Marco quoted above.Uri Blass wrote:The more interesting question is which houdini does not use that formula because there are more than one houdini version.
Houdini uses an aspiration window that widens much faster.
Robert
My intuition also tells me that it is better to have less researches(increasing delta faster by a different formula may be a way to do it)
Best Regards,
Karlo Balla Jr.
Karlo Balla Jr.
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: optimal aspiration window for stockfish question
Since it was a trivial change, I took the above and rewrote it to fit Crafty's search. Absolutely no change, either up or down. A couple of things I do not like about it however. 1. Several re-searches to win a pawn or more, and 2. way too many re-searches to find a forced mate.Don wrote:That's what we do. We will run a few hundred games at various fixed depth levels and have the instrumentation that gives us the average time per move up to some arbitrary move number. I think we currently use move 70. We think this is superior to just running 100 positions and timing them. We only do this when we want data on something that is supposed to be a speedup - normally we just run time games to measure general improvements and things that involve trade-offs.mcostalba wrote:Yes, we have.Rebel wrote:I am sure the SF team have done considerable testing here.
At the end of endless trials with differnt formulas and values we end up in starting with a small window value:
That is increased after a fail low/high with the following formula:Code: Select all
delta = Value(16); alpha = RootMoves[PVIdx].prevScore - delta; beta = RootMoves[PVIdx].prevScore + delta;
Code: Select all
if (bestValue >= beta) { beta += delta; delta += delta / 2; } else if (bestValue <= alpha) { alpha -= delta; delta += delta / 2; }
I want to add that this is the Ippo* formula and I think that very probably it is what is used in _all_ the top engines from Rybka 3 to Houdini. Although we knew the Ippo formula since when sources were published we moved to that only one year later, after having tried all the possible different combinations: some are weaker, some are equivalent ELO wise, but more complex, so this is the simplest formula (we know) that guarantees top performance.
Answering to Uri: I am not interested in tweaking the engine on a sample position. I only use real games to validate a change.
We've used several different approaches. In Cray Blitz we did more "bumps". In Crafty I settled on fewer. But it actually doesn't seem to matter one bit based on the testing I just did, which was run at fast games and at 1min+1sec games as well. Absolutely no +/- change when switching from what I did to this, and back...