What should not be forgotten is the saturation of the hash table. With an almost full hash table researches will become very expensive and it would make sense to widen the window.bob wrote:A quick looks says that the average is in the 13-14-15 range. I looked at a few logs and there are plenty of re-searches going on, so it is having a chance to exert influence.Karlo Bala wrote:What is the average depth?bob wrote:Here is the output from BayesElo first:
2 Crafty-23.5R06-200 2650 4 4 30000 64% 2539 22%
3 Crafty-23.5R06-24 2650 4 4 30000 64% 2539 22%
4 Crafty-23.5R06-100 2649 4 4 30000 63% 2539 22%
5 Crafty-23.5R06-30 2648 4 4 30000 63% 2539 22%
6 Crafty-23.5R06-50 2648 4 4 30000 63% 2539 22%
7 Crafty-23.5R06-300 2648 4 4 30000 63% 2539 22%
8 Crafty-23.5R06-20 2648 4 4 30000 63% 2539 22%
9 Crafty-23.5R06-10 2645 4 4 30000 63% 2539 22%
10 Crafty-23.5-2 2645 4 4 30000 63% 2539 22%
11 Crafty-23.5R06-1 2645 4 4 30000 63% 2539 22%
12 Crafty-23.5R06-8 2644 4 4 30000 63% 2539 22%
13 Crafty-23.5R06-5 2643 4 4 30000 63% 2539 22%
14 Crafty-23.5-1 2641 4 4 30000 63% 2539 22%
15 Crafty-23.5R06-2 2636 4 4 30000 62% 2539 22%
version 23.5-1 and 23.5-2 are simply two consecutive runs with the same version to provide a normal result. The rest of the tests are version 23.5R06 and were tested where the -n is the aspiration window (delta value in the code posted yesterday). 23.5R06-1 means the aspiration window was +/- 1 with delta=1. 1 and 2 are a bit low, and by the time it gets to 10, it is pretty optimal. Bigger doesn't seem to hurt at all up to +/- 3.0 pawns... I was expecting a better result in the 20-40 range, the reason I ran the big numbers was to produce some worse results so that there is a recognizable curve with a clear optimal value, and worse results on either side. Didn't get exactly what I expected, as you can see...
optimal aspiration window for stockfish question
Moderators: hgm, Rebel, chrisw
-
- Posts: 6995
- Joined: Thu Aug 18, 2011 12:04 pm
Re: test results
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: test results
That's one problem I really don't have to deal with. I don't hash the q-search, so I really don't see much in terms of saturation. When I was testing this stuff, I found that hashing the q-search reduced the total tree size by about 10%, but it slowed the program down by almost exactly the same amount. A wash. But by not hashing the q-search, the stress on the ttable is greatly reduced, which tends to make this pay off (not hashing qsearch) in real long games without sufficient memory for the ttable. 8 gigs gives 512 million entries, which is a lot even for a long 40/2hr type game...Rebel wrote:What should not be forgotten is the saturation of the hash table. With an almost full hash table researches will become very expensive and it would make sense to widen the window.bob wrote:A quick looks says that the average is in the 13-14-15 range. I looked at a few logs and there are plenty of re-searches going on, so it is having a chance to exert influence.Karlo Bala wrote:What is the average depth?bob wrote:Here is the output from BayesElo first:
2 Crafty-23.5R06-200 2650 4 4 30000 64% 2539 22%
3 Crafty-23.5R06-24 2650 4 4 30000 64% 2539 22%
4 Crafty-23.5R06-100 2649 4 4 30000 63% 2539 22%
5 Crafty-23.5R06-30 2648 4 4 30000 63% 2539 22%
6 Crafty-23.5R06-50 2648 4 4 30000 63% 2539 22%
7 Crafty-23.5R06-300 2648 4 4 30000 63% 2539 22%
8 Crafty-23.5R06-20 2648 4 4 30000 63% 2539 22%
9 Crafty-23.5R06-10 2645 4 4 30000 63% 2539 22%
10 Crafty-23.5-2 2645 4 4 30000 63% 2539 22%
11 Crafty-23.5R06-1 2645 4 4 30000 63% 2539 22%
12 Crafty-23.5R06-8 2644 4 4 30000 63% 2539 22%
13 Crafty-23.5R06-5 2643 4 4 30000 63% 2539 22%
14 Crafty-23.5-1 2641 4 4 30000 63% 2539 22%
15 Crafty-23.5R06-2 2636 4 4 30000 62% 2539 22%
version 23.5-1 and 23.5-2 are simply two consecutive runs with the same version to provide a normal result. The rest of the tests are version 23.5R06 and were tested where the -n is the aspiration window (delta value in the code posted yesterday). 23.5R06-1 means the aspiration window was +/- 1 with delta=1. 1 and 2 are a bit low, and by the time it gets to 10, it is pretty optimal. Bigger doesn't seem to hurt at all up to +/- 3.0 pawns... I was expecting a better result in the 20-40 range, the reason I ran the big numbers was to produce some worse results so that there is a recognizable curve with a clear optimal value, and worse results on either side. Didn't get exactly what I expected, as you can see...
I run fast games with a modest hash on the cluster, just to try to keep things within perspective...
-
- Posts: 1822
- Joined: Thu Mar 09, 2006 11:54 pm
- Location: The Netherlands
Re: optimal aspiration window for stockfish question
Cray Blitz still alive?bob wrote:I thought I had said that I tested up to 1 minute + 1 second. I did not go beyond that, and found absolutely no difference, which was surprising. The only thing I didnt like was the excessive re-searches to reach a mate. But at that point, it doesn't affect the game result at all of course...mcostalba wrote:Have you tested at longer TC ? It would be interesting to know how scales at longer TC, with your cluster should be feasible to test say at 1' TC.bob wrote: Absolutely no change, either up or down.
When we first started doing this in Cray Blitz, we tried lots of ideas. Our aspiration window was roughly 1/3 of a pawn, so we relaxed alpha/beta to +1.0, then +3.0, then +9.0 and then all the way to infinite. In Crafty, I eliminated that +3.0 and have been using +1, +9 and +infinite for the longest...
I am also running a test (since I was not testing anything else) on various aspiration window widths as well. I'll post those results when they finish...
In diep i'm starting search with (-inf,+inf) for a simple reason: hashtable will directly give back a great bound anyway and you nullwindow around that.
Only when your PV is a total mess i assume that aspiration search is a great idea, shouldn't happen for a proper YBW search.
Maybe that's why crafty doesn't suffer from the same problem there.
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: optimal aspiration window for stockfish question
Depends on what you mean by "alive". The last 3 years of Cray Blitz development was lost. I have a 1991 or so version that is on the net and compiles/runs just fine (no parallel search however as the Cray "task common" is not a standard fortran feature.diep wrote:Cray Blitz still alive?bob wrote:I thought I had said that I tested up to 1 minute + 1 second. I did not go beyond that, and found absolutely no difference, which was surprising. The only thing I didnt like was the excessive re-searches to reach a mate. But at that point, it doesn't affect the game result at all of course...mcostalba wrote:Have you tested at longer TC ? It would be interesting to know how scales at longer TC, with your cluster should be feasible to test say at 1' TC.bob wrote: Absolutely no change, either up or down.
When we first started doing this in Cray Blitz, we tried lots of ideas. Our aspiration window was roughly 1/3 of a pawn, so we relaxed alpha/beta to +1.0, then +3.0, then +9.0 and then all the way to infinite. In Crafty, I eliminated that +3.0 and have been using +1, +9 and +infinite for the longest...
I am also running a test (since I was not testing anything else) on various aspiration window widths as well. I'll post those results when they finish...
In diep i'm starting search with (-inf,+inf) for a simple reason: hashtable will directly give back a great bound anyway and you nullwindow around that.
Only when your PV is a total mess i assume that aspiration search is a great idea, shouldn't happen for a proper YBW search.
Maybe that's why crafty doesn't suffer from the same problem there.
I'm beginning to think that the +/- infinity idea is just as good as aspiration search. Most likely because we all use PVS (null-window, not null-move) anyway. I did not go all the way to infinity in my test results I posted here, but I will try it to see if it makes any difference. (I suspect there will be no change at all).
-
- Posts: 2684
- Joined: Sat Jun 14, 2008 9:17 pm
Re: optimal aspiration window for stockfish question
A little detail: aspiration window should not start from iteration 1, in Stockfish it starts from iteration 5 and before we use +-inf.bob wrote: I'm beginning to think that the +/- infinity idea is just as good as aspiration search. Most likely because we all use PVS (null-window, not null-move) anyway. I did not go all the way to infinity in my test results I posted here, but I will try it to see if it makes any difference. (I suspect there will be no change at all).
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: optimal aspiration window for stockfish question
Why would it matter? Those iterations take microseconds to complete. Doesn't matter how many re-searches you do down there...mcostalba wrote:A little detail: aspiration window should not start from iteration 1, in Stockfish it starts from iteration 5 and before we use +-inf.bob wrote: I'm beginning to think that the +/- infinity idea is just as good as aspiration search. Most likely because we all use PVS (null-window, not null-move) anyway. I did not go all the way to infinity in my test results I posted here, but I will try it to see if it makes any difference. (I suspect there will be no change at all).
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: optimal aspiration window for stockfish question
I confirm what Robery is saying. I've done some testing in DoubleCheck with that, and found the minimum depth condition to be useless. However, if it doesn't help, it cannot hurt, and it reduces the nodecount on a "./DC bench" (I haven't tried with "./stockfish bench" but I'm almost certain it's the same). But I kept the condition anyway, as it may have an impact at stupidly fast games (sometimes I test without time control and just a node limit of 50k nodes for example).bob wrote:Why would it matter? Those iterations take microseconds to complete. Doesn't matter how many re-searches you do down there...mcostalba wrote:A little detail: aspiration window should not start from iteration 1, in Stockfish it starts from iteration 5 and before we use +-inf.bob wrote: I'm beginning to think that the +/- infinity idea is just as good as aspiration search. Most likely because we all use PVS (null-window, not null-move) anyway. I did not go all the way to infinity in my test results I posted here, but I will try it to see if it makes any difference. (I suspect there will be no change at all).
But there is another condition in StockFish which might make more sense. If the score is a mate score (or a winning endgame by EG knowledge which returnsd high scores) then SF doesn't do aspiration. I haven't done any serious measurement on that, but it makes sense. Although I expect no elo improvement from it, it should make SF waste less time in researches when it's finding a mate, which is nice for analysis.
-
- Posts: 613
- Joined: Sun Jan 18, 2009 7:03 am
Re: optimal aspiration window for stockfish question
We don't want to use aspiration search in endgame like KBNK, because TT is filled with mate scores.lucasart wrote: But there is another condition in StockFish which might make more sense. If the score is a mate score (or a winning endgame by EG knowledge which returnsd high scores) then SF doesn't do aspiration. I haven't done any serious measurement on that, but it makes sense. Although I expect no elo improvement from it, it should make SF waste less time in researches when it's finding a mate, which is nice for analysis.
Joona Kiiski