Hi everyone,
I have been following Stockfish code since about version 1.4 and I guess that this question would most likely be appropriate for Tord and Marco, but I would be interested in hearing from anyone who has been working with Stockfish in depth...
Overall I have been quite impressed with the amount of refactoring and the shift to templates for compile-time optimization (among other things) that have been taking place from version to version. I haven't really been following any changes to search or eval but I am curious as to the overall approach to determining areas of improvement, what you all are focusing on, any testing methodologies, profiling, etc you all are using to keep up this rate of improvement?
Specifically, I'm hoping to gain some technical insight into how you all are approaching improving search and eval and any other areas that may be running at sub-optimal levels.
Of course I would also have to say thank you to Tord (for publishing Glaurung to begin with), Marco, and all who have contributed to Stockfish for pushing the envelope and keeping the code open so that everyone can benefit. I truly admire that, and I do look forward to your response
Stockfish direction and improvements
Moderators: hgm, Rebel, chrisw
-
- Posts: 2684
- Joined: Sat Jun 14, 2008 9:17 pm
Re: Stockfish direction and improvements
Thanks for your kind words. Well there have been many (around 100) functionality changes apart from optimizations and cleanups.
The biggest booster has been evaluation tuning because it has a leverage effect also on other changes: if your evaluation is well tuned then you can also improve pruning or, for instance, a change in move ordering will give you more ELO then in case eval is not well tuned.
In SF 1.6 also the search is changed a lot. Anyhow the only _secret_ is to devote 90% of time and resources to testing becasue we have seen many, really many, apparently good ideas to crash against the testing wall, so also if something seems obviously good we test anyway with real games...and this takes soooo long
The biggest booster has been evaluation tuning because it has a leverage effect also on other changes: if your evaluation is well tuned then you can also improve pruning or, for instance, a change in move ordering will give you more ELO then in case eval is not well tuned.
In SF 1.6 also the search is changed a lot. Anyhow the only _secret_ is to devote 90% of time and resources to testing becasue we have seen many, really many, apparently good ideas to crash against the testing wall, so also if something seems obviously good we test anyway with real games...and this takes soooo long
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: Stockfish direction and improvements
What time limit or limits do you do most of your testing at?
Re: Stockfish direction and improvements
And since we are on the subject, could you please explain to me how the option "Randomness" works?
-
- Posts: 1808
- Joined: Wed Mar 08, 2006 9:19 pm
- Location: Oslo, Norway
Re: Stockfish direction and improvements
Thanks for your kind words -- but don't forget Joona, whose contributions to the last few versions have been just as important as Marco's, and far more important than mine.Venom wrote:Of course I would also have to say thank you to Tord (for publishing Glaurung to begin with), Marco, and all who have contributed to Stockfish for pushing the envelope and keeping the code open so that everyone can benefit. I truly admire that, and I do look forward to your response
-
- Posts: 1808
- Joined: Wed Mar 08, 2006 9:19 pm
- Location: Oslo, Norway
Re: Stockfish direction and improvements
Almost all our tests are at 1+0 (1000 games at 1+0 when testing whether some new feature improves the strength) or even faster levels (thousands of games at super-fast time controls when auto-tuning parameters). Ideally we would of course prefer to test at slower time controls, but because of severely limited hardware resources and the huge number of things we want to test, this just isn't possible.lkaufman wrote:What time limit or limits do you do most of your testing at?
-
- Posts: 1808
- Joined: Wed Mar 08, 2006 9:19 pm
- Location: Oslo, Norway
Re: Stockfish direction and improvements
Sure, that's very simple. When "Randomness" is greater than zero, a small random number is added to each entry of all piece square tables at the beginning of every new game. The idea is to make it a more interesting and less predictable opponent for human players. Of course, the parameter should be left at the default value of 0 for optimal strength.Hart wrote:And since we are on the subject, could you please explain to me how the option "Randomness" works?
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: Stockfish direction and improvements
That's pretty much what we do with Doch, but I often wonder whether the optimum values for positional features, even for material, might be very different if we could test at tournament levels.
-
- Posts: 2272
- Joined: Mon Sep 29, 2008 1:50 am
Re: Stockfish direction and improvements
Sorry for my ignorance but do you mean one second or one minute games?Almost all our tests are at 1+0 (1000 games at 1+0 when testing whether some new feature improves the strength)
I assume you mean one second but I just wanted to be sure.
-
- Posts: 613
- Joined: Sun Jan 18, 2009 7:03 am
Re: Stockfish direction and improvements
He meant one minute / game, no increment.Sorry for my ignorance but do you mean one second or one minute games?Almost all our tests are at 1+0 (1000 games at 1+0 when testing whether some new feature improves the strength)
I assume you mean one second but I just wanted to be sure.
Joona Kiiski