Stockfish's tuning method

[Rémi Coulom]

The name "ampli-bias knobs" was chosen as an analogy to old analog TV where you can control the full image view (made of many pixels) using only two knobs: contrast (ampli) and luminance (bias).

Thanks. I understand now.

Rémi

[jdart]

How many CPUs, and how much time, do you need for 30,000 games?

[mjlef]

and were these fixed depth or timed matches?

[mcostalba]

Thanks. I understand now.

My mathematics skills are very rusty these days, but more or less (forgive me if I am not precise), if we consider the N parameters to tune in parallel a vector of dimension N, then we tune instead the coefficients of the matrix for which the vector is an eigenvector.

[zamar]

The method is a practical approach and not mathematically very sound. Because algorithm is very simple, it's very
likely already invented a long time ago.

It is not so unsound. It is like the SPSA algorithm, except SPSA does not use self-play. You can read about SPSA there, if you are interested:
http://www.jhuapl.edu/SPSA/

Thanks for the link Remi. Yes, it looks like that our method is SPSA with the exception of self-play.

As I mentioned in my paper, SPSA has the potential to be close in performance to CLOP, but its main weakness (as Joona says) is that it is very difficult to choose good values for all its meta-parameters. In my experiments, SPSA with optimal meta-parameters performs like CLOP. But in practice, it is not possible to find the optimal meta-parameters of SPSA, so I'd prefer using CLOP.

The attractiveness in SPSA for us is that when we already have a very good starting value, it immediately starts to improve it. While CLOP (if I've understood it correctly) always starts from scratch. Most of the SF tuning was done using only one QUAD-core computer, so we could only use at maximum 100'000 games for each set of variables (set containing 7-30 variables).

But I don't know, it's of course possible that CLOP could have done better...

[zamar]

How many CPUs, and how much time, do you need for 30,000 games?

We used time controls to reach 100'000 games/1 week. Only one quad core computer was used. We ran 1CPU matches on parallel although this adds some extra noise to the results.

[zamar]

and were these fixed depth or timed matches?

We used time controls 5s+0.1s, 10s+0.1s and 20s+0.1s. If we suspected that variables are dependant on the time control, we used longer time controls and ran fewer matches on parallel.

[ethanara]

Can you tell which variables are tuned in stockfish? Or if there are more tuned than untuned, then which are untuned?
Regards
Ethan

[zamar]

Created quickly a page in wiki:

https://chessprogramming.wikispaces.com ... ing+method

Oups sorry, I didn't recognize it was you (not aware of all you confusing alias names), but thought somebody else had copied/pasted the text without quoting original source - so I deleted the page but restored it later

I have edited it slightly. Thank you!

Cheers,
Gerd

Thanks for cleaning up the page and linking it to the right places, Gerd!

Cheers,
Joona

[mcostalba]

Can you tell which variables are tuned in stockfish? Or if there are more tuned than untuned, then which are untuned?
Regards
Ethan

Because you are a little kid (12 year's old if I guess right) I take the chance to do a small game with you (no offence, just for playing), because I think it is important you learn early on to understand what your questions means (and this is important not only related to chess engines). This is a prerequisite to understand the corresponding answers.

So please, as a little homework, if you wish write here below the list of what you think can be "tunable variables" in SF and I will tell to you for each one of them if are already tuned of not. Just to be clear I mean the real variable names used in the program, not general, grouping names.