It's very easy to use, requiring only a few python packages to run, but no external tools or "connection scripts".
I used it to tune my new fastText & mcts based chess engine: https://github.com/thomasahle/fastchess and gained about 200 ELO (according to a cutechess tournament run afterwards.)
I just wrote it yesterday (and last night), but it already has a great number of features:
- Smart: Uses Gaussian processes for picking parameters, or any other optimization supported by skopt.
- Easy: Automatically parses option ranges from uci engines.
- Fast: Runs many matches in parallel.
- Resilient: Stores results in a log file. Kill the script and simply rerun script to start from where you left off.
New versions of the engine are played against the default parameters, or against another engine of you choice.
Below is an example of optimizing four parameters:
Code: Select all
$ py tune.py fastchess -nodes 800 -concurrency=8 -book ../cutechess/lines.pgn -games-file tune.pgn -opt MilliCPUCT -opt LegalPolicyTreshold -opt CapturePolicyTreshold -opt CheckPolicyTreshold -n 1000 -log-file data.log
/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
warnings.warn(msg, category=DeprecationWarning)
Loaded book with 173 positions
Loading engines
Parsing options
Reading data.log
Using [2166.0, 7214.0, -6494.0, -2943.0] => -2.0 from log-file
Using [4923.0, 527.0, 4660.0, 7250.0] => -2.0 from log-file
Using [2766.0, 8864.0, 2955.0, 6891.0] => -2.0 from log-file
Using [4429.0, -7236.0, -8824.0, 9966.0] => -2.0 from log-file
Using [3759.0, 5304.0, 4815.0, -4245.0] => -2.0 from log-file
Using [402.0, 4140.0, -2505.0, 7399.0] => -2.0 from log-file
Using [3191.0, 4132.0, -7215.0, 7277.0] => -2.0 from log-file
Using [9191.0, -5090.0, 2703.0, -6528.0] => -2.0 from log-file
Using [6606.0, -4668.0, 3898.0, 8736.0] => -2.0 from log-file
Using [4112.0, -4567.0, -3716.0, -8980.0] => 0.0 from log-file
Using [3728.0, -8675.0, -759.0, 5232.0] => -2.0 from log-file
Using [111.0, -3209.0, -5932.0, -6504.0] => -2.0 from log-file
Using [8791.0, -4908.0, -7604.0, 7226.0] => -2.0 from log-file
Using [5570.0, 1106.0, 9823.0, 4585.0] => -2.0 from log-file
Using [0.0, 10000.0, -10000.0, -10000.0] => -2.0 from log-file
Using [3100.0, 885.0, -4304.0, 2987.0] => -2.0 from log-file
Starting game 1/984 with {'MilliCPUCT': 1666, 'LegalPolicyTreshold': 2260, 'CapturePolicyTreshold': -7713, 'CheckPolicyTreshold': -8541}
Starting game 2/984 with {'MilliCPUCT': 3571, 'LegalPolicyTreshold': -3567, 'CapturePolicyTreshold': -4507, 'CheckPolicyTreshold': -8864}
Starting game 3/984 with {'MilliCPUCT': 0, 'LegalPolicyTreshold': 10000, 'CapturePolicyTreshold': -10000, 'CheckPolicyTreshold': -8143}
...
Starting game 818/984 with {'MilliCPUCT': 3641, 'LegalPolicyTreshold': -2983, 'CapturePolicyTreshold': -627, 'CheckPolicyTreshold': -7392}
[4787, -3097, -3065, -6430] => 0
KeyboardInterrupt
Summarizing best values
Best expectation (κ=0): [ 4299. -3568. -1639. -7074.] = 0.162 ± 0.025 (ELO-diff 56.920 ± 9.071)
If this seems like a useful tool for anyone, even just for people who are too lazy to install CLOP or SPSA, it could be given its own repository instead of being part of fastchess.