New Tool for Tuning with Skopt

[thomasahle]

I wrote a new tuning script using Scikit-Optimize, and thought you might find it interesting.
It's very easy to use, requiring only a few python packages to run, but no external tools or "connection scripts".
I used it to tune my new fastText & mcts based chess engine: https://github.com/thomasahle/fastchess and gained about 200 ELO (according to a cutechess tournament run afterwards.)

I just wrote it yesterday (and last night), but it already has a great number of features:

• Smart: Uses Gaussian processes for picking parameters, or any other optimization supported by skopt.
• Easy: Automatically parses option ranges from uci engines.
• Fast: Runs many matches in parallel.
• Resilient: Stores results in a log file. Kill the script and simply rerun script to start from where you left off.

It doesn't do anything complicated chess wise. Just a bridge between skopt and python-chess by Niklas Fiekas.
New versions of the engine are played against the default parameters, or against another engine of you choice.

Below is an example of optimizing four parameters:

Code: Select all

$ py tune.py fastchess -nodes 800 -concurrency=8 -book ../cutechess/lines.pgn -games-file tune.pgn -opt MilliCPUCT -opt LegalPolicyTreshold -opt CapturePolicyTreshold -opt CheckPolicyTreshold -n 1000 -log-file data.log

/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
Loaded book with 173 positions
Loading engines
Parsing options
Reading data.log
Using [2166.0, 7214.0, -6494.0, -2943.0] => -2.0 from log-file
Using [4923.0, 527.0, 4660.0, 7250.0] => -2.0 from log-file
Using [2766.0, 8864.0, 2955.0, 6891.0] => -2.0 from log-file
Using [4429.0, -7236.0, -8824.0, 9966.0] => -2.0 from log-file
Using [3759.0, 5304.0, 4815.0, -4245.0] => -2.0 from log-file
Using [402.0, 4140.0, -2505.0, 7399.0] => -2.0 from log-file
Using [3191.0, 4132.0, -7215.0, 7277.0] => -2.0 from log-file
Using [9191.0, -5090.0, 2703.0, -6528.0] => -2.0 from log-file
Using [6606.0, -4668.0, 3898.0, 8736.0] => -2.0 from log-file
Using [4112.0, -4567.0, -3716.0, -8980.0] => 0.0 from log-file
Using [3728.0, -8675.0, -759.0, 5232.0] => -2.0 from log-file
Using [111.0, -3209.0, -5932.0, -6504.0] => -2.0 from log-file
Using [8791.0, -4908.0, -7604.0, 7226.0] => -2.0 from log-file
Using [5570.0, 1106.0, 9823.0, 4585.0] => -2.0 from log-file
Using [0.0, 10000.0, -10000.0, -10000.0] => -2.0 from log-file
Using [3100.0, 885.0, -4304.0, 2987.0] => -2.0 from log-file
Starting game 1/984 with {'MilliCPUCT': 1666, 'LegalPolicyTreshold': 2260, 'CapturePolicyTreshold': -7713, 'CheckPolicyTreshold': -8541}
Starting game 2/984 with {'MilliCPUCT': 3571, 'LegalPolicyTreshold': -3567, 'CapturePolicyTreshold': -4507, 'CheckPolicyTreshold': -8864}
Starting game 3/984 with {'MilliCPUCT': 0, 'LegalPolicyTreshold': 10000, 'CapturePolicyTreshold': -10000, 'CheckPolicyTreshold': -8143}
...
Starting game 818/984 with {'MilliCPUCT': 3641, 'LegalPolicyTreshold': -2983, 'CapturePolicyTreshold': -627, 'CheckPolicyTreshold': -7392}
[4787, -3097, -3065, -6430] => 0
KeyboardInterrupt
Summarizing best values
Best expectation (κ=0): [ 4299. -3568. -1639. -7074.] = 0.162 ± 0.025 (ELO-diff 56.920 ± 9.071)

I wouldn't take the ELO too seriously. In particular, the standard deviation bounds come from the Gaussian process estimate, which seems to usually be too low.

If this seems like a useful tool for anyone, even just for people who are too lazy to install CLOP or SPSA, it could be given its own repository instead of being part of fastchess.

[thomasahle]

I should link directly to the script: https://github.com/thomasahle/fastchess ... ls/tune.py

A fun feature is confidence plots when optimizing a single parameter:

Code: Select all

py tools/tune.py fastchess -nodes 100 -book ../cutechess/lines.pgn -opt MilliCpuct -n 40

[zenpawn]

Looks interesting. Could you please say something about how to use it? Thanks!

[thomasahle]

Looks interesting. Could you please say something about how to use it? Thanks!

Right! Make sure you have an engines.json file, like with cutechess. Then simply run

"python tune.py engine-name -opt OptionToBeTuned".

If you don't already have skopt and python-chess you will need to run "pip install skopt python-chess" first.

[zenpawn]

Looks interesting. Could you please say something about how to use it? Thanks!

Right! Make sure you have an engines.json file, like with cutechess. Then simply run

"python tune.py engine-name -opt OptionToBeTuned".

If you don't already have skopt and python-chess you will need to run "pip install skopt python-chess" first.

I guess it was the OptionToBeTuned that was unclear, but I gather now that you're using the Winboard protocol to set those for your engine? And, for multiple options, it's comma-delimited?

[Joerg Oster]

Interesting. Will definitely try it out in the next days.
Thanks for sharing!

[thomasahle]

I guess it was the OptionToBeTuned that was unclear, but I gather now that you're using the Winboard protocol to set those for your engine? And, for multiple options, it's comma-delimited?

It should work with both the xboard/winboard and uci protocols, though I've mostly tested it with the later.
You just add one `-opt OptionToBeTuned` for each option to tune. (See e.g. my first example.) You can also add a custom range, like `-opt OptionToBeTuned lower upper`. For categorical options you can do `-c-opt OptionToBeTuned` or give the values you are interested in `-c-opt OptionToBeTuned value1 value2 value3`.

[thomasahle]

I did a quick test and found the following problems:

my engines.json

Code: Select all

[
	{
		"command" : "danasah730.exe",
		"name" : "danasah730_uci",
...
FileNotFoundError: [WinError 2] El sistema no puede encontrar el archivo especificado

Is it possible to make an executable of tune.py?

At least on unix systems you need to add './' to execute a binary from its directory. That is write './danasah730.exe'.
You can also try specifying your binary with an absolute path.

[pedrox]

I can't get it to work.

In Windows I have installed scikit-optimize and I already had python-chess installed. I am using Python 3.7 32 bits. I have created the engines.json of 2 versions of my engine and to the second I have added the options uci Parameter1 and Parameter2

python tune.py danasah800_uci -movetime 1000 -games-file tune.pgn -opt Parameter1 -opt Parameter2

Code: Select all

C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\sklearn\externals\joblib\__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
Starting every game from initial position
Loading engines
Traceback (most recent call last):
  File "tune.py", line 368, in <module>
    asyncio.run(main())
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 584, in run_until_complete
    return future.result()
  File "tune.py", line 264, in main
    for _ in range(args.concurrency)))
  File "tune.py", line 97, in load_engine
    _, engine = await chess.engine.popen_uci(cmd, **popen_args)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\chess\engine.py", line 2233, in popen_uci
    transport, protocol = await UciProtocol.popen(command, setpgrp=setpgrp, loop=loop, **popen_args)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\chess\engine.py", line 894, in popen
    return await loop.subprocess_exec(cls, *command, **popen_args)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 1533, in subprocess_exec
    bufsize, **kwargs)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\windows_events.py", line 371, in _make_subprocess_transport
    **kwargs)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_subprocess.py", line 37, in __init__
    stderr=stderr, bufsize=bufsize, **kwargs)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\windows_events.py", line 792, in _start
    bufsize=bufsize, **kwargs)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\windows_utils.py", line 155, in __init__
    stderr=stderr_wfd, **kwds)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] El sistema no puede encontrar el archivo especificado

Is it possible for someone to do an exe from tune.py?

[pedrox]

I did a quick test and found the following problems:

my engines.json

Code: Select all

[
	{
		"command" : "danasah730.exe",
		"name" : "danasah730_uci",
...
FileNotFoundError: [WinError 2] El sistema no puede encontrar el archivo especificado

Is it possible to make an executable of tune.py?

At least on unix systems you need to add './' to execute a binary from its directory. That is write './danasah730.exe'.
You can also try specifying your binary with an absolute path.

I deleted my first message

I already used a engines.json file that works with cutechess, in Windows it is enough to indicate the working directory and you can directly put the executable.

Code: Select all

[
	{
		"command" : "danasah730.exe",
		"name" : "danasah730_uci",
		"options" : [
			{
				"alias" : "",
				"default" : "128",
				"name" : "Hash",
				"type" : "text",
				"value" : "128"
			},
			{
				"alias" : "",
				"default" : "d:\\gtb\\",
				"name" : "GaviotaTbPath",
				"type" : "text",
				"value" : "d:\\gtb\\"
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"workingDirectory" : "d:\\cutechess\\engines\\danasah7"
	}
]

But with your suggestion it seems to work!