New Tool for Tuning with Skopt

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

New Tool for Tuning with Skopt

Post by thomasahle » Sun Aug 25, 2019 10:05 am

I wrote a new tuning script using Scikit-Optimize, and thought you might find it interesting.
It's very easy to use, requiring only a few python packages to run, but no external tools or "connection scripts".
I used it to tune my new fastText & mcts based chess engine: https://github.com/thomasahle/fastchess and gained about 200 ELO (according to a cutechess tournament run afterwards.)

I just wrote it yesterday (and last night), but it already has a great number of features:
  • Smart: Uses Gaussian processes for picking parameters, or any other optimization supported by skopt.
  • Easy: Automatically parses option ranges from uci engines.
  • Fast: Runs many matches in parallel.
  • Resilient: Stores results in a log file. Kill the script and simply rerun script to start from where you left off.
It doesn't do anything complicated chess wise. Just a bridge between skopt and python-chess by Niklas Fiekas.
New versions of the engine are played against the default parameters, or against another engine of you choice.

Below is an example of optimizing four parameters:

Code: Select all

$ py tune.py fastchess -nodes 800 -concurrency=8 -book ../cutechess/lines.pgn -games-file tune.pgn -opt MilliCPUCT -opt LegalPolicyTreshold -opt CapturePolicyTreshold -opt CheckPolicyTreshold -n 1000 -log-file data.log

/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
Loaded book with 173 positions
Loading engines
Parsing options
Reading data.log
Using [2166.0, 7214.0, -6494.0, -2943.0] => -2.0 from log-file
Using [4923.0, 527.0, 4660.0, 7250.0] => -2.0 from log-file
Using [2766.0, 8864.0, 2955.0, 6891.0] => -2.0 from log-file
Using [4429.0, -7236.0, -8824.0, 9966.0] => -2.0 from log-file
Using [3759.0, 5304.0, 4815.0, -4245.0] => -2.0 from log-file
Using [402.0, 4140.0, -2505.0, 7399.0] => -2.0 from log-file
Using [3191.0, 4132.0, -7215.0, 7277.0] => -2.0 from log-file
Using [9191.0, -5090.0, 2703.0, -6528.0] => -2.0 from log-file
Using [6606.0, -4668.0, 3898.0, 8736.0] => -2.0 from log-file
Using [4112.0, -4567.0, -3716.0, -8980.0] => 0.0 from log-file
Using [3728.0, -8675.0, -759.0, 5232.0] => -2.0 from log-file
Using [111.0, -3209.0, -5932.0, -6504.0] => -2.0 from log-file
Using [8791.0, -4908.0, -7604.0, 7226.0] => -2.0 from log-file
Using [5570.0, 1106.0, 9823.0, 4585.0] => -2.0 from log-file
Using [0.0, 10000.0, -10000.0, -10000.0] => -2.0 from log-file
Using [3100.0, 885.0, -4304.0, 2987.0] => -2.0 from log-file
Starting game 1/984 with {'MilliCPUCT': 1666, 'LegalPolicyTreshold': 2260, 'CapturePolicyTreshold': -7713, 'CheckPolicyTreshold': -8541}
Starting game 2/984 with {'MilliCPUCT': 3571, 'LegalPolicyTreshold': -3567, 'CapturePolicyTreshold': -4507, 'CheckPolicyTreshold': -8864}
Starting game 3/984 with {'MilliCPUCT': 0, 'LegalPolicyTreshold': 10000, 'CapturePolicyTreshold': -10000, 'CheckPolicyTreshold': -8143}
...
Starting game 818/984 with {'MilliCPUCT': 3641, 'LegalPolicyTreshold': -2983, 'CapturePolicyTreshold': -627, 'CheckPolicyTreshold': -7392}
[4787, -3097, -3065, -6430] => 0
KeyboardInterrupt
Summarizing best values
Best expectation (κ=0): [ 4299. -3568. -1639. -7074.] = 0.162 ± 0.025 (ELO-diff 56.920 ± 9.071)
I wouldn't take the ELO too seriously. In particular, the standard deviation bounds come from the Gaussian process estimate, which seems to usually be too low.

If this seems like a useful tool for anyone, even just for people who are too lazy to install CLOP or SPSA, it could be given its own repository instead of being part of fastchess.

thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

Re: New Tool for Tuning with Skopt

Post by thomasahle » Sun Aug 25, 2019 1:04 pm

I should link directly to the script: https://github.com/thomasahle/fastchess ... ls/tune.py

A fun feature is confidence plots when optimizing a single parameter:

Code: Select all

py tools/tune.py fastchess -nodes 100 -book ../cutechess/lines.pgn -opt MilliCpuct -n 40
Image

zenpawn
Posts: 296
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

Re: New Tool for Tuning with Skopt

Post by zenpawn » Mon Aug 26, 2019 12:51 am

Looks interesting. Could you please say something about how to use it? :) Thanks!
Erin Dame
Author of RookieMonster

thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

Re: New Tool for Tuning with Skopt

Post by thomasahle » Mon Aug 26, 2019 9:31 am

zenpawn wrote:
Mon Aug 26, 2019 12:51 am
Looks interesting. Could you please say something about how to use it? :) Thanks!
Right! Make sure you have an engines.json file, like with cutechess. Then simply run

"python tune.py engine-name -opt OptionToBeTuned".

If you don't already have skopt and python-chess you will need to run "pip install skopt python-chess" first.

zenpawn
Posts: 296
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

Re: New Tool for Tuning with Skopt

Post by zenpawn » Mon Aug 26, 2019 12:49 pm

thomasahle wrote:
Mon Aug 26, 2019 9:31 am
zenpawn wrote:
Mon Aug 26, 2019 12:51 am
Looks interesting. Could you please say something about how to use it? :) Thanks!
Right! Make sure you have an engines.json file, like with cutechess. Then simply run

"python tune.py engine-name -opt OptionToBeTuned".

If you don't already have skopt and python-chess you will need to run "pip install skopt python-chess" first.
I guess it was the OptionToBeTuned that was unclear, but I gather now that you're using the Winboard protocol to set those for your engine? And, for multiple options, it's comma-delimited?
Erin Dame
Author of RookieMonster

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Mon Aug 26, 2019 1:30 pm

Interesting. Will definitely try it out in the next days.
Thanks for sharing!
Jörg Oster

thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

Re: New Tool for Tuning with Skopt

Post by thomasahle » Mon Aug 26, 2019 2:40 pm

zenpawn wrote:
Mon Aug 26, 2019 12:49 pm
I guess it was the OptionToBeTuned that was unclear, but I gather now that you're using the Winboard protocol to set those for your engine? And, for multiple options, it's comma-delimited?
It should work with both the xboard/winboard and uci protocols, though I've mostly tested it with the later.
You just add one `-opt OptionToBeTuned` for each option to tune. (See e.g. my first example.) You can also add a custom range, like `-opt OptionToBeTuned lower upper`. For categorical options you can do `-c-opt OptionToBeTuned` or give the values you are interested in `-c-opt OptionToBeTuned value1 value2 value3`.

thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

Re: New Tool for Tuning with Skopt

Post by thomasahle » Mon Aug 26, 2019 6:58 pm

pedrox wrote:
Mon Aug 26, 2019 6:52 pm
I did a quick test and found the following problems:

my engines.json

Code: Select all

[
	{
		"command" : "danasah730.exe",
		"name" : "danasah730_uci",
...
FileNotFoundError: [WinError 2] El sistema no puede encontrar el archivo especificado
Is it possible to make an executable of tune.py?
At least on unix systems you need to add './' to execute a binary from its directory. That is write './danasah730.exe'.
You can also try specifying your binary with an absolute path.

User avatar
pedrox
Posts: 992
Joined: Fri Mar 10, 2006 5:07 am
Location: Basque Country (Spain)
Contact:

Re: New Tool for Tuning with Skopt

Post by pedrox » Mon Aug 26, 2019 7:07 pm

I can't get it to work.

In Windows I have installed scikit-optimize and I already had python-chess installed. I am using Python 3.7 32 bits. I have created the engines.json of 2 versions of my engine and to the second I have added the options uci Parameter1 and Parameter2

python tune.py danasah800_uci -movetime 1000 -games-file tune.pgn -opt Parameter1 -opt Parameter2

Code: Select all

C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\sklearn\externals\joblib\__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
Starting every game from initial position
Loading engines
Traceback (most recent call last):
  File "tune.py", line 368, in <module>
    asyncio.run(main())
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 584, in run_until_complete
    return future.result()
  File "tune.py", line 264, in main
    for _ in range(args.concurrency)))
  File "tune.py", line 97, in load_engine
    _, engine = await chess.engine.popen_uci(cmd, **popen_args)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\chess\engine.py", line 2233, in popen_uci
    transport, protocol = await UciProtocol.popen(command, setpgrp=setpgrp, loop=loop, **popen_args)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\chess\engine.py", line 894, in popen
    return await loop.subprocess_exec(cls, *command, **popen_args)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 1533, in subprocess_exec
    bufsize, **kwargs)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\windows_events.py", line 371, in _make_subprocess_transport
    **kwargs)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_subprocess.py", line 37, in __init__
    stderr=stderr, bufsize=bufsize, **kwargs)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\windows_events.py", line 792, in _start
    bufsize=bufsize, **kwargs)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\asyncio\windows_utils.py", line 155, in __init__
    stderr=stderr_wfd, **kwds)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "C:\Users\pecas\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] El sistema no puede encontrar el archivo especificado
Is it possible for someone to do an exe from tune.py?

User avatar
pedrox
Posts: 992
Joined: Fri Mar 10, 2006 5:07 am
Location: Basque Country (Spain)
Contact:

Re: New Tool for Tuning with Skopt

Post by pedrox » Mon Aug 26, 2019 7:14 pm

thomasahle wrote:
Mon Aug 26, 2019 6:58 pm
pedrox wrote:
Mon Aug 26, 2019 6:52 pm
I did a quick test and found the following problems:

my engines.json

Code: Select all

[
	{
		"command" : "danasah730.exe",
		"name" : "danasah730_uci",
...
FileNotFoundError: [WinError 2] El sistema no puede encontrar el archivo especificado
Is it possible to make an executable of tune.py?
At least on unix systems you need to add './' to execute a binary from its directory. That is write './danasah730.exe'.
You can also try specifying your binary with an absolute path.
I deleted my first message

I already used a engines.json file that works with cutechess, in Windows it is enough to indicate the working directory and you can directly put the executable.

Code: Select all

[
	{
		"command" : "danasah730.exe",
		"name" : "danasah730_uci",
		"options" : [
			{
				"alias" : "",
				"default" : "128",
				"name" : "Hash",
				"type" : "text",
				"value" : "128"
			},
			{
				"alias" : "",
				"default" : "d:\\gtb\\",
				"name" : "GaviotaTbPath",
				"type" : "text",
				"value" : "d:\\gtb\\"
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"workingDirectory" : "d:\\cutechess\\engines\\danasah7"
	}
]
But with your suggestion it seems to work!

Post Reply