New Tool for Tuning with Skopt

[thomasahle]

Once I have played a series of games and I have the file data.log, if I run the test again and use the file, in each run I have different results and the results seem quite random.

This happens because summarizing works by sampling a number of random values. As far as I can see the arguments you get out are quite similar.

I realize this might actually be annoying, so I've pushed a new version that summarizes using gradient descent and should be more deterministic (though still not entirely, as it picks a 100 random starting points for the decent.)

[voffka]

Hello thomasahle,

Once I have played a series of games and I have the file data.log, if I run the test again and use the file, in each run I have different results and the results seem quite random.

This happens because summarizing works by sampling a number of random values. As far as I can see the arguments you get out are quite similar.

I realize this might actually be annoying, so I've pushed a new version that summarizes using gradient descent and should be more deterministic (though still not entirely, as it picks a 100 random starting points for the decent.)

Thanks for the tool!

After some manual hand tuning I decided it is time to master your tool and I got the following error on a latest version from master:

Code: Select all

Starting 2 games 504/1000 with {'Option_1': 376, 'Option_2': 11}
Summarizing best values
play_game Cancelled
play_game Cancelled
play_game Cancelled
play_game Cancelled
Traceback (most recent call last):
  File "tune.py", line 523, in <module>
    asyncio.run(main())
  File "C:\Users\volodymyr\AppData\Local\Programs\Python\Python37-32\lib\asyncio\runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "C:\Users\volodymyr\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 579, in run_until_complete
    return future.result()
  File "tune.py", line 497, in main
    summarize(opt, steps=1)
  File "tune.py", line 332, in summarize
    new_opt.tell(opt.Xi, opt.yi)
  File "C:\Users\volodymyr\AppData\Local\Programs\Python\Python37-32\lib\site-packages\skopt\optimizer\optimizer.py", line 443, in tell
    return self._tell(x, y, fit=fit)
  File "C:\Users\volodymyr\AppData\Local\Programs\Python\Python37-32\lib\site-packages\skopt\optimizer\optimizer.py", line 502, in _tell
    acq_func_kwargs=self.acq_func_kwargs)
  File "C:\Users\volodymyr\AppData\Local\Programs\Python\Python37-32\lib\site-packages\skopt\acquisition.py", line 42, in _gaussian_acquisition
    func_and_grad = gaussian_lcb(X, model, kappa, return_grad)
  File "C:\Users\volodymyr\AppData\Local\Programs\Python\Python37-32\lib\site-packages\skopt\acquisition.py", line 142, in gaussian_lcb
    mu, std = model.predict(X, return_std=True)
  File "C:\Users\volodymyr\AppData\Local\Programs\Python\Python37-32\lib\site-packages\skopt\learning\gaussian_process\gpr.py", line 324, in predict
    y_var -= np.einsum("ki,kj,ij->k", K_trans, K_trans, K_inv)
  File "<__array_function__ internals>", line 6, in einsum
  File "C:\Users\volodymyr\AppData\Local\Programs\Python\Python37-32\lib\site-packages\numpy\core\einsumfunc.py", line 1356, in einsum
    return c_einsum(*operands, **kwargs)
ValueError: iterator is too large

My command line is the following:

Code: Select all

python tune.py igel -concurrency=4 -nodes 3000 -book C:\Users\volodymyr\Documents\Sources\fastchess\tools\gaviota-starters.pgn -games-file tune.pgn -opt Option_1 -opt Option_2 -n 1000 -base-estimator GBRT -acq-noise 5 -log-file data.log