New Tool for Tuning with Skopt

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

Re: New Tool for Tuning with Skopt

Post by thomasahle » Sun Sep 01, 2019 1:12 am

Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Sun Sep 01, 2019 8:18 am

thomasahle wrote:
Sun Sep 01, 2019 1:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Jörg Oster

Michel
Posts: 2053
Joined: Sun Sep 28, 2008 11:50 pm

Re: New Tool for Tuning with Skopt

Post by Michel » Sun Sep 01, 2019 9:06 am

Joerg Oster wrote:
Sun Sep 01, 2019 8:18 am
thomasahle wrote:
Sun Sep 01, 2019 1:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Maybe you are running out of memory?
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Sun Sep 01, 2019 10:00 am

Joerg Oster wrote:
Sun Sep 01, 2019 8:18 am
thomasahle wrote:
Sun Sep 01, 2019 1:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
So I just tried 'GBRT' as base estimator (with 'LCB' as acq_func, although this seems to be the default anyway).
Good news, from begin to the end it was running smoothly and at the same speed! :D

Here is the final result:

Code: Select all

Summarizing best values
Best expectation (κ=0): [ 8 48 28 99  9 35 69  7  0  0] = -0.000 ± 0.824 (ELO-diff -0.000 ± 405.717)
Best expectation (κ=1): [34 18 46 81 28 36  8 80  3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Best expectation (κ=2): [34 18 46 81 28 36  8 80  3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Best expectation (κ=3): [34 18 46 81 28 36  8 80  3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Next I will try 'RF'.
Jörg Oster

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Sun Sep 01, 2019 10:07 am

Michel wrote:
Sun Sep 01, 2019 9:06 am
Joerg Oster wrote:
Sun Sep 01, 2019 8:18 am
thomasahle wrote:
Sun Sep 01, 2019 1:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Maybe you are running out of memory?
How can I find out?
Total sytem memory is 8 GB.

Note, a testrun with 'GBRT' as base estimator just finished smoothly!
And from the API documentation here https://scikit-optimize.github.io/, I don't immediately see why the 'GP' method should take so much more computation time.
(But I am only a noob, of course!)
Jörg Oster

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Sun Sep 01, 2019 10:33 am

Michel wrote:
Sun Sep 01, 2019 9:06 am
Joerg Oster wrote:
Sun Sep 01, 2019 8:18 am
thomasahle wrote:
Sun Sep 01, 2019 1:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Maybe you are running out of memory?
'top' tells me the python3 process stays permanently below 5% of memory usage. :)
And at that time the process already started to show a noticeable slowdown.

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     ZEIT+ BEFEHL   
28328 joster    20   0  807180 367740  31776 R 150,5  4,5   8:29.91 python3     
Jörg Oster

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Sun Sep 01, 2019 10:44 am

thomasahle wrote:
Sun Sep 01, 2019 1:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Is this of any help?

Code: Select all

Summarizing best values
unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<main() done, defined at tune.py:299> exception=ValueError('math domain error')>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
    self._run_once()
  File "/usr/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
    handle._run()
  File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "tune.py", line 271, in on_done
    opt.tell(x, -y)  # opt is minimizing
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/optimizer/optimizer.py", line 443, in tell
    return self._tell(x, y, fit=fit)
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/optimizer/optimizer.py", line 502, in _tell
    acq_func_kwargs=self.acq_func_kwargs)
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/acquisition.py", line 42, in _gaussian_acquisition
    func_and_grad = gaussian_lcb(X, model, kappa, return_grad)
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/acquisition.py", line 142, in gaussian_lcb
    mu, std = model.predict(X, return_std=True)
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/learning/gaussian_process/gpr.py", line 324, in predict
    y_var -= np.einsum("ki,kj,ij->k", K_trans, K_trans, K_inv)
  File "<__array_function__ internals>", line 6, in einsum
  File "/home/joster/.local/lib/python3.7/site-packages/numpy/core/einsumfunc.py", line 1356, in einsum
    return c_einsum(*operands, **kwargs)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tune.py", line 414, in main
    pm = max(abs(score_to_elo(y_pred[i] / 2 + sigma[i] / 2) - elo),
  File "tune.py", line 412, in score_to_elo
    return - 400 * math.log10(2 / (score + 1) - 1)
ValueError: math domain error
This came up after cancelling the script while tracking the memory usage with 'top'.
Jörg Oster

Ratosh
Posts: 71
Joined: Mon Apr 16, 2018 4:56 pm

Re: New Tool for Tuning with Skopt

Post by Ratosh » Sun Sep 01, 2019 10:49 am

I had the same error using RF as -base-estimator.

Other issue i had: When data file has 100 valid lines and the number of iterations (-n) option is 100, it will give me 'Not enought data to summarize results.'

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: New Tool for Tuning with Skopt

Post by Ferdy » Sun Sep 01, 2019 12:08 pm

Tried to monitor memory and cpu usage of tune.py. I tuned 6 eval parameters, concurrency is 6 and set at 1000 games and 50ms/move, others are default.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.

This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.

Image

mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s

I think with this setting my hardware can safely reach 600 games without problems.

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Sun Sep 01, 2019 12:50 pm

Ferdy wrote:
Sun Sep 01, 2019 12:08 pm
Tried to monitor memory and cpu usage of tune.py. I tuned 6 eval parameters, concurrency is 6 and set at 1000 games and 50ms/move, others are default.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.

This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.

Image

mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s

I think with this setting my hardware can safely reach 600 games without problems.
I observe similar memory consumption wth base estimator set to 'RF'.

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     ZEIT+ BEFEHL   
 1097 joster    20   0 3933736 3,363g  30580 R  99,7 43,5  92:34.85 python3     
This was after about 1,300 iterations, and free memory had gone down to 7% of total memory,
so I stopped here.
Jörg Oster

Post Reply