New Tool for Tuning with Skopt

[thomasahle]

Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.

[Joerg Oster]

Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.

Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.

[Michel]

Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.

Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.

Maybe you are running out of memory?

[Joerg Oster]

Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.

Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.

So I just tried 'GBRT' as base estimator (with 'LCB' as acq_func, although this seems to be the default anyway).
Good news, from begin to the end it was running smoothly and at the same speed!

Here is the final result:

Code: Select all

Summarizing best values
Best expectation (κ=0): [ 8 48 28 99  9 35 69  7  0  0] = -0.000 ± 0.824 (ELO-diff -0.000 ± 405.717)
Best expectation (κ=1): [34 18 46 81 28 36  8 80  3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Best expectation (κ=2): [34 18 46 81 28 36  8 80  3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Best expectation (κ=3): [34 18 46 81 28 36  8 80  3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)

Next I will try 'RF'.

[Joerg Oster]

Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.

Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.

Maybe you are running out of memory?

How can I find out?
Total sytem memory is 8 GB.

Note, a testrun with 'GBRT' as base estimator just finished smoothly!
And from the API documentation here https://scikit-optimize.github.io/, I don't immediately see why the 'GP' method should take so much more computation time.
(But I am only a noob, of course!)

[Joerg Oster]

Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.

Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.

I also tried playing games without a book, but this also made no noticeable difference in terms of speed.

And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.

Maybe you are running out of memory?

'top' tells me the python3 process stays permanently below 5% of memory usage.
And at that time the process already started to show a noticeable slowdown.

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     ZEIT+ BEFEHL   
28328 joster    20   0  807180 367740  31776 R 150,5  4,5   8:29.91 python3     

[Joerg Oster]

Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.

But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.

Is this of any help?

Code: Select all

Summarizing best values
unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<main() done, defined at tune.py:299> exception=ValueError('math domain error')>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
    self._run_once()
  File "/usr/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
    handle._run()
  File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "tune.py", line 271, in on_done
    opt.tell(x, -y)  # opt is minimizing
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/optimizer/optimizer.py", line 443, in tell
    return self._tell(x, y, fit=fit)
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/optimizer/optimizer.py", line 502, in _tell
    acq_func_kwargs=self.acq_func_kwargs)
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/acquisition.py", line 42, in _gaussian_acquisition
    func_and_grad = gaussian_lcb(X, model, kappa, return_grad)
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/acquisition.py", line 142, in gaussian_lcb
    mu, std = model.predict(X, return_std=True)
  File "/home/joster/.local/lib/python3.7/site-packages/skopt/learning/gaussian_process/gpr.py", line 324, in predict
    y_var -= np.einsum("ki,kj,ij->k", K_trans, K_trans, K_inv)
  File "<__array_function__ internals>", line 6, in einsum
  File "/home/joster/.local/lib/python3.7/site-packages/numpy/core/einsumfunc.py", line 1356, in einsum
    return c_einsum(*operands, **kwargs)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tune.py", line 414, in main
    pm = max(abs(score_to_elo(y_pred[i] / 2 + sigma[i] / 2) - elo),
  File "tune.py", line 412, in score_to_elo
    return - 400 * math.log10(2 / (score + 1) - 1)
ValueError: math domain error

This came up after cancelling the script while tracking the memory usage with 'top'.

[Ratosh]

I had the same error using RF as -base-estimator.

Other issue i had: When data file has 100 valid lines and the number of iterations (-n) option is 100, it will give me 'Not enought data to summarize results.'

[Ferdy]

Tried to monitor memory and cpu usage of tune.py. I tuned 6 eval parameters, concurrency is 6 and set at 1000 games and 50ms/move, others are default.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.

This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.



mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s

I think with this setting my hardware can safely reach 600 games without problems.

[Joerg Oster]

Tried to monitor memory and cpu usage of tune.py. I tuned 6 eval parameters, concurrency is 6 and set at 1000 games and 50ms/move, others are default.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.

This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.



mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s

I think with this setting my hardware can safely reach 600 games without problems.

I observe similar memory consumption wth base estimator set to 'RF'.

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     ZEIT+ BEFEHL   
 1097 joster    20   0 3933736 3,363g  30580 R  99,7 43,5  92:34.85 python3     

This was after about 1,300 iterations, and free memory had gone down to 7% of total memory,
so I stopped here.