Page 4 of 7
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 3:12 am
by thomasahle
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 10:18 am
by Joerg Oster
thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 11:06 am
by Michel
Joerg Oster wrote: ↑Sun Sep 01, 2019 10:18 am
thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Maybe you are running out of memory?
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 12:00 pm
by Joerg Oster
Joerg Oster wrote: ↑Sun Sep 01, 2019 10:18 am
thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
So I just tried 'GBRT' as base estimator (with 'LCB' as acq_func, although this seems to be the default anyway).
Good news, from begin to the end it was running smoothly and at the same speed!
Here is the final result:
Code: Select all
Summarizing best values
Best expectation (κ=0): [ 8 48 28 99 9 35 69 7 0 0] = -0.000 ± 0.824 (ELO-diff -0.000 ± 405.717)
Best expectation (κ=1): [34 18 46 81 28 36 8 80 3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Best expectation (κ=2): [34 18 46 81 28 36 8 80 3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Best expectation (κ=3): [34 18 46 81 28 36 8 80 3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Next I will try 'RF'.
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 12:07 pm
by Joerg Oster
Michel wrote: ↑Sun Sep 01, 2019 11:06 am
Joerg Oster wrote: ↑Sun Sep 01, 2019 10:18 am
thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Maybe you are running out of memory?
How can I find out?
Total sytem memory is 8 GB.
Note, a testrun with 'GBRT' as base estimator just finished smoothly!
And from the API documentation here
https://scikit-optimize.github.io/, I don't immediately see why the 'GP' method should take so much more computation time.
(But I am only a noob, of course!)
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 12:33 pm
by Joerg Oster
Michel wrote: ↑Sun Sep 01, 2019 11:06 am
Joerg Oster wrote: ↑Sun Sep 01, 2019 10:18 am
thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Hi Thomas,
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Maybe you are running out of memory?
'top' tells me the python3 process stays permanently below 5% of memory usage.
And at that time the process already started to show a noticeable slowdown.
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM ZEIT+ BEFEHL
28328 joster 20 0 807180 367740 31776 R 150,5 4,5 8:29.91 python3
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 12:44 pm
by Joerg Oster
thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am
Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Is this of any help?
Code: Select all
Summarizing best values
unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<main() done, defined at tune.py:299> exception=ValueError('math domain error')>
Traceback (most recent call last):
File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
self.run_forever()
File "/usr/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
self._run_once()
File "/usr/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
handle._run()
File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "tune.py", line 271, in on_done
opt.tell(x, -y) # opt is minimizing
File "/home/joster/.local/lib/python3.7/site-packages/skopt/optimizer/optimizer.py", line 443, in tell
return self._tell(x, y, fit=fit)
File "/home/joster/.local/lib/python3.7/site-packages/skopt/optimizer/optimizer.py", line 502, in _tell
acq_func_kwargs=self.acq_func_kwargs)
File "/home/joster/.local/lib/python3.7/site-packages/skopt/acquisition.py", line 42, in _gaussian_acquisition
func_and_grad = gaussian_lcb(X, model, kappa, return_grad)
File "/home/joster/.local/lib/python3.7/site-packages/skopt/acquisition.py", line 142, in gaussian_lcb
mu, std = model.predict(X, return_std=True)
File "/home/joster/.local/lib/python3.7/site-packages/skopt/learning/gaussian_process/gpr.py", line 324, in predict
y_var -= np.einsum("ki,kj,ij->k", K_trans, K_trans, K_inv)
File "<__array_function__ internals>", line 6, in einsum
File "/home/joster/.local/lib/python3.7/site-packages/numpy/core/einsumfunc.py", line 1356, in einsum
return c_einsum(*operands, **kwargs)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tune.py", line 414, in main
pm = max(abs(score_to_elo(y_pred[i] / 2 + sigma[i] / 2) - elo),
File "tune.py", line 412, in score_to_elo
return - 400 * math.log10(2 / (score + 1) - 1)
ValueError: math domain error
This came up after cancelling the script while tracking the memory usage with 'top'.
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 12:49 pm
by Ratosh
I had the same error using RF as -base-estimator.
Other issue i had: When data file has 100 valid lines and the number of iterations (-n) option is 100, it will give me 'Not enought data to summarize results.'
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 2:08 pm
by Ferdy
Tried to monitor memory and cpu usage of tune.py. I tuned 6 eval parameters, concurrency is 6 and set at 1000 games and 50ms/move, others are default.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.
This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.
mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s
I think with this setting my hardware can safely reach 600 games without problems.
Re: New Tool for Tuning with Skopt
Posted: Sun Sep 01, 2019 2:50 pm
by Joerg Oster
Ferdy wrote: ↑Sun Sep 01, 2019 2:08 pm
Tried to monitor memory and cpu usage of tune.py. I tuned 6 eval parameters, concurrency is 6 and set at 1000 games and 50ms/move, others are default.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.
This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.
mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s
I think with this setting my hardware can safely reach 600 games without problems.
I observe similar memory consumption wth base estimator set to 'RF'.
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM ZEIT+ BEFEHL
1097 joster 20 0 3933736 3,363g 30580 R 99,7 43,5 92:34.85 python3
This was after about 1,300 iterations, and free memory had gone down to 7% of total memory,
so I stopped here.