Page 5 of 7

Re: New Tool for Tuning with Skopt

Posted: Mon Sep 02, 2019 12:49 pm
by thomasahle
Ratosh wrote:
Fri Aug 30, 2019 7:09 am
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
I have pushed a new version that should fix these issues. Thank you for the suggestions!

It turns out that skopt saves all previously "fitted models" for the gaussian optimization. I now delete all but the latest, which will hopefully mitigate Joerg Oster's memory leak problem.

Re: New Tool for Tuning with Skopt

Posted: Mon Sep 02, 2019 1:12 pm
by Joerg Oster
thomasahle wrote:
Mon Sep 02, 2019 12:49 pm
Ratosh wrote:
Fri Aug 30, 2019 7:09 am
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
I have pushed a new version that should fix these issues. Thank you for the suggestions!

It turns out that skopt saves all previously "fitted models" for the gaussian optimization. I now delete all but the latest, which will hopefully mitigate Joerg Oster's memory leak problem.
Thanks, will try it.

Re: New Tool for Tuning with Skopt

Posted: Mon Sep 02, 2019 7:45 pm
by Ratosh
Seems way better!

Thank you.

Re: New Tool for Tuning with Skopt

Posted: Thu Sep 05, 2019 1:16 am
by voffka
thomasahle,

Very nice tool! Thx a lot for sharing it.

jorose,
jorose wrote:
Fri Aug 30, 2019 5:08 am
Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.
How many games did you have to run during tunning to get this elo? I am trying to tune SEE evals in Igel, not clear yet how many games I need to run. Also, I presume for search it is better to use nodes limit per move I assume?

Thanks!

Re: New Tool for Tuning with Skopt

Posted: Thu Sep 05, 2019 3:44 am
by jorose
voffka wrote:
Thu Sep 05, 2019 1:16 am
thomasahle,

Very nice tool! Thx a lot for sharing it.

jorose,
jorose wrote:
Fri Aug 30, 2019 5:08 am
Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.
How many games did you have to run during tunning to get this elo? I am trying to tune SEE evals in Igel, not clear yet how many games I need to run. Also, I presume for search it is better to use nodes limit per move I assume?

Thanks!
I ran it for roughly 800 games if I recall correctly. I was running the games with time per move.

Re: New Tool for Tuning with Skopt

Posted: Thu Sep 05, 2019 7:57 am
by voffka
Thanks jorose,

Currently I am running 5000 games with nodes=800 to tune 4 parameters at once. I will report the status here once it is done.

Re: New Tool for Tuning with Skopt

Posted: Thu Sep 05, 2019 12:19 pm
by thomasahle
voffka wrote:
Thu Sep 05, 2019 7:57 am
Currently I am running 5000 games with nodes=800 to tune 4 parameters at once. I will report the status here once it is done.
Note that if you are doing that many games, it might be better to use the GBRT estimator by Jorg's experiements. This is done with the `-base-estimator GBRT` flag. See https://github.com/thomasahle/fastchess/issues/10 for the discussion.

I'm excited to hear your results!

Re: New Tool for Tuning with Skopt

Posted: Sun Sep 08, 2019 10:53 am
by xr_a_y
Hi,

I'm trying it now, using xboard protocol, and I'm not really successful because python-chess seems to be rejecting "usermove=1".
Anyone experience this before ?

Re: New Tool for Tuning with Skopt

Posted: Sun Sep 08, 2019 4:58 pm
by JVMerlino
xr_a_y wrote:
Sun Sep 08, 2019 10:53 am
Hi,

I'm trying it now, using xboard protocol, and I'm not really successful because python-chess seems to be rejecting "usermove=1".
Anyone experience this before ?
Soon I'll be trying it with Myrddin, also an xboard engine. Hoping you have some success!

Re: New Tool for Tuning with Skopt

Posted: Sun Sep 08, 2019 5:12 pm
by xr_a_y
JVMerlino wrote:
Sun Sep 08, 2019 4:58 pm
xr_a_y wrote:
Sun Sep 08, 2019 10:53 am
Hi,

I'm trying it now, using xboard protocol, and I'm not really successful because python-chess seems to be rejecting "usermove=1".
Anyone experience this before ?
Soon I'll be trying it with Myrddin, also an xboard engine. Hoping you have some success!
Nope ... just switched to UCI and it works well. I'm trying to tune qfutilityMargin right now ... we'll see

I've done some kind of generic option implementation for both UCI and Xboard (and still compatible with my json / CLI thing).

Regards