New Tool for Tuning with Skopt

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

Re: New Tool for Tuning with Skopt

Post by thomasahle » Mon Sep 02, 2019 12:49 pm

Ratosh wrote:
Fri Aug 30, 2019 7:09 am
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
I have pushed a new version that should fix these issues. Thank you for the suggestions!

It turns out that skopt saves all previously "fitted models" for the gaussian optimization. I now delete all but the latest, which will hopefully mitigate Joerg Oster's memory leak problem.

Joerg Oster
Posts: 684
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Mon Sep 02, 2019 1:12 pm

thomasahle wrote:
Mon Sep 02, 2019 12:49 pm
Ratosh wrote:
Fri Aug 30, 2019 7:09 am
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
I have pushed a new version that should fix these issues. Thank you for the suggestions!

It turns out that skopt saves all previously "fitted models" for the gaussian optimization. I now delete all but the latest, which will hopefully mitigate Joerg Oster's memory leak problem.
Thanks, will try it.
Jörg Oster

Ratosh
Posts: 71
Joined: Mon Apr 16, 2018 4:56 pm

Re: New Tool for Tuning with Skopt

Post by Ratosh » Mon Sep 02, 2019 7:45 pm

Seems way better!

Thank you.

voffka
Posts: 64
Joined: Sat Jun 30, 2018 8:58 pm
Location: Ukraine
Full name: Volodymyr Shcherbyna

Re: New Tool for Tuning with Skopt

Post by voffka » Thu Sep 05, 2019 1:16 am

thomasahle,

Very nice tool! Thx a lot for sharing it.

jorose,
jorose wrote:
Fri Aug 30, 2019 5:08 am
Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.
How many games did you have to run during tunning to get this elo? I am trying to tune SEE evals in Igel, not clear yet how many games I need to run. Also, I presume for search it is better to use nodes limit per move I assume?

Thanks!

jorose
Posts: 260
Joined: Thu Jan 22, 2015 2:21 pm
Location: Zurich, Switzerland
Full name: Jonathan Rosenthal

Re: New Tool for Tuning with Skopt

Post by jorose » Thu Sep 05, 2019 3:44 am

voffka wrote:
Thu Sep 05, 2019 1:16 am
thomasahle,

Very nice tool! Thx a lot for sharing it.

jorose,
jorose wrote:
Fri Aug 30, 2019 5:08 am
Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.
How many games did you have to run during tunning to get this elo? I am trying to tune SEE evals in Igel, not clear yet how many games I need to run. Also, I presume for search it is better to use nodes limit per move I assume?

Thanks!
I ran it for roughly 800 games if I recall correctly. I was running the games with time per move.
-Jonathan

voffka
Posts: 64
Joined: Sat Jun 30, 2018 8:58 pm
Location: Ukraine
Full name: Volodymyr Shcherbyna

Re: New Tool for Tuning with Skopt

Post by voffka » Thu Sep 05, 2019 7:57 am

Thanks jorose,

Currently I am running 5000 games with nodes=800 to tune 4 parameters at once. I will report the status here once it is done.

thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

Re: New Tool for Tuning with Skopt

Post by thomasahle » Thu Sep 05, 2019 12:19 pm

voffka wrote:
Thu Sep 05, 2019 7:57 am
Currently I am running 5000 games with nodes=800 to tune 4 parameters at once. I will report the status here once it is done.
Note that if you are doing that many games, it might be better to use the GBRT estimator by Jorg's experiements. This is done with the `-base-estimator GBRT` flag. See https://github.com/thomasahle/fastchess/issues/10 for the discussion.

I'm excited to hear your results!

User avatar
xr_a_y
Posts: 732
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: New Tool for Tuning with Skopt

Post by xr_a_y » Sun Sep 08, 2019 10:53 am

Hi,

I'm trying it now, using xboard protocol, and I'm not really successful because python-chess seems to be rejecting "usermove=1".
Anyone experience this before ?

User avatar
JVMerlino
Posts: 1003
Joined: Wed Mar 08, 2006 9:15 pm
Location: San Francisco, California

Re: New Tool for Tuning with Skopt

Post by JVMerlino » Sun Sep 08, 2019 4:58 pm

xr_a_y wrote:
Sun Sep 08, 2019 10:53 am
Hi,

I'm trying it now, using xboard protocol, and I'm not really successful because python-chess seems to be rejecting "usermove=1".
Anyone experience this before ?
Soon I'll be trying it with Myrddin, also an xboard engine. Hoping you have some success!

User avatar
xr_a_y
Posts: 732
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: New Tool for Tuning with Skopt

Post by xr_a_y » Sun Sep 08, 2019 5:12 pm

JVMerlino wrote:
Sun Sep 08, 2019 4:58 pm
xr_a_y wrote:
Sun Sep 08, 2019 10:53 am
Hi,

I'm trying it now, using xboard protocol, and I'm not really successful because python-chess seems to be rejecting "usermove=1".
Anyone experience this before ?
Soon I'll be trying it with Myrddin, also an xboard engine. Hoping you have some success!
Nope ... just switched to UCI and it works well. I'm trying to tune qfutilityMargin right now ... we'll see

I've done some kind of generic option implementation for both UCI and Xboard (and still compatible with my json / CLI thing).

Regards

Post Reply