New Tool for Tuning with Skopt

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
JVMerlino
Posts: 1003
Joined: Wed Mar 08, 2006 9:15 pm
Location: San Francisco, California

Re: New Tool for Tuning with Skopt

Post by JVMerlino » Wed Aug 28, 2019 2:56 pm

Hi Thomas,

I have a version of Myrddin ready for you to test if you want. PM me your e-mail address and I'll send it off.

Thanks for your help!
jm

User avatar
pedrox
Posts: 992
Joined: Fri Mar 10, 2006 5:07 am
Location: Basque Country (Spain)
Contact:

Re: New Tool for Tuning with Skopt

Post by pedrox » Thu Aug 29, 2019 2:11 am

I find the new version that gives more comments in pgn file as interesting pgn.

But I see that PV is also shown on the console and this is not good for me, I try to play 7 concurrent games and litte movetime, so seeing all the PV seems more annoying than interesting information, besides not being able to see the boards . In addition, I believe that if PV information is printed in console, this possibly slows the process.

thomasahle
Posts: 71
Joined: Thu Feb 27, 2014 7:19 pm

Re: New Tool for Tuning with Skopt

Post by thomasahle » Thu Aug 29, 2019 4:10 am

pedrox wrote:
Thu Aug 29, 2019 2:11 am
But I see that PV is also shown on the console and this is not good for me,
That was never supposed to be there. It's gone now :-)

jorose
Posts: 267
Joined: Thu Jan 22, 2015 2:21 pm
Location: Zurich, Switzerland
Full name: Jonathan Rosenthal

Re: New Tool for Tuning with Skopt

Post by jorose » Fri Aug 30, 2019 5:08 am

Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.
-Jonathan

Ratosh
Posts: 71
Joined: Mon Apr 16, 2018 4:56 pm

Re: New Tool for Tuning with Skopt

Post by Ratosh » Fri Aug 30, 2019 7:09 am

Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Fri Aug 30, 2019 8:35 am

Ratosh wrote:
Fri Aug 30, 2019 7:09 am
Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
What happens with more than 100 games?
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.
Jörg Oster

User avatar
pedrox
Posts: 992
Joined: Fri Mar 10, 2006 5:07 am
Location: Basque Country (Spain)
Contact:

Re: New Tool for Tuning with Skopt

Post by pedrox » Fri Aug 30, 2019 1:57 pm

jorose wrote:
Fri Aug 30, 2019 5:08 am
Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.
Can you indicate what parameters you used when calling tune.py?

I was also tuning those 2 parameters

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Fri Aug 30, 2019 4:33 pm

Joerg Oster wrote:
Fri Aug 30, 2019 8:35 am
Ratosh wrote:
Fri Aug 30, 2019 7:09 am
Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
What happens with more than 100 games?
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.
It looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.

Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)
Jörg Oster

Michel
Posts: 2047
Joined: Sun Sep 28, 2008 11:50 pm

Re: New Tool for Tuning with Skopt

Post by Michel » Sat Aug 31, 2019 1:21 pm

Joerg Oster wrote:
Fri Aug 30, 2019 4:33 pm
Joerg Oster wrote:
Fri Aug 30, 2019 8:35 am
Ratosh wrote:
Fri Aug 30, 2019 7:09 am
Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
What happens with more than 100 games?
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.
It looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.

Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)
I think that Bayesian optimization is computationally very heavy. From the description here https://arxiv.org/pdf/1807.02811.pdf it appears to involve at every step the inversion of a matrix whose size is the number of observations up to then.
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

Joerg Oster
Posts: 691
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: New Tool for Tuning with Skopt

Post by Joerg Oster » Sat Aug 31, 2019 4:38 pm

Michel wrote:
Sat Aug 31, 2019 1:21 pm
Joerg Oster wrote:
Fri Aug 30, 2019 4:33 pm
Joerg Oster wrote:
Fri Aug 30, 2019 8:35 am
Ratosh wrote:
Fri Aug 30, 2019 7:09 am
Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
What happens with more than 100 games?
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.
It looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.

Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)
I think that Bayesian optimization is computationally very heavy. From the description here https://arxiv.org/pdf/1807.02811.pdf it appears to involve at every step the inversion of a matrix whose size is the number of observations up to then.
So it's eventually better suited for tuning search parameters where you don't want to play ultrafast games.
Maybe I will give this another try.
Jörg Oster

Post Reply