New Tool for Tuning with Skopt

[JVMerlino]

Hi Thomas,

I have a version of Myrddin ready for you to test if you want. PM me your e-mail address and I'll send it off.

Thanks for your help!
jm

[pedrox]

I find the new version that gives more comments in pgn file as interesting pgn.

But I see that PV is also shown on the console and this is not good for me, I try to play 7 concurrent games and litte movetime, so seeing all the PV seems more annoying than interesting information, besides not being able to see the boards . In addition, I believe that if PV information is printed in console, this possibly slows the process.

[thomasahle]

But I see that PV is also shown on the console and this is not good for me,

That was never supposed to be there. It's gone now

[jorose]

Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.

[Ratosh]

Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.

[Joerg Oster]

Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.

What happens with more than 100 games?
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.

[pedrox]

Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.

Can you indicate what parameters you used when calling tune.py?

I was also tuning those 2 parameters

[Joerg Oster]

Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.

What happens with more than 100 games?
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.

It looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.

Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)

[Michel]

Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.

What happens with more than 100 games?
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.

It looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.

Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)

I think that Bayesian optimization is computationally very heavy. From the description here https://arxiv.org/pdf/1807.02811.pdf it appears to involve at every step the inversion of a matrix whose size is the number of observations up to then.

[Joerg Oster]

Hello,

It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.

What happens with more than 100 games?
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.

It looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.

Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)

I think that Bayesian optimization is computationally very heavy. From the description here https://arxiv.org/pdf/1807.02811.pdf it appears to involve at every step the inversion of a matrix whose size is the number of observations up to then.

So it's eventually better suited for tuning search parameters where you don't want to play ultrafast games.
Maybe I will give this another try.