howto for automated parameter tuning

[benstoker]

How is automated parameter tuning done?

Is it as simple as the following: I imagine a bash script that loops an engine engine match, incrementing or decrementing a single parameter value by x amount after playing each round of 1000 or so games, and passing that parameter to the engine via uci setoption for instance. One could use cutechess-cli, for instance, or xboard. Each parameter increment also changes the name passed to cutechess-cli, so that after it's all done, you end up with a pgn file to run bayeselo and discover the so-called best value.

OR, do you all code parameter tuning code directly into the engine and let 'er rip?

[hgm]

I never believed much in automatic parameter tuning this way. If having a Queen value off by two Pawn units (750 in stead of 950) only costs you 10 Elo, (as, according to Bob, it does), and thus requires gauntlets of 6,400 games to even detect with 95% confience that it is worse, you could not even get a Queen value that is anywhere near the correct one (say within 25 cP). Let alone for the more subtle parameters...

[Dann Corbit]

How is automated parameter tuning done?

Is it as simple as the following: I imagine a bash script that loops an engine engine match, incrementing or decrementing a single parameter value by x amount after playing each round of 1000 or so games, and passing that parameter to the engine via uci setoption for instance. One could use cutechess-cli, for instance, or xboard. Each parameter increment also changes the name passed to cutechess-cli, so that after it's all done, you end up with a pgn file to run bayeselo and discover the so-called best value.

OR, do you all code parameter tuning code directly into the engine and let 'er rip?

You have to tune with games or you get tactical settings (e.g. if you tune with an EPD set, you will solve the set like gangbusters, but it will not play as well in games).

[jhaglund]

Either method would do the trick...

The more automated the better

I would suggest a utility that adjusts values based on results for UCI engines. Store & compare automatically...

WB engines, I would suggest multiple parameter changes done automatically by ... Store & compare here also...

Then you could even have; edit & recompile your engine automatically after a result... for another test.

...but the engine has to have those parameter options coded to be manipulated first...

[bob]

I never believed much in automatic parameter tuning this way. If having a Queen value off by two Pawn units (750 in stead of 950) only costs you 10 Elo, (as, according to Bob, it does), and thus requires gauntlets of 6,400 games to even detect with 95% confience that it is worse, you could not even get a Queen value that is anywhere near the correct one (say within 25 cP). Let alone for the more subtle parameters...

Depends on the number of games. I believe ours is pretty reasonable (queen value and other pieces) = 1050 where pawn = 100.

Note that the value of a queen for an idiot player might actually only be +300, so there is no "correct" value in general. There is an optimal value for a particular player. For one that attacks a lot, it is probably higher. For one that plays slow positional stuff, it would probably be lower.

[AlvaroBegue]

Rémi Coulom has done some work that seems perfect for this problem:
http://www.mail-archive.com/computer-go ... 00053.html

It doesn't consider the possibility of having a draw as a result, but it shouldn't be too hard to adapt his method to accommodate draws.

[bob]

Either method would do the trick...

The more automated the better

I would suggest a utility that adjusts values based on results for UCI engines. Store & compare automatically...

WB engines, I would suggest multiple parameter changes done automatically by ... Store & compare here also...

Then you could even have; edit & recompile your engine automatically after a result... for another test.

...but the engine has to have those parameter options coded to be manipulated first...

Most common approach is simulated annealing. The issue is, how do you test after the annealer has made a pass over the values? Cozzie worked on this a while, but for Crafty, it was never able to produce anything better. It would modify values all over the place, but the final result was no change in real games...

[Mangar]

Hi,

I had some tests with a gen-pool related optimization. Having a population of about 1000 parameter vectors that are dying after too many lost games or getting childs with mixted or mutated parameters if successful.

This was very fun to do but absolutely useless. You can´t get enough testgames in your lifetime.

Greetings Volker

[zamar]

Is it as simple as the following: I imagine a bash script that loops an engine engine match, incrementing or decrementing a single parameter value by x amount after playing each round of 1000 or so games, and passing that parameter to the engine via uci setoption for instance. One could use cutechess-cli, for instance, or xboard. Each parameter increment also changes the name passed to cutechess-cli, so that after it's all done, you end up with a pgn file to run bayeselo and discover the so-called best value.

I won't go into the details, but I can say that what we've done with Stockfish is not much more complicated than that. If you do not happen to have a cluster (and cannot afford to spend 10 years for tuning) what you are suggesting is a very reasonable starting point. There are many good sides in this basic method:

* It's based on real games
* It's very scalable on the number of variables
* Meaningful variables are adjusting quickly while less meaningful suffer only from small random noise
* It's not going to find perfect solution, but often good enough solutions
* It's ugly and unscientific, but who cares if it can give you elo boost.

IMO genetic algorithms are hopeless thing to try. They work only if you can order the population reliably and this is impossible with real games in reasonable time scale.

This being said, I hope that Remi's work will show us more scientific and better way for doing the parameter optimization...

[rbarreira]

incrementing or decrementing a single parameter value by x amount after playing each round of 1000 or so games, and passing that parameter to the engine via uci setoption for instance.

I won't go into the details, but I can say that what we've done with Stockfish is not much more complicated than that.

For the eval related params, how were the test games played? Very fast timecontrol, or perhaps even just a 1-ply / 2-ply search? If that is enough, it's not hard to make it play thousands of games.