Query for bob

[mridul]

Hi,

I saw a few comments [1] by bob about automatic tuning of parameters, etc in crafty - based on games/tournament results.
On looking at the latest crafty source [2], I did not find any code, infrastructure or the like to allow for this sort of testing/tuning to be done ...

So wondering if crafty still relying on manual tuning ?
If not, is this framework/infra available some place else ? Curious about how things are done nowadays ... Or did I miss something in the code ?


Thanks,
Mridul



[1] In my random walk through the top 20 odd threads in CCC

[2] Assuming I did not miss anythign !

[bob]

Hi,

I saw a few comments [1] by bob about automatic tuning of parameters, etc in crafty - based on games/tournament results.
On looking at the latest crafty source [2], I did not find any code, infrastructure or the like to allow for this sort of testing/tuning to be done ...

So wondering if crafty still relying on manual tuning ?
If not, is this framework/infra available some place else ? Curious about how things are done nowadays ... Or did I miss something in the code ?


Thanks,
Mridul



[1] In my random walk through the top 20 odd threads in CCC

[2] Assuming I did not miss anythign !

I have not done "automatic tuning" in the classic simulated annealing approach. What I do is run a big match of games, repeatedly, changing individual scoring parameters for each match. The "personality command" can be used to do this. Play a match and save the results, then add a personality command to the end of the crafty.rc/.craftyrc file that changes some value, and run the match again. I did add a "scale" command in crafty that I modify in option.c from time to time. My tester can emit "scale xx" and then "scale yy" and such for each match. That lets me custom-define what "scale xx" means for this test run.

But that's as automatic as I get. Simulated annealing (Cozzie worked on this a good while) never worked for Crafty.

[mridul]

Thanks for clarifying Bob !
I assumed that automatic tuning was the rage in chess nowadays, so kind of read too much into what you wrote I guess :-)

It is not practical for the erstwhile programs I authored (way too many parameters - which interoperate in non-trivial ways), but I wanted to see if there are other high quality original programs which did do it successfully.


I did get the source of QLR that Rémi Coulom posted (Mathieu Pagé was kind enough to send it to me), and I had had previous thoughts on this subject in a different life (none of which had worked) : just was wondering if things were different now.

Regards,
Mridul

[bob]

Thanks for clarifying Bob !
I assumed that automatic tuning was the rage in chess nowadays, so kind of read too much into what you wrote I guess

It is not practical for the erstwhile programs I authored (way too many parameters - which interoperate in non-trivial ways), but I wanted to see if there are other high quality original programs which did do it successfully.


I did get the source of QLR that Rémi Coulom posted (Mathieu Pagé was kind enough to send it to me), and I had had previous thoughts on this subject in a different life (none of which had worked) : just was wondering if things were different now.

Regards,
Mridul

From what I saw, you generate mostly noise. If you use very fast games to do the tuning, there are literally hundreds or thousands of local maxima that you can reach. Some with _very_ bizarre scoring values. All we did was generate dozens of new "improved tunings" that were typically worse, and never any better than defaults, when tested on the cluster...