Yet another parameter tuner using optuna framework

[Joerg Oster]

Hi Jörg

Thanks a lot for this tunable Stockfish. If possible can you also put NNUE inside, ( whitout) embeeded net

For what purpose?
If you simply want a configurable Stockfish, for the handcrafted eval that is,
there are some forks around which offer this functionality.

[Joerg Oster]

Those are the bishops.

Now if we look at the slice plot, the tuner is somewhat confused, the trials are all over the place for the BishopValueEn. Trials are somewhat divided in the 320 to 340 range and also in the 340 to 360 range. The number of trials for those plots are 300 from 24 games/trial at tc=15s+100ms. In this situation I can continue the trials to perhaps another 100 trials and see what happens to importances and slice plot. After 300 trials I get around +12 wins in favor of the tuned param values over the default, at around 800 games in the actual game match at tc=15s+100ms. I discontinued this study.

Does this happen because you're tuning against a moving target?

[Ferdy]

Those are the bishops.

Now if we look at the slice plot, the tuner is somewhat confused, the trials are all over the place for the BishopValueEn. Trials are somewhat divided in the 320 to 340 range and also in the 340 to 360 range. The number of trials for those plots are 300 from 24 games/trial at tc=15s+100ms. In this situation I can continue the trials to perhaps another 100 trials and see what happens to importances and slice plot. After 300 trials I get around +12 wins in favor of the tuned param values over the default, at around 800 games in the actual game match at tc=15s+100ms. I discontinued this study.

Does this happen because you're tuning against a moving target?

I don't know.

[Ferdy]

Got promising results when optimizing search parameters.

[No4b]

I tried to use this tuner for my Drofa engine with some nice results.

My goal was to tune piece values.
Initial engine piece values:

Code: Select all

vBishopEG: 315
vBishopOP: 315
vKnightEG: 300
vKnightOP: 300
vPawnEG: 100
vQueenEG: 950
vQueenOP': 950
vRookEG: 500
vRookOP': 500

I did not enter value of the Pawn_Opening in the list becasue i wanted it to act as reference point for me to understand final params better.

I ran a test using your tuner (slightly modified in order to run it on my system), i used initial-best-value of 0.54, with test of 100 games (10`` + 0.1`` TC), threshold-pruner result 0.35 and acquisition_function LCB
I did 231 trials overall, and then used best parameters suggested by algorithm (params in the 225th trial) - it took overall ~30h.

Parameters suggested by the tuner:

Code: Select all

vBishopEG: 357
vBishopOP': 336
vKnightEG': 328
vKnightOP': 304
vPawnEG': 86
vQueenEG': 995
vQueenOP': 1190
vRookEG': 565
vRookOP': 465

It is interesting to me that in the the couple of the engines with tapered eval i looked up vPawnEG > vPawnOP, so i can hope it give my engine interesting gimmick

Then I ran a match of 3000 games against previous master version with different opening book in order to validate results:

Code: Select all

Score of Drofa_dev vs Drofa_1.2.9: 1128 - 950 - 922  [0.530] 3000
Elo difference: 20.64 +/- 10.35
Finished match

I am very pleased with the final result, i like both elo-gain (obviously) and play-style change (now Drofa often fearlessly sac Rook for Bishop+Pawn in the early middle game and than use that Bishop to attack/pick up more pawns).
Maybe a bit later i will try to tune search params and more evaluation terms with it.
Thank you for your great tool!

[Ferdy]

I tried to use this tuner for my Drofa engine with some nice results.

Thanks for info. Looking forward for your next test.

In case you are not aware, I run a fixed depth test comparing different algorithms. Would like to do a test with time control next.

[JohnWoe]

The real question is there a way to tune without massive computation? Take the cheapest pinebook and tune some engine in 6 hours?