Page 4 of 4

Re: Yet another parameter tuner using optuna framework

Posted: Sat Sep 19, 2020 9:41 am
by Joerg Oster
Damir wrote:
Sat Sep 19, 2020 8:51 am
Hi Jörg

Thanks a lot for this tunable Stockfish. If possible can you also put NNUE inside, ( whitout) embeeded net :) :)
For what purpose?
If you simply want a configurable Stockfish, for the handcrafted eval that is,
there are some forks around which offer this functionality.

Re: Yet another parameter tuner using optuna framework

Posted: Sat Sep 19, 2020 10:08 am
by Joerg Oster
Ferdy wrote:
Fri Sep 18, 2020 1:01 pm

Those are the bishops.

Now if we look at the slice plot, the tuner is somewhat confused, the trials are all over the place for the BishopValueEn. Trials are somewhat divided in the 320 to 340 range and also in the 340 to 360 range. The number of trials for those plots are 300 from 24 games/trial at tc=15s+100ms. In this situation I can continue the trials to perhaps another 100 trials and see what happens to importances and slice plot. After 300 trials I get around +12 wins in favor of the tuned param values over the default, at around 800 games in the actual game match at tc=15s+100ms. I discontinued this study.
Does this happen because you're tuning against a moving target?

Re: Yet another parameter tuner using optuna framework

Posted: Sat Sep 19, 2020 10:19 am
by Ferdy
Joerg Oster wrote:
Sat Sep 19, 2020 10:08 am
Ferdy wrote:
Fri Sep 18, 2020 1:01 pm

Those are the bishops.

Now if we look at the slice plot, the tuner is somewhat confused, the trials are all over the place for the BishopValueEn. Trials are somewhat divided in the 320 to 340 range and also in the 340 to 360 range. The number of trials for those plots are 300 from 24 games/trial at tc=15s+100ms. In this situation I can continue the trials to perhaps another 100 trials and see what happens to importances and slice plot. After 300 trials I get around +12 wins in favor of the tuned param values over the default, at around 800 games in the actual game match at tc=15s+100ms. I discontinued this study.
Does this happen because you're tuning against a moving target?
I don't know.

Re: Yet another parameter tuner using optuna framework

Posted: Mon Sep 21, 2020 12:11 pm
by Ferdy
Got promising results when optimizing search parameters.

Re: Yet another parameter tuner using optuna framework

Posted: Sat Oct 10, 2020 1:58 pm
by No4b
I tried to use this tuner for my Drofa engine with some nice results.

My goal was to tune piece values.
Initial engine piece values:

Code: Select all

vBishopEG: 315
vBishopOP: 315
vKnightEG: 300
vKnightOP: 300
vPawnEG: 100
vQueenEG: 950
vQueenOP': 950
vRookEG: 500
vRookOP': 500
I did not enter value of the Pawn_Opening in the list becasue i wanted it to act as reference point for me to understand final params better.

I ran a test using your tuner (slightly modified in order to run it on my system), i used initial-best-value of 0.54, with test of 100 games (10`` + 0.1`` TC), threshold-pruner result 0.35 and acquisition_function LCB
I did 231 trials overall, and then used best parameters suggested by algorithm (params in the 225th trial) - it took overall ~30h.

Parameters suggested by the tuner:

Code: Select all

vBishopEG: 357
vBishopOP': 336
vKnightEG': 328
vKnightOP': 304
vPawnEG': 86
vQueenEG': 995
vQueenOP': 1190
vRookEG': 565
vRookOP': 465
It is interesting to me that in the the couple of the engines with tapered eval i looked up vPawnEG > vPawnOP, so i can hope it give my engine interesting gimmick :D

Then I ran a match of 3000 games against previous master version with different opening book in order to validate results:

Code: Select all

Score of Drofa_dev vs Drofa_1.2.9: 1128 - 950 - 922  [0.530] 3000
Elo difference: 20.64 +/- 10.35
Finished match
I am very pleased with the final result, i like both elo-gain (obviously) and play-style change (now Drofa often fearlessly sac Rook for Bishop+Pawn in the early middle game and than use that Bishop to attack/pick up more pawns).
Maybe a bit later i will try to tune search params and more evaluation terms with it.
Thank you for your great tool!

Re: Yet another parameter tuner using optuna framework

Posted: Sun Oct 11, 2020 6:31 am
by Ferdy
No4b wrote:
Sat Oct 10, 2020 1:58 pm
I tried to use this tuner for my Drofa engine with some nice results.
Thanks for info. Looking forward for your next test.

In case you are not aware, I run a fixed depth test comparing different algorithms. Would like to do a test with time control next.

Re: Yet another parameter tuner using optuna framework

Posted: Sun Oct 11, 2020 3:40 pm
by JohnWoe
The real question is there a way to tune without massive computation? Take the cheapest pinebook and tune some engine in 6 hours?