Yet another parameter tuner using optuna framework

Discussion of anything and everything relating to chess playing software and machines.

Moderators: Harvey Williamson, Dann Corbit, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Joerg Oster
Posts: 792
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: Yet another parameter tuner using optuna framework

Post by Joerg Oster » Sat Sep 19, 2020 9:41 am

Damir wrote:
Sat Sep 19, 2020 8:51 am
Hi Jörg

Thanks a lot for this tunable Stockfish. If possible can you also put NNUE inside, ( whitout) embeeded net :) :)
For what purpose?
If you simply want a configurable Stockfish, for the handcrafted eval that is,
there are some forks around which offer this functionality.
Jörg Oster

Joerg Oster
Posts: 792
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: Yet another parameter tuner using optuna framework

Post by Joerg Oster » Sat Sep 19, 2020 10:08 am

Ferdy wrote:
Fri Sep 18, 2020 1:01 pm

Those are the bishops.

Now if we look at the slice plot, the tuner is somewhat confused, the trials are all over the place for the BishopValueEn. Trials are somewhat divided in the 320 to 340 range and also in the 340 to 360 range. The number of trials for those plots are 300 from 24 games/trial at tc=15s+100ms. In this situation I can continue the trials to perhaps another 100 trials and see what happens to importances and slice plot. After 300 trials I get around +12 wins in favor of the tuned param values over the default, at around 800 games in the actual game match at tc=15s+100ms. I discontinued this study.
Does this happen because you're tuning against a moving target?
Jörg Oster

Ferdy
Posts: 4309
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Yet another parameter tuner using optuna framework

Post by Ferdy » Sat Sep 19, 2020 10:19 am

Joerg Oster wrote:
Sat Sep 19, 2020 10:08 am
Ferdy wrote:
Fri Sep 18, 2020 1:01 pm

Those are the bishops.

Now if we look at the slice plot, the tuner is somewhat confused, the trials are all over the place for the BishopValueEn. Trials are somewhat divided in the 320 to 340 range and also in the 340 to 360 range. The number of trials for those plots are 300 from 24 games/trial at tc=15s+100ms. In this situation I can continue the trials to perhaps another 100 trials and see what happens to importances and slice plot. After 300 trials I get around +12 wins in favor of the tuned param values over the default, at around 800 games in the actual game match at tc=15s+100ms. I discontinued this study.
Does this happen because you're tuning against a moving target?
I don't know.

Ferdy
Posts: 4309
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Yet another parameter tuner using optuna framework

Post by Ferdy » Mon Sep 21, 2020 12:11 pm

Got promising results when optimizing search parameters.

No4b
Posts: 48
Joined: Thu Jun 18, 2020 1:21 pm
Location: Moscow
Full name: Alexander Litov

Re: Yet another parameter tuner using optuna framework

Post by No4b » Sat Oct 10, 2020 1:58 pm

I tried to use this tuner for my Drofa engine with some nice results.

My goal was to tune piece values.
Initial engine piece values:

Code: Select all

vBishopEG: 315
vBishopOP: 315
vKnightEG: 300
vKnightOP: 300
vPawnEG: 100
vQueenEG: 950
vQueenOP': 950
vRookEG: 500
vRookOP': 500
I did not enter value of the Pawn_Opening in the list becasue i wanted it to act as reference point for me to understand final params better.

I ran a test using your tuner (slightly modified in order to run it on my system), i used initial-best-value of 0.54, with test of 100 games (10`` + 0.1`` TC), threshold-pruner result 0.35 and acquisition_function LCB
I did 231 trials overall, and then used best parameters suggested by algorithm (params in the 225th trial) - it took overall ~30h.

Parameters suggested by the tuner:

Code: Select all

vBishopEG: 357
vBishopOP': 336
vKnightEG': 328
vKnightOP': 304
vPawnEG': 86
vQueenEG': 995
vQueenOP': 1190
vRookEG': 565
vRookOP': 465
It is interesting to me that in the the couple of the engines with tapered eval i looked up vPawnEG > vPawnOP, so i can hope it give my engine interesting gimmick :D

Then I ran a match of 3000 games against previous master version with different opening book in order to validate results:

Code: Select all

Score of Drofa_dev vs Drofa_1.2.9: 1128 - 950 - 922  [0.530] 3000
Elo difference: 20.64 +/- 10.35
Finished match
I am very pleased with the final result, i like both elo-gain (obviously) and play-style change (now Drofa often fearlessly sac Rook for Bishop+Pawn in the early middle game and than use that Bishop to attack/pick up more pawns).
Maybe a bit later i will try to tune search params and more evaluation terms with it.
Thank you for your great tool!

Ferdy
Posts: 4309
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Yet another parameter tuner using optuna framework

Post by Ferdy » Sun Oct 11, 2020 6:31 am

No4b wrote:
Sat Oct 10, 2020 1:58 pm
I tried to use this tuner for my Drofa engine with some nice results.
Thanks for info. Looking forward for your next test.

In case you are not aware, I run a fixed depth test comparing different algorithms. Would like to do a test with time control next.

JohnWoe
Posts: 258
Joined: Sat Mar 02, 2013 10:31 pm

Re: Yet another parameter tuner using optuna framework

Post by JohnWoe » Sun Oct 11, 2020 3:40 pm

The real question is there a way to tune without massive computation? Take the cheapest pinebook and tune some engine in 6 hours?

Post Reply