tuning info

[elcabesa]

Hi everyone,
I just finished a good session of tuning of my engine using a technique very similar to texel method, minimising the error of evaluation against game results.

I'd like now to try to tune some other parameters using an spsa like approach.
I have studied the perl spsa tuner and looked at fishtest code.
The spsa Perl tuner doesn't work very well with vajolet and I don't like to debug the uci interaction between them. We already have a very good tournament software, cutechess and I was asking myself if already exist some tool or fork of cutechess that allow to tune an engine.

Are you aware of some project or tuning script already using cutechess? Otherwise my idea is to add a tuner to cutechess.

Thank you all

[stegemma]

Hi everyone,
I just finished a good session of tuning of my engine using a technique very similar to texel method, minimising the error of evaluation against game results.

I'd like now to try to tune some other parameters using an spsa like approach.
I have studied the perl spsa tuner and looked at fishtest code.
The spsa Perl tuner doesn't work very well with vajolet and I don't like to debug the uci interaction between them. We already have a very good tournament software, cutechess and I was asking myself if already exist some tool or fork of cutechess that allow to tune an engine.

Are you aware of some project or tuning script already using cutechess? Otherwise my idea is to add a tuner to cutechess.

Thank you all

Does Vajolet have a parameter on the commadn line to load parameters from a file? I could test genetic optimization using Sabrina tournaments mode. I don't know SPSA but it could be added as a learning algorithm.

[Daniel Shawul]

Clop + cutechess-cli is good for tuning search parameters.

It could be slow if you have too many parameters but often times there are just a handful of search parameters.

For example, I only have one futility_pruning_margin to tune in search.

Daniel

[elcabesa]

I don't like clop, but probably I can his approach.
write a python script for spsa that use cutechess to play game.. I'll give it a try

[jdart]

There is a script that comes with CLOP that you can probably modify.

You might also take a look also at:

https://github.com/jdart1/arasan-chess/ ... sitions.py

Which shows how to invoke cutechess-cli and parse the results.

--Jon

[Daniel Shawul]

I think clop may support different optimization algorithms.

I see an C2SPSA.cpp in there though i am not sure what it is used for.

P.S: Does anybody know why latest cutechess-cli does not display results on stdout ?
It plays the game and saves it to a pgn but there is no result output on the screen.
I tried cutechess-cli 1.0 and 0.85, on ubuntu 17.10 -- Qt version 5.9.1

Daniel

[petero2]

P.S: Does anybody know why latest cutechess-cli does not display results on stdout ?
It plays the game and saves it to a pgn but there is no result output on the screen.
I tried cutechess-cli 1.0 and 0.85, on ubuntu 17.10 -- Qt version 5.9.1

I had this problem in fedora 24. I solved it by adding the following to ~/.config/QtProject/qtlogging.ini:

Code: Select all

[Rules]
*.debug=true

[Daniel Shawul]

It works like a charm!

Thanks a lot Peter!

Daniel

[Ferdy]

The spsa Perl tuner doesn't work very well with vajolet and I don't like to debug the uci interaction between them.

Couple of notes when using spsa from
https://github.com/zamar/spsa/blob/master/spsa.pl

1. Be aware of the game adjudication on stalemate.
When your engine received a stalemate position what bestmove it prints?
Stockfish would print

Code: Select all

bestmove (none)

If your engine returns bestmove 0000,
then the code in line 446,

Code: Select all

$flag_stalemate = 1 if ($array[1] eq '(none)');

should be change to:

Code: Select all

$flag_stalemate = 1 if ($array[1] eq '0000');

2. When parsing engine output info line, make sure the splitted line converted into an array is properly read. At line 452,

Code: Select all

# Check for mate in one
if ($#array >= 9 && $array[0] eq 'info' && $array[1] eq 'depth' &&
    $array[7] eq 'score' && $array[8] eq 'mate' && $array[9] eq '1') 
{
    $flag_mate = 1;
    $winner = $engine_to_move;
}

With typical uci engine search output for example.

Code: Select all

info depth 1 score mate 1 nodes 20 nps 1818 tbhits 0 time 11 pv e2e4

The code above must be revised to:

Code: Select all

if ($#array >= 5 && $array[0] eq 'info' && $array[1] eq 'depth' &&
    $array[3] eq 'score' && $array[4] eq 'mate' && $array[5] eq '1') 
{
...
}

3. There is an unsafe code in line 461, revised this depending on your engine output.

Code: Select all

# Record score
if ($#array >= 7 && $array[0] eq 'info' && $array[1] eq 'depth' &&
    $array[7] eq 'score') 
{    
    $score = $array[9] if ($array[8] eq 'cp');
    $score = +100000   if ($array[8] eq 'mate' && $array[9] > 0);
    $score = -100000   if ($array&#91;8&#93; eq 'mate' && $array&#91;9&#93; < 0&#41;;
&#125;

That,

Code: Select all

$#array >= 7

should be change to:

Code: Select all

$#array >= 9

[elcabesa]

I don't like reinventing the well, we already have the beatiful cutechess-cli to play games.
I think I'll probably port spsa script to python and use cutechess to play games