txt: automated chess engine tuning

[Ferdy]

I wonder why you chose such an asymmetric score window [-50/+3000]. Is there a good reason behind it?

The idea is not to train my eval on positions that are too inferior, say less than -50, it might begin to like it . If the score is too big on the other hand, there is no point in tuning anymore, it is already winning, your K may vary here. With that window, the positions there are from won games. I also don't use positions from drawn games, and don't train my eval on drawn positions either. Only black to play and win and white to play and win positions.

I do not restrict it. Even if winning, Gaviota needs to know it is winning by far, so it will score it heavily. It needs to know the value of a Queen advantage, for instance.

Miguel

Right, in my case I am using 30 to 50 ms thinking time. I will try the depth 1 movetime 30 command next and pure eval too, when I got some quiet positions.

I ended up creating a multi-threaded interface for uci engines in python, before it was handled by the engine internally.
param_to_tune.txt

Code: Select all

RMg 504
REg 498

Sample run.

Code: Select all

engine param: {'RMg': 504, 'REg': 498}
tolerance: +/-2
maxTrials: 25

trial 1 ...
param: RMg, value: 504
newValue: 503
newKeyValue: RMg503

param: REg, value: 498
newValue: 497
newKeyValue: RMg503REg497

newParamdict {'RMg': 503, 'REg': 497}
md5hash: d0a399a4f1e3613fecdac0fa6d785c89

Thread 2 setoption name RMg value 503
Thread 1 setoption name RMg value 503


Thread 2 setoption name REg value 497
Thread 1 setoption name REg value 497


Thread 2 is running engine d96.exe ...
Thread 1 is running engine d96.exe ...


Thread 2, aveError: 0.3278691984
Thread 1, aveError: 0.3242161028


Thread 2, epd: test_10000.epd, numLines: 5000
Thread 1, epd: test_5000.epd, numLines: 5000


Thread 2 is done running engine d96.exe ...
Thread 1 is done running engine d96.exe ...


Thread 2, time: 81sec
Thread 1, time: 81sec


aveCurErr: 0.3260426506

trial 2 ...
param: RMg, value: 504
newValue: 506
newKeyValue: RMg506

param: REg, value: 498
newValue: 500
newKeyValue: RMg506REg500

newParamdict {'RMg': 506, 'REg': 500}
md5hash: ce6eede9365c269be52afc46dbf1031e

Thread 1 setoption name RMg value 506
Thread 2 setoption name RMg value 506
[...]

[michiguel]

Gaviota uses ALL positions that are quiet (no captures in the next 6 plies of the game) from engines games. I have like 3,000,000 positions. I do not think that being selective is a good idea. The whole thing is probabilistic, so the more the merrier. It should not be only games from top engines! that would completely defeat the purpose and narrow the landscape.

Why do you search only _very_ quiet moves?

Because the Gaviota system uses eval, not qsearch. It is not quiet moves, it is quiet positions, in which you are not caught in the middle of a recapture.

Miguel

Why only evaluate on quiet position? Is it to remove the effect of search?

The static evaluation is only valid for a quiescent position by definition. If you evaluate the position statically after e4 e5/ Nf3 Nc6/ Bb5 a6/ Bxc6 you will get that black is up by a piece, which is highly inaccurate.

Miguel
PS: Texel system uses qsearch, then quiet positions are not needed. Gaviota system uses eval, then quiet positions are needed.

[Insignificant correction: You probably meant "white is up by a piece".]

One possible problem with requiring no captures in the next six plies is that it will not work for some of the more tactical terms in the evaluation function, like pins or hanging pieces, because those are generally resolved quickly and the resolution involves captures. This would also be a problem for king safety evaluation, since strong attacks rarely have six plies without captures.

It did work for me to tune those type of terms. Most of those terms have effects that are felt in few plies. I think that if there are positions in which a term is important in 4 plies but not in six, then it capture by search. Of course, i could be wrong, and something else could have worked even better. I could also choose quiet positions for less plies than six.

Miguel

[Joerg Oster]

P. S. extract.sh only gives errors.

I'll post an epd line when I get home.

For the errors please file a bug with a sample .pgn and include the error message too. I know that pgn-extract complains for cutechess produced .pgn files because the FENs are incorrect because they lack halfmove clock and fullmove number. Nevertheless, epds are still produced.

It seems you only want to extract games from one player.
This only works for me (pgn-extract 17-19) after adding a 'p' after the Tcriterion option.

Code: Select all

# Drop comments.
pgn-extract -s --nocomments -Tp$result -o$tmp1 $games
# Extract the FEN of every position in all games.

Now a epd-line looks like this:

• 1b1r1k2/p1rn1b1p/2p1p1p1/2p1PpPP/2Pp1P2/qP1P1BN1/P5Q1/R2RB1K1 b - - hmvc 4; fmvn 39; c9 "Stockfish";

Result in c9 is missing ...

[brtzsnr]

It seems you only want to extract games from one player.
This only works for me (pgn-extract 17-19) after adding a 'p' after the Tcriterion option.

Code: Select all

# Drop comments.
pgn-extract -s --nocomments -Tp$result -o$tmp1 $games
# Extract the FEN of every position in all games.

Now a epd-line looks like this:

• 1b1r1k2/p1rn1b1p/2p1p1p1/2p1PpPP/2Pp1P2/qP1P1BN1/P5Q1/R2RB1K1 b - - hmvc 4; fmvn 39; c9 "Stockfish";

Result in c9 is missing ...

The scripts extracts all positions from both player. Do you suggest that it's better to keep positions from a single player? I pushed a small code change which suppresses the error messages. I also increased the default score limit to +-2500 (from +-600).


An EPD lines looks like this:

Code: Select all

r4rk1/1pp2pp1/p2bpq1p/8/2PPQ1P1/P6P/1P3P2/R1B2RK1 b - c3 hmvc 0; fmvn 16; c9 "1/2-1/2";

hmvc and fmvn are irrelevant for the eval, and so is bm (missing here). c9 contains the result of the game, here it was a draw. The result is needed to compute Ri in the description of the Texel tuning method. If it is necessary, I can put the result to c8, for example.

[Joerg Oster]

It seems you only want to extract games from one player.
This only works for me (pgn-extract 17-19) after adding a 'p' after the Tcriterion option.

Code: Select all

# Drop comments.
pgn-extract -s --nocomments -Tp$result -o$tmp1 $games
# Extract the FEN of every position in all games.

Now a epd-line looks like this:

• 1b1r1k2/p1rn1b1p/2p1p1p1/2p1PpPP/2Pp1P2/qP1P1BN1/P5Q1/R2RB1K1 b - - hmvc 4; fmvn 39; c9 "Stockfish";

Result in c9 is missing ...

The scripts extracts all positions from both player. Do you suggest that it's better to keep positions from a single player? I pushed a small code change which suppresses the error messages. I also increased the default score limit to +-2500 (from +-600).


An EPD lines looks like this:

Code: Select all

r4rk1/1pp2pp1/p2bpq1p/8/2PPQ1P1/P6P/1P3P2/R1B2RK1 b - c3 hmvc 0; fmvn 16; c9 "1/2-1/2";

hmvc and fmvn are irrelevant for the eval, and so is bm (missing here). c9 contains the result of the game, here it was a draw. The result is needed to compute Ri in the description of the Texel tuning method. If it is necessary, I can put the result to c8, for example.

Here is what I get when I call your updated script:

Code: Select all

./extract.sh Komodo.pgn epd
./extract.sh: 12: ./extract.sh: function: not found
Extracting Komodo.pgn from Komodo.pgn to epd
./extract.sh: 30: ./extract.sh: Syntax error: "}" unexpected

Somehow I need to specify which pgn-file to process, right?
Obviously I must be doing something wrong here.

[brtzsnr]

Here is what I get when I call your updated script:

Code: Select all

./extract.sh Komodo.pgn epd
./extract.sh: 12: ./extract.sh: function: not found
Extracting Komodo.pgn from Komodo.pgn to epd
./extract.sh: 30: ./extract.sh: Syntax error: "}" unexpected

Somehow I need to specify which pgn-file to process, right?
Obviously I must be doing something wrong here.

That's weird. `./extract.sh Komodo.pgn epd` is the right way to call it. I changed the interpreter to `/bin/bash` maybe that's the issue. The new code is here: https://bitbucket.org/brtzsnr/txt/src/e ... extract.sh

Just as an example:

Code: Select all

$ ./extract.sh zurichess.pgn epds                                                                              
Extracting 1-0 from zurichess.pgn to epds
Extracting 0-1 from zurichess.pgn to epds
Extracting 1/2-1/2 from zurichess.pgn to epds
1302326 epds

$ head -n3 epds
3r2k1/1pp1r1p1/p1n2nBp/3p4/3P4/PP2PR1P/1B1N2Pb/1K2R3 w - - c9 "1/2-1/2";
4r1k1/1ppb1ppp/1rn1p3/8/3P2P1/P3PN1P/1B3KB1/1R2R3 b - - c9 "1-0";
r1bqk2r/1p3pp1/p3pn2/3pB2p/1b1p4/2NBP3/PPP2PPP/R2Q1RK1 w kq - c9 "1-0";

[brtzsnr]

Here is what I get when I call your updated script:

Code: Select all

./extract.sh Komodo.pgn epd
./extract.sh: 12: ./extract.sh: function: not found
Extracting Komodo.pgn from Komodo.pgn to epd
./extract.sh: 30: ./extract.sh: Syntax error: "}" unexpected

Somehow I need to specify which pgn-file to process, right?
Obviously I must be doing something wrong here.

I see. Something is fishy here. You modified the script. It should not be Extracting Komodo.pgn from Komodo.pgn to epd it should be instead Extracting 1-0 from Komodo.pgn to epd.

The script works in three phases: First extracts positions from all white wins, then extracts positions from all black wins, then from all draws. I do this logic because pgn-extract does not provide a way to annotate the outputted FENs with the result.

[Joerg Oster]

Here is what I get when I call your updated script:

Code: Select all

./extract.sh Komodo.pgn epd
./extract.sh: 12: ./extract.sh: function: not found
Extracting Komodo.pgn from Komodo.pgn to epd
./extract.sh: 30: ./extract.sh: Syntax error: "}" unexpected

Somehow I need to specify which pgn-file to process, right?
Obviously I must be doing something wrong here.

That's weird. `./extract.sh Komodo.pgn epd` is the right way to call it. I changed the interpreter to `/bin/bash` maybe that's the issue. The new code is here: https://bitbucket.org/brtzsnr/txt/src/e ... extract.sh

Just as an example:

Code: Select all

$ ./extract.sh zurichess.pgn epds                                                                              
Extracting 1-0 from zurichess.pgn to epds
Extracting 0-1 from zurichess.pgn to epds
Extracting 1/2-1/2 from zurichess.pgn to epds
1302326 epds

$ head -n3 epds
3r2k1/1pp1r1p1/p1n2nBp/3p4/3P4/PP2PR1P/1B1N2Pb/1K2R3 w - - c9 "1/2-1/2";
4r1k1/1ppb1ppp/1rn1p3/8/3P2P1/P3PN1P/1B3KB1/1R2R3 b - - c9 "1-0";
r1bqk2r/1p3pp1/p3pn2/3pB2p/1b1p4/2NBP3/PPP2PPP/R2Q1RK1 w kq - c9 "1-0";

That was the issue!
Now it works as expected. Many thanks!
(You see, I'm a noob with shell scripts ...)

First, I will now generate a new epd collection with a smaller number of positions and see, how long it takes to find k.
After that, I will try a first tuning session to see where that leads.

[Joerg Oster]

I ran findk for both, Zurichess and Stockfish.
Each iteration with Zurichess took about 14 minutes, while Stockfish needed about 4 minutes.
Here the results.

Zurichess

Code: Select all

2015/03/21 03:22:03 found: k = +0.80362; score = +0.10471; best: k = +0.85885 score = +0.10470

Stockfish

Code: Select all

2015/03/20 21:26:41 found: k = +1.43501; score = +0.09129; best: k = +1.22653 score = +0.09087

[brtzsnr]

I got similar values for Stockfish and Zurichess.

Unrelated, I changed the default eval depth to 0. Stockfish doesn't output a score for `go depth 0` so it won't work with stockfish. Also I changed the script to only output quiet positions as suggested earlier in this thread.