txt: automated chess engine tuning

[brtzsnr]

Could you explain these 2 lines of code, please?

Code: Select all

=>			mutationRate := ft / (ft + float64(len(vals)))
			for has := false; !has; { // loop until at least one value was modified
				for i, o := range opts {
=>					if i == 0 || rand.Float64() < mutationRate {

mutationRate is the probability to select each value to mutate. For large N (the number of parameters) it tends to t/N. The extra +t term from t/(N+t) is just so that if N is smaller that T the mutation probability is not >1. mutationRate is somewhere in interval (0, 1).

Second line is basically: do the following with probability mutationRate. rand.Float64() returns a uniformly distributed number between 0 and 1.

[brtzsnr]

Hi, all!

I wrote a small framework to do automated chess tuning based on Texel's Tuning method (https://chessprogramming.wikispaces.com ... ing+Method). You can find the source code and instructions how to use it here https://bitbucket.org/brtzsnr/txt.

I'm still experimenting with it, so I cannot yet report any success. Nevertheless, feel free to experiment with txt and if you find it useful, please consider contributing.

Regards,

After almost a month of testing and tuning I glad to say that zurichess is 40ELO stronger now. I have a few more ideas from Joerg that I'll include in txt.

For reference here is the change that I've just pushed: https://bitbucket.org/brtzsnr/zurichess ... ?at=master

[Joerg Oster]

Hi, all!

I wrote a small framework to do automated chess tuning based on Texel's Tuning method (https://chessprogramming.wikispaces.com ... ing+Method). You can find the source code and instructions how to use it here https://bitbucket.org/brtzsnr/txt.

I'm still experimenting with it, so I cannot yet report any success. Nevertheless, feel free to experiment with txt and if you find it useful, please consider contributing.

Regards,

After almost a month of testing and tuning I glad to say that zurichess is 40ELO stronger now. I have a few more ideas from Joerg that I'll include in txt.

For reference here is the change that I've just pushed: https://bitbucket.org/brtzsnr/zurichess ... ?at=master

Congrats! Glad to hear.

[jdart]

There is a huge literature on this stuff.

SPSA, which randomly mutates all variables, has some pretty strong theoretical guarantees of convergence even in the presence of noise. It can be slow in the final phases of convergence though.

If you an experimentally proven algorithm that mutates only some variables you might look at MVMO (https://www.uni-due.de/mvmo/). There is also a cluster-wise version of SPSA (http://www.sciencedirect.com/science/ar ... 0X15000248) for high-dimensional problems.

--Jon

[Engin]

i will test your GO txt program too with my engine, but i can not get it run on windows.

Can you give me a installation and configuration to run your code with any UCI engine and a set of PGN games extract to FEN/EPD ?

[Joerg Oster]

Could you explain these 2 lines of code, please?

Code: Select all

=>			mutationRate := ft / (ft + float64(len(vals)))
			for has := false; !has; { // loop until at least one value was modified
				for i, o := range opts {
=>					if i == 0 || rand.Float64() < mutationRate {

mutationRate is the probability to select each value to mutate. For large N (the number of parameters) it tends to t/N. The extra +t term from t/(N+t) is just so that if N is smaller that T the mutation probability is not >1. mutationRate is somewhere in interval (0, 1).

Second line is basically: do the following with probability mutationRate. rand.Float64() returns a uniformly distributed number between 0 and 1.

Recently you changed a lot of code in your framework, so I wanted to give it another try with Stockfish.
Unfortunately, I get an error message after

Code: Select all

go get -u bitbucket.org/brtzsnr/txt/...
package bitbucket.org/brtzsnr/txt/...
	imports bitbucket.org/brtzsnr/txt/example
	imports bitbucket.org/brtzsnr/txt/txt
	imports bitbucket.org/brtzsnr/txt/txt
	imports bitbucket.org/brtzsnr/txt/txt: cannot find package "bitbucket.org/brtzsnr/txt/txt" in any of:
	/usr/src/pkg/bitbucket.org/brtzsnr/txt/txt (from $GOROOT)
	/home/oster/gows1/src/bitbucket.org/brtzsnr/txt/txt (from $GOPATH)

I guess example.go needs to be updated, since bitbucket.org/brtzsnr/txt/txt no longer exists?!

[brtzsnr]

I pushed new code. The new version has a simplified mutating algorithm and its simpler to use overall. There is support for saving/loading but there is one small bug that the wrong values are written. I'll try to fix it today.

I cannot give a set of pgn because my file with 200.000 games takes 220MB compressed. However you can generate one with the following script:

Code: Select all

nice cutechess-cli \
        -srand $RANDOM \
        -pgnout games.pgn \
        -repeat \
        -recover \
        -rounds 500000 \
        -concurrency 2 \
        -ratinginterval 50 \
        -draw movenumber=60 movecount=8 score=20 \
        -openings file=2moves_v1.pgn format=pgn order=random \
        -engine cmd=path/to/engine name=z4a option.Hash=512 \
        -engine cmd=path/to/engine name=z4b option.Hash=512 \
        -each tc=40/1+0.01 proto=uci timemargin=5000

Use 2moves_v1.pgn as opening book.
Use a very fast time control to make search less relevant.
Use high timemargin to prevent timeouts at short time control.
Use high concurrency >= number of logical processors.

On my 2 core cpu it took about 1.5 days to generate 200.000 games. I actually used -concurrecy=6 for some of the games.

[Joerg Oster]

Big thanks again!
Games at ultrasfast tc have already been played.

I'm still undecided whether to use 'go depth 1' as recommended by original Texel tuning method, or SF's internal 'eval' command (with slightly modified output).
The first will take longer, but i'm not bound to only use quiet positions ...

[brtzsnr]

Big thanks again!
Games at ultrasfast tc have already been played.

I'm still undecided whether to use 'go depth 1' as recommended by original Texel tuning method, or SF's internal 'eval' command (with slightly modified output).
The first will take longer, but i'm not bound to only use quiet positions ...

Yesterday I pushed the fix and some slight improvements. Saving/loading is more important than flexibility of setting parameters (e.g. multiple names) so the config file has changed again. I hope it won't change in the near future.

After giving some though I realized why it doesn't work for stockfish. Remember the constant k? It's a scaling factor. The returned values depend highly on the scaling factor. So changing k will give you different values which is why it makes sense to tune all parameters at once.

[Joerg Oster]

Big thanks again!
Games at ultrasfast tc have already been played.

I'm still undecided whether to use 'go depth 1' as recommended by original Texel tuning method, or SF's internal 'eval' command (with slightly modified output).
The first will take longer, but i'm not bound to only use quiet positions ...

Yesterday I pushed the fix and some slight improvements. Saving/loading is more important than flexibility of setting parameters (e.g. multiple names) so the config file has changed again. I hope it won't change in the near future.

Great stuff!
This plot

is from a still ongoing tuning session with Stockfish, 40 parameters at once.
No sign of convergence so far. And it seems this will take some more hours until there's no further improvement.
And one estimation takes only 6 seconds, and the evaluation of the large set about 70 seconds! (I'm using SFs internal eval command)

Now I wonder how you can get tuned values in 3 - 5 hours ...

Do you have a link to the hill-climbing algorithm you are using?
How do you pick the values from the pool?

After giving some though I realized why it doesn't work for stockfish. Remember the constant k? It's a scaling factor. The returned values depend highly on the scaling factor. So changing k will give you different values which is why it makes sense to tune all parameters at once.