txt: automated chess engine tuning

[brtzsnr]

Congratulations!
How many iterations did it take? And how many hours?

I also found a higher number of positions seems to work better. Currently I have 5 million positions.
But then one iteration takes more than an hour with Stockfish, which is simply impractical.

I use 8 million _quiet_ positions. Zurichess takes 11m15s to go through all of them and txt needed about 40 iterations to find an improvement. I left it running overnight, so it ran for about 9h in total.

The time difference between Stockfish and Zurichess is explained by the fact that Stockfish doesn't output the score at depth 0 (see below). At depth 1 Stockfish has to go through several moves.

Additionally, did you extract the set of positions with the latest version of the script https://bitbucket.org/brtzsnr/txt/src/d ... ?at=master? If you have non-quiet positions in your set then probably quiescence search or other search extensions are activated.

Code: Select all

$ stockfish 
Stockfish 09-06-13 64bit by Tord Romstad, Marco Costalba and Joona Kiiski
position startpos

go depth 0
info nodes 0 time 1
bestmove a2a3 ponder (none)

go depth 1
info depth 1 seldepth 1 score cp 76 nodes 28 nps 28000 time 1 multipv 1 pv b1c3
info nodes 28 time 1
bestmove b1c3 ponder (none)

[Joerg Oster]

Congratulations!
How many iterations did it take? And how many hours?

I also found a higher number of positions seems to work better. Currently I have 5 million positions.
But then one iteration takes more than an hour with Stockfish, which is simply impractical.

I use 8 million _quiet_ positions. Zurichess takes 11m15s to go through all of them and txt needed about 40 iterations to find an improvement. I left it running overnight, so it ran for about 9h in total.

The time difference between Stockfish and Zurichess is explained by the fact that Stockfish doesn't output the score at depth 0 (see below). At depth 1 Stockfish has to go through several moves.

Additionally, did you extract the set of positions with the latest version of the script https://bitbucket.org/brtzsnr/txt/src/d ... ?at=master? If you have non-quiet positions in your set then probably quiescence search or other search extensions are activated.

Code: Select all

$ stockfish 
Stockfish 09-06-13 64bit by Tord Romstad, Marco Costalba and Joona Kiiski
position startpos

go depth 0
info nodes 0 time 1
bestmove a2a3 ponder (none)

go depth 1
info depth 1 seldepth 1 score cp 76 nodes 28 nps 28000 time 1 multipv 1 pv b1c3
info nodes 28 time 1
bestmove b1c3 ponder (none)

Yes, I used the latest version of extract.sh. So only quiet positions are used.
About 40 iterations only sounds less, but I guess this highly depends on the range of values used for the parameters to be tuned.

[brtzsnr]

Yes, I used the latest version of extract.sh. So only quiet positions are used.
About 40 iterations only sounds less, but I guess this highly depends on the range of values used for the parameters to be tuned.

I pushed a new version of txt. New the evaluator uses two sets a large set (same as before) to optimize and a small set (40x-60x smaller) to guide the optimization.

You can split your set of positions using shuf like this:

Code: Select all

$ shuf epds | head -n 250000 > small
$ shuf epds | head -n 8000000 > large

Here is an experiment with 1.000.000 and 25.000 positions. You can see that txt can optimize a quiet fast now.

[brtzsnr]

Here is an experiment with 1.000.000 and 25.000 positions. You can see that txt can optimize a quiet fast now.

Here is the direct link http://picpaste.com/pics/nL4cIxja.1427542076.jpeg

[Joerg Oster]

Yes, I used the latest version of extract.sh. So only quiet positions are used.
About 40 iterations only sounds less, but I guess this highly depends on the range of values used for the parameters to be tuned.

I pushed a new version of txt. New the evaluator uses two sets a large set (same as before) to optimize and a small set (40x-60x smaller) to guide the optimization.

You can split your set of positions using shuf like this:

Code: Select all

$ shuf epds | head -n 250000 > small
$ shuf epds | head -n 8000000 > large

Here is an experiment with 1.000.000 and 25.000 positions. You can see that txt can optimize a quiet fast now.

Wow, this looks like a nice idea.
Did you have another tuning success with this in the meantime?

With Stockfish, I still suffer from the same problem that the computed score is always the same.
You already set HashTable size to 1MB and send a Clear Hash command.
Additionally, I disabled Pawn Hash and Material Hash probing.
Maybe someone can point me to what I must be missing.

Here a testrun with a value range from 0 to 200:

Code: Select all

./example -- $GOPATH/bin/eval --large ../misc/large200 --small ../misc/small10 --binary ./stockfish --k +1.0
12:49:36 eval.go:241: hello, world!
12:49:41 eval.go:248: read 200000 positions from ../misc/large200
12:49:41 eval.go:254: read 10000 positions from ../misc/small10
12:49:41 eval.go:261: engine ./stockfish is ready
12:52:23 eval.go:297: computed 0.08783270402132677 in 2m41.531353s
12:52:31 eval.go:304: estimated 0.08859369813031666 in 8.091452s
12:52:39 eval.go:304: estimated 0.08859369813031666 in 8.075842s
12:52:47 eval.go:304: estimated 0.08859369813031666 in 8.085928s
12:52:55 eval.go:304: estimated 0.08859369813031666 in 8.167851s
12:53:03 eval.go:304: estimated 0.08859369813031666 in 8.014409s
12:55:44 eval.go:297: computed 0.08783270402132677 in 2m40.98506s
12:55:52 eval.go:304: estimated 0.08859369813031666 in 8.115297s

This is my example.go

Code: Select all

package main

import (
	"bitbucket.org/brtzsnr/txt/txt"
	// "fmt"
)

func RookBonus(opts []txt.Option) []txt.Option {
	opts = append(opts, txt.Option{[]string{"rookmg"}, 0, 200})   // RookOnPawn
	opts = append(opts, txt.Option{[]string{"rookeg"}, 0, 200})  // RookOnPawn
	opts = append(opts, txt.Option{[]string{"rook1mg"}, 0, 200})   // RookOnOpenFile
	opts = append(opts, txt.Option{[]string{"rook1eg"}, 0, 200})  // RookOnOpenFile
	opts = append(opts, txt.Option{[]string{"rook2mg"}, 0, 200})   // RookOnSemiopenFile
	opts = append(opts, txt.Option{[]string{"rook2eg"}, 0, 200})  // RookOnSemiopenFile
	return opts
}

func main() {
	// opts := knightPsqt(nil)
        opts := RookBonus(nil)
	txt.Run(`setvalue {{.Name}} {{printf "%.0f" .Value}}`, "go depth 1", opts)
}

[brtzsnr]

Yes, I used the latest version of extract.sh. So only quiet positions are used.
About 40 iterations only sounds less, but I guess this highly depends on the range of values used for the parameters to be tuned.

I pushed a new version of txt. New the evaluator uses two sets a large set (same as before) to optimize and a small set (40x-60x smaller) to guide the optimization.

You can split your set of positions using shuf like this:

Code: Select all

$ shuf epds | head -n 250000 > small
$ shuf epds | head -n 8000000 > large

Here is an experiment with 1.000.000 and 25.000 positions. You can see that txt can optimize a quiet fast now.

Wow, this looks like a nice idea.
Did you have another tuning success with this in the meantime?

With Stockfish, I still suffer from the same problem that the computed score is always the same.
You already set HashTable size to 1MB and send a Clear Hash command.
Additionally, I disabled Pawn Hash and Material Hash probing.
Maybe someone can point me to what I must be missing.

Here a testrun with a value range from 0 to 200:

Code: Select all

./example -- $GOPATH/bin/eval --large ../misc/large200 --small ../misc/small10 --binary ./stockfish --k +1.0
12:49:36 eval.go:241: hello, world!
12:49:41 eval.go:248: read 200000 positions from ../misc/large200
12:49:41 eval.go:254: read 10000 positions from ../misc/small10
12:49:41 eval.go:261: engine ./stockfish is ready
12:52:23 eval.go:297: computed 0.08783270402132677 in 2m41.531353s
12:52:31 eval.go:304: estimated 0.08859369813031666 in 8.091452s
12:52:39 eval.go:304: estimated 0.08859369813031666 in 8.075842s
12:52:47 eval.go:304: estimated 0.08859369813031666 in 8.085928s
12:52:55 eval.go:304: estimated 0.08859369813031666 in 8.167851s
12:53:03 eval.go:304: estimated 0.08859369813031666 in 8.014409s
12:55:44 eval.go:297: computed 0.08783270402132677 in 2m40.98506s
12:55:52 eval.go:304: estimated 0.08859369813031666 in 8.115297s

This is my example.go

Code: Select all

package main

import (
	"bitbucket.org/brtzsnr/txt/txt"
	// "fmt"
)

func RookBonus(opts []txt.Option) []txt.Option {
	opts = append(opts, txt.Option{[]string{"rookmg"}, 0, 200})   // RookOnPawn
	opts = append(opts, txt.Option{[]string{"rookeg"}, 0, 200})  // RookOnPawn
	opts = append(opts, txt.Option{[]string{"rook1mg"}, 0, 200})   // RookOnOpenFile
	opts = append(opts, txt.Option{[]string{"rook1eg"}, 0, 200})  // RookOnOpenFile
	opts = append(opts, txt.Option{[]string{"rook2mg"}, 0, 200})   // RookOnSemiopenFile
	opts = append(opts, txt.Option{[]string{"rook2eg"}, 0, 200})  // RookOnSemiopenFile
	return opts
}

func main() {
	// opts := knightPsqt(nil)
        opts := RookBonus(nil)
	txt.Run(`setvalue {{.Name}} {{printf "%.0f" .Value}}`, "go depth 1", opts)
}

You need to modify the template (first .Run argument) to set the value. From your output it looks like the evaluation parameters are not modified at atll.

For zurichess I pass options as in the following example.

Code: Select all

setvalue EndGameMaterial.FigureBonus.1 125

How do you pass options to stockfish? For stockfish setvalue doesn't work:

Code: Select all

$ stockfish 
Stockfish 09-06-13 64bit by Tord Romstad, Marco Costalba and Joona Kiiski
setvalue rookmg 34
Unknown command: setvalue rookmg 34
setoption rookmg 34
No such option: 34
setoption name rookmg value 34
No such option: rookmg
setoption name RookMg value 34
No such option: RookMg

Judging from fishtest code (https://github.com/glinscott/fishtest/b ... es.py#L289) I think it should be setoption name RookMg value 34, however it doesn't work. Maybe you need to compile stockfish with special flags?

[Joerg Oster]

Ah, ok, now I see ...

Stockfish obeys UCI protocol, option changes must be sent with

• setoption name rookmg value 34

Stockfish 290315 64 POPCNT by Tord Romstad, Marco Costalba and Joona Kiiski
uci
id name Stockfish 290315 64 POPCNT
id author Tord Romstad, Marco Costalba and Joona Kiiski

option name Write Debug Log type check default false
option name Contempt type spin default 0 min -100 max 100
option name Min Split Depth type spin default 0 min 0 max 12
option name Threads type spin default 1 min 1 max 128
option name Hash type spin default 1 min 1 max 1048576
option name Clear Hash type button
option name Ponder type check default true
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 30 min 0 max 5000
option name Minimum Thinking Time type spin default 20 min 0 max 5000
option name Slow Mover type spin default 80 min 10 max 1000
option name UCI_Chess960 type check default false
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 6 min 0 max 6
option name rookmg type spin default 14 min 0 max 200
option name rookeg type spin default 50 min 0 max 200
option name rook1mg type spin default 80 min 0 max 200
option name rook1eg type spin default 40 min 0 max 200
option name rook2mg type spin default 38 min 0 max 200
option name rook2eg type spin default 20 min 0 max 200
uciok
setoption name rookmg value 34

Of course, those parameters must be made visible in ucioption.cpp.


So changing this line in example.go should work, right?

Code: Select all

   txt.Run(`setoption name {{.Name}} value {{printf "%.0f" .Value}}`, "go depth 1", opts) 

[brtzsnr]

So changing this line in example.go should work, right?

Code: Select all

   txt.Run(`setoption name {{.Name}} value {{printf "%.0f" .Value}}`, "go depth 1", opts) 

LGTM.

[Joerg Oster]

So changing this line in example.go should work, right?

Code: Select all

   txt.Run(`setoption name {{.Name}} value {{printf "%.0f" .Value}}`, "go depth 1", opts) 

LGTM.

Big thanks!
Now the tuning can begin ...

[Joerg Oster]

Next tuning try, still more or less a testrun, with the three rook parameters is up and running.
Looks good so far, score is slowly decreasing.

I use 200,000 positions for the small set, and 4,000,000 positions for the large one.
I'll let you know about the end result.