txt: automated chess engine tuning

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Wed Jun 17, 2015 6:04 pm

You cannot unfortunately divide an array by 2. You need to create a new one:

Code: Select all

e := (e1+e2)/2
v := make([]int32, len(v1))
And then fill it in with the correct values

Code: Select all

for i := range v {
v[i] = (v1[i]+v2[i])/2
}
Computing the estimation for the new set like this is a bit bogus, but it's not really used except for logging so it should be fine.

Code: Select all

e := (e1+e2)/2

Joerg Oster
Posts: 703
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: txt: automated chess engine tuning

Post by Joerg Oster » Wed Jun 17, 2015 6:42 pm

brtzsnr wrote:You cannot unfortunately divide an array by 2. You need to create a new one:

Code: Select all

e := (e1+e2)/2
v := make([]int32, len(v1))
And then fill it in with the correct values

Code: Select all

for i := range v {
v[i] = (v1[i]+v2[i])/2
}
Computing the estimation for the new set like this is a bit bogus, but it's not really used except for logging so it should be fine.

Code: Select all

e := (e1+e2)/2
Thanks, highly appreciated. :D

A testrun with only a small number of positions is up and running.
My first impression is, that now the changed values are more evenly distributed, and the resulting best values look much better, not so erratic.

I will let it run for a while and post the graph later.
Jörg Oster

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Wed Jun 17, 2015 6:53 pm

Joerg Oster wrote:I will let it run for a while and post the graph later.
Cool. Can you post two graphs: one with the improvement and one without the improvement so we can compare?

Also, I pushed a change which I think you'll like: you can now set the number of cpus to use. In the example (update) if you pass --cpus 12 it will attempt to use 12 threads for evaluation. It doesn't scale linearly so you want to use a number higher than the actual number of cores available.

Joerg Oster
Posts: 703
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: txt: automated chess engine tuning

Post by Joerg Oster » Wed Jun 17, 2015 8:00 pm

brtzsnr wrote:
Joerg Oster wrote:I will let it run for a while and post the graph later.
Cool. Can you post two graphs: one with the improvement and one without the improvement so we can compare?

Also, I pushed a change which I think you'll like: you can now set the number of cpus to use. In the example (update) if you pass --cpus 12 it will attempt to use 12 threads for evaluation. It doesn't scale linearly so you want to use a number higher than the actual number of cores available.
Woooow :!:
Now this sounds great.
This should significantly speed up the computation. 8-)
Jörg Oster

Joerg Oster
Posts: 703
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: txt: automated chess engine tuning

Post by Joerg Oster » Wed Jun 17, 2015 9:06 pm

brtzsnr wrote:
Joerg Oster wrote:I will let it run for a while and post the graph later.
Cool. Can you post two graphs: one with the improvement and one without the improvement so we can compare?
OK, both testruns are finished.
First, the graph with the default algorithm.

Image

Nothing really remarkable, so far.
Maybe except the little detail, that the gap between computed and estimated gets wider with more time.


And now for the modified algorithm with the average of 2 sets.

Image

Big difference!
Now computed, estimated and pool are staying much closer to each other.
And at the end we see the curves flattening, indicating we almost reached convergence!
And the final values look much more reasonable than before!

Another interesting detail: computed score is close to 0.08453, while with default setting it already is much lower. Suggesting a much better possible result, but only a non-existing one I guess, because the resulting values are much more erratic.

I'm really happy with this result. :D
Jörg Oster

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Wed Jun 17, 2015 9:31 pm

Since you average you could change here
https://bitbucket.org/brtzsnr/txt/src/0 ... ster#cl-58

with

Code: Select all

stddev = math.Sqrt(stddev)
to make the changes bigger.

EvgeniyZh
Posts: 42
Joined: Fri Sep 19, 2014 2:54 pm
Location: Israel
Contact:

Re: txt: automated chess engine tuning

Post by EvgeniyZh » Thu Aug 20, 2015 6:07 pm

Hi!
First of all thanks for your project.
Second, I meet some problems while using it, and I'd like to suggest fixes.

You have changed repo placement, so neither link in your post, nor README have right link, and go get cant get dependencies too. I opened issue about it.

Also, my engine sometimes didn't have enough time to print the output, then the writer was sending it next position, and this caused problems. So I rewrote the work with writer, though it might be my engine problem and not problem of the txt. If you are interested, check the code.

EDIT: Also my engine has info messages without score in it, which caused bunch of errors, so I commented that line.

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Sun Aug 30, 2015 7:18 am

Yeah, not all changes were committed. The example was not compiling because the import path was wrong. I pushed everything, except updating the readme file. It should work now.

Code: Select all

go get -u bitbucket.org/zurichess/txt/...

Sergei S. Markoff
Posts: 224
Joined: Mon Sep 12, 2011 9:27 pm
Location: Moscow, Russia
Contact:

Re: txt: automated chess engine tuning

Post by Sergei S. Markoff » Mon Feb 15, 2016 12:36 pm

It's funny that in SmarThink I'm using nearly the same tuning method.
I have about 30 mln relatively "quiet" positions extracted from SmarThink games. I used a very simple criteria: position is "quiet" if the move for this position was not a capture or promotion.

Then I use 1.0 + pow(10.0, -((double)eval) / 400.0) sigmoid, it's funny that thare are no K coeff in my sigmoid whan Texel has and that's only difference.

But what I have is error weight that depends on number of plies until game end. The main idea is that we're trying to obtain some approximation of "true" position eval from game result, but game result can be caused by some error made *after* a position occurence in the game.

remaining plies 1.0 - mean absolute error
0 0,91502994
1 0,91340348
2 0,90854397
3 0,90694811
4 0,90077156
5 0,89893914
6 0,89295661
7 0,89087872
8 0,88610333
9 0,88570349
10 0,87991417
11 0,87757882
12 0,87145933
13 0,86850624
14 0,86266026
15 0,85930579
16 0,85357369
...

So I just used this value to weight errors.
The Force Be With You!

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Mon Feb 15, 2016 1:54 pm

I haven't seen any improvement tuning with more than 500k positions. I normally tune with 1mln and just before releasing I retune with 2mln positions.

I also use captures (but not checks). Looking at quiet moves only is a significant regression for zurichess (20-30 ELO).

K was discussed in another thread. Is basically a scaling coefficient if you only want to add new feautures without retuning the old parameters. I usually tune everything so I don't care.

Post Reply