txt: automated chess engine tuning

[brtzsnr]

You cannot unfortunately divide an array by 2. You need to create a new one:

Code: Select all

e := (e1+e2)/2
v := make([]int32, len(v1))

And then fill it in with the correct values

Code: Select all

for i := range v {
v[i] = (v1[i]+v2[i])/2
}

Computing the estimation for the new set like this is a bit bogus, but it's not really used except for logging so it should be fine.

Code: Select all

e := (e1+e2)/2

[Joerg Oster]

You cannot unfortunately divide an array by 2. You need to create a new one:

Code: Select all

e := (e1+e2)/2
v := make([]int32, len(v1))

And then fill it in with the correct values

Code: Select all

for i := range v {
v[i] = (v1[i]+v2[i])/2
}

Computing the estimation for the new set like this is a bit bogus, but it's not really used except for logging so it should be fine.

Code: Select all

e := (e1+e2)/2

Thanks, highly appreciated.

A testrun with only a small number of positions is up and running.
My first impression is, that now the changed values are more evenly distributed, and the resulting best values look much better, not so erratic.

I will let it run for a while and post the graph later.

[brtzsnr]

I will let it run for a while and post the graph later.

Cool. Can you post two graphs: one with the improvement and one without the improvement so we can compare?

Also, I pushed a change which I think you'll like: you can now set the number of cpus to use. In the example (update) if you pass --cpus 12 it will attempt to use 12 threads for evaluation. It doesn't scale linearly so you want to use a number higher than the actual number of cores available.

[Joerg Oster]

I will let it run for a while and post the graph later.

Cool. Can you post two graphs: one with the improvement and one without the improvement so we can compare?

Also, I pushed a change which I think you'll like: you can now set the number of cpus to use. In the example (update) if you pass --cpus 12 it will attempt to use 12 threads for evaluation. It doesn't scale linearly so you want to use a number higher than the actual number of cores available.

Woooow
Now this sounds great.
This should significantly speed up the computation.

[Joerg Oster]

I will let it run for a while and post the graph later.

Cool. Can you post two graphs: one with the improvement and one without the improvement so we can compare?

OK, both testruns are finished.
First, the graph with the default algorithm.



Nothing really remarkable, so far.
Maybe except the little detail, that the gap between computed and estimated gets wider with more time.


And now for the modified algorithm with the average of 2 sets.



Big difference!
Now computed, estimated and pool are staying much closer to each other.
And at the end we see the curves flattening, indicating we almost reached convergence!
And the final values look much more reasonable than before!

Another interesting detail: computed score is close to 0.08453, while with default setting it already is much lower. Suggesting a much better possible result, but only a non-existing one I guess, because the resulting values are much more erratic.

I'm really happy with this result.

[brtzsnr]

Since you average you could change here
https://bitbucket.org/brtzsnr/txt/src/0 ... ster#cl-58

with

Code: Select all

stddev = math.Sqrt(stddev)

to make the changes bigger.

[EvgeniyZh]

Hi!
First of all thanks for your project.
Second, I meet some problems while using it, and I'd like to suggest fixes.

You have changed repo placement, so neither link in your post, nor README have right link, and go get cant get dependencies too. I opened issue about it.

Also, my engine sometimes didn't have enough time to print the output, then the writer was sending it next position, and this caused problems. So I rewrote the work with writer, though it might be my engine problem and not problem of the txt. If you are interested, check the code.

EDIT: Also my engine has info messages without score in it, which caused bunch of errors, so I commented that line.

[brtzsnr]

Yeah, not all changes were committed. The example was not compiling because the import path was wrong. I pushed everything, except updating the readme file. It should work now.

Code: Select all

go get -u bitbucket.org/zurichess/txt/...

[Sergei S. Markoff]

It's funny that in SmarThink I'm using nearly the same tuning method.
I have about 30 mln relatively "quiet" positions extracted from SmarThink games. I used a very simple criteria: position is "quiet" if the move for this position was not a capture or promotion.

Then I use 1.0 + pow(10.0, -((double)eval) / 400.0) sigmoid, it's funny that thare are no K coeff in my sigmoid whan Texel has and that's only difference.

But what I have is error weight that depends on number of plies until game end. The main idea is that we're trying to obtain some approximation of "true" position eval from game result, but game result can be caused by some error made *after* a position occurence in the game.

remaining plies 1.0 - mean absolute error
0 0,91502994
1 0,91340348
2 0,90854397
3 0,90694811
4 0,90077156
5 0,89893914
6 0,89295661
7 0,89087872
8 0,88610333
9 0,88570349
10 0,87991417
11 0,87757882
12 0,87145933
13 0,86850624
14 0,86266026
15 0,85930579
16 0,85357369
...

So I just used this value to weight errors.

[brtzsnr]

I haven't seen any improvement tuning with more than 500k positions. I normally tune with 1mln and just before releasing I retune with 2mln positions.

I also use captures (but not checks). Looking at quiet moves only is a significant regression for zurichess (20-30 ELO).

K was discussed in another thread. Is basically a scaling coefficient if you only want to add new feautures without retuning the old parameters. I usually tune everything so I don't care.