How to tune the bonus for having the move.

[michiguel]

Quick and dirty test:
Gaviota, 134 positions (Dann Corbit's "silent but deadly"), depth = 6

Code: Select all

 Bonus    Tree size  
(Pawns)  (relative, %)

 0/32      100.0
 1/32       94.2
 2/32       92.4
 3/32       90.1
 4/32       92.0
 5/32       93.9
 6/32       96.2

3/32 pawns seems to be optimal (my granularity is 1/256 pawns but I did not go to test that). This is similar to your results (0.10 pawns).
This way of testing is really interesting. I wonder what would happen with other terms.

Using an online multiple regression app

http://www.shodor.org/chemviz/tools/multireg/index.html

to produce a quadratic approximation polynomial, I determined by extrapolation that 27/256 seems to be the optimal value for your program. FYI FWIW

Thanks Ricardo!

In fact, I just tested 27/256 pawns and it *is* better. It gives a relative tree size of 89.3%

Later, I am going to do a more thorough test at deeper plies.

Miguel

[Don]

Here is another idea that Bob and Chris can shoot down:

Base the "having the move bonus" on the number of legal moves available, or the mobility count. The theory here is that the more active your position is, the more likely it is that you will have a static improvement.

I came up with this when thinking about closed positions. In closed positions you probably don't need or want much of a bonus. Closed positions usually have low mobility counts. This would also degrade the bonus as you get into the end-game.

- Don

Quick and dirty test:
Gaviota, 134 positions (Dann Corbit's "silent but deadly"), depth = 6

Code: Select all

 Bonus    Tree size  
(Pawns)  (relative, %)

 0/32      100.0
 1/32       94.2
 2/32       92.4
 3/32       90.1
 4/32       92.0
 5/32       93.9
 6/32       96.2

3/32 pawns seems to be optimal (my granularity is 1/256 pawns but I did not go to test that). This is similar to your results (0.10 pawns).
This way of testing is really interesting. I wonder what would happen with other terms.

Using an online multiple regression app

http://www.shodor.org/chemviz/tools/multireg/index.html

to produce a quadratic approximation polynomial, I determined by extrapolation that 27/256 seems to be the optimal value for your program. FYI FWIW

Thanks Ricardo!

In fact, I just tested 27/256 pawns and it *is* better. It gives a relative tree size of 89.3%

Later, I am going to do a more thorough test at deeper plies.

Miguel

[bob]

In how many of those 100 positions is the side to move in check? How many are endings where zugzwang may be a factor? How many are closed positions? I would expect that any of the above would "depress" the value of the tempo bonus.

BTW, it might be interesting to determine what the optimal tempo bonus should be for each individual position in the test to see if there are any interesting patterns that could be used to make the tempo bonus system smarter.

My testing set is not very complete, I just use 100 of the opening positions that I use for auto-testing. These positions are all exactly 10 ply deep into the game, so it's probably not perfect for this and certainly doesn't have huge variety. At some point I will generate a more comprehensive set with positions from all stages of the middlegame.

I forgot to mention that right now I cut the stand pat bonus in half for the endgame - and this value is arbitrary. It could be "calibrated" using the same method I suggest here of course, assuming this is sound.

I like your idea of testing various types of positions individually and looking for common patterns to improve on this.

I have another idea of making this adaptive. As the game progresses just monitor the odd/even differnences in the highest 2 plys completed and use this value for the next search. There are some problems with this however. First of all, the odd/even differences are influenced by the values you are already using, but you could figure that into the calculation. The other problem is that this could have a negative effect on the hash tables, assuming you do not clear them between searches (which I don't) and I think most people don't. You would also have to deal with tactical shots that would skew the results and so on. Probably simple is to force the delta of change to move very slowly.

Usually stuff like this isn't worth obsessing over, I'm just throwing out some ideas!

This is one reason I don't like test sets for tuning. Tempo computation in the opening is very important, but the results of this tuning are not felt for quite a few moves into the game, far enough that you might not see any influence at all just searching from a position and then stopping...

[Onno Garms]

0.05 is surprisingly small (but almost matches Strelka's value of 0.03).

As I have been taught that three tempi are a pawn in the opening and having the move is half a tempo, I would expect 1/6th of a pawn to be a good value. I would expect that this is fairly independent of the evaluation.

[Don]

In how many of those 100 positions is the side to move in check? How many are endings where zugzwang may be a factor? How many are closed positions? I would expect that any of the above would "depress" the value of the tempo bonus.

BTW, it might be interesting to determine what the optimal tempo bonus should be for each individual position in the test to see if there are any interesting patterns that could be used to make the tempo bonus system smarter.

My testing set is not very complete, I just use 100 of the opening positions that I use for auto-testing. These positions are all exactly 10 ply deep into the game, so it's probably not perfect for this and certainly doesn't have huge variety. At some point I will generate a more comprehensive set with positions from all stages of the middlegame.

I forgot to mention that right now I cut the stand pat bonus in half for the endgame - and this value is arbitrary. It could be "calibrated" using the same method I suggest here of course, assuming this is sound.

I like your idea of testing various types of positions individually and looking for common patterns to improve on this.

I have another idea of making this adaptive. As the game progresses just monitor the odd/even differnences in the highest 2 plys completed and use this value for the next search. There are some problems with this however. First of all, the odd/even differences are influenced by the values you are already using, but you could figure that into the calculation. The other problem is that this could have a negative effect on the hash tables, assuming you do not clear them between searches (which I don't) and I think most people don't. You would also have to deal with tactical shots that would skew the results and so on. Probably simple is to force the delta of change to move very slowly.

Usually stuff like this isn't worth obsessing over, I'm just throwing out some ideas!

This is one reason I don't like test sets for tuning. Tempo computation in the opening is very important, but the results of this tuning are not felt for quite a few moves into the game, far enough that you might not see any influence at all just searching from a position and then stopping...

Bob,

This is just an optimization - a pre-test. I agree that game playing is the better test. I simply suggest that this "MIGHT" be a simple way to zero in on numbers that you would then want to expose to your more comprehensive auto-testing procedure.

If I had a whole bunch of things to test, I would be first looking for ways to eliminate some of them before wasting trillions of cycles on them. Then I would very thoroughly test the more interesting of these ideas.

There is a way to make massive auto-testing a lot more efficient but I'm getting a little gun shy about sharing ideas. I don't have that many anyway, so I'll just hang out here and be a leech.

[michiguel]

What you mention about closed positions is actually common knowledge among chess players. A tempo is less advantageous in close positions.
I think it makes a lot of sense.
Material (endgame) should be included into this equation. Tempo advantage is more chaotic there.

Miguel

Here is another idea that Bob and Chris can shoot down:

Base the "having the move bonus" on the number of legal moves available, or the mobility count. The theory here is that the more active your position is, the more likely it is that you will have a static improvement.

I came up with this when thinking about closed positions. In closed positions you probably don't need or want much of a bonus. Closed positions usually have low mobility counts. This would also degrade the bonus as you get into the end-game.

- Don

Quick and dirty test:
Gaviota, 134 positions (Dann Corbit's "silent but deadly"), depth = 6

Code: Select all

 Bonus    Tree size  
(Pawns)  (relative, %)

 0/32      100.0
 1/32       94.2
 2/32       92.4
 3/32       90.1
 4/32       92.0
 5/32       93.9
 6/32       96.2

3/32 pawns seems to be optimal (my granularity is 1/256 pawns but I did not go to test that). This is similar to your results (0.10 pawns).
This way of testing is really interesting. I wonder what would happen with other terms.

Using an online multiple regression app

http://www.shodor.org/chemviz/tools/multireg/index.html

to produce a quadratic approximation polynomial, I determined by extrapolation that 27/256 seems to be the optimal value for your program. FYI FWIW

Thanks Ricardo!

In fact, I just tested 27/256 pawns and it *is* better. It gives a relative tree size of 89.3%

Later, I am going to do a more thorough test at deeper plies.

Miguel

[michiguel]

0.05 is surprisingly small (but almost matches Strelka's value of 0.03).

As I have been taught that three tempi are a pawn in the opening and having the move is half a tempo, I would expect 1/6th of a pawn to be a good value. I would expect that this is fairly independent of the evaluation.

0.05 would be an "average" of all the situations during different games. In close positions and endgames would be worth less, but in open, populated positions would be worth more.

Miguel

[rjgibert]

I used multiple regression to find an approximating polynomial, but I couldn't even get a 4th degree polynomial to fit particularly well. It gives .11 as the "optimal" value, but this result is not very trustworthy. FYI FWIW

[bob]

In how many of those 100 positions is the side to move in check? How many are endings where zugzwang may be a factor? How many are closed positions? I would expect that any of the above would "depress" the value of the tempo bonus.

BTW, it might be interesting to determine what the optimal tempo bonus should be for each individual position in the test to see if there are any interesting patterns that could be used to make the tempo bonus system smarter.

My testing set is not very complete, I just use 100 of the opening positions that I use for auto-testing. These positions are all exactly 10 ply deep into the game, so it's probably not perfect for this and certainly doesn't have huge variety. At some point I will generate a more comprehensive set with positions from all stages of the middlegame.

I forgot to mention that right now I cut the stand pat bonus in half for the endgame - and this value is arbitrary. It could be "calibrated" using the same method I suggest here of course, assuming this is sound.

I like your idea of testing various types of positions individually and looking for common patterns to improve on this.

I have another idea of making this adaptive. As the game progresses just monitor the odd/even differnences in the highest 2 plys completed and use this value for the next search. There are some problems with this however. First of all, the odd/even differences are influenced by the values you are already using, but you could figure that into the calculation. The other problem is that this could have a negative effect on the hash tables, assuming you do not clear them between searches (which I don't) and I think most people don't. You would also have to deal with tactical shots that would skew the results and so on. Probably simple is to force the delta of change to move very slowly.

Usually stuff like this isn't worth obsessing over, I'm just throwing out some ideas!

This is one reason I don't like test sets for tuning. Tempo computation in the opening is very important, but the results of this tuning are not felt for quite a few moves into the game, far enough that you might not see any influence at all just searching from a position and then stopping...

Bob,

This is just an optimization - a pre-test. I agree that game playing is the better test. I simply suggest that this "MIGHT" be a simple way to zero in on numbers that you would then want to expose to your more comprehensive auto-testing procedure.

If I had a whole bunch of things to test, I would be first looking for ways to eliminate some of them before wasting trillions of cycles on them. Then I would very thoroughly test the more interesting of these ideas.

There is a way to make massive auto-testing a lot more efficient but I'm getting a little gun shy about sharing ideas. I don't have that many anyway, so I'll just hang out here and be a leech.

There's no reason to be gun-shy. My comments came from experience because we have been trying various ways of evaluating changes for a couple of years. With a goal of trying to do this as quickly as possible. And the primary conclusion we have reached is that doing this quickly is not possible. And finally, testing with positions tends to make you handle those positions better, but whether this carries over into games may or may not be the case.

Bottom line is we have found no quick way to pull this off except for cases where the change is amazingly good, or horribly bad. But I have come to the conclusion that testing using positions is good for debugging or sanity testing, but not for evaluating changes, unless you are working on some specific tactical issue and have positions that you don't do well in normally. But even with those, it is critical to play a bunch of games to verify things, as some extensions improve tactical test results but hurt in real games...

[bob]

0.05 is surprisingly small (but almost matches Strelka's value of 0.03).

As I have been taught that three tempi are a pawn in the opening and having the move is half a tempo, I would expect 1/6th of a pawn to be a good value. I would expect that this is fairly independent of the evaluation.

Different issue. In Crafty, I have an EvaluateDevelopment() module that tries to handle the opening tempo issue. This stm bonus (.05) is applied throughout the game, from opening to endgame. I've not tried to adjust it as the game wears on, but the idea is interesting enough to test one day. But clearly a single move in the opening is often critical, while in the middlegame it is much less important.