What would you do differently

[lucasart]

Even the Bruce Moreland no risk way iintroduces inefficiency with fail soft. I use fail soft and no lazy eval: that's what works best for me. And I conjecture that lazy eval will not scale, like anything that compromises accuracy for a pesky speed gain(which gains less at LTC...)

How can it introduce an efficiency. You NEVER use a lazy cutoff test that is not 100% accurate, because you know the largest possible positional score for each piece type. If you have no bishops, there can be no score for bishops, the lazy eval cutoff window shrinks.

If you can fail lower or higher by not doing lazy eval, you will get more cutoffs.

[hgm]

Usually investing in things that might happen is a losing deal in Chess engines. You also don't go on searching moves in a node after you find one that beats beta, just because a still higher score for that node might give you more cutoffs in case of transpositions. And because of the lazy nature of alpha-beta pruning, scores tend to rapidly converge towards the window when you get closer to the root, even if you would produce tighter bounds in eval.

[mvk]

Yes, when I had lazy eval, I passed in the bounds.

But if you call it multiple times with different bounds, you can't cache and re-use the result. And this negates some of the benefit of lazy eval. Now I don't use it but the result of eval is cached (and stored in the hashtable).

--Jon

I see lazy eval as a pitfall. If you find it helps a lot, you are searching the wrong part of the tree and should work on that first. And then it keeps working against you when tuning the evaluation weights, and the tree shaping in general because of the lost fidelity: every change in the right direction now first must overcome a barrier raised by this premature optimisation. Development-wise, it therefore tends to keep you locked in a local optimum of program strength. I still have it in the code, but with huge bounds so it is effectively off. This might be different if you're within 30 elo of the top and must scrape the barrel.

I don't view it quite the same. In Crafty, we have several sorts of eval terms.

1. Those that produce big score changes

2. Those that produce small score changes but are computationally expensive.

Don't you think that Crafty is stuck at a local optimum, evaluation-wise, with such approach? Have you given your weights tuner all the room it asks?

[bob]

Yes, when I had lazy eval, I passed in the bounds.

But if you call it multiple times with different bounds, you can't cache and re-use the result. And this negates some of the benefit of lazy eval. Now I don't use it but the result of eval is cached (and stored in the hashtable).

--Jon

I see lazy eval as a pitfall. If you find it helps a lot, you are searching the wrong part of the tree and should work on that first. And then it keeps working against you when tuning the evaluation weights, and the tree shaping in general because of the lost fidelity: every change in the right direction now first must overcome a barrier raised by this premature optimisation. Development-wise, it therefore tends to keep you locked in a local optimum of program strength. I still have it in the code, but with huge bounds so it is effectively off. This might be different if you're within 30 elo of the top and must scrape the barrel.

I don't view it quite the same. In Crafty, we have several sorts of eval terms.

1. Those that produce big score changes

2. Those that produce small score changes but are computationally expensive.

Don't you think that Crafty is stuck at a local optimum, evaluation-wise, with such approach? Have you given your weights tuner all the room it asks?

It is trivial to turn off. What you are asking is, "is it safe to assume that not all eval terms are going to reach their max or min in the same position?" You accept some risk, you gain some speed. To date, getting rid of it has ALWAYS hurt by at least 10-20 Elo. I don't do CLOPS type tuning at all, I've never had any luck with either that or the annealing stuff I did with Cozzie years back. Never got any results that tested better than or even as good as the normal hand-tuned eval terms.

[bob]

Even the Bruce Moreland no risk way iintroduces inefficiency with fail soft. I use fail soft and no lazy eval: that's what works best for me. And I conjecture that lazy eval will not scale, like anything that compromises accuracy for a pesky speed gain(which gains less at LTC...)

How can it introduce an efficiency. You NEVER use a lazy cutoff test that is not 100% accurate, because you know the largest possible positional score for each piece type. If you have no bishops, there can be no score for bishops, the lazy eval cutoff window shrinks.

If you can fail lower or higher by not doing lazy eval, you will get more cutoffs.

You will have to run that one by me again. Fail-soft doesn't guarantee anything, and in general, you cut off whether the score is == beta, or > beta, how much doesn't matter. Only place I have EVER seen fail-soft useful is in mtd(f) since anything that gets you closer to the true score with fewer re-searches is a good thing...

[mvk]

every change in the right direction now first must overcome a barrier raised by this premature optimisation. Development-wise, it therefore tends to keep you locked in a local optimum of program strength. I still have it in the code, but with huge bounds so it is effectively off. This might be different if you're within 30 elo of the top and must scrape the barrel.

I don't view it quite the same. In Crafty, we have several sorts of eval terms.

1. Those that produce big score changes

2. Those that produce small score changes but are computationally expensive.

Don't you think that Crafty is stuck at a local optimum, evaluation-wise, with such approach? Have you given your weights tuner all the room it asks?

It is trivial to turn off. What you are asking is, "is it safe to assume that not all eval terms are going to reach their max or min in the same position?" You accept some risk, you gain some speed. To date, getting rid of it has ALWAYS hurt by at least 10-20 Elo. I don't do CLOPS type tuning at all, I've never had any luck with either that or the annealing stuff I did with Cozzie years back. Never got any results that tested better than or even as good as the normal hand-tuned eval terms.

When you tried to auto-tune the weights, did you gauge progress against a version with lazy evaluation, or relative to a version without such cut-offs? In the first case, you might have been tuning against the wall of the local optimum.

[PK]

Perhaps an aspiration window widened in several stages (more than two or three) might behave more like MTD(f) and benefit from fail-soft + eager eval (is it a correct antonym?)

Anyhow, my problem with lazy eval is different: statistically it helps my engine a lot, but at the same time it causes it to misevaluate positions with really high king safety scores. Increasing the margin (going towards Bruce Moreland's optimum) solves this problem but decreases Elo. Doing king safety before lazy eval cutoff and mobility afterwards would mean looping through the pieces twice and generating similar bitboards twice (added complexity, negating some of speed benefits). Perhaps foregoing lazy eval together with its benefits is a better decision for a casual programmer.

[bob]

Perhaps an aspiration window widened in several stages (more than two or three) might behave more like MTD(f) and benefit from fail-soft + eager eval (is it a correct antonym?)

Anyhow, my problem with lazy eval is different: statistically it helps my engine a lot, but at the same time it causes it to misevaluate positions with really high king safety scores. Increasing the margin (going towards Bruce Moreland's optimum) solves this problem but decreases Elo. Doing king safety before lazy eval cutoff and mobility afterwards would mean looping through the pieces twice and generating similar bitboards twice (added complexity, negating some of speed benefits). Perhaps foregoing lazy eval together with its benefits is a better decision for a casual programmer.

I do the aspiration widening in several stages. Here is an example from current Crafty:

Code: Select all

         23->   9.39/54.19    7.90   73. Rd5 Rb2 74. Be8+ Kc7 75. e5 Rf2+ 76.
                                     Kg3 Rf8 77. Bb5 Rg8+ 78. Kf4 Rg6 79. Bd7
                                     Rh6 80. Kg5 Rh8 81. Bf5 Kc6 82. Rd7 Rg8+
                                     83. Kf6 Rf8+ 84. Rf7 Rh8 85. e6 Rh6+ 86.
                                     Kg5 Rh2 87. Bg4
         24     9.56/54.19      ++   73. Rd5! (>+8.06)
         24     9.73/54.19      ++   73. Rd5! (>+8.22)
         24     9.91/54.19      ++   73. Rd5! (>+8.54)
         24    10.13/54.19      ++   73. Rd5! (>+9.18)
         24    11.04/54.19   10.23   73. Rd5 Rb2 74. Be8+ Kc7 75. e5 Rf2+ 76.
                                     Kg3 Rf8 77. Bb5 Rg8+ 78. Kf4 Rg6 79. Bd7
                                     Rh6 80. Bg4 Rh7 81. e6 Rg7 82. Rd3 Re7
                                     83. Kg5 Kc6 84. Rd7 Rxe6 85. Bxe6 Kc5 86.
                                     Rd5+ Kc4 87. Rf5+ Kd3

That ">xxx" on the end of the fail high lines shows the current beta value before it is increased. The increase is somewhat geometric in nature.

[bob]

every change in the right direction now first must overcome a barrier raised by this premature optimisation. Development-wise, it therefore tends to keep you locked in a local optimum of program strength. I still have it in the code, but with huge bounds so it is effectively off. This might be different if you're within 30 elo of the top and must scrape the barrel.

I don't view it quite the same. In Crafty, we have several sorts of eval terms.

1. Those that produce big score changes

2. Those that produce small score changes but are computationally expensive.

Don't you think that Crafty is stuck at a local optimum, evaluation-wise, with such approach? Have you given your weights tuner all the room it asks?

It is trivial to turn off. What you are asking is, "is it safe to assume that not all eval terms are going to reach their max or min in the same position?" You accept some risk, you gain some speed. To date, getting rid of it has ALWAYS hurt by at least 10-20 Elo. I don't do CLOPS type tuning at all, I've never had any luck with either that or the annealing stuff I did with Cozzie years back. Never got any results that tested better than or even as good as the normal hand-tuned eval terms.

When you tried to auto-tune the weights, did you gauge progress against a version with lazy evaluation, or relative to a version without such cut-offs? In the first case, you might have been tuning against the wall of the local optimum.

No lazy for that. I just never got anything better from auto-tuning. Got some pretty strange looking values, but they were slightly worse. I believe the best I have seen ever was like -5-6 Elo. Cozzie had similar results last time we talked. He thought his annealing was going to be "the greatest thing ever in computer chess" but then along came a reality check.