lazy eval discussion

[bob]

Last week we discussed lazy eval and I said I would try to test it to see what the gain is in Crafty. We've been going thru yet another cluster fiasco and over the week I have finally gotten a test run that worked.

First, we have two lazy eval cutoffs. One right at the top, which avoids doing everything if the score is a piece or so outside the AB window. One near the bottom after pawns and passed pawns are evaluated, to bypass the individual piece scoring if the score at that point is outside a somewhat narrower window.

I first disabled the first test only, leaving the second. The net loss was -13 Elo. I am trying to run with the second test disabled, and then with both disabled. I will report as the results come in...

[FlavusSnow]

Could this be caused by a bad range for the window?

[UncombedCoconut]

Would it be interesting to quantify the benefit from the speed-up and the cost of the error separately? (This would involve an asm hack to produce a Crafty that does the full eval's calculations every time, but returns the same result as default Crafty.) With margins on the scale you've mentioned, I'm guessing it wouldn't be, but I figured I would ask.

[Ferdy]

Last week we discussed lazy eval and I said I would try to test it to see what the gain is in Crafty. We've been going thru yet another cluster fiasco and over the week I have finally gotten a test run that worked.

First, we have two lazy eval cutoffs. One right at the top, which avoids doing everything if the score is a piece or so outside the AB window. One near the bottom after pawns and passed pawns are evaluated, to bypass the individual piece scoring if the score at that point is outside a somewhat narrower window.

I first disabled the first test only, leaving the second. The net loss was -13 Elo. I am trying to run with the second test disabled, and then with both disabled. I will report as the results come in...

At one point in my development, I tried lazy eval same as your first one with additional conditions, one side should not do lazy eval successively, and both sides should not do lazy eval successively - I call this limited lazy eval. The idea is to prevent total dependency on lazy eval which probably at times will miss winning or drawing oppurtunities. I got around +3 elo on this from time control of 40 moves / 20 sec (repeating).

example:
white: lazy
black: no lazy (because white used lazy)

white: no lazy (because white used lazy previously)
black: lazy

white: no lazy (because black used lazy)
black: no lazy (because black used lazy previously)

[bob]

Could this be caused by a bad range for the window?

What?

Lazy eval is good. Removing the first test in Crafty drops Elo by 13. I am testing dropping the second test, and then dropping both. But just dropping the first early exit loses 13. Did I not write that clearly enough???

[bob]

Would it be interesting to quantify the benefit from the speed-up and the cost of the error separately? (This would involve an asm hack to produce a Crafty that does the full eval's calculations every time, but returns the same result as default Crafty.) With margins on the scale you've mentioned, I'm guessing it wouldn't be, but I figured I would ask.

This technique adds error and saves time. Within reason, the time saved more than offsets the error. But you can tweak the lazy eval margin to be more aggressive and it certainly plays worse. We tuned that value to its optimal setting with our cluster testing...

[bob]

Last week we discussed lazy eval and I said I would try to test it to see what the gain is in Crafty. We've been going thru yet another cluster fiasco and over the week I have finally gotten a test run that worked.

First, we have two lazy eval cutoffs. One right at the top, which avoids doing everything if the score is a piece or so outside the AB window. One near the bottom after pawns and passed pawns are evaluated, to bypass the individual piece scoring if the score at that point is outside a somewhat narrower window.

I first disabled the first test only, leaving the second. The net loss was -13 Elo. I am trying to run with the second test disabled, and then with both disabled. I will report as the results come in...

At one point in my development, I tried lazy eval same as your first one with additional conditions, one side should not do lazy eval successively, and both sides should not do lazy eval successively - I call this limited lazy eval. The idea is to prevent total dependency on lazy eval which probably at times will miss winning or drawing oppurtunities. I got around +3 elo on this from time control of 40 moves / 20 sec (repeating).

example:
white: lazy
black: no lazy (because white used lazy)

white: no lazy (because white used lazy previously)
black: lazy

white: no lazy (because black used lazy)
black: no lazy (because black used lazy previously)

When you say +3 elo, that implies around 50K games total to measure within +/-3???

[Ferdy]

Last week we discussed lazy eval and I said I would try to test it to see what the gain is in Crafty. We've been going thru yet another cluster fiasco and over the week I have finally gotten a test run that worked.

First, we have two lazy eval cutoffs. One right at the top, which avoids doing everything if the score is a piece or so outside the AB window. One near the bottom after pawns and passed pawns are evaluated, to bypass the individual piece scoring if the score at that point is outside a somewhat narrower window.

I first disabled the first test only, leaving the second. The net loss was -13 Elo. I am trying to run with the second test disabled, and then with both disabled. I will report as the results come in...

At one point in my development, I tried lazy eval same as your first one with additional conditions, one side should not do lazy eval successively, and both sides should not do lazy eval successively - I call this limited lazy eval. The idea is to prevent total dependency on lazy eval which probably at times will miss winning or drawing oppurtunities. I got around +3 elo on this from time control of 40 moves / 20 sec (repeating).

example:
white: lazy
black: no lazy (because white used lazy)

white: no lazy (because white used lazy previously)
black: lazy

white: no lazy (because black used lazy)
black: no lazy (because black used lazy previously)

When you say +3 elo, that implies around 50K games total to measure within +/-3???

It's only 16k both engines actually and I give up further testing. I tested like 2k, then 2k, then 2k ... for both. The engine with lazy eval just don't go down, but could not gain a sizeable advantage either.

[Mincho Georgiev]

Using Lazy evaluation to prune for me was always "yet another reasonable idea". Unfortunately, last time I've tried it, it was expensive enough to eat up to eventual gain that would give. So what I did was to add a lazy (material only) evaluation and still there was a gain.
It was while ago, so maybe I will give it a shot again. A bitboard based evaluation however is another animal. Since there is no piece list loops, the range of the evaluation terms included could be much wider, including mobility. So I'm not really sure if it would be so "Lazy" as mine

[FlavusSnow]

I reread your post. I had originally taken it to mean you had just added two break points to the eval function to exit early and you were testing them to find if there was any ELO. Now I see you had those two points in there all along and were trying to put an ELO value to them.