Comparative nodes per second

[jdart]

This is the current logic:

Code: Select all

    bool doNull = !in_check &&
        !node->PV() &&
        !(node->flags & (IID|VERIFY)) &&
        !IsNull((node-1)->last_move) &&
        !Scoring::mateScore(node->alpha) &&
        board.state.moveCount <= 98 &&
        &#40;board.getMaterial&#40;board.sideToMove&#40;)).pieceCount&#40;) >= 1&#41;;

No eval test, as you can see. I haven't re-tested this recently but the last I tried it was no better to include it.

--Jon

[BubbaTough]

Don't you have to do eval on the last four mainsearch plies in order to do futility pruning?

IIRC Crafty does futility based on material score only.

--Jon

Additionally, for programs with lazy eval (not Stockfish) most calls to eval in the last few ply are really very similar to simple material balance checks.

-Sam

[lkaufman]

Don't you have to do eval on the last four mainsearch plies in order to do futility pruning?

IIRC Crafty does futility based on material score only.

--Jon

Additionally, for programs with lazy eval (not Stockfish) most calls to eval in the last few ply are really very similar to simple material balance checks.

-Sam

That is a very interesting observation. This could account for some of the NPS gap between SF and IH. What I still don't understand is why lazy eval helps the ippos (and Rybka) but does not help Stockfish or Komodo. Previously I thought maybe we were missing something, but now it seems to me that the Ippos are doing something wrong if lazy eval helps them. I think lazy eval is the solution for the lazy programmer! He should really cover every case where it is used separately. But I don't quite get why lazy eval works in Ivanhoe, since it seems they already do have margins for each separate case.

[jdart]

What I still don't understand is why lazy eval helps the ippos (and Rybka) but does not help Stockfish or Komodo. Previously I thought maybe we were missing something, but now it seems to me that the Ippos are doing something wrong if lazy eval helps them. I think lazy eval is the solution for the lazy programmer! He should really cover every case where it is used separately. But I don't quite get why lazy eval works in Ivanhoe, since it seems they already do have margins for each separate case.

FYI I recently took out lazy eval except for cutoff when material is far outside the alpha-beta window, and it wasn't really any slower, and it performed better. It is hard to get a good gain from lazy eval when you have reasonably large positional scores.

--Jon

[Uri Blass]

What I still don't understand is why lazy eval helps the ippos (and Rybka) but does not help Stockfish or Komodo. Previously I thought maybe we were missing something, but now it seems to me that the Ippos are doing something wrong if lazy eval helps them. I think lazy eval is the solution for the lazy programmer! He should really cover every case where it is used separately. But I don't quite get why lazy eval works in Ivanhoe, since it seems they already do have margins for each separate case.

FYI I recently took out lazy eval except for cutoff when material is far outside the alpha-beta window, and it wasn't really any slower, and it performed better. It is hard to get a good gain from lazy eval when you have reasonably large positional scores.

--Jon

I think that the main question is not if you have large positional scores(that I think that every good program has in some positions) but if you can have something fast that gives a good estimate for the large positional scores.

For example I imagine that every good program has a large positionl score for white in the following position when material is equal but I think that it is also easy to get large positional scores also for fast lazy evaluation and it is going to be stupid to have lazy evaluation of only material here.

[D]4r2k/PP5p/2PP3p/7p/7p/8/5K2/R7 b - - 0 1

[bob]

Don't you have to do eval on the last four mainsearch plies in order to do futility pruning?

IIRC Crafty does futility based on material score only.

--Jon

What about for deciding whether to try null move or not? I don't see how Crafty could avoid calling the eval in the main search.

I always do null-move. I've tried all the alternatives and the only exception is that I use the hash table trick where entries with a draft that is too shallow can't be used to terminate the search, but they might have draft sufficient to tell me that a null-move search is hopeless.

[jdart]

The slow part of my eval is piece scoring, including king safety calculation, and there is no reasonable quick estimate for this. So the lazy margin had to be large (3 pawns or so).

I had some debugging code to print out positions where quick eval would have caused cutoff but a full eval wouldn't. This is one position that caused problems:

[D] 1r4k1/1r5p/2pq1p2/p2p1Pp1/P1nPp1P1/4P1P1/2p1N3/K2RQB1R b - - 0 1

--Jon

[Uri Blass]

Here is another example for the huge positional score of chess programs(probably also includes IvanHoe that I did not install in my computer and at least it includes robbolito0.09w32 and houdini that gives more than +4 for white at depth 1 when the program cannot see the promotion by search because it is hidden by a lot of checks)

Toga1.2.1a gives 1.17 for black at depth 1 but toga is inferior old program
and I believe that all the programs that are at the playing strength of Rybka3 or higher give a big advantage for white when they search to depth 1 inspite of nominal 6 pawns advantage by black.

[D]7k/PP5p/2PP3p/7p/7p/7p/6r1/K7 b - - 0 1

[bob]

Don't you have to do eval on the last four mainsearch plies in order to do futility pruning?

IIRC Crafty does futility based on material score only.

--Jon

Additionally, for programs with lazy eval (not Stockfish) most calls to eval in the last few ply are really very similar to simple material balance checks.

-Sam

That is a very interesting observation. This could account for some of the NPS gap between SF and IH. What I still don't understand is why lazy eval helps the ippos (and Rybka) but does not help Stockfish or Komodo. Previously I thought maybe we were missing something, but now it seems to me that the Ippos are doing something wrong if lazy eval helps them. I think lazy eval is the solution for the lazy programmer! He should really cover every case where it is used separately. But I don't quite get why lazy eval works in Ivanhoe, since it seems they already do have margins for each separate case.

I can only see a few ways lazy eval doesn't help.

(1) it is done incorrectly and has too much error. That is, you lazy exit too frequently, and get evals that are simply wrong.

(2) it is done incorrectly and has too much error. That is, you lazy eval almost never, because your margins are too conservative, and you see no savings in time (but of course, you also get no errors).

The benefit comes when you get the margins right, so that you save the effort when it won't help, while keeping the error rate low enough that the overall speedup makes the program stronger. This is a direct influence on NPS, plain and simple. One can do lazy eval with zero error. Ferret did this. I didn't like the complexity and discovered that a tuned margin was just as good, and less complicated, and more importantly, did not need adjusting every time an eval term is modified, added or deleted...

If your eval is so entangled that you can't safely take an early exit and you end up doing most of the eval code before you can exit, then the savings are going to be small, as will the elo gain. I suppose I could turn lazy eval off to see what the gain/loss is. I'll start that test right now, just for information...

Here's a quick speed test for comparison:

log.001: time=36.15 mat=0 n=64797986 fh=93% nps=1.8M
log.002: time=48.42 mat=0 n=64299103 fh=94% nps=1.3M

The second has no lazy exit. Significant change in search speed (.5M nps slower) not much change in tree size. scores and PV are the same:

20-> 36.15 -0.11 1. ... Bxe4 2. Bxe4 Qxc4 3. Qxc4 Rxc4
4. Nxb6 Rxe4 5. Nxd7 Nxd7 6. Rac1 Nb6
7. Bd4 Rb8 8. Rc6 Nd5 9. b6 Rxd4 10.
Nxd4 Rxb6 11. Rxb6 Nxb6

20-> 48.41 -0.11 1. ... Bxe4 2. Bxe4 Qxc4 3. Qxc4 Rxc4
4. Nxb6 Rxe4 5. Nxd7 Nxd7 6. Rac1 Nb6
7. Bd4 Rb8 8. Rc6 Nd5 9. b6 Rxd4 10.
Nxd4 Rxb6 11. Rxb6 Nxb6

A note. We have two lazy exits in Crafty. With different margins. The last exit is the one that skips all the individual piece loops (except for pawns) as that is where most of the heavy lifting is done, looping over all the pieces and such. A significant speed advantage.

[bob]

What I still don't understand is why lazy eval helps the ippos (and Rybka) but does not help Stockfish or Komodo. Previously I thought maybe we were missing something, but now it seems to me that the Ippos are doing something wrong if lazy eval helps them. I think lazy eval is the solution for the lazy programmer! He should really cover every case where it is used separately. But I don't quite get why lazy eval works in Ivanhoe, since it seems they already do have margins for each separate case.

FYI I recently took out lazy eval except for cutoff when material is far outside the alpha-beta window, and it wasn't really any slower, and it performed better. It is hard to get a good gain from lazy eval when you have reasonably large positional scores.

--Jon

I think that the main question is not if you have large positional scores(that I think that every good program has in some positions) but if you can have something fast that gives a good estimate for the large positional scores.

For example I imagine that every good program has a large positionl score for white in the following position when material is equal but I think that it is also easy to get large positional scores also for fast lazy evaluation and it is going to be stupid to have lazy evaluation of only material here.

[D]4r2k/PP5p/2PP3p/7p/7p/8/5K2/R7 b - - 0 1

Also, as I mentioned, one can take multiple exits. Do the big positional score terms first, such as passed pawns and the like, then you can have a smaller margin for the code that takes more of the time..