Verification of pruning techniques

[Ferdy]

Result plot with futility pruning at depth 2 and 1.

[zd3nik]

You would indeed have spent your time better re-reading what I actually wrote, than shooting off your mouth about what you imagine I wrote. Especially after I pointed out that you did not understand a word of it...

Belligerent much?

This entire fragment of the conversation is obviously a digression stemming from poor communication. Here is what you wrote:

I suppose 'checks' here means that you are in check in the current node, so that this move really is an evasion

And the code in question is

Code: Select all

!(checks | child->checks ...

You clearly state in the quote above that you think this code is written to activate when the side to move is in check and this move is really an evasion. But the code clearly shows it's activating the razoring routine when not in check.

Take some time to re-read your own words and then tell me how you didn't overlook the negation.

When in check, futile moves deserve to be pruned just as much as when not in check.

I disagree entirely. You should never do any kind of pruning when in check.

And, based on the code misinterpretation bit above I thought you were also saying you should never prune when in check. And I was trying to tell you "I know, pruning while in check is BAD, and I'm *not* doing it".

You have now made it perfectly clear that you think futility pruning while in check is GOOD. Hopefully now we can at least agree we understand each other: you say futility pruning while in check is GOOD, I say it's BAD, and I still say you overlooked the negation in the code (but that part is immaterial at this point).

It will not cause a 100-Elo slowdown. But note that it is not merely an inefficiency because it executes some pointless instructions, but actually prevents futility prunings that should logically have been made.

I will remove the redundant statement and we'll see just how much of a difference it makes.

I'm using a very standard scale. Pawn=100, minor pieces=320-350, rook=500, queen=950 (or something like that).

Well, then Michel's so far unwithspoken remark that 800 is "only about the value of a Bishop" doesn't cut wood, and I should revive the original criticism: why do you use such a ridiculously large margin before deciding that non-captures are not going to get even in a single ply (i.e. at d=1)? Do you really expect there to be non-captures that up the evaluation by 799 centi-Pawn? If so, what eval term would be responsible for this?

I never said bishop=800. Nor did the 800 margin I'm using come from stockfish. If stockfish is using a similar delta it's purely coincidence. The razoring routine came from stockfish, the deltas I'm plugging into that routine come from me. And as I've said multiple times I've tried many different deltas. All will bad results. And yes, the deltas I posted are unusually large, unnecessarily large perhaps. I've said multiple times now, but you refuse to listen/acknowledge, what I've posted here is the most conservative version of alpha pruning I have ever written. And it is just as an attempt to rule out the possibility that the deltas are the problem - higher deltas mean less pruning but also fewer bad prunings. So let me spell it out for you: the deltas are not the issue. I get similarly bad results whether I'm using large deltas or using deltas that are less offensive to you.

And no I don't expect non-captures to overcome a deficit of 800 centipawns. The routine drops into qsearch. Have you forgotten that qsearch only searches captures, promotions, and checks? If there are any captures/promos/checks that can overcome the deficit qsearch will find them. And if qsearch doesn't find any saving cap/promo/check then razoring has successfully avoided doing a full depth search on a futile line.

Simply not true, for the reason I've already stated: checks.

Well, as I explained for checks you should have a separate generator. As long as you have that, futility pruning cannot be expected to work, so that it should not come as a surprise to you that it indeed does not work.

"As long as you have [a separate generator for checking moves], futility pruning cannot be expected to work" ... That makes no sense. I don't even know how to respond to this.

And you're apparently ignoring what I said in a previous post. This engine uses a dedicated check move generator in qsearch. Whether that's actually relevant I don't know. Because I have no idea what you're trying to say here.

I don't allow stand pat when in check. I don't understand the point you're trying to make here.

The point is that the conditions for futility pruning should be an exact match to the conditions of your stand-pat. [snip] So you should prune when in check.

I must assume you're talking about exclusively about the razoring routine. Because if you're suggesting one should prune futile looking moves in qsearch when in check - which can easily result in standing pat (while in check) - I can safely "prune" all further conversation with you because you have no idea what you're talking about.

See, now you've got me being belligerent. Now it's just one big happy belligerent party!

I disagree. As with all other forms of pruning the point is to prune entire branches of the search tree, not simply avoid doing makes and/or evals here and there.

Hard fact is that at d=1 your "entire branches" are never more than a single node. If you would have not prunded the node, the daughter would have stand pat based on the lazy eval. That is a fixed amount of work that you might be able to save by pruning that node if the pruning decision itself was not more expensive. Imagining that you could ever gain more is a delusion.

Perhaps it wasn't clear to you that I was referring to qsearch pruning here. Not razoring (i.e. d=1). And perhaps it wasn't clear to me that you were talking about razoring. I suspect this context problem has occurred several times throughout our conversation.

But you and I obviously have our minds made up on this point so I see little reason to continue discussing it further.

Well, if you are not willing to learn, posing questions here is pretty useless, I would think.

Posting a question doesn't endow anyone that answers with infinite wisdom. I am free and able to question any answer that is given, the same as anyone else.

And it is only in the interest of saving time and energy that I said we might as well stop arguing this point. But we've obviously thrown that strategy out the window.

Again, you dismiss the possibility of checks. There is no way to "correctly predict which lines would fail low due to stand pat" when the move gives check.

I did not mention checks at all here. If a move delivers check, you can correctly predict it wil NOT fail low due to stand pat. This is why you should NOT prune checks; they are NEVER futile. Nothing of what I said contradicts this. You seem to have a false association between doing making moves and searching checks. When I say you cannot expect any gain from futility pruning when you make all pruned moves, it in no way implies that you should not search checks. It just means that you should determine what are checks without making any moves. And if your engine is not at the stage yet where it can do this, you are certainly not in a position where you should attempt futility pruning. Even without any futility pruning you should have a huge speedup from not making/unmaking all non-captures in QS.

Now you've switch back to qsearch, when my statement was in response to something you where saying about razoring (you know, d=1). I'm tempted to try and clarify what I was saying (put it back into the correct context). But it's clear at this point the effort would be a waste of time.

some more jabbering about nodes/sec optimization

I will say this only one more time - you can counter with all the arguments you wish to the benefit/detriment of everyone reading this thread, but I'm not going to waste any more time on this conversation: The routines I've posted have inconsequential effect on node rate. And futility pruning is in my opinion not about node rate. Like all other forms of pruning it's about "pruning" entire branches of the search tree. You continually state there is only one move to prune, but that is nonsense. There are checks - which should never be examined without at least one more ply of search. There are low value captures and promos that generate new threats which lead to higher gains. All of these things require searching a "branch" of the tree to determine their outcome. If you think you can assume the knowledge necessary to deem any move as futile can always be achieved without searching you're welcome to that foolish belief - good luck with it. Futility pruning is a gamble. You're only "betting" that a move is futile when you prune it. You can't "know" it's futile without searching it.

It will take time, but I will try testing all the suggestions given in this thread (including yours HGM) and I will post the results. Then we will see what works and what doesn't. And then we can bicker about whether the results of my tests are valid or not. Sound fun?

[zd3nik]

Result plot with futility pruning at depth 2 and 1.

Thank you Ferdy, for doing something productive with this thread. I can't say I've provided anything useful thus far, that's for sure.

[elpapa]

I suppose 'checks' here means that you are in check in the current node, so that this move really is an evasion

And the code in question is

Code: Select all

!(checks | child->checks ...

You clearly state in the quote above that you think this code is written to activate when the side to move is in check and this move is really an evasion.

You're reading something into that quote that just isn't there. He's been saying all along that you *do not* prune check evasions, but that you *should*.

[hgm]

This entire fragment of the conversation is obviously a digression stemming from poor communication. Here is what you wrote:

I suppose 'checks' here means that you are in check in the current node, so that this move really is an evasion

And the code in question is

Code: Select all

!(checks | child->checks ...

Yes, that was a guess as to what your variables meant. So was it correct or not?

You clearly state in the quote above that you think this code is written to activate when the side to move is in check and this move is really an evasion. But the code clearly shows it's activating the razoring routine when not in check.

Take some time to re-read your own words and then tell me how you didn't overlook the negation.

My words were:

I am also not sure why you would exempt check evasions. The idea of futility pruning is that the move is guaranteed to fail low to a stand-pat reply. That is just as true for check evasions.

which obviously means

I am also not sure why you would exempt check evasions [from futility pruning, which is what we are talking about]. The idea of futility pruning is that the [pruned] move is guaranteed to fail low to a stand-pat reply. That is just as true for check evasions [so that it should be pruned too].

You get it now? Then tell me where I overlooked a negation. Do you exempt check evasions from futility pruning or not?

When in check, futile moves deserve to be pruned just as much as when not in check.

I disagree entirely. You should never do any kind of pruning when in check.

So you are WRONG. For one, alpha-beta pruning prunes move when in check, showing how non-sensical your statement is. When you find one evasion that scores above beta, all other evasions will be pruned. Fact is that every move that will fail low can ALWAYS be pruned without any consequences whatsoever. And, unsurprisingly, "always" includes "when in check".

And, based on the code misinterpretation bit above I thought you were also saying you should never prune when in check. And I was trying to tell you "I know, pruning while in check is BAD, and I'm *not* doing it".

You have now made it perfectly clear that you think futility pruning while in check is GOOD. Hopefully now we can at least agree we understand each other: you say futility pruning while in check is GOOD, I say it's BAD, and I still say you overlooked the negation in the code (but that part is immaterial at this point).

Well, the immaterial part has already been dealt with above, and you are wrong about th pruning. If you don't prune the futile check evasion, you would have to make it, do the evaluation, conclude that it is above beta (which it is guaranteed to be), return the stand-pat score, unmake the move, and conclude in the in-check node that the score is below alpha, so that you complete ignore the move (alpha will not be raised). So you will have done all that completely in vain, and there is absolutely no difference with the case where you would not have done it at all (except for the passing of time). Now do you continue to insist that wasting time is GOOD?

I never said bishop=800.

No, Michel said that. I first interpreted the margin in centi-Pawn, and remarked that this is ridiculously high, and would practically disable the code completely.

So let me spell it out for you: the deltas are not the issue. I get similarly bad results whether I'm using large deltas or using deltas that are less offensive to you.

I already commented on that. If you get bad results by adding practically unreachable code (because of asymptoticaly high deltas) something very fishy is going on. It must either be blamed on a compiler fluke, or you are not doing what you think you are doing. Normally adding dead code should neither cause a slowdown nor a speedup. For asymptotically large margin the change should approach neutrality. If for realistic margins (which cause the pruning to actually be done often enough to have some impact) the effect can be positive or negative depending on the quality of the implementation.

And no I don't expect non-captures to overcome a deficit of 800 centipawns. The routine drops into qsearch. <Some babble about qsearch>

So if you don't expect them to overcome a deficit of 799, why don't you prune them when the deficit is 799? Why search moves that that you don't expect to overcome the deficit? (And "expect" here of course is a weak qualification: there is no hope in hell that any move that was NOT searched in QS would overcome even a deficit of 150...)

"As long as you have [a separate generator for checking moves], futility pruning cannot be expected to work" ... That makes no sense. I don't even know how to respond to this.

Sorry, my bad. I intended to write "As long as you don't have that".

And you're apparently ignoring what I said in a previous post. This engine uses a dedicated check move generator in qsearch. Whether that's actually relevant I don't know. Because I have no idea what you're trying to say here.

I admit that this had sort of slipped from my mind by now. That doesn't change my claim that you cannot expect any gains from futility pruning if you don't have it. But it does of course make it less relevant for your case. So in summary, the gain by switching to QS when non-captures are futile would come from the use of that check generator to replace generation of non-captures and their testing for being checks. So whether this gives a gain or not depends on the efficiency of that special checks generator. I don't know how you did that. A reasonably efficient way to do this in a mailbox engine is to have a list of checking opportunities as a function of the relative positioning of king and piece, one for each piece type, and then verify each opportunity for the to-square being on-board, and (in case of sliders) for the two legs being unobstructed. But is should be just as easy to do that for capture and non-capture checks. In fact you would get that sort of automatically in this method, as it would first point you to the possible checking square, and when it is on-board and reachable you should still test that for its contents to see if the opportunity is real. (It could be occupied by your own piece, after all.) You could test for emptiness of the checking square, but you could almost as easily test for futility (futValue[stm][board[toSqr]] < threshold).

I must assume you're talking about exclusively about the razoring routine. Because if you're suggesting one should prune futile looking moves in qsearch when in check - which can easily result in standing pat (while in check) - I can safely "prune" all further conversation with you because you have no idea what you're talking about.

Apparently not. When there exist ANY 'futile looking' moves, stand-pat will certainly be futile, as your condition for futility is standPat + victimValue + margin < alpha, and victimValue >= 0. So you cannot stand pat if any move meets the criteria for futility pruning, and the node will simply fail low. It would fail low when you do the 'futile looking' evasion (because that is guaranteed to fail low too, as the opponent would now stand pat in reply to it), and it would fail low if you pruned it. It would not have made the slightest difference whether you pruned it or not, except that not pruning it would have meant extra work.

Perhaps it wasn't clear to you that I was referring to qsearch pruning here. Not razoring (i.e. d=1). And perhaps it wasn't clear to me that you were talking about razoring. I suspect this context problem has occurred several times throughout our conversation.

Well, so semantics is more important than you thought, after all... I always tried to explicitly write the d=1, if I thought there could be any doubt. It is well known (and you can find this remark in many places) that d=1 futility pruning is not a true pruning, but just a speed optimization. So implementing it in an efficient way is crucial for it to work.

Now you've switch back to qsearch, when my statement was in response to something you where saying about razoring (you know, d=1).

Well, you handle d=1 by doing qsearch, right? As concluded above, I might has misunderstood that the make/unmake you do for check testing only applied to the captures, and that you do have a selective generator for non-captures. What had stuck inmy mind was your remark:

I make/unmake the move because I have no other way of knowing if the move gives check or not.

So I don't understand why you think I switched; the cases are interchangeable, and what I say applies to all. Whether you are in QS or at d=1, you have to search both all moves that capture enough to approach alpha, and all checks. All other moves are predicted to fail low, and correctly so if your margin was realistic.

I will say this only one more time - you can counter with all the arguments you wish to the benefit/detriment of everyone reading this thread, but I'm not going to waste any more time on this conversation: The routines I've posted have inconsequential effect on node rate. And futility pruning is in my opinion not about node rate. Like all other forms of pruning it's about "pruning" entire branches of the search tree. You continually state there is only one move to prune, but that is nonsense. There are checks - which should never be examined without at least one more ply of search. There are low value captures and promos that generate new threats which lead to higher gains. All of these things require searching a "branch" of the tree to determine their outcome. If you think you can assume the knowledge necessary to deem any move as futile can always be achieved without searching you're welcome to that foolish belief - good luck with it. Futility pruning is a gamble. You're only "betting" that a move is futile when you prune it. You can't "know" it's futile without searching it.

Just let the reader be warned then that none of what you say here is true at d<=1. Checks will not be searched in a single node, but they will also never be pruned, (and exactly for that reason), so that is quite irrelevant. All moves that are pruned will prune a single, cheap node.

It will take time, but I will try testing all the suggestions given in this thread (including yours HGM) and I will post the results.

Well, my most important recommendation is to test the d=1 case seperately from the 1<d<4 case, with a separate margin. Much more can happen in 3 ply than in 1 ply, and the values Heinz' classic paper recommends for razoring (i.e. d=3) are indeed on the order of a Queen, while such a large margin would render the d=1 stuff practically ineffective (also because it would never get to d=1 if it was already pruned at d=3), and uses only 1 or 2 Pawns.

First you should understand why your d=1 case by itself doesn't give you anything but a speedup. Note that there is one difference so far not discussed: when you only search checks in a limited number of QS levels, replacing the d=1 search with futile non-captures by QS will effectively strip one layer of checks off your tree.

A useful test could be to not return the QS score, but remember it and continue the d=1 search. And then at the end of the node compare if the scores are indeed equal (or at least equivalent, when they are out of window). And print the scores, window and position when they are not equal, so that you can verify that the pruning never prunes anything significant, or investigate what tactics you mist by pruning.

[bob]

Code: Select all

    // forward pruning prerequisites
    if (pvNode || !(pruneOK & !checks & (depthChange <= 0))) {
      goto search_main;  // yeah that's a goto, horrible I know
    }

    // razoring (fail low pruning)
    if ((depth < 4) & (abs(alpha) < WinningScore) &
        !parent->checks & ((eval + _razorDelta[depth]) <= alpha))
    {
      _stats.rzrCount++;
      if ((depth <= 1) && ((eval + _razorDelta[3 * depth]) <= alpha)) {
        _stats.rzrEarlyOut++;
        return QSearch<color>(alpha, beta, 0);
      }
      const int ralpha = (alpha - _razorDelta[depth]);
      const int val = QSearch<color>(ralpha, (ralpha + 1), 0);
      if (_stop) {
        return beta;
      }
      if (val <= ralpha) {
        _stats.rzrCutoffs++;
        return val;
      }
    }

Btw, this code makes very little sense to me. I know razoring in the definition of Heinz, where it is just futility pruning at d=3 with a larger margin. I.e. it would be applied to the individual moves, and not to the node as a whole, unless you could prove there were no moves that could possibly beat the margin.

I've posted the same argument on this forum before: razoring is just the inverse of futility pruning, e.g. one does the pruning after the move, the other does it before. But this topic isn't about the semantics of the terms futility pruning and razoring.

I don't understand how with depth <= 1 razorDelta[0] would be NOT_USED.

_razorDelta[0] is not used because this routine is in the primary search function where depth is always > 0. And for that matter _razorDelta[4] is also not used in this particular implementation.

How much is 3*depth if depth=0? And if depth can only be 1, why not use depth[3] or a hard-coded 800 here?

I could have had-coded _razorDelta[3*depth] to 800 or _razorDelta[3]. it doesn't matter. the net result is the same. the extra couple cycles isn't the problem. I'll optimize away the extra cycles later.

The depth <= 1 part seems regular futility pruning, not razoring, but implemented in an inefficient way, by delegating it to QS. In the first place this would re-calculate eval, while this was established to be already so much below alpha that there is no hope for a stand-pat at all.

It drops into qsearch not in hopes of getting a standPat. It drops into qsearch to see if there are any checks/captures/promos that can save the day.

(But perhaps it relies on an eval cache to ameliorate this waste of time.) In the second place it would still test all the individual captures (after making them) to see if they get close to alpha, despite the fact that they start from a position hugely below alpha. Basically only capture of a Queen or promotion could beat the margin of 800. And there might not even been a Queen on the board, so that the entire exercise is a pure waste of time.

Even a pawn capture can beat the margin of 800, the game doesn't stop after 1 move. One capture can expose other threats that lead to larger gains.

Anyway you're arguing against the logic of a routine used in one of the most successful chess engines on the planet. All I'm looking for is flaws in my implementation of that well proven logic. Perhaps I'm using an AND somewhere that I should be using an OR, or I've inverted some logic somewhere.

When you reach a point where something seems odd, best advice is to test it. IE remove it from "the strongest program..." and see what the effect is. On occasion you might be surprised. I've added something new to Crafty, more than once, and got an improvement. And later discovered something old was now worthless. Sometimes hurting, sometimes just doing nothing.

An example might be reductions vs extensions. Suppose you take program A and extend all checks, and do nothing else unusual. Then you take program B that reduces all non-checks and does nothing else. You can't compare the same depth, but then the depth doesn't mean the same thing anyway.

But search a position where you have 3 moves at the root, one is a check and it leads to mate. A program with no extensions or reductions requires 2 plies to see the mate, assuming you don't do any cute q-search stuff. At depth=1 you search all three moves, but don't see it is mate until you do the depth = 2 search and find no legal moves after that check.

Now take program B. At 1 ply it searches the three moves and nothing else. At 2 plies it searches the first 2 moves to depth=1 since it reduces non-checks, but it searches move 3 to depth=2 (no reduce on checks) and discovers the mate. It searched just the three nodes for the three moves and the successor to the check node where it was mated.

Program A finds the mate at depth=1. It searches the first two and then extends the third where it discovers the mate. Both searched the same tree. But not at the same iteration #.

Now add extensions to B. and you can get into trouble since you are effectively extending a check twice, once by not reducing and once by the extension. In pathological cases that leads to a deep or sometimes non-terminating search. But remove the reductions and you are back in business.

Some of the changes I have seen appear to be "test it now" and then test the inverse of this at the top of the next ply. Usually is redundant, except that you might have more accurate information at the top of the next ply if your MakeMove() updates are incomplete. But that seems to be more an issue of implementation that one of the idea being good to do it twice...

One lesson to keep in mind is to test everything. You can't assume everyone knows what they are doing with 100% accuracy. Otherwise there would be little room for improvement...

[zd3nik]

Result plot with futility pruning at depth 2 and 1.

I just realized something strange here. The CDrill Razor 1 scores are different in the top vs bottom graphs. Are the scores relative or absolute?

Also I was wondering if the test positions you're using here are freely available. I'd like to run the same tests.

[zd3nik]

You're reading something into that quote that just isn't there. He's been saying all along that you *do not* prune check evasions, but that you *should*.

I did misinterpret the whole of what he said. And admit as much in my last post. The entire paragraph beings with " 'checks' here means that you are in check in the current node, so that this move really is an evasion", which to me means the code was interpreted to mean "you are in check in the current node, so that this move really is an evasion" which is exactly the opposite of what the code is doing.

I then interpretted everything that followed from the perspective that he had inverted the logic.

So yes, I misunderstood what he was saying. I am now admitting that a second time. But my misinterpretation was caused by the quoted section above. Which, even now, still looks to me like it is saying that the code is activating razoring when "in" check - which is the opposite of what it is doing.

[zd3nik]

So you are WRONG. For one, alpha-beta pruning prunes move when in check, showing how non-sensical your statement is. When you find one evasion that scores above beta, all other evasions will be pruned. Fact is that every move that will fail low can ALWAYS be pruned without any consequences whatsoever. And, unsurprisingly, "always" includes "when in check".

You obviously don't know the difference between backward pruning (alpha/beta) and forward pruning (razoring, futility pruning, null move pruning, etc)

Backward pruning has complete enough knowledge to prune any kind of move without error. Forward pruning does not. Using the behavior of a backward pruning technique as proof of your "logic" concerning forward pruning only convinces me even more that I don't need to waste any more time on your mad ramblings.

Good night. I have lots of testing to do.

Oh and btw, the bullet tournament that I've been waiting to finish - so I could begin new testing - has finally completed. Here are the results.

Code: Select all

   Engine         Score         Cl     Cl     Cl     Cl     Cl     Cl     Cl     Ki     Cl     Cl     Cl     Cl     Cl     Fa     TS    S-B
01: Clunk ALL      207.5/280 ······  9-5-6 10-7-3 11-4-5 10-3-7 11-3-6 13-2-5 12-6-2 14-4-2 14-2-4 17-1-2 13-2-5 11-2-7 13-0-7 19-1-0  26929.
02: Clunk NO ALPHA 199.5/280  5-9-6 ······  9-3-8  9-5-6 12-6-2 10-4-6 8-1-11 13-4-3 11-3-6 10-2-8 13-4-3 14-2-4 14-0-6 17-0-3 17-0-3  25555.
03: Clunk NO LMR   186.0/280 7-10-3  3-9-8 ······  9-8-3 12-4-4 11-4-5  9-5-6  7-7-6 14-2-4 11-4-5 16-1-3 12-1-7 12-3-5 15-3-2 16-1-3  23711.
04: Clunk NMP      169.0/280 4-11-5  5-9-6  8-9-3 ······  5-8-7  6-9-5  7-6-7  9-7-4 10-6-4 10-4-6  9-2-9 14-3-3 12-1-7 17-1-2 18-0-2  20893.
05: Clunk LMR      169.0/280 3-10-7 6-12-2 4-12-4  8-5-7 ······  7-8-5  9-6-5  6-8-6 11-5-4 12-3-5  9-3-8 11-2-7 12-1-7 18-0-2 17-0-3  20786.
06: Clunk BETA     165.5/280 3-11-6 4-10-6 4-11-5  9-6-5  8-7-5 ······  8-9-3  9-7-4 11-7-2 12-4-4 13-4-3 12-3-5 13-3-4 15-3-2 16-1-3  20624.
07: Clunk NO BETA  160.5/280 2-13-5 1-8-11  5-9-6  6-7-7  6-9-5  9-8-3 ······ 10-4-6  8-6-6 10-2-8 13-2-5 11-5-4 12-4-4 11-3-6 18-1-1  19927.
08: KingSlayer     155.0/280 6-12-2 4-13-3  7-7-6  7-9-4  8-6-6  7-9-4 4-10-6 ······ 10-9-1  9-8-3  8-4-8 13-5-2 12-4-4 17-1-2 16-1-3  19307.
09: Clunk FUTILITY 137.5/280 4-14-2 3-11-6 2-14-4 6-10-4 5-11-4 7-11-2  6-8-6 9-10-1 ······  8-7-5 13-4-3  9-6-5  9-5-6 12-5-3 18-0-2  16722.
10: Clunk          124.5/280 2-14-4 2-10-8 4-11-5 4-10-6 3-12-5 4-12-4 2-10-8  8-9-3  7-8-5 ······ 12-6-2  8-8-4 10-5-5 11-4-5 13-2-5  15358.
11: Clunk ALPHA    109.5/280 1-17-2 4-13-3 1-16-3  2-9-9  3-9-8 4-13-3 2-13-5  4-8-8 4-13-3 6-12-2 ······  9-6-5  8-4-8 10-4-6 19-1-0  12794.
12: Clunk DELTA    109.0/280 2-13-5 2-14-4 1-12-7 3-14-3 2-11-7 3-12-5 5-11-4 5-13-2  6-9-5  8-8-4  6-9-5 ······  9-7-4 11-5-4 16-3-1  13170.
13: Clunk RZR      101.5/280 2-11-7 0-14-6 3-12-5 1-12-7 1-12-7 3-13-4 4-12-4 4-12-4  5-9-6 5-10-5  4-8-8  7-9-4 ······  8-5-7 16-1-3  12300.
14: FairyMax       75.0/280  0-13-7 0-17-3 3-15-2 1-17-2 0-18-2 3-15-2 3-11-6 1-17-2 5-12-3 4-11-5 4-10-6 5-11-4  5-8-7 ······ 15-4-1  8787.2
15: TSCP64         31.0/280  1-19-0 0-17-3 1-16-3 0-18-2 0-17-3 1-16-3 1-18-1 1-16-3 0-18-2 2-13-5 1-19-0 3-16-1 1-16-3 4-15-1 ······  4216.2

I already gave a description of the names in a prior post. But it's probably worth mentioning that we've been talking about the "RZR" (primary search routine) and "DELTA" (qsearch routine) configurations. And "ALPHA" is both of those routines enabled.

The "FUTILITY" config is poorly named. It really should be named "STATIC_NMP" - static null move pruning.

[Ferdy]

I just realized something strange here. The CDrill Razor 1 scores are different in the top vs bottom graphs. Are the scores relative or absolute?

1. The top plot is Razor vs No Razor.
Silver opening set has 50 games, the test was done with reversed start color which would produce 100 games. Razor scored 55 points, and 45 points for No Razor, these figures are indicated on the bar chart.
Same thing happened for Noomen gambit, SALC and stonewall opening sets.
Noomen has 50 games, SALC has 10000 games but only use the first 50, stonewall is an epd file, also use the first 50.

2. The bottom plot is Razor + Futility vs Razor.
This is a test of how my futility implementation is worth when there is Razor.
The test sets are constant. Razor looks good vs No Razor in the top plot but is punished by futility in the bottom plot.

Also I was wondering if the test positions you're using here are freely available. I'd like to run the same tests.

All are free. I created stonewall epd and already has a download link somewhere here.

Check the link below, these are all the sets I use, also sorted randomly the original SALC.

Download:
http://www.mediafire.com/download/p870a ... g_sets.rar