Verification of pruning techniques

[Ferdy]

Excellent feedback. That gives pretty substantial proof that razoring can work. Do you see any fundamental differences in CDrill's razoring implementation vs the code that I've posted?

I think they are similar, except my alpha has no protection against extreme values, and a first attempt margin estimate of 150.
I use this.

Code: Select all

// Razoring 
   if (depth < 2 && !pv_node && !incheck){ 
      int seval = eval(); 
      if (seval + 150 <= alpha){ 
         return qsearch(alpha, beta, 0); 
      } 
   }

[hgm]

I've posted the same argument on this forum before: razoring is just the inverse of futility pruning, e.g. one does the pruning after the move, the other does it before. But this topic isn't about the semantics of the terms futility pruning and razoring.

Well, this is not what Heinz calls 'razoring'. It seems to me that agreeing about the semantics of the terms used is quite essential to having a meaningful discussion about it. It also seems to me that having a different name for a technique that prunes exactly the same moves is a very bad idea. Especially if the synonym that is taken for it already has another established meaning...

_razorDelta[0] is not used because this routine is in the primary search function where depth is always > 0.

So the depth <= 1 is quite misleading. What is intended is really depth == 1. When you present a code snippet that takes something from an array one would assume that this is done for a reason, and that the array index is really a variable.

It drops into qsearch not in hopes of getting a standPat. It drops into qsearch to see if there are any checks/captures/promos that can save the day.

But because stand-pat is part of QS, it will try it anyway. And re-calculate eval for the purpose of doing that, and re-establishes if it is in check or not... My point was that this is a sloppy implementation that does a lot of stuff in duplicate. If you are trying a modification that is only a speedup, and doesn't change the tree (like futility pruning), using a time-wasting implementation for it is like throwing away the baby with the bath water. No wonder you won't see any effect of it. You delegate doing the futility pruning to QS, which causes a lot of duplicate work, and implements the futility pruning on the captures in a non-speed-gaining way, so that you really only benefit from bulk-pruning the captures. (If your QS really does that; if you generate checking non-captures by first generating all non-captures and then testing them for delivering check, you would of course not have gained anything at all.)

Even a pawn capture can beat the margin of 800, the game doesn't stop after 1 move.

The whole idea of futility pruning is in fact that it does. At d=1 the game will stop after one move. That is what d=1 means. The opponent will stand pat after this one move if it is not good enough. No matter how interesting it is. I might fork two Queens with a Pawn move, and he will still happily stand pat.

Anyway you're arguing against the logic of a routine used in one of the most successful chess engines on the planet.

I am singularly unimpressed by that. A poor implementation remains a poor implementation. Wasting time by doing things in duplicat is a poor implementation, and lowing things down will certainly not contribute to the strength of a Chess engine. If the engine containing time-wasting code happens to be strong, it will not be because of this, but due tounrelated features despite of this.

All I'm looking for is flaws in my implementation of that well proven logic. Perhaps I'm using an AND somewhere that I should be using an OR, or I've inverted some logic somewhere.

Well, if you copy-pasted the code from Stockfish, that doesn't seem very likely. The reason seems more that the way you implement the pruning in QS doesn't save any work, as you are making the moves anyway.

[Michel]

But because stand-pat is part of QS, it will try it anyway. And re-calculate eval for the purpose of doing that, and re-establishes if it is in check or not... My point was that this is a sloppy implementation that does a lot of stuff in duplicate.

SF caches all those things.

Wasting time by doing things in duplicat is a poor implementation, and lowing things down will certainly not contribute to the strength of a Chess engine

You keep ignoring that this was _tested_ to contribute elo. Moreover Lucas tried to remove this code _twice_ (which is statistically unsound) using a no-regression test and failed.

[Michel]

By interesting moves I mean moves which can cause large eval swings without being captures.

But if non-captures exist where you can gain on the order of a Bishop, not considering them in QS would have the same effect as not considering Bishop captures in QS. And when you consider them in QS, there wouldn't be any need to not do QS instead of d=1 already at a much lower margin. I would say the margin should always be smaller as a Pawn, to avoid severe horizon effect. When I gave castling a bonus of more than a Pawn, my engine would make non-sensical Pawn sacs in the opening all the time, to push the opponent's castling over the horizon.

You seem to be using a strange kind of logic here. The point is precisely that you do _not_ expect there to be non-captures of the value of a Bishop. That's why you skip all safe guards.

As I repeatedly said, the value of 800 was not tuned and perhaps it can be lower. But I would be surprised if it can be as low as the value of a pawn. But who knows?

Another issues is, even if the value can be lower, would it make enough of a difference to be detectable in testing?

[hgm]

I think you've overlooked the negation. It's not evading checks, it's making sure it only does pruning when *not* (in check or giving check with the current move or promoting to a queen with the current move or in a mate search).

I think you've also overlooked the winning score comparison is against alpha, not standPat. standPat will indeed never be within the realm of winningScore, but alpha could be if we're in a mate search.

I don't think I overlooked anything. You do *not* prune futile moves when in check. That is what 'exempt' means, right?: *not* doing something. And when in check, every legalmove is by definition an evasion. So you do *not* prune futile check evasions. The question was: "why do you *not* prune them?"

The point with WinningScore is that the condition abs(alpha) >= WinningScore causes you to *not* prune when alpha <= -WinningScore or alpha >= WinningScore. The first case is redundant, because when alpha <= -WinningScore, standPat + victimValue + margin will *always* be > alpha, so that you won't prune anyway based on the condition after &. And *not* pruning when alpha >= WinningScore seems a mistake, as it lead you to search moves that will most certainly be refuted by stand pat. In the daughter node you will have beta <= -WinningScore, and eval will most certainly be larger than that.

The thinking is that the farther away from the frontier node you get the less risky pruning gets. Perhaps that reasoning is false.

It most certainly is.

but it's not worse that using a static delta of say 150 (the recommended value from what I've seen).

Well, as you seem to use a non-standard (i.e. non-centi-Pawn) score scale, I cannot really comment on that.

The moves are sorted MVV/LVA.

Well, then as soon as you reach your first futile capture, the entire remaining part of the list must be futile, and you could bulk-prune it. That seems the only way to make this pruning save you any time.

I make/unmake the move because I have no other way of knowing if the move gives check or not. And moves that give check throw all assumptions about material value out the window because they could be mate, or lead to mate.

It is more because even at d=0 you would not allow stand pat when in check.

In any case, I have done the more traditional version where you stop searching on the first move that fits the pruning bill. And the results were much worse, for obvious reasons: you skip searching all the checks.

Be that as it may, this should be clear: futility pruning is a method to speed up the engine by not doing make/unmake and stand-pat test with full evaluation on moves where you can be certain they would pass the stand-pat test based on a lazy estimate of the evaluation. If you are going to make/unmake the move, and your evaluation is not very heavy or you allow the stand-pat test to be lazy as well, you cannot expect any gain from it.

Good point. I do have a check move generator, but it only generates non-capture, non-promo checks. This is because it's very easy in this engine to generate captures so I do that first then look for the non-capture checks. I could try reversing that, generate all checking moves then find the non-checking captures/promos. That way I can mark the moves that give check and I won't have to make the moves to know.

But this is getting into nodes/sec optimization.

The point is that futility pruning (i.e. the normal d=1 stuff, not deep futility or razoring) is a nodes/sec optimalization. It is not really a pruning, it just handles the stand-pat cutoffs in a more efficient way, by doing them before makemove.

The way I've implemented it doesn't have any noticeable detrimental effect on nodes/sec.

That might be true without futility pruning, but it isn't true with futility pruning. That is the key issue here. You were lucky that the time wasted by inefficient check generation happened to coincide with the time wasted on inefficient stand-pat cutoff. So fixing one or the other won't result in any noticeable speedup. You can only expect a speedup if you fix both.

Trying to squeeze more Nodes/sec isn't going to correct the weaknesses this routine introduces in the search path. That's what this topic is all about: why does this implementation cause so much ELO drop? I think I can safely say it's not because it's effecting nodes/sec; it's because it's pruning a higher number of important lines.than it should be for some reason - even though I've made it exceedingly conservative about it.

I think we must separate two issues here: the d=1 futility pruning and the deeper stuff. The d=1 futility should not really prune any line; it would just correctly predict which lines would fail low due to stand pat. There should be no Elo drop other than caused by the speed loss due to the implementation. (Node that nps might be misleading here, as moving the stand-pat cutoff to the parent would no longer count the nodes where you do stand pat, which you might have counted before.) I don't think there should be any large Elo drop because of this.

One caveat: If you do fail-soft you should take care how to increment your upper score bound after futility pruning; the score of the pruned move should be counted as currEval + victimValue + margin. Otherwise you will get a way-too-low upper bound, which might cause wron prunings when retrieved from the hash in cases where the pruned moves wer not futile (i.e. with lower alpha).

The only real pruning takes place for the deeper stuff. This has more the character of a conditional reduction than a pruning: you first search reduced (by doing QS instead of d=2 or d=3), and if that fails low you leave it at that (and otherwise you continue with the normal search).

This will of course prune many important lines: basically it says "if I am very much behind, and still have 3 ply to recover, I am going to give up already if I cannot recover in a single ply". An example of something that would be unjustly pruned by this is when you have a Knight fork on two Rooks, and the opponent tries to push the exchange loss over the horizon by trading a Queen. You might be down by more than a Queen at that point, and the best you can achieve in QS is recapturing the Queen. The opponent would then stand pat, so the recapture fails low, and with it the entire QS, so that you prune. Would you have searched d=2 or d=3 you would have noticed that whatever the opponent played after you recaptured the Queen, you would have played NxR to cash the exchange, which gained you the 200cP you needed to end above alpha, rather than below it.

Reducing moves because your surrent score is bad is often a self-fulfilling profecy: You get even less time to get even, and so you won't succeed. In most cases it would be better to prune the move outright. What intuitively would make sense is to say: "Hey, I am at -800, and will get only two opportunities to get even. The second opportunity will likely gain me less than the first, as this is how I set my priorities, so if I cannot get at least halfway in my first attempt things will look pretty bleak. So if QS doesn't get me above at least -400, I give up."

Note that QS allows the opponent to bail out early only after a tempo-losing gain, such as the Q recapture in the example. If you have multiple exchange gains QS will typically cash them all, when you cash them in MVV order.

[Rebel]

I have this uci engine called CDrill without search reduction and pruning not even NMP, it only has hash table. In the evaluation only piece values and pst for P, N, B and K. No mobility, no passers, no king shelter, no weak pawns eval nothing.
It scored 100 (+61,=12,-27), 67.0 % vs Robin 0.983 (CCRL40/4 1755).

Created a branch from this engine and added razoring at depth 1. Run some self-test of 100 games each (TC 40moves/60s) from different test suites. Here is the result.

Add futility pruning to it and depending on its implementation the good results of razoring might melt as snow for the sun.

[Rebel]

I've posted the same argument on this forum before: razoring is just the inverse of futility pruning,

No it isn't, like extensions are not the inverse of reductions.

What is true that they bite each other and it will be a hard job to get the maximum elo out of it using both.

[zd3nik]

Ordinarily I would have better things to do than banter with you over these mostly unimportant details. But since I don't want to do anything too heavy (like compiling and running epd tests) while the round-robin is running I'll kill some time volleying with you.

Exec<color>(*move, *child);
if ( this is a *not* ---> ! <--- (checks | child->checks | (move->Promo() == (color|Queen)) | (abs(alpha) >= WinningScore)) &
((standPat + ValueOf(move->Cap()) + _qsearchDelta[-depth]) < alpha))
{
_stats.deltaCount++;
Undo<color>(*move);
continue;
}
move->Score() = -child->QSearch<!color>(-beta, -alpha, (depth - 1));
Undo<color>(*move);

I suppose 'checks' here means that you are in check in the current node, so that this move really is an evasion

I think you've overlooked the negation. It's not evading checks, it's making sure it only does pruning when *not* (in check or giving check with the current move or promoting to a queen with the current move or in a mate search).

I don't think I overlooked anything. You do *not* prune futile moves when in check.

I think you very clearly overlooked the negation. I am *not* pruning futile moves when in check.

The point with WinningScore is that the condition abs(alpha) >= WinningScore causes you to *not* prune when alpha <= -WinningScore or alpha >= WinningScore. The first case is redundant, because when alpha <= -WinningScore, standPat + victimValue + margin will *always* be > alpha, so that you won't prune anyway based on the condition after &.

You are absolutely correct that testing for (alpha <= -WinningScore) is redundant. But is the routine going to be so much slower because of this that it causes it to play 100+ elo weaker? I don't think so. I use this simple pattern (alpha or beta compared to abs(winningScore)) in places where I want to avoid triggering code during a mate search. It's not going to wreck the whole thing having a redundant test.

And *not* pruning when alpha >= WinningScore seems a mistake, as it lead you to search moves that will most certainly be refuted by stand pat. In the daughter node you will have beta <= -WinningScore, and eval will most certainly be larger than that.

Indeed, when (alpha >= WinningScore) most moves will simply be refuted in the child node - in qsearch. That's why it doesn't drop into qsearch. Instead it proceeds with search at normal depth without pruning so it can have some chance of finding a move that has a score better than the current alpha. Sure, it will be slower than simply saying "A winning PV has already been found, so I'm just going to bail out". But when a winning PV has already been found there's no need to prune, we can waste time looking for a better move and still win.

but it's not worse that using a static delta of say 150 (the recommended value from what I've seen).

Well, as you seem to use a non-standard (i.e. non-centi-Pawn) score scale, I cannot really comment on that.

I'm using a very standard scale. Pawn=100, minor pieces=320-350, rook=500, queen=950 (or something like that).

I only have the deltas set to such large (e.g. conservative) values because in prior tests where I have used a static delta of 150 (or 200, or 250, or 500 - yes I've tried many different static delta values) the results were just as bad. So I am just trying something different - a more conservative delta at the beginning of each qsearch branch, with that delta getting logarithmically less as the branch gets longer (because along the branch more pieces should be coming off the board reducing the volatility of the position, making smaller deltas less risky).

The moves are sorted MVV/LVA.

Well, then as soon as you reach your first futile capture, the entire remaining part of the list must be futile, and you could bulk-prune it.

Simply not true, for the reason I've already stated: checks.

I make/unmake the move because I have no other way of knowing if the move gives check or not. And moves that give check throw all assumptions about material value out the window because they could be mate, or lead to mate.

It is more because even at d=0 you would not allow stand pat when in check.

I don't allow stand pat when in check. I don't understand the point you're trying to make here.

futility pruning is a method to speed up the engine by not doing make/unmake and stand-pat test with full evaluation on moves where you can be certain they would pass the stand-pat test based on a lazy estimate of the evaluation. If you are going to make/unmake the move, and your evaluation is not very heavy or you allow the stand-pat test to be lazy as well, you cannot expect any gain from it.

I disagree. As with all other forms of pruning the point is to prune entire branches of the search tree, not simply avoid doing makes and/or evals here and there. But you and I obviously have our minds made up on this point so I see little reason to continue discussing it further.

That being said I can and will do things to avoid doing unnecessary makes/unmakes in order to know if a move gives check or not. This engine is only a month old. It simply doesn't have all the minor optimizations that you deem so important (yet).

I think we must separate two issues here: the d=1 futility pruning and the deeper stuff. The d=1 futility should not really prune any line; it would just correctly predict which lines would fail low due to stand pat. There should be no Elo drop other than caused by the speed loss due to the implementation. (Node that nps might be misleading here, as moving the stand-pat cutoff to the parent would no longer count the nodes where you do stand pat, which you might have counted before.) I don't think there should be any large Elo drop because of this.

Again, you dismiss the possibility of checks. There is no way to "correctly predict which lines would fail low due to stand pat" when the move gives check.

As for the nps rate being misleading, all my engines calculate node rate from the number of Exec(move) calls that are made, not the number of times search is called. So the node rates reported by my engines are effected by activity in qsearch, not just what happens in the primary search function. The node rate will also be properly effected by all the makes/unmakes that you're talking about avoiding.

One caveat: If you do fail-soft you should take care how to increment your upper score bound after futility pruning; the score of the pruned move should be counted as currEval + victimValue + margin. Otherwise you will get a way-too-low upper bound, which might cause wron prunings when retrieved from the hash in cases where the pruned moves wer not futile (i.e. with lower alpha).

Duly noted. Good advice.

The only real pruning takes place for the deeper stuff. This has more the character of a conditional reduction than a pruning: you first search reduced (by doing QS instead of d=2 or d=3), and if that fails low you leave it at that (and otherwise you continue with the normal search).

This will of course prune many important lines: basically it says "if I am very much behind, and still have 3 ply to recover, I am going to give up already if I cannot recover in a single ply". An example of something that would be unjustly pruned by this is when you have a Knight fork on two Rooks, and the opponent tries to push the exchange loss over the horizon by trading a Queen. You might be down by more than a Queen at that point, and the best you can achieve in QS is recapturing the Queen. The opponent would then stand pat, so the recapture fails low, and with it the entire QS, so that you prune. Would you have searched d=2 or d=3 you would have noticed that whatever the opponent played after you recaptured the Queen, you would have played NxR to cash the exchange, which gained you the 200cP you needed to end above alpha, rather than below it.

Reducing moves because your surrent score is bad is often a self-fulfilling profecy: You get even less time to get even, and so you won't succeed.

This is why I think alpha pruning doesn't work very well. But other engines seem to have great success with it, which is why I started this thread - to try and understand why.

Maybe I'm not giving your arguments about futility pruning being all about node rate increase enough consideration. I will dwell on it more and do more testing with your suggestions (most of which I've already done in other engines, but not in Clunk). Though I can tell you I have not seen a noticeable node rate drop in the instances with alpha pruning enabled. They search a little deeper but play a lot weaker and they do it all at the same node rate.

[hgm]

Ordinarily I would have better things to do than banter with you over these mostly unimportant details. But since I don't want to do anything too heavy (like compiling and running epd tests) while the round-robin is running I'll kill some time volleying with you.

You would indeed have spent your time better re-reading what I actually wrote, than shooting off your mouth about what you imagine I wrote. Especially after I pointed out that you did not understand a word of it...

I think you very clearly overlooked the negation. I am *not* pruning futile moves when in check.

Which is of course exactly what I said. And that seems WRONG, which is why I brought it up. When in check, futile moves deserve to be pruned just as much as when not in check.

You are absolutely correct that testing for (alpha <= -WinningScore) is redundant. But is the routine going to be so much slower because of this that it causes it to play 100+ elo weaker? I don't think so. I use this simple pattern (alpha or beta compared to abs(winningScore)) in places where I want to avoid triggering code during a mate search. It's not going to wreck the whole thing having a redundant test.

It will not cause a 100-Elo slowdown. But note that it is not merely an inefficiency because it executes some pointless instructions, but actually prevents futility prunings that should logically have been made.

Indeed, when (alpha >= WinningScore) most moves will simply be refuted in the child node - in qsearch. That's why it doesn't drop into qsearch. Instead it proceeds with search at normal depth without pruning so it can have some chance of finding a move that has a score better than the current alpha.

And that is a pure waste of time, because the 'chance' that any non-checking move will score above WinningScore is an exect mathematical zero...

Sure, it will be slower than simply saying "A winning PV has already been found, so I'm just going to bail out". But when a winning PV has already been found there's no need to prune, we can waste time looking for a better move and still win.

Except of course that this winning PV might disappear again because the opponent switches away from the blunder that would get him mated in a node closer to the root, and you would have to fight hard for a draw after having wasted all this time on searching futile moves in an irrelevant branch...

I'm using a very standard scale. Pawn=100, minor pieces=320-350, rook=500, queen=950 (or something like that).

Well, then Michel's so far unwithspoken remark that 800 is "only about the value of a Bishop" doesn't cut wood, and I should revive the original criticism: why do you use such a ridiculously large margin before deciding that non-captures are not going to get even in a single ply (i.e. at d=1)? Do you really expect there to be non-captures that up the evaluation by 799 centi-Pawn? If so, what eval term would be responsible for this?

I only have the deltas set to such large (e.g. conservative) values because in prior tests where I have used a static delta of 150 (or 200, or 250, or 500 - yes I've tried many different static delta values) the results were just as bad. So I am just trying something different - a more conservative delta at the beginning of each qsearch branch, with that delta getting logarithmically less as the branch gets longer (because along the branch more pieces should be coming off the board reducing the volatility of the position, making smaller deltas less risky).

Well, obviously letting the margins approach infinity should make all effects of the pruning go away, as it would be never done. If you would still see a 100-Elo decrease in strength by adding non-functional code, that would be very fishy. Note that it is not completely impossible; when I was writing qperft at some point it contained some statements that were redundant, and could not be executed anymore. (I verified the unreachability by incrementing a global counter variablein it, and asserting that it was still at zero after peft(8).) And when I deleted those unreachable statements, qperft ran 20% slower! So yes, adding code that is never executed can have an impact on the speed of code that is executed, simply by taking up space, and moving the executed code to other addresses, which happen to be more favorable (e.g. better alignment of branch targets with cache lines). I would only expect that to matter much in very tight loops, however.

Simply not true, for the reason I've already stated: checks.

Well, as I explained for checks you should have a separate generator. As long as you have that, futility pruning cannot be expected to work, so that it should not come as a surprise to you that it indeed does not work.

I don't allow stand pat when in check. I don't understand the point you're trying to make here.

The point is that the conditions for futility pruning should be an exact match to the conditions of your stand-pat. They should not be based on vague notions like "this move could lead to mate" or "I have already a mating score in the current PV". When the daughter can stand pat you should prune. When it cannot, you should not prune. E.g. when you are in check now, and evade, the daughter can stand pat after the evasion. So you should prune when in check.

I disagree. As with all other forms of pruning the point is to prune entire branches of the search tree, not simply avoid doing makes and/or evals here and there.

Hard fact is that at d=1 your "entire branches" are never more than a single node. If you would have not prunded the node, the daughter would have stand pat based on the lazy eval. That is a fixed amount of work that you might be able to save by pruning that node if the pruning decision itself was not more expensive. Imagining that you could ever gain more is a delusion.

But you and I obviously have our minds made up on this point so I see little reason to continue discussing it further.

Well, if you are not willing to learn, posing questions here is pretty useless, I would think.

Again, you dismiss the possibility of checks. There is no way to "correctly predict which lines would fail low due to stand pat" when the move gives check.

I did not mention checks at all here. If a move delivers check, you can correctly predict it wil NOT fail low due to stand pat. This is why you should NOT prune checks; they are NEVER futile. Nothing of what I said contradicts this. You seem to have a false association between doing making moves and searching checks. When I say you cannot expect any gain from futility pruning when you make all pruned moves, it in no way implies that you should not search checks. It just means that you should determine what are checks without making any moves. And if your engine is not at the stage yet where it can do this, you are certainly not in a position where you should attempt futility pruning. Even without any futility pruning you should have a huge speedup from not making/unmaking all non-captures in QS.

As for the nps rate being misleading, all my engines calculate node rate from the number of Exec(move) calls that are made, not the number of times search is called. So the node rates reported by my engines are effected by activity in qsearch, not just what happens in the primary search function. The node rate will also be properly effected by all the makes/unmakes that you're talking about avoiding.

Well, in that case you would see the drop in nps that accompanies the speedup only after you found out a way to do futility pruning and QS without making all moves (and in particular non-captures) that you are not going to search.

Though I can tell you I have not seen a noticeable node rate drop in the instances with alpha pruning enabled. They search a little deeper but play a lot weaker and they do it all at the same node rate.

This is because you count makemoves, rather than nodes. This will cause your nps to drop only when eleminiting redundant makemoves. But that should of course not deter you from doing so.

[hgm]

SF caches all those things.

Well, that is what I suspected. So the damage is not very large. But that doesn't mean you can just transplant that code to an arbitrary other engine and expect it to be beneficial. As is the case here, that other engine probably does NOT cache these things, and furhermore has an awful implementation of QS to which you defer the pruning, by making and unmaking all non-captures even in QS.

You keep ignoring that this was _tested_ to contribute elo. Moreover Lucas tried to remove this code _twice_ (which is statistically unsound) using a no-regression test and failed.

I did not ignore that at all; I already commented on it. That you have a regression on removing it doesn't prove a thing if it is not specified what the alternative was that they tested. Just removing the code and searching all non-captures at d=1 when they are futile would obviously not be the proper thing to do. If I have a crappy implementation of one algorithm, and replace it by a good implementation of another algorithm, and this is shown to be detrimental, it doesn't prove in any way that the crappy implementaion was optimal.

Also note the qperft experience I describe in the previous post: modern CPUs are such complex machines that sometimes unexpected speed differences result from non-functional changes, like a 20% slowdown from removing some unreachable code. Regressions can also be due to such random 'compiler noise', and make algorithms that on average (i.e. averaged over many different compiles with many independent changes) would be better (because they cause lower CPU loading) would sometimes run slower. I don't think that it would be good strategy to stick with the less efficient, but accidentally faster code in that case.