Magic SEE

Discussion of chess software programming and technical issues.

Moderators: hgm, Dann Corbit, Harvey Williamson

mvk
Posts: 589
Joined: Tue Jun 04, 2013 10:15 pm

Re: Magic SEE

Post by mvk »

hgm wrote:
mvk wrote:The hit rate for the imperfect hash of attacks^defenders is good enough, where attackers and defenders are just the linear combination of the piece counts, each of 12 bits if I recall correctly.
This means you store a signature in the hash table to verify a hit,and rehash on a miss?

I like the idea of hashing on some difference measure of the attacker and protector set, as often equal pieces would cancel.
I have to double-check, because this is old stuff. There is no rehash on miss. Just a calculation and a store (overwrite). For convenience, this is the implementation.

Code: Select all

/*----------------------------------------------------------------------+
 |      exchange_evaluate                                               |
 +----------------------------------------------------------------------*/

/*
 *  Static Exchange Evaluation (SEE).
 *
 *  We assume that the attacker has just moved, thus the defending side
 *  is to move next. This function calculates the maximum retaliation that
 *  the defender can obtain by following an optimal exchange sequence.
 *
 *  Input are the two piece sets, each represented by a compact word.
 *  Returned is the static exchange evaluation, scaled by EXCHANGE_UNIT
 *  for convenience.
 *
 *  Defenders
 *                14-13      12               11-0
 *              +-------+---------+-------------------------------+
 *              |   0   |last_rank|    set of defending pieces    |
 *              +-------+---------+-------------------------------+
 *
 *  'defenders' represents the set of defending pieces. 0 means none.
 *  The 'last_rank' flag indicates that the exchange occurs on the last rank.
 *  We assume that the defender will always capture with its least-valued
 *  piece. Interestingly, this is also the optimal strategy on the last rank.
 *
 *  Attackers
 *                14-13      12               11-0
 *              +-------+---------+-------------------------------+
 *              |upfront|last_rank|    set of attacking pieces    |
 *              +-------+---------+-------------------------------+
 *
 *  'attackers' represents the set of attacking pieces. This set can't be
 *  empty because of the assumption that this side has just moved.
 *  This set has a 2-bit field 'first' which indicates what piece moved
 *  just before, because that is the first to get captured by the defender.
 *
 *  The piece encoded in 'first' is not present in the 12 bit set. The
 *  reason is that this allows for 3 pawns (one just moved, 2 defending)
 *  while the set only has space for 2 (otherwise more than 12 bits are
 *  needed for the set and everything doesn't fit anymore).
 *  Therefore in this set 0 means: one unsupported pawn.
 *
 *  The returned result can't be negative because the side to move has the
 *  right not to continue the exchange sequence.
 *
 *  The function caches earlier results for improved performance.
 */

int exchange_evaluate_fn(int defenders, int attackers)
{
        /*
         *  Input check
         */
        xassert(defenders != 0);

        xassert((defenders & 0x1fff) == defenders);
        xassert((attackers & 0x7fff) == attackers);

        xassert((defenders & attackers & EXCHANGE_LAST_RANK) == 0);

        int store = defenders & 0x0fff;

        int hash = &#40;store << 3&#41; ^ defenders ^ attackers;

        xassert&#40;hash >= 0&#41;;
        xassert&#40;hash < 0x8000&#41;;

        /*
         *  Hashing should be such that we can reconstruct the two inputs
         *  from the hash and the stored lower 12 bits of the defenders.
         *  There is one ambiguity allowed&#58; which of the two inputs has the
         *  LAST_RANK bit set, if any.
         *  Verify the hashing inverse with assertions.
         */
        xassert&#40;(&#40;hash ^ &#40;store << 3&#41;)   & EXCHANGE_LAST_RANK&#41; ==
                (&#40;attackers ^ defenders&#41; & EXCHANGE_LAST_RANK&#41;);
        xassert&#40;(&#40;hash ^ &#40;store << 3&#41; ^ store&#41; & ~EXCHANGE_LAST_RANK&#41; ==
                &#40;attackers                     & ~EXCHANGE_LAST_RANK&#41;);

        /*
         *  Consult the lookup table first
         */
        unsigned short lookup = exchange_table&#91;hash&#93;; /* Thread-safe */

        if ((&#40;lookup ^ defenders&#41; & 0x0fff&#41; == 0&#41; &#123;
                xassert&#40;EXCHANGE_UNIT == 0x0100&#41;;
                return &#40;lookup & 0xf000&#41; >> 4;
        &#125;

        /*
         *  Table missed, calculate exchange sequence result
         */
        board_exchange_table_miss_counter++;

        /*
         *  Value of captured piece
         */

        static const int victim_value&#91;4&#93; = &#123;
                1 * EXCHANGE_UNIT,        // pawn
                3 * EXCHANGE_UNIT,        // minor &#40;bishop or knight&#41;
                5 * EXCHANGE_UNIT,        // rook
                9 * EXCHANGE_UNIT,        // royal &#40;king or queen&#41;
        &#125;;
        int result = victim_value&#91; attackers >> 13 &#93;;

        /*
         *  Fix up a false LAST_RANK flag in some promotions captures
         */

        if &#40;defenders == EXCHANGE_LAST_RANK&#41; &#123;
                return 0; // TODO&#58; fix the flow of control &#40;and cache&#41;
        &#125;

        /*
         *  Select next capturer
         */

        if (&#40;defenders & EXCHANGE_LAST_RANK&#41; != 0&#41; &#123;

                /*
                 *  Capture with pawn and promotion
                 */
                switch (&#40;defenders & 0x0fff&#41; % RANGE_PAWN&#41; &#123;
                case 0&#58;
                        /* No pawn present... Clear the LAST_RANK flag and proceed as normal */
                        defenders = next_upfront&#40;defenders & ~EXCHANGE_LAST_RANK&#41;;
                        break;
                case 1&#58;
                        result += 8 * EXCHANGE_UNIT;
                        defenders += &#40;3 << 13&#41; - EXCHANGE_LIST_PAWN - EXCHANGE_LAST_RANK;
                        xassert&#40;&#40;defenders & EXCHANGE_LAST_RANK&#41; == 0&#41;;
                        break;
                default&#58;
                        result += 8 * EXCHANGE_UNIT;
                        defenders += &#40;3 << 13&#41; - EXCHANGE_LIST_PAWN;
                        xassert&#40;&#40;defenders & EXCHANGE_LAST_RANK&#41; != 0&#41;;
                        break;
                &#125;
        &#125; else &#123;

                /*
                 *  Regular capture
                 */
                defenders = next_upfront&#40;defenders&#41;;
                xassert&#40;defenders >= 0&#41;;
        &#125;

        /*
         *  Recursive evaluation
         */
        attackers &= 0x1fff; /* remove the upfront piece */
        if &#40;attackers != 0&#41; &#123;
                result -= exchange_evaluate_fn&#40;attackers, defenders&#41;;

                /*
                 *  If result is negative, don't capture but stand pat
                 */
                if &#40;result < 0&#41; &#123;
                        result = 0;
                &#125;
        &#125;

        xassert&#40;result >= 0&#41;;

        /*
         *  Clip at +14 so that the SEE result always fits in 5 bits
         *      0xfe00 -> +14 == maximum gain
         *      0xf000 ->  +0 == neutral with capture
         *      0x0f00 ->  -0 == neutral, no capture
         *      0x0100 -> -14 == maximum loss
         */
        if &#40;result > 14 * EXCHANGE_UNIT&#41; &#123;
                result = 14 * EXCHANGE_UNIT;
        &#125;

        /*
         *  Update the lookup table and return
         */

        exchange_table&#91;hash&#93; = &#40;result << 4&#41; | store; /* Thread-safe */

        return result;
&#125;
[Account deleted]
User avatar
hgm
Posts: 27701
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Magic SEE

Post by hgm »

D Sceviour wrote:Your numbers simply do not add up. The fact is the top programs exhibit incredible speed. Every doubling of speed results in an extra ply depth of search (assuming a branching factor of 2). And every extra ply is worth 100-150 elo.
That is a bit optimistic. Usually people assume 70 Elo for a speed doubling, (100*ln(T)) and even that tails off when you you get to very high depth. And the numbers I give are consistent with that: a factor 100 increase in speed is slightly under 7 doublings, and 6.5*70 Elo = 455 Elo. In your estimate one would even gain 1000 Elo.
If what you said were true, then threading would be a waste of time. Maybe it is for other reasons, but not for improvement with speed. There is no substitution for depth of search.
The latter is true, but the point was that the the depth of search increases much more by reducing your node count needed to reach the same depth by a factor 100, than it would by doubling nps. So yes, threading is basically a waste of time, unless you have really nothing else to improve on your search.
Node counts are important, but if you look at Schooner, you will find the node counts low. Much lower than Crafty for example, but Crafty exhibits speeds of 200 million moves per second (i7 - 3.4 GHz).
Crafty v23.4 PS (1 cpus)

White(1): perf
generated 80000000 moves, time=0.41 seconds
generated 197530864 moves per second
generated/made/unmade 80000000 moves, time=1.61 seconds
generated/made/unmade 49689440 moves per second
Moves are not nodes. Divide it by 40 for the typical number of moves per position,and you are left with 5M nps. Micro-Max 1.6 does about 6M nodes/sec. Because it has no hash table, and the hash probe slows it down to about 1M nps (in micro-Max 4.8). Crafty also does no hash probe in QS, which makes it easier to reach high nps (but of course then uses many more nodes for the same search, because it doesn't get any hash cutoffs). In perft it does not use any hashing at all, AFAIK. And it also doesn't evaluate. Perft speeds are just totally meaningless.
Crafty could not possibly do that with strenuous SEE calculations for move ordering.
Indeed. I don't think it orders moves at all during perft. No need for SEE then. It is just one of these other parts of the engine that take no time at all during perft because they are not used. Very easy to get a high nps that way...
Your dead-end advice seems more suitable for an engine playing 1500 strength. Maybe it would be better increasing the number of nodes searched slightly to reduce errors and improve play.
Oh, it also still holds for engines playing at 2400 strength. Of course eventually you will reach some absolute limit, where the search is 100% efficient, you make all prunings that can be afforded, the evaluation is close to perfect etc., and improving the speed is the only thing left to do. But you know 2400 is still very far from that, as there are engines around that perform at 3400 Elo. You would have to speed up nps by a factor 10,000 to make up for that with speed, using my 70Elo/doubling estimate. There is no way you are going to get that kind of a factor by changing to more efficient move generation or multi-threading...
User avatar
hgm
Posts: 27701
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Magic SEE

Post by hgm »

mvk wrote:I have to double-check, because this is old stuff. There is no rehash on miss. Just a calculation and a store (overwrite). For convenience, this is the implementation.
Ah, OK, you hash it dynamically. I was imagining a read-only hash table initialized during startup. I guess your system would work very well, as the complex exchanges very rarely occur in practice, and all the common ones would quickly get into the table without colliding. The only drawback is that there is some overhead in checking the signature.

I am thinking now of resorting to the 'zero-order approximation' that human beginners typically use:
1) more attackers than protectors => you gain the piece
2) equal or fewer attackers => you trade the piece
But then only apply it to the tail of the SEE, and do the first few steps taking account of piece identity and value. Case 1 is dependent on thre never being a stage where the value of the defender's first N+1 pieces is less than that of the attacker's first N pieces, and this can even cause case (1) to turn into a bad capture.

But in any case, it seems relevant to not lose the information on how many attackers and protectors there are; this is more important than howmuch exactly the later pieces are worth. This thought led to the following system:

For each side you encode the two lowest attackers and the total number. Sometimes this implies what the remaining piece is: if Q is part of the lowest two, and you had 3 attackers, the third must be K. So I decided to make the Q+K part exact, if it participates in the first two steps. That means there are 9 possibilities for it, taking note of x-rays: Q, QB, QR, QRR (where the latter 3 are x-rays), then the same 4 with additional K, and finally K-only. Each of these 9 combinations occurring as second piece can be combined with P, R or minor as the first piece (25 possibilities, as QRR and QRRK cannot combine with R). There are 8 cases of a single attacker (where x-rays count as single): P, m, R, Q, QB, QR, QRR, K, and 1 case of no attackers at all; these 9 cases imply the totalnumber of attackers. This is also true for all cases where K or Q belong to the first two.

So there only is uncertainty on the tota lnumber of pieces when you the lowest two pieces are both P, minor or R. This can happen in 6 ways. For each of the 6 cases you then supply 4 versions: where they occur as only two, and where there 1, 2 or more additional attackers of unspecified type. That needs 6x4 = 24 codes. So the attackers state is summarized by 25 + 9 + 24 = 58 different codes. These can then be used to index a 2d table of 58 x 58 ~ 3.5 KB with SEE results. The very deep SEEs will not be entirely reliable, and at some point (with 4-or-more pieces on both sides) even completely unkown. In those cases you just play the move to let QS figure it out.

Note that this table is intended to judge the value of a recapture, so that the original victim and capturer are not part of the lists. (The capturer would be removed before encoding the attackers set.) So at least 6 pieces involved in the exchage would be considered explicitly, and two more just by number.
User avatar
hgm
Posts: 27701
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Magic SEE

Post by hgm »

I think I will finally do it the following way:

Use weights P=1, m=3, R=12, K=36, Q=72, RxQ=72 (on top of Q, like it is a second Queen), RxRxQ = 24 (on top of Q + RxQ, like a second Queen with 2 Rooks present, which could not happen when some of the Rooks are x-raying the Queen). That uses all codes up to 216.

Unfortunately it does not yet handle the BxQ case. This can occur in 54 different combinations of other material. But only 40 codes are available before we overflow a byte. I don't want the counters to be word size, as that would drive up the time needed to reset the counters to 0 before you start counting (which can be done in groups of 8 when they are bytes). I'd rather do something special in the rare case there actually is a BxQ x-ray.

So rather than adding something to the attackers counter for the to-square, I will just remember the square that is hit by the BxQ x-ray (which can only be one square). The x-rays are handled by special, rarely executed code anyway, so it is no burdonto test for this case and treat it in a special way.

In the rare case that after move generation this BxQ square is set (it will be initialized to an invalid square number, so we can see that), the counters for that square will be modified to add the BxQ x-ray to the already recorder attackers. As only 40 codes are available, some of the 54 cases of already-recorded material will have to be re-mapped to collide with other cases. Knowing that we are going to use the counters to determine lowest two pieces plus total number, we will map the QxB case to the QR case whenever QB is not part of the lowest two pieces.

This only leaves the cases where QxB is amongst the lowest two. This happens when there are no P, minor or R attackers, just one of those, or 2 Rooks (for which there existed no RxQ code). That is 5 cases, with and 5 without King as additional, last attacker. So we only need 10 new codes, which brings the coding range to 0-225.

This code will then be used to access a 226-entry table to assign a number 0-57 to it, encoding lowest two attackers, and together with a similar code for the protectors this will then be used to index a 58 x 58 table of SEE values.

Code: Select all

char seeValue&#91;NrPieces&#93; = &#123; 0, 1, 3, 3, 5, 9, 100, ... &#125;;     // empty, P, N, B, R, Q, K, ... &#40;black omitted&#41;
char attackWeight&#91;NrPieces&#93; = &#123; 0, 1, 3, 3, 12, 72, 36, ...&#125;; // weights in attackers&#91;&#93; and protectors&#91;&#93;

char bqEncode&#91;72&#93;;       // adds a B-Q x-ray to given attackers set, mapping it to B-R when Q is not amongst two lowest attackers
char attackersCode&#91;226&#93;; // convert attackers set to a code 0-57 specifying lowest two and total number
char seeTable&#91;58&#93;&#91;58&#93;;   // tabulates loss for moving into square with lowest attacker

// one-time Bishop x-ray correction, done after move generation that counted attackers&#91;&#93;
if&#40;xSqr >= 0&#41; &#123; // there was B-through-Q x-ray attack on xSqr
  attackers&#91;xSqr&#93; = bqEncode&#91;attackers&#91;xSqr-72&#93;&#93;; // only codes 72-143 occur, as there must be Q attacker
&#125;

int FreeSEE&#40;int victim, int attackers, int protectors&#41;
&#123; // returns how much best capture of 'victim' will gain, negated
  int att = attackersCode&#91;attackers&#93;;
  int prot = attackersCode&#91;protectors&#93;;
  int see = seeTable&#91;att&#93;&#91;prot&#93; - seeValue&#91;victim&#93;; // loss incurred by capturing
  see &= see >> 31; // clip positive loss to 0, as we would then refrain from capture
  return see;
&#125;

int SEE&#40;int fromSqr, int toSqr&#41;
&#123; // return gain from a give capture
  int piece = board&#91;fromSqr&#93;;
  int victim = board&#91;toSqr&#93;;
  int att = attackers&#91;toSqr&#93; - piece2bit&#91;piece&#93;; // initial capturer has already been used
  int recapt = FreeSEE&#40;piece, protectors&#91;toSqr&#93;, att&#41;; // swap role of attack and protect, gives &#40;negated&#41; gain of recapture
  return seeValue&#91;victim&#93; + recapt; // recapt <= 0
&#125;
This seems an affordable amount of work for a SEE, with no excessive load on the L1 cache.

The FreeSEE() can also be used for judging threats against the side to move. A non-zero return value (always negative) would mean we stand to lose something on that square.
jwes
Posts: 778
Joined: Sat Jul 01, 2006 7:11 am

Re: Magic SEE

Post by jwes »

To go back to the original question:
If you represent the possible combinations of attackers on a square by a number, can you find a simple combination of operations that will give the result of a SEE on that square, e.g. multiply by a constant, shift, mask, and 256 byte table lookup.