Progress On Bricabrac

[Mike Sherwin]

Well that was easy! Just increase the bonus for a lesser value piece attacking a more valuable piece and now the algorithmically generated tables are at least on par with PeSTO.

Bricabrac - Tscp181 : 59.5/100 53-34-13 (110101=11101=0=11=1100=011001111111100==101010101=0=11110=10111010111101=01100001100110=010011=01101) 60% +70

I know that 100 games is not enough but it is good enough for now. It is a lot better than the first attempt for sure.

Bricabrac - Tscp181 : 9.0/100 6-88-6 (0100000000000000001000000000000000010000000000000=000000000=000000=000000000=000=000010100000=100000) 9% -402

The resulting PSTs are generated by really crude code. For example, wpPST[sq] = wrow * wrow * wrow + cenCol; cannot really be tuned. And it is quite arbitrary to begin with. If wrow was floating point then at least it could be multiplied by another floating point so it can be tuned. Or is there just a much better formula to begin with. Any suggestions? Or maybe just load values into an array and use an automated tuning method. I really could use some advice on this.

Anyway, if one wants to see the crude eval code that at least equaled PeSTO in my engine here it is.

Code: Select all

void InitSearch(Thread* t) {
  u64 pieces, moves, captures;
  u32 sq;
  s32 col, wrow, brow, cenCol, cenRow, sum, x, y;
  u64 wPawns, wKnights, wBishops, wRooks, wQueens;
  u64 bPawns, bKnights, bBishops, bRooks, bQueens;
  u64 allPieces = piece[WHITE] | piece[BLACK];
  SplitBB blocks;

  u64* type[] = {
    nullptr,
    &wPawns, &wKnights, &wBishops, &wRooks, &wRooks, &wQueens, nullptr, nullptr,
    &bPawns, &bKnights, &bBishops, &bRooks, &bRooks, &bQueens, nullptr, nullptr,
  };

  wPawns = wKnights = wBishops = wRooks = wQueens = 0;
  bPawns = bKnights = bBishops = bRooks = bQueens = 0;

  for (sq = A1; sq <= H8; sq++) {
    if (type[board[sq]] != nullptr) *type[board[sq]] |= one << sq;
  }

  for (sq = A1; sq <= H8; sq++) {

    col = x = sq & 7;
    wrow = y = sq >> 3;
    brow = 7 - wrow;
    cenCol = col < 4 ? col : 7 - col;
    cenRow = wrow < 4 ? wrow : 7 - wrow;

    // white pawns
    if (sq > H1 && sq < A8) {
      if (wrow == 1) wpPST[sq] = 40;
      else wpPST[sq] = wrow * wrow * wrow + cenCol;
      wpPST[sq] += ply * wrow / 32;
    }

    // black pawns
    if (sq < A8 && sq > H1) {
      if (brow == 1) bpPST[sq] = 40;
      else bpPST[sq] = brow * brow * brow + cenCol;
      bpPST[sq] += ply * brow / 32;
    }

    // knights
    moves = knightMoves[sq] & ~allPieces;

    // white knights
    captures = knightMoves[sq] & (bRooks | bQueens | king[BLACK]);
    wnPST[sq] = __popcnt64(moves) * 4 + __popcnt64(captures) * 32;
    wnPST[sq] += cenRow * cenRow + cenCol * cenCol + wrow;
    wnPST[sq] -= (!cenRow || !cenCol) * 30;

    // black knights
    captures = knightMoves[sq] & (wRooks | wQueens | king[WHITE]);
    bnPST[sq] = __popcnt64(moves) * 4 + __popcnt64(captures) * 32;
    bnPST[sq] += cenRow * cenRow + cenCol * cenCol + brow;
    bnPST[sq] -= (!cenRow || !cenCol) * 20;

    // bishops
    blocks.bb = allPieces & b7e[sq];
    blocks.bb = ((blocks.l32 >> 8) | blocks.h32);
    moves = bss[sq][blocks.r1][0]
      & bss[sq][blocks.r2][1]
      & bss[sq][blocks.r3][2];

    // white bishops
    captures = moves & (bRooks | bQueens | king[BLACK]);
    moves ^= captures;
    wbPST[sq] = __popcnt64(moves) * 3 + __popcnt64(captures) * 32;
    // wbPST[sq] += cenCol * cenRow * cenRow;

    // black bishops
    captures = moves & (wRooks | wQueens | king[WHITE]);
    moves ^= captures;
    bbPST[sq] = __popcnt64(moves) * 3 + __popcnt64(captures) * 32;
    // bbPST[sq] += cenCol * cenRow * cenRow;

    // rooks
    blocks.bb = allPieces & rob[sq];
    moves = qss[sq][(blocks.bb >> (sq & 56)) & 127][0]
      & qss[sq][blocks.r2][1]
      & qss[sq][blocks.r3][2]
      & qss[sq][blocks.r4][3]
      & qss[sq][blocks.r5][4]
      & qss[sq][blocks.r6][5]
      & qss[sq][blocks.r7][6]
      & rob[sq];

    // white rooks
    captures = moves & (bQueens | king[BLACK]);
    moves ^= captures;
    wrPST[sq] = __popcnt64(moves) * 2 + __popcnt64(captures) * 32;
    wrPST[sq] += cenRow + wrow == 6 * 40;

    // black rooks
    captures = moves & (wQueens | king[WHITE]);
    moves ^= captures;
    brPST[sq] = __popcnt64(moves) * 2 + __popcnt64(captures) * 32;
    brPST[sq] += cenRow + brow == 6 * 40;

    // queens
    blocks.bb = allPieces & rob[sq];
    moves = qss[sq][(blocks.bb >> (sq & 56)) & 127][0]
      & qss[sq][blocks.r2][1]
      & qss[sq][blocks.r3][2]
      & qss[sq][blocks.r4][3]
      & qss[sq][blocks.r5][4]
      & qss[sq][blocks.r6][5]
      & qss[sq][blocks.r7][6];

    // white queens
    captures = moves & king[BLACK];
    moves ^= captures;
    wqPST[sq] = __popcnt64(moves) + __popcnt64(captures) * 32;

    //black queens
    captures = moves & king[WHITE];
    moves ^= captures;
    bqPST[sq] = __popcnt64(moves) + __popcnt64(captures) * 32;

    // white king
    sum = (5000 - mat[BLACK]) >> 6;
    wkPST[sq] = (cenCol + cenRow) * sum;

    // black king
    sum = (5000 - mat[WHITE]) >> 6;
    bkPST[sq] = (cenCol + cenRow) * sum;
  }

  // white pawn specifics
  wpPST[E2] -= (board[E1] == WC) * 40;
  wpPST[E3] += (board[E1] == WC) * 20;
  wpPST[E4] += (board[E1] == WC) * 40;
  wpPST[D2] -= (board[E1] == WC) * 40;
  wpPST[D3] += (board[E1] == WC) * 20;
  wpPST[D4] += (board[E1] == WC) * 40;

  // black pawn specifics
  bpPST[E7] -= (board[E8] == BC) * 40;
  bpPST[E6] += (board[E8] == BC) * 20;
  bpPST[E5] += (board[E8] == BC) * 40;
  bpPST[D7] -= (board[E8] == BC) * 40;
  bpPST[D6] += (board[E8] == BC) * 20;
  bpPST[D5] += (board[E8] == BC) * 40;

  // white knight specifics
  wnPST[G1] -= (board[E1] == WC) * 20;
  wnPST[B1] -= (board[E1] == WC) * 20;
  wnPST[C3] -= (board[C2] == WP) * 40;

  // black knight specifics
  bnPST[G8] -= (board[E8] == BC) * 20;
  bnPST[B8] -= (board[E8] == BC) * 20;
  bnPST[C6] -= (board[C7] == BP) * 40;

  // white bishop specifics
  wbPST[F1] -= (board[E1] == WC) * 20;
  wbPST[C1] -= (board[E1] == WC) * 20;
  wbPST[D3] -= (board[D2] == WP) * 60;
  wbPST[E3] -= (board[E2] == WP) * 60;

  // black bishop specifics
  bbPST[F8] -= (board[E8] == BC) * 20;
  bbPST[C8] -= (board[E8] == BC) * 20;
  bbPST[D6] -= (board[D7] == BP) * 60;
  bbPST[E6] -= (board[E7] == BP) * 60;

  // white queen specifics
  wqPST[E2] -= (board[F1] == WB) * 40;
  wqPST[D3] -= (board[F1] == WB) * 40;
  wqPST[D2] -= (board[C1] == WB) * 40;
  wqPST[E3] -= (board[C1] == WB) * 40;
  wqPST[D1] += (board[B1] == WN) * 10;
  wqPST[D1] += (board[C1] == WB) * 10;
  wqPST[D1] += (board[F1] == WB) * 10;
  wqPST[D1] += (board[G1] == WN) * 10;

  // black queen specifics
  bqPST[E7] -= (board[F8] == BB) * 40;
  bqPST[D6] -= (board[F8] == BB) * 40;
  bqPST[D7] -= (board[C8] == BB) * 40;
  bqPST[E6] -= (board[C8] == BB) * 40;
  bqPST[D8] += (board[B8] == BN) * 10;
  bqPST[D8] += (board[C8] == BB) * 10;
  bqPST[D8] += (board[F8] == BB) * 10;
  bqPST[D8] += (board[G8] == BN) * 10;

  // white king specifics
  wkPST[G1] += (board[E1] == WC) * 50;
  wrPST[F1] += (board[E1] == WC) * 50;

  // black king specifics
  bkPST[G8] += (board[E8] == BC) * 50;
  brPST[F8] += (board[E8] == BC) * 50;

  pos[WHITE] = 0;
  pos[BLACK] = 0;
  mat[WHITE] = 0;
  mat[BLACK] = 0;

  pieces = piece[WHITE];
  while (pieces) {
    _BitScanForward64(&sq, pieces);
    pieces ^= one << sq;
    mat[WHITE] += value[board[sq]];
    pos[WHITE] += pcePST[board[sq]][sq];
  }

  pieces = piece[BLACK];
  while (pieces) {
    _BitScanForward64(&sq, pieces);
    pieces ^= one << sq;
    mat[BLACK] += value[board[sq]];
    pos[BLACK] += pcePST[board[sq]][sq];
  }
}

[Carbec]

Hello,

I am developping my own little program, and was wondering how you test/evaluate your elo ?

At now, I use perft (to see it the speed didn't went down), WAC positions, and some positions
from the Silver suite. I also did some matches against several engines I downloaded with Arena.
Games were 5 min/game. And its very slow.

Im not sure i use the right method. Could you give me some advice ?

Thanks a lot

Philippe

[algerbrex]

Hello,

I am developping my own little program, and was wondering how you test/evaluate your elo ?

At now, I use perft (to see it the speed didn't went down), WAC positions, and some positions
from the Silver suite. I also did some matches against several engines I downloaded with Arena.
Games were 5 min/game. And its very slow.

Im not sure i use the right method. Could you give me some advice ?

Thanks a lot

Philippe

Hi Philippe, I'm not the original author, but hopefully I can still help you out.

There are a couple of different ways to concretely test Elo gains. A very common method is to play many thousands of games at a short time control (like 10+0.1s or 60+0.6s) using a program like Cutechess, which would then calculate the Elo gain/loss from the WDL results.

Here's an example.

Let's say I have two versions of my chess engine Blunder. Blunder 7.5.0 and Blunder 7.5.0. Let's assume Blunder 7.6.0 is the newer version of Blunder that I think I've added strength gaining features to. To test this belief, I can use the following Cutechess command

Code: Select all

cutechess-cli.exe ^
-srand %RANDOM% ^
-engine cmd=blunder-7.6.0.exe ^
-engine cmd=blunder-7.5.0.exe ^
-openings file=C:%HOMEPATH%\2moves_v2a.pgn format=pgn order=random ^
-each tc=inf/10+0.1 option.Hash=16 proto=uci ^
-games 2 -rounds 2000 -repeat 2 ^
-concurrency 8 ^
-ratinginterval 50 ^
-tournament gauntlet

Which takes the newest version (Blunder 7.6.0) and pits it against the older version (Blunder 7.5.0) in a 4000 game match, with a variety of random openings used to ensure game diversity. This command would be a very basic way to test the Elo difference between two versions of your engine. After the match is completed, you'll get output looking something like this:

Code: Select all

Score of Blunder 7.6.1 vs Blunder 7.6.0: 1947 - 1779 - 2474  [0.514] 6200
...      Blunder 7.6.1 playing White: 1040 - 822 - 1238  [0.535] 3100
...      Blunder 7.6.1 playing Black: 907 - 957 - 1236  [0.492] 3100
...      White vs Black: 1997 - 1729 - 2474  [0.522] 6200
Elo difference: 9.4 +/- 6.7, LOS: 99.7 %, DrawRatio: 39.9 %
SPRT: llr 2.19 (74.5%), lbound -2.94, ubound 2.94
Finished match

A couple of things to note.

First, instead of running a fixed number of games, many people use SPRT testing instead, which is a way to end the match earlier if Cutechess establishes that one instance of an engine is stronger than another with a certain level of confidence.

Second, self-play will often give inflated results in comparison to playing your engine against other opponents. This is because you're only testing your engine against itself, which can bias things since you're specifically exploiting a feature you believe an old version of your engine doesn't have. So many people will either exclusive test against other opponents, or use some combination of self-play testing and testing against other opponents.

See here and here for more details on the above: forum3/viewtopic.php?t=76225, https://rustic-chess.org/progress/sprt_testing.html

(Small disclaimer for the second link: The website author, Marcel Vanthoor has explained elsewhere that he's been contacted and informed that the information on his page isn't exactly accurate. Taking note of this, I still believe it's a good resource for a beginner to get a simple idea of the goals and usefulness of SPRT testing)

P.S: In the future, if you have anymore questions, feel free to start your own thread, since it's usually best to create your own dedicated topic since this thread is specifically meant for discussing progress on Bricabrac.

[Mike Sherwin]

Hello,

I am developping my own little program, and was wondering how you test/evaluate your elo ?

At now, I use perft (to see it the speed didn't went down), WAC positions, and some positions
from the Silver suite. I also did some matches against several engines I downloaded with Arena.
Games were 5 min/game. And its very slow.

Im not sure i use the right method. Could you give me some advice ?

Thanks a lot

Philippe

Bric is not far enough along to start testing against a multitude of engines and playing thousands of games. Right now I'm just using what I call the gatekeeper method. On the Mount Olympus of chess TSCP181 is the first gatekeeper. Faile1.4 is the second. Etc. Once Bric can score 80% or better against TSCP then Bric will move on to Faile.

[mvanthoor]

Bric is not far enough along to start testing against a multitude of engines and playing thousands of games. Right now I'm just using what I call the gatekeeper method. On the Mount Olympus of chess TSCP181 is the first gatekeeper. Faile1.4 is the second. Etc. Once Bric can score 80% or better against TSCP then Bric will move on to Faile.

I've also considered TSCP to be the first engine to beat. Many other people seem to.

Why is that? To be honest, TSCP is not a sepcial engine anymore todaty, EXCEPT for the fact that it basically acquires all of its strength through pawn trickery. TSCP is not fast, and my first public version of Rustic (Alpha 1) just fell short by 50 Elo against TSCP 1.81, reaching its strength by brute speed. Rustic Alpha 2, which did nothing more than add a transposition table and TT-move sorting, exceeded TSCP's playing strength by about 100 Elo.

Therefore TSCP is not hard to beat even with a very basic engine + TT. (If I had implemented Previous PV Move Sorting, I probably wouldn't even have needed a TT, because lots of playing strength in an early engine comes from this enhanced Previous PV Move Sorting or TT-move sorting, not from the TT cuts themselves.)

For me, the second "gatekeeper" would be Vice; not because it is a super-special engine with regard to programming, but because it (or more correctly, it's legendary video series made by its author) re-taught me many chess programming concepts. I knew most of those already before I started, but my knowledge was sketchy. Those video's filled in the gaps in that basic knowledge. Vice is about 300 points stronger than TSCP, which is a nice step-up.

My current development version of Rustic is about +120 against Vice; I have not yet determined my next gatekeeper. I'm just working up the ladder now.

For me, my "final" gatekeeper is Fritz 11, because I'm a fan of the Frans Morsch Fritz versions, and Fritz 11 has been my main chess engine and GUI for a LONG time. Fritz 11 has a CCRL-rating of 2890. When I exceed that rating and can beat Fritz in a 100 game match using my own engine, I feel I'm "done". I'll probably try to push for 3000 Elo if at all possible, and then add multi-cpu search. After that I'll probably not try to develop my engine any further, aside from (maybe) adding NNUE.

[Mike Sherwin]

TSCP is a perfect first gate keeper because it is just at the start of being good. A stronger engine might not allow one to see if simple changes are good or not. Weaker engines, well what's the point? Another nearby engine might work just as well but TSCP to me is a known quantity.

[mvanthoor]

TSCP is a perfect first gate keeper because it is just at the start of being good. A stronger engine might not allow one to see if simple changes are good or not. Weaker engines, well what's the point? Another nearby engine might work just as well but TSCP to me is a known quantity.

That is certainly true enough: TSCP is a very well-known engine. The one strength it has is all of its pawn trickery. It will create isolated, doubled and tripled pawns on your side, and use that fact to create one or more passers for itself. Most engines don't see this because they lack the depth. However, as soon as you add a transposition table and TT-move ordering, engines will reach enough depth in the end-game to see what TSCP is up to and then they'll easily become stronger.

I used this fact to actually test the impact of the transposition table. The moment I added the TT and TT-move ordering, my engine became +100 stronger than TSCP, where it had been -50 behind. In short: if you have an engine that only has the very basics (a/b, qsearch, mvv-lva) + TT / TT-move sorting, you should easily be able to defeat TSCP, or something is wrong.