Efficiently Generated Legal moves question

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

pedrojdm2021
Posts: 157
Joined: Fri Apr 30, 2021 7:19 am
Full name: Pedro Duran

Re: Efficiently Generated Legal moves question

Post by pedrojdm2021 »

algerbrex wrote: Thu Jul 08, 2021 10:43 pm
pedrojdm2021 wrote: Thu Jul 08, 2021 6:52 pm Hello, i have been doing some research into my engine's code, and i have found that one of the main causes of the slowness (that the speed actually is nearly 300.000 nps) caused by the move generation:
I would agree with what Marcel said. I'm also in the same position as you and very much still a novice when it comes to chess programming. But one important lesson I've learned from talking with more experienced chess programmers on here and from my own research is that move generation speed isn't nearly as important as you might think it is when trying to create a decently strong engine.

Instead, focus on getting a working move generator, and then focus your efforts on creating a solid search and evaluation phase using alpha-beta pruning with good move ordering (e.g. Most-valuable-victim, Least-valuable-attacker), material count, and piece-square tables. You might be surprised at how much quicker your engine seems during the search phase, since, if you have good pruning, your engine has to search only a fraction of the full move tree to get the same quality results as searching the whole tree.

For example, in one of my previous iterations of Blunder (my engine), when I went from pure negamax, to adding alpha-beta pruning and MvvLva move ordering, my engine went from searching ~4M nodes in the Kikipete position to only searching ~4k, a fraction of the move tree. And it still found the best move.

So my advice to you as someone who's in a similar position is to focus right now on getting a working move generator. And really make sure it's working. If you google "perft test positions" or "perft test suite", you'll find different collections of positions to test your engine with. Once you finish your move generator, write a little testing script to go through each position and make sure you get the correct numbers at each depth.

Or if you'd like, I'd also be happy to post the testing suite I've been using. It's an amalgamation of tests I've found myself online and it seemed to work well and rooting out any nasty bugs I had left in the move generator.
pedrojdm2021 wrote: Thu Jul 08, 2021 6:52 pm my question is, any of you have implemented the "pin rays" ? if yes how did you do that? i have some ideas but i don't have a very clear solution in mind, thank you all!
I'm currently in the process of re-writing Blunder since I wasn't happy with the code base, and because I realized I needed a better plan for developing Blunder. But anyway, the way I'm detecting and finding pin rays in this version is by having a 64x64 array. The array has a pre-calculated bitboard containing the line between any two squares that are aligned. Here's the relevant code:

Code: Select all

func sqOnBoard(sq int) bool {
	return sq >= 0 && sq <= 63
}

func distance(sq1, sq2 int) int {
	return max(abs(rankOf(sq2)-rankOf(sq1)), abs(fileOf(sq2)-fileOf(sq1)))
}

func pseduoSlidingAttacks(deltas []int, square int) (attacks Bitboard) {
	for _, delta := range deltas {
		for to := square + delta; sqOnBoard(to) &&
			distance(to, to-delta) == 1; to += delta {
			setBit(&attacks, to)
		}
	}
	return attacks
}

func lineBetween(deltas []int, sq1, sq2 int) (line Bitboard) {
	for _, delta := range deltas {
		for to := sq1 + delta; sqOnBoard(to) &&
			distance(to, to-delta) == 1; to += delta {
			setBit(&line, to)
			if to == sq2 {
				break
			}
		}
		if hasBitSet(line, sq2) {
			break
		} else {
			line = 0
		}
	}
	return line
}

var LinesBewteen [64][64]Bitboard
var Directions []int = []int{North, South, East, West, NorthEast, SouthEast, NorthWest, SouthWest}

func init() {
	for from := 0; from < 64; from++ {
		attacks := pseduoSlidingAttacks(Directions, from)
		for attacks != 0 {
			to, _ := popLSB(&attacks)
			LinesBewteen[from][to] = lineBetween(Directions, from, to)
		}
	}
}
And here's how I use it to detect pinned pieces:

Code: Select all

func genPinnedPieces(board *Board) (pinnedBB uint64) {
	enemyBB := board.Sides[board.ColorToMove^1]
	enemyBishops := board.Pieces[board.ColorToMove^1][Bishop]
	enemyRooks := board.Pieces[board.ColorToMove^1][Rook]
	enemyQueens := board.Pieces[board.ColorToMove^1][Queen]

	usBB := board.Sides[board.ColorToMove]
	kingBB := board.Pieces[board.ColorToMove][King]

	pinnersBB := (genIntercardianlMovesBB(kingBB, enemyBB)&(enemyBishops|enemyQueens) |
		genCardianlMovesBB(kingBB, enemyBB)&(enemyRooks|enemyQueens))
	kingPos := getLSBPos(kingBB)
	for pinnersBB != 0 {
		pinnerPos, _ := popLSB(&pinnersBB)
		pinnedBB |= LinesBewteen[kingPos][pinnerPos] & usBB
	}
	return pinnedBB
}
The nice thing about this approach is that both of the arrays used can be pre-computed before the engine starts, so the pin ray doesn't need to be generated on the fly.

I can't say this approach is original to me though. I got the idea from reading through the source code of Stockfish and hearing some members on here talk about similar ideas.
Hello, thank you for your response, and yeah i know that a proper pruning is the best.

Some things that i actually have:
- I did the perft test, they all pointed out correct results.

- i am not using pure negamax with alpha-beta, i am using a negamax version with this set of features:
> Principal variation search
> Quiescense search
> Null Move pruning
> Late move reductions
> Transposition tables
> Futility pruning

the number of nodes searched are between 1 to 2 millons depending on the position, and even with 1millon, the application searches in 3.60 seconds 1.4 millons of positions :/ and it is clearly a slowness of: move generator or search
pedrojdm2021
Posts: 157
Joined: Fri Apr 30, 2021 7:19 am
Full name: Pedro Duran

Re: Efficiently Generated Legal moves question

Post by pedrojdm2021 »

mvanthoor wrote: Thu Jul 08, 2021 8:22 pm
pedrojdm2021 wrote: Thu Jul 08, 2021 6:52 pm Hello, i have been doing some research into my engine's code, and i have found that one of the main causes of the slowness (that the speed actually is nearly 300.000 nps) caused by the move generation...
This should not be the cause of a slow engine. Rustic generates and executes its moves in exactly the same way, and on my 5 year old i7-6700K, it manages 5.000.000 nodes per second. (With just a tapered evaluation). Perft speed is 32 million leaves/second (without hash table and bulk counting). Rustic keeps both sets of PST's and the Zobrist key incrementally. This slows down playing the moves because the PST values are changed at that point. This negatively impacts perft. However, because only two values are changed when moving a piece (subtract when picking it up, add when putting it down, and maybe a third when capturing a piece), this speeds up the evaluation by a lot, because you don't have to re-add up to 32 values there.
my question is, any of you have implemented the "pin rays" ? if yes how did you do that? i have some ideas but i don't have a very clear solution in mind, thank you all!
It's on my todo list, but because Rustic already runs fast enough for now, it's way down the list. The move generator is only a small part of the engine's speed. The evaluation function has much more impact. (If you generate 35 moves, you'll only run the move generator once, but when evaluating, you'll run the evaluation up to 35 times.)
My perft is only about 8,000,000 leaves per second, it calculates around 120,000,000 leaves in about 30 seconds in an AMD FX 8350 CPU, so i think one of the main causes are the move generator, visual studio points to the "in_check" function to be the cause, i do that with the "Square Attacked By" technique: https://www.chessprogramming.org/Square_Attacked_By :

Code: Select all

[MethodImpl(MethodImplOptions.AggressiveInlining)]
        public bool IsSquareAttacked(byte _square, byte _currentPlayer) 
        {
           
            // Attacked by bishops
            if((ChessData.GetBishopMoves(_square, totalOccupancy) & bitboards[_currentPlayer ^ 1][ChessPiece.Bishop]) > 0 )
                return true;
            // Attacked by rooks    
            if((ChessData.GetRookMoves(_square, totalOccupancy) & bitboards[_currentPlayer ^ 1][ChessPiece.Rook]) > 0)
                return true; 
            // Attacked by queens
            if((ChessData.GetQueenMoves(_square, totalOccupancy) & bitboards[_currentPlayer ^ 1][ChessPiece.Queen]) > 0)
                return true;
            // Attacked by knights
            if((ChessData.GetKnightMoves(_square) & bitboards[_currentPlayer ^ 1][ChessPiece.Knight]) > 0 )
                return true;
            // Attacked by kings
            if((ChessData.GetKingMoves(_square) & bitboards[_currentPlayer ^ 1][ChessPiece.King]) > 0)
                return true;
            // Attacked by pawns 
            if ((ChessData.GetPawnAttacks(_square, _currentPlayer) & bitboards[_currentPlayer ^ 1][ChessPiece.Pawn]) > 0)
                return true;

            return false;
        }
User avatar
hgm
Posts: 27808
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Efficiently Generated Legal moves question

Post by hgm »

mvanthoor wrote: Thu Jul 08, 2021 8:22 pmThe evaluation function has much more impact. (If you generate 35 moves, you'll only run the move generator once, but when evaluating, you'll run the evaluation up to 35 times.)
That is an oversimplification. The leaf nodes of the search are QS nodes. You might run the evaluation there, but if that is not above beta, you would run the move generator for generating captures. Even if there are no captures that are worth searching, you would still have to run the move generator to know that.

And the evaluation would not be above beta very often: a serious engine would have futility pruning, and would have already seen from the incremental eval that you would end so high above beta that calculatig the full evaluation would fail high, pruning the node. On the other hand, nodes that are not pruned, but have the incremental eval too much below alpha, can 'prune' the evaluation itself. So not many leaf nodes do a real evaluation. But about half of those that do, and all that do not, do a move generation.
User avatar
MartinBryant
Posts: 69
Joined: Thu Nov 21, 2013 12:37 am
Location: Manchester, UK
Full name: Martin Bryant

Re: Efficiently Generated Legal moves question

Post by MartinBryant »

pedrojdm2021 wrote: Thu Jul 08, 2021 6:52 pm and the number of illegal moves can be really big when playing higher level depth's (depth 9 and up based of pruning techniques)
so that can be very bad when one look for speed...
You didn't state if you have a separate function for generating moves when you are in check?
When you're not in check, most pseudo legal moves will be fully legal.
When you're in check, most pseudo legal moves will be illegal.
A separate function to generate moves out of check can save a lot of validating of pseudo legal moves. It doesn't even have to generate 100% legal moves as there are many shortcuts to massively reduce the number of pseudo legal moves you generate.
pedrojdm2021 wrote: Thu Jul 08, 2021 6:52 pm I found this interesting article on internet:
https://peterellisjones.com/posts/gener ... ficiently/

my question is, any of you have implemented the "pin rays" ? if yes how did you do that? i have some ideas but i don't have a very clear solution in mind, thank you all!
I've seen that article and it was very useful to me when I switched to fully legal move generation.
The following functions generate the pinners bitboards...

Code: Select all

UINT64 RankFilePinnersBB(int kingSquare, int sideToMove)
{
	UINT64 occupiedBB = PiecesBB[0][AllPieces] | PiecesBB[1][AllPieces];
	UINT64 attacksBB = RookAttacksBB(kingSquare, occupiedBB);
	UINT64 potentiallyPinnedBB = attacksBB & PiecesBB[sideToMove][AllPieces]; // Find friendly pieces that may be pinned
	return (attacksBB ^ RookAttacksBB(kingSquare, occupiedBB ^ potentiallyPinnedBB)) & (PiecesBB[sideToMove ^ 1][Rook] | PiecesBB[sideToMove ^ 1][Queen]);
}

UINT64 DiagonalPinnersBB(int kingSquare, int sideToMove)
{
	UINT64 occupiedBB = PiecesBB[0][AllPieces] | PiecesBB[1][AllPieces];
	UINT64 attacksBB = BishopAttacksBB(kingSquare, occupiedBB);
	UINT64 potentiallyPinnedBB = attacksBB & PiecesBB[sideToMove][AllPieces]; // Find friendly pieces that may be pinned
	return (attacksBB ^ BishopAttacksBB(kingSquare, occupiedBB ^ potentiallyPinnedBB)) & (PiecesBB[sideToMove ^ 1][Bishop] | PiecesBB[sideToMove ^ 1][Queen]);
}
You can then use the 'pinners' bitboards to generate your 'pinned' bitboards which you then use in your move generation to just generate fully legal moves.
Ras
Posts: 2488
Joined: Tue Aug 30, 2016 8:19 pm
Full name: Rasmus Althoff

Re: Efficiently Generated Legal moves question

Post by Ras »

pedrojdm2021 wrote: Fri Jul 09, 2021 11:13 amMy perft is only about 8,000,000 leaves per second
On my hardware, my engine has roughly 20M NPS for perft in the initial position, and 1.8M NPS for regular search. The move generator logic is the same as yours, with make/unmake, and the "in check" detection is without bitboards. Instead, with dedicated offset checks or even loops for sliders.

If you get from 8M in perft down to 300k in search, I don't think it's the move generation. Did you profile your code to see where the time really gets spent? With a similar perft-to-search ratio as I have, your search should be around 700k NPS, not 300k.
Rasmus Althoff
https://www.ct800.net
pedrojdm2021
Posts: 157
Joined: Fri Apr 30, 2021 7:19 am
Full name: Pedro Duran

Re: Efficiently Generated Legal moves question

Post by pedrojdm2021 »

Ras wrote: Fri Jul 09, 2021 9:26 pm
pedrojdm2021 wrote: Fri Jul 09, 2021 11:13 amMy perft is only about 8,000,000 leaves per second
On my hardware, my engine has roughly 20M NPS for perft in the initial position, and 1.8M NPS for regular search. The move generator logic is the same as yours, with make/unmake, and the "in check" detection is without bitboards. Instead, with dedicated offset checks or even loops for sliders.

If you get from 8M in perft down to 300k in search, I don't think it's the move generation. Did you profile your code to see where the time really gets spent? With a similar perft-to-search ratio as I have, your search should be around 700k NPS, not 300k.
sure i did profiling, with the following results:

Image

My evaluation implementation is this:

Code: Select all

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public int Evaluate()
        {
            mg_eval[ChessPiece.White] = 0;
            mg_eval[ChessPiece.Black] = 0;
            eg_eval[ChessPiece.White] = 0; 
            eg_eval[ChessPiece.Black] = 0;
            int gamePhase = 0;
            ulong piece_bitboard = 0ul;
            int bitCount_dummy = 0;
            int extra_score = 0;

            //byte square = 0;
            byte square_side = 0,player = 0, piece = 0, square = 0;

            for(player = 0; player <= 1; ++player)
            {
                for (piece = 1; piece <= 6; ++piece)
                {
                    piece_bitboard = board.bitboards[player][piece];
                    while(piece_bitboard > 0)
                    {
                        // update game-phase state
                        gamePhase += GAMEPHASE_WEIGHTS[piece];

                        square = BitUtility.GetLS1BIndex(piece_bitboard);
                        BitUtility.RemoveLS1B(ref piece_bitboard);
                        square_side = AISettings.square_table[player][square];

                        // Evaluate material
                        mg_eval[player] += AISettings.mg_material_score[piece];
                        eg_eval[player] += AISettings.eg_material_score[piece];

                        // Evaluate piece positional scores
                        switch(piece)
                        {
                            case ChessPiece.Pawn:
                            
                            extra_score = 0;

                            // ---------- [Pawn structure] -------------

                            // Double pawns penalty
                            
                            // Get the double pawns bit count
                            bitCount_dummy = BitUtility.CountBits(board.bitboards[player][piece] & ChessData.File_masks[square]);
                            // Verify existence of double pawns and apply penalty
                            if (bitCount_dummy > 0)
                            {
                                mg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_opening;
                                eg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_ending;
                            }

                            // Isolated pawns penalty

                            // verify existence of isolated pawns and apply penalty
                            if ((board.bitboards[player][piece] & ChessData.Isolated_masks[square]) == 0)
                            {
                                mg_eval[player] += AIData.Isolated_pawn_penalty_opening;
                                eg_eval[player] += AIData.Isolated_pawn_penalty_ending;
                            }
                            
                            // Passed pawns bonus

                            // Verify existence of passed pawns and apply bonus
                            if ((board.bitboards[player ^ 1][piece] & ChessData.Passed_masks[player][square]) == 0)
                            {
                                extra_score += AIData.PassedPawnBonus[ChessData.RanksIndices[square_side]];
                            }

                            // ---------------------------------------

                            // Pawn square tables
                            mg_eval[player] += AISettings.mg_pawn_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_pawn_table[square_side] + extra_score;

                            break;

                            case ChessPiece.Bishop:

                            // ---------- [Bishop Mobility] ------------

                            // Get mobility count
                            bitCount_dummy = BitUtility.CountBits(ChessData.GetBishopMoves(square , 
                                board.totalOccupancy));
                            
                            // apply mobility bonus
                            mg_eval[player] += (bitCount_dummy - 4) * 5;
                            eg_eval[player] += (bitCount_dummy - 4) * 5;

                            // ------------------------------------------

                            // Bishop square tables
                            mg_eval[player] += AISettings.mg_bishop_table[square_side];
                            eg_eval[player] += AISettings.eg_bishop_table[square_side];

                            break;

                            case ChessPiece.Rook:

                            // ---------- [Rook Structure] ------------------

                            extra_score = 0;

                            // Semi open file scoring
                            if ((board.bitboards[player][ChessPiece.Pawn] & ChessData.File_masks[square]) == 0)
                                extra_score += AIData.Semi_open_file_score;

                            // Open file scoreing
                            if (((board.bitboards[player][ChessPiece.Pawn] | board.bitboards[player ^ 1][ChessPiece.Pawn]) 
                                & ChessData.File_masks[square]) == 0)
                                extra_score += AIData.Open_file_score;
                            
                            // ----------------------------------------------
                            
                            // Rook square tables
                            mg_eval[player] += AISettings.mg_rook_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_rook_table[square_side] + extra_score;

                            break;

                            case ChessPiece.Knight:

                            // Knight square tables
                            mg_eval[player] += AISettings.mg_knight_table[square_side];
                            eg_eval[player] += AISettings.eg_knight_table[square_side];

                            break;

                            case ChessPiece.Queen:

                            // ------------ [ Queen mobility ] ------------

                            // Get mobility count
                            bitCount_dummy = BitUtility.CountBits(ChessData.GetQueenMoves(square , 
                                board.totalOccupancy));
                                                        
                            // Apply mobility bonus
                            mg_eval[player] += (bitCount_dummy - 9);
                            eg_eval[player] += (bitCount_dummy - 9) * 2;

                            // ----------------------------------------------

                            // Queen square tables
                            mg_eval[player] += AISettings.mg_queen_table[square_side];
                            eg_eval[player] += AISettings.eg_queen_table[square_side];

                            break;

                            case ChessPiece.King:

                            extra_score = 0;

                            // ------------ [ King Structure ] ------------

                            // Semi open file scoring (penalty)
                            if ((board.bitboards[player][ChessPiece.Pawn] & ChessData.File_masks[square]) == 0)
                                extra_score -= AIData.Semi_open_file_score;

                            // Open file scoreing (penalty)
                            if (((board.bitboards[player][ChessPiece.Pawn] | board.bitboards[player ^ 1][ChessPiece.Pawn]) 
                                & ChessData.File_masks[square]) == 0)
                                extra_score -= AIData.Open_file_score;
                            
                            // ------------ [ King Safety ] ----------------

                            extra_score += BitUtility.CountBits(ChessData.GetKingMoves(square) & 
                                board.occupancies[player]) * AIData.King_shield_bonus;

                            // ---------------------------------------------

                            // King square tables
                            mg_eval[player] += AISettings.mg_king_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_king_table[square_side] + extra_score;

                            break;
                        }

                    } 
                }
            }

            if (gamePhase > 24) gamePhase = 24;

            int mg_score = (mg_eval[ChessPiece.White] - mg_eval[ChessPiece.Black]) * GetPlayerSide(board.sideToMove);
            int eg_score = (eg_eval[ChessPiece.White] - eg_eval[ChessPiece.Black]) * GetPlayerSide(board.sideToMove);

            int eg_phase = 24 - gamePhase;
            
            return (mg_score * gamePhase + eg_score * eg_phase) / 24;
        }
but in visual studio it does not point to any relevant "hot point". In visual studio even doing an bitwise and operation (&) and performing a count bits lookup it represents a 5% in computation time? it was not supposed to be a really fast operation?

Image
User avatar
algerbrex
Posts: 596
Joined: Sun May 30, 2021 5:03 am
Location: United States
Full name: Christian Dean

Re: Efficiently Generated Legal moves question

Post by algerbrex »

mvanthoor wrote: Fri Jul 09, 2021 9:56 am Rust isn't an easy language to learn if you're coming from garbage collected languages. C seems easier to start with, but it's much harder to make memory-related mistakes. C is easy to learn, but hard to write correctly. Rust is hard to learn, but easy to write correctly. As soon as you do something that's not memory-safe or not thread-safe, the compiler will slap you over the head with it.
Yeah, I have some experience programming in C, like making a basic compiler, so I know how much of a pain writing good C code is. Rust definitely, sounds interesting, but of course as you mention that austere-ness of the compiler would probably take some getting used too.
User avatar
algerbrex
Posts: 596
Joined: Sun May 30, 2021 5:03 am
Location: United States
Full name: Christian Dean

Re: Efficiently Generated Legal moves question

Post by algerbrex »

pedrojdm2021 wrote: Fri Jul 09, 2021 10:16 am
algerbrex wrote: Thu Jul 08, 2021 10:43 pm
pedrojdm2021 wrote: Thu Jul 08, 2021 6:52 pm Hello, i have been doing some research into my engine's code, and i have found that one of the main causes of the slowness (that the speed actually is nearly 300.000 nps) caused by the move generation:
I would agree with what Marcel said. I'm also in the same position as you and very much still a novice when it comes to chess programming. But one important lesson I've learned from talking with more experienced chess programmers on here and from my own research is that move generation speed isn't nearly as important as you might think it is when trying to create a decently strong engine.

Instead, focus on getting a working move generator, and then focus your efforts on creating a solid search and evaluation phase using alpha-beta pruning with good move ordering (e.g. Most-valuable-victim, Least-valuable-attacker), material count, and piece-square tables. You might be surprised at how much quicker your engine seems during the search phase, since, if you have good pruning, your engine has to search only a fraction of the full move tree to get the same quality results as searching the whole tree.

For example, in one of my previous iterations of Blunder (my engine), when I went from pure negamax, to adding alpha-beta pruning and MvvLva move ordering, my engine went from searching ~4M nodes in the Kikipete position to only searching ~4k, a fraction of the move tree. And it still found the best move.

So my advice to you as someone who's in a similar position is to focus right now on getting a working move generator. And really make sure it's working. If you google "perft test positions" or "perft test suite", you'll find different collections of positions to test your engine with. Once you finish your move generator, write a little testing script to go through each position and make sure you get the correct numbers at each depth.

Or if you'd like, I'd also be happy to post the testing suite I've been using. It's an amalgamation of tests I've found myself online and it seemed to work well and rooting out any nasty bugs I had left in the move generator.
pedrojdm2021 wrote: Thu Jul 08, 2021 6:52 pm my question is, any of you have implemented the "pin rays" ? if yes how did you do that? i have some ideas but i don't have a very clear solution in mind, thank you all!
I'm currently in the process of re-writing Blunder since I wasn't happy with the code base, and because I realized I needed a better plan for developing Blunder. But anyway, the way I'm detecting and finding pin rays in this version is by having a 64x64 array. The array has a pre-calculated bitboard containing the line between any two squares that are aligned. Here's the relevant code:

Code: Select all

func sqOnBoard(sq int) bool {
	return sq >= 0 && sq <= 63
}

func distance(sq1, sq2 int) int {
	return max(abs(rankOf(sq2)-rankOf(sq1)), abs(fileOf(sq2)-fileOf(sq1)))
}

func pseduoSlidingAttacks(deltas []int, square int) (attacks Bitboard) {
	for _, delta := range deltas {
		for to := square + delta; sqOnBoard(to) &&
			distance(to, to-delta) == 1; to += delta {
			setBit(&attacks, to)
		}
	}
	return attacks
}

func lineBetween(deltas []int, sq1, sq2 int) (line Bitboard) {
	for _, delta := range deltas {
		for to := sq1 + delta; sqOnBoard(to) &&
			distance(to, to-delta) == 1; to += delta {
			setBit(&line, to)
			if to == sq2 {
				break
			}
		}
		if hasBitSet(line, sq2) {
			break
		} else {
			line = 0
		}
	}
	return line
}

var LinesBewteen [64][64]Bitboard
var Directions []int = []int{North, South, East, West, NorthEast, SouthEast, NorthWest, SouthWest}

func init() {
	for from := 0; from < 64; from++ {
		attacks := pseduoSlidingAttacks(Directions, from)
		for attacks != 0 {
			to, _ := popLSB(&attacks)
			LinesBewteen[from][to] = lineBetween(Directions, from, to)
		}
	}
}
And here's how I use it to detect pinned pieces:

Code: Select all

func genPinnedPieces(board *Board) (pinnedBB uint64) {
	enemyBB := board.Sides[board.ColorToMove^1]
	enemyBishops := board.Pieces[board.ColorToMove^1][Bishop]
	enemyRooks := board.Pieces[board.ColorToMove^1][Rook]
	enemyQueens := board.Pieces[board.ColorToMove^1][Queen]

	usBB := board.Sides[board.ColorToMove]
	kingBB := board.Pieces[board.ColorToMove][King]

	pinnersBB := (genIntercardianlMovesBB(kingBB, enemyBB)&(enemyBishops|enemyQueens) |
		genCardianlMovesBB(kingBB, enemyBB)&(enemyRooks|enemyQueens))
	kingPos := getLSBPos(kingBB)
	for pinnersBB != 0 {
		pinnerPos, _ := popLSB(&pinnersBB)
		pinnedBB |= LinesBewteen[kingPos][pinnerPos] & usBB
	}
	return pinnedBB
}
The nice thing about this approach is that both of the arrays used can be pre-computed before the engine starts, so the pin ray doesn't need to be generated on the fly.

I can't say this approach is original to me though. I got the idea from reading through the source code of Stockfish and hearing some members on here talk about similar ideas.
Hello, thank you for your response, and yeah i know that a proper pruning is the best.

Some things that i actually have:
- I did the perft test, they all pointed out correct results.

- i am not using pure negamax with alpha-beta, i am using a negamax version with this set of features:
> Principal variation search
> Quiescense search
> Null Move pruning
> Late move reductions
> Transposition tables
> Futility pruning

the number of nodes searched are between 1 to 2 millons depending on the position, and even with 1millon, the application searches in 3.60 seconds 1.4 millons of positions :/ and it is clearly a slowness of: move generator or search
Well in that case, I would look at the move generator again. You said you've started profiling, so that's good. Make sure you're being smart about not doing expensive operations during move generation. If there's anything that can be precomputed (e.g. pin rays), do that before move generation.
Pio
Posts: 334
Joined: Sat Feb 25, 2012 10:42 pm
Location: Stockholm

Re: Efficiently Generated Legal moves question

Post by Pio »

pedrojdm2021 wrote: Fri Jul 09, 2021 10:51 pm
Ras wrote: Fri Jul 09, 2021 9:26 pm
pedrojdm2021 wrote: Fri Jul 09, 2021 11:13 amMy perft is only about 8,000,000 leaves per second
On my hardware, my engine has roughly 20M NPS for perft in the initial position, and 1.8M NPS for regular search. The move generator logic is the same as yours, with make/unmake, and the "in check" detection is without bitboards. Instead, with dedicated offset checks or even loops for sliders.

If you get from 8M in perft down to 300k in search, I don't think it's the move generation. Did you profile your code to see where the time really gets spent? With a similar perft-to-search ratio as I have, your search should be around 700k NPS, not 300k.
sure i did profiling, with the following results:

Image

My evaluation implementation is this:

Code: Select all

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public int Evaluate()
        {
            mg_eval[ChessPiece.White] = 0;
            mg_eval[ChessPiece.Black] = 0;
            eg_eval[ChessPiece.White] = 0; 
            eg_eval[ChessPiece.Black] = 0;
            int gamePhase = 0;
            ulong piece_bitboard = 0ul;
            int bitCount_dummy = 0;
            int extra_score = 0;

            //byte square = 0;
            byte square_side = 0,player = 0, piece = 0, square = 0;

            for(player = 0; player <= 1; ++player)
            {
                for (piece = 1; piece <= 6; ++piece)
                {
                    piece_bitboard = board.bitboards[player][piece];
                    while(piece_bitboard > 0)
                    {
                        // update game-phase state
                        gamePhase += GAMEPHASE_WEIGHTS[piece];

                        square = BitUtility.GetLS1BIndex(piece_bitboard);
                        BitUtility.RemoveLS1B(ref piece_bitboard);
                        square_side = AISettings.square_table[player][square];

                        // Evaluate material
                        mg_eval[player] += AISettings.mg_material_score[piece];
                        eg_eval[player] += AISettings.eg_material_score[piece];

                        // Evaluate piece positional scores
                        switch(piece)
                        {
                            case ChessPiece.Pawn:
                            
                            extra_score = 0;

                            // ---------- [Pawn structure] -------------

                            // Double pawns penalty
                            
                            // Get the double pawns bit count
                            bitCount_dummy = BitUtility.CountBits(board.bitboards[player][piece] & ChessData.File_masks[square]);
                            // Verify existence of double pawns and apply penalty
                            if (bitCount_dummy > 0)
                            {
                                mg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_opening;
                                eg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_ending;
                            }

                            // Isolated pawns penalty

                            // verify existence of isolated pawns and apply penalty
                            if ((board.bitboards[player][piece] & ChessData.Isolated_masks[square]) == 0)
                            {
                                mg_eval[player] += AIData.Isolated_pawn_penalty_opening;
                                eg_eval[player] += AIData.Isolated_pawn_penalty_ending;
                            }
                            
                            // Passed pawns bonus

                            // Verify existence of passed pawns and apply bonus
                            if ((board.bitboards[player ^ 1][piece] & ChessData.Passed_masks[player][square]) == 0)
                            {
                                extra_score += AIData.PassedPawnBonus[ChessData.RanksIndices[square_side]];
                            }

                            // ---------------------------------------

                            // Pawn square tables
                            mg_eval[player] += AISettings.mg_pawn_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_pawn_table[square_side] + extra_score;

                            break;

                            case ChessPiece.Bishop:

                            // ---------- [Bishop Mobility] ------------

                            // Get mobility count
                            bitCount_dummy = BitUtility.CountBits(ChessData.GetBishopMoves(square , 
                                board.totalOccupancy));
                            
                            // apply mobility bonus
                            mg_eval[player] += (bitCount_dummy - 4) * 5;
                            eg_eval[player] += (bitCount_dummy - 4) * 5;

                            // ------------------------------------------

                            // Bishop square tables
                            mg_eval[player] += AISettings.mg_bishop_table[square_side];
                            eg_eval[player] += AISettings.eg_bishop_table[square_side];

                            break;

                            case ChessPiece.Rook:

                            // ---------- [Rook Structure] ------------------

                            extra_score = 0;

                            // Semi open file scoring
                            if ((board.bitboards[player][ChessPiece.Pawn] & ChessData.File_masks[square]) == 0)
                                extra_score += AIData.Semi_open_file_score;

                            // Open file scoreing
                            if (((board.bitboards[player][ChessPiece.Pawn] | board.bitboards[player ^ 1][ChessPiece.Pawn]) 
                                & ChessData.File_masks[square]) == 0)
                                extra_score += AIData.Open_file_score;
                            
                            // ----------------------------------------------
                            
                            // Rook square tables
                            mg_eval[player] += AISettings.mg_rook_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_rook_table[square_side] + extra_score;

                            break;

                            case ChessPiece.Knight:

                            // Knight square tables
                            mg_eval[player] += AISettings.mg_knight_table[square_side];
                            eg_eval[player] += AISettings.eg_knight_table[square_side];

                            break;

                            case ChessPiece.Queen:

                            // ------------ [ Queen mobility ] ------------

                            // Get mobility count
                            bitCount_dummy = BitUtility.CountBits(ChessData.GetQueenMoves(square , 
                                board.totalOccupancy));
                                                        
                            // Apply mobility bonus
                            mg_eval[player] += (bitCount_dummy - 9);
                            eg_eval[player] += (bitCount_dummy - 9) * 2;

                            // ----------------------------------------------

                            // Queen square tables
                            mg_eval[player] += AISettings.mg_queen_table[square_side];
                            eg_eval[player] += AISettings.eg_queen_table[square_side];

                            break;

                            case ChessPiece.King:

                            extra_score = 0;

                            // ------------ [ King Structure ] ------------

                            // Semi open file scoring (penalty)
                            if ((board.bitboards[player][ChessPiece.Pawn] & ChessData.File_masks[square]) == 0)
                                extra_score -= AIData.Semi_open_file_score;

                            // Open file scoreing (penalty)
                            if (((board.bitboards[player][ChessPiece.Pawn] | board.bitboards[player ^ 1][ChessPiece.Pawn]) 
                                & ChessData.File_masks[square]) == 0)
                                extra_score -= AIData.Open_file_score;
                            
                            // ------------ [ King Safety ] ----------------

                            extra_score += BitUtility.CountBits(ChessData.GetKingMoves(square) & 
                                board.occupancies[player]) * AIData.King_shield_bonus;

                            // ---------------------------------------------

                            // King square tables
                            mg_eval[player] += AISettings.mg_king_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_king_table[square_side] + extra_score;

                            break;
                        }

                    } 
                }
            }

            if (gamePhase > 24) gamePhase = 24;

            int mg_score = (mg_eval[ChessPiece.White] - mg_eval[ChessPiece.Black]) * GetPlayerSide(board.sideToMove);
            int eg_score = (eg_eval[ChessPiece.White] - eg_eval[ChessPiece.Black]) * GetPlayerSide(board.sideToMove);

            int eg_phase = 24 - gamePhase;
            
            return (mg_score * gamePhase + eg_score * eg_phase) / 24;
        }
but in visual studio it does not point to any relevant "hot point". In visual studio even doing an bitwise and operation (&) and performing a count bits lookup it represents a 5% in computation time? it was not supposed to be a really fast operation?

Image
I think you can speed up the evaluation with some small tricks:

1) You should be able to speed up the two dimensional arrays to one dimensional ones. I guess that is why the profiler said that your bitcount function was slow, not because it was so slow but because you did the two dimensional access within the function call.

2) You can reduce a couple of branches such as

if (bitCount_dummy > 0)
{
mg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_opening;
eg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_ending;
}
Just remove the if case since multiplying with 0 is zero. Test to see which is faster.

3) Remove the array access for example mg_eval[player] and replace it with mg_eval_white and mg_eval_black. Then you can remove unnecessary array accesses. If you want the code tidier I would probably just put a bool in the Evaluate function and written Evaluate(isWhite)-Evaluate(!isWhite)
pedrojdm2021
Posts: 157
Joined: Fri Apr 30, 2021 7:19 am
Full name: Pedro Duran

Re: Efficiently Generated Legal moves question

Post by pedrojdm2021 »

Pio wrote: Sat Jul 10, 2021 1:35 am
pedrojdm2021 wrote: Fri Jul 09, 2021 10:51 pm
Ras wrote: Fri Jul 09, 2021 9:26 pm
pedrojdm2021 wrote: Fri Jul 09, 2021 11:13 amMy perft is only about 8,000,000 leaves per second
On my hardware, my engine has roughly 20M NPS for perft in the initial position, and 1.8M NPS for regular search. The move generator logic is the same as yours, with make/unmake, and the "in check" detection is without bitboards. Instead, with dedicated offset checks or even loops for sliders.

If you get from 8M in perft down to 300k in search, I don't think it's the move generation. Did you profile your code to see where the time really gets spent? With a similar perft-to-search ratio as I have, your search should be around 700k NPS, not 300k.
sure i did profiling, with the following results:

Image

My evaluation implementation is this:

Code: Select all

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public int Evaluate()
        {
            mg_eval[ChessPiece.White] = 0;
            mg_eval[ChessPiece.Black] = 0;
            eg_eval[ChessPiece.White] = 0; 
            eg_eval[ChessPiece.Black] = 0;
            int gamePhase = 0;
            ulong piece_bitboard = 0ul;
            int bitCount_dummy = 0;
            int extra_score = 0;

            //byte square = 0;
            byte square_side = 0,player = 0, piece = 0, square = 0;

            for(player = 0; player <= 1; ++player)
            {
                for (piece = 1; piece <= 6; ++piece)
                {
                    piece_bitboard = board.bitboards[player][piece];
                    while(piece_bitboard > 0)
                    {
                        // update game-phase state
                        gamePhase += GAMEPHASE_WEIGHTS[piece];

                        square = BitUtility.GetLS1BIndex(piece_bitboard);
                        BitUtility.RemoveLS1B(ref piece_bitboard);
                        square_side = AISettings.square_table[player][square];

                        // Evaluate material
                        mg_eval[player] += AISettings.mg_material_score[piece];
                        eg_eval[player] += AISettings.eg_material_score[piece];

                        // Evaluate piece positional scores
                        switch(piece)
                        {
                            case ChessPiece.Pawn:
                            
                            extra_score = 0;

                            // ---------- [Pawn structure] -------------

                            // Double pawns penalty
                            
                            // Get the double pawns bit count
                            bitCount_dummy = BitUtility.CountBits(board.bitboards[player][piece] & ChessData.File_masks[square]);
                            // Verify existence of double pawns and apply penalty
                            if (bitCount_dummy > 0)
                            {
                                mg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_opening;
                                eg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_ending;
                            }

                            // Isolated pawns penalty

                            // verify existence of isolated pawns and apply penalty
                            if ((board.bitboards[player][piece] & ChessData.Isolated_masks[square]) == 0)
                            {
                                mg_eval[player] += AIData.Isolated_pawn_penalty_opening;
                                eg_eval[player] += AIData.Isolated_pawn_penalty_ending;
                            }
                            
                            // Passed pawns bonus

                            // Verify existence of passed pawns and apply bonus
                            if ((board.bitboards[player ^ 1][piece] & ChessData.Passed_masks[player][square]) == 0)
                            {
                                extra_score += AIData.PassedPawnBonus[ChessData.RanksIndices[square_side]];
                            }

                            // ---------------------------------------

                            // Pawn square tables
                            mg_eval[player] += AISettings.mg_pawn_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_pawn_table[square_side] + extra_score;

                            break;

                            case ChessPiece.Bishop:

                            // ---------- [Bishop Mobility] ------------

                            // Get mobility count
                            bitCount_dummy = BitUtility.CountBits(ChessData.GetBishopMoves(square , 
                                board.totalOccupancy));
                            
                            // apply mobility bonus
                            mg_eval[player] += (bitCount_dummy - 4) * 5;
                            eg_eval[player] += (bitCount_dummy - 4) * 5;

                            // ------------------------------------------

                            // Bishop square tables
                            mg_eval[player] += AISettings.mg_bishop_table[square_side];
                            eg_eval[player] += AISettings.eg_bishop_table[square_side];

                            break;

                            case ChessPiece.Rook:

                            // ---------- [Rook Structure] ------------------

                            extra_score = 0;

                            // Semi open file scoring
                            if ((board.bitboards[player][ChessPiece.Pawn] & ChessData.File_masks[square]) == 0)
                                extra_score += AIData.Semi_open_file_score;

                            // Open file scoreing
                            if (((board.bitboards[player][ChessPiece.Pawn] | board.bitboards[player ^ 1][ChessPiece.Pawn]) 
                                & ChessData.File_masks[square]) == 0)
                                extra_score += AIData.Open_file_score;
                            
                            // ----------------------------------------------
                            
                            // Rook square tables
                            mg_eval[player] += AISettings.mg_rook_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_rook_table[square_side] + extra_score;

                            break;

                            case ChessPiece.Knight:

                            // Knight square tables
                            mg_eval[player] += AISettings.mg_knight_table[square_side];
                            eg_eval[player] += AISettings.eg_knight_table[square_side];

                            break;

                            case ChessPiece.Queen:

                            // ------------ [ Queen mobility ] ------------

                            // Get mobility count
                            bitCount_dummy = BitUtility.CountBits(ChessData.GetQueenMoves(square , 
                                board.totalOccupancy));
                                                        
                            // Apply mobility bonus
                            mg_eval[player] += (bitCount_dummy - 9);
                            eg_eval[player] += (bitCount_dummy - 9) * 2;

                            // ----------------------------------------------

                            // Queen square tables
                            mg_eval[player] += AISettings.mg_queen_table[square_side];
                            eg_eval[player] += AISettings.eg_queen_table[square_side];

                            break;

                            case ChessPiece.King:

                            extra_score = 0;

                            // ------------ [ King Structure ] ------------

                            // Semi open file scoring (penalty)
                            if ((board.bitboards[player][ChessPiece.Pawn] & ChessData.File_masks[square]) == 0)
                                extra_score -= AIData.Semi_open_file_score;

                            // Open file scoreing (penalty)
                            if (((board.bitboards[player][ChessPiece.Pawn] | board.bitboards[player ^ 1][ChessPiece.Pawn]) 
                                & ChessData.File_masks[square]) == 0)
                                extra_score -= AIData.Open_file_score;
                            
                            // ------------ [ King Safety ] ----------------

                            extra_score += BitUtility.CountBits(ChessData.GetKingMoves(square) & 
                                board.occupancies[player]) * AIData.King_shield_bonus;

                            // ---------------------------------------------

                            // King square tables
                            mg_eval[player] += AISettings.mg_king_table[square_side] + extra_score;
                            eg_eval[player] += AISettings.eg_king_table[square_side] + extra_score;

                            break;
                        }

                    } 
                }
            }

            if (gamePhase > 24) gamePhase = 24;

            int mg_score = (mg_eval[ChessPiece.White] - mg_eval[ChessPiece.Black]) * GetPlayerSide(board.sideToMove);
            int eg_score = (eg_eval[ChessPiece.White] - eg_eval[ChessPiece.Black]) * GetPlayerSide(board.sideToMove);

            int eg_phase = 24 - gamePhase;
            
            return (mg_score * gamePhase + eg_score * eg_phase) / 24;
        }
but in visual studio it does not point to any relevant "hot point". In visual studio even doing an bitwise and operation (&) and performing a count bits lookup it represents a 5% in computation time? it was not supposed to be a really fast operation?

Image
I think you can speed up the evaluation with some small tricks:

1) You should be able to speed up the two dimensional arrays to one dimensional ones. I guess that is why the profiler said that your bitcount function was slow, not because it was so slow but because you did the two dimensional access within the function call.

2) You can reduce a couple of branches such as

if (bitCount_dummy > 0)
{
mg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_opening;
eg_eval[player] += bitCount_dummy * AIData.Double_pawn_penalty_ending;
}
Just remove the if case since multiplying with 0 is zero. Test to see which is faster.

3) Remove the array access for example mg_eval[player] and replace it with mg_eval_white and mg_eval_black. Then you can remove unnecessary array accesses. If you want the code tidier I would probably just put a bool in the Evaluate function and written Evaluate(isWhite)-Evaluate(!isWhite)
Hey! thank you for the very useful info! i am going to try that, in C# when you have arr[X][Y] is usually an "Array inside array" aka "Jagged arrays" multi-dimensional arrays in C# are: arr[X,Y], but i do agree that a single-dimensional array will be better.

about the branches, yes, i am going to review it.

another question, do you know an efficient way to detect if the king is in check?

i use this technique:
https://www.chessprogramming.org/Square ... ck_by_Side

It works great, it is bug free
but i feel that is very slow

i did a benchmark in kiwipete position for that function call ( IsSquareAttacked(sq,player) ) and it reports nearly 40 cpu ticks for a single call :oops: