My transposition tables make things slower

[JVMerlino]

FEN[226] = "6r1/p1pq1p1p/1p1p1Qnk/3PrR2/2n1P1PP/P1P5/4R3/6K1 w - - 0 1"; // f5h5 e5h5 g4g5 h5g5 h4g5 h6h5 e2h2 h5g4 h2g2 g4h5 f6f3 h5h4 f3g3 h4h5 g2h2 d7h3 g3h3 g6h4 h3h4 h5g6 h4h6

Using the FEN above, I tested the solution with and without the TT. Results as follows:

1) With TT: time to solution: 161.085 seconds; nodes searched = 310,019,642
2) Without TT: time to solution = 1244.05 seconds; nodes searched = 1,556,880,385

The TT table reduced time to solution by 80% and the number of nodes searched by 87%.

My TT table includes 1) hash key 2) success/failure 3) ply. If a collusion occurs, per your suggestion, I just throw ignore the hash key. Also, I have enough space so the keys can't overlap. I use checkmate solutions instead of perft, and have found at least 80% reduction in time and about 92 % reduction in nodes searched. However, the TT gets much more efficient at higher ply levels. The one thing I found is you need larger hash table for deep ply solutions. Without a TT the time to solution on some problems could be months or even years. I can't even imagine running without a TT unless the solution is only a few ply deep.

Are you talking about finding the Mate in 11, or just finding Qh5? Either way, this is not a hard position to solve at all. Even if I turn off EVERYTHING (hash, history, aspiration, SEE, pruning, null move, IIR, etc.), my mediocre Myrddin finds Qh5 in 1.6 seconds at depth 5, although it's a LONG way from finding mate.

If I turn off only hash, it finds Mate in 12 in 19 seconds and Mate in 11 in 67 seconds.

If I enable hash, it finds Mate in 11 in 0.2 seconds.

So there must be several methods of improving the time to find mates that you are not making use of.

[Chessnut1071]

FEN[226] = "6r1/p1pq1p1p/1p1p1Qnk/3PrR2/2n1P1PP/P1P5/4R3/6K1 w - - 0 1"; // f5h5 e5h5 g4g5 h5g5 h4g5 h6h5 e2h2 h5g4 h2g2 g4h5 f6f3 h5h4 f3g3 h4h5 g2h2 d7h3 g3h3 g6h4 h3h4 h5g6 h4h6

Using the FEN above, I tested the solution with and without the TT. Results as follows:

1) With TT: time to solution: 161.085 seconds; nodes searched = 310,019,642
2) Without TT: time to solution = 1244.05 seconds; nodes searched = 1,556,880,385

The TT table reduced time to solution by 80% and the number of nodes searched by 87%.

My TT table includes 1) hash key 2) success/failure 3) ply. If a collusion occurs, per your suggestion, I just throw ignore the hash key. Also, I have enough space so the keys can't overlap. I use checkmate solutions instead of perft, and have found at least 80% reduction in time and about 92 % reduction in nodes searched. However, the TT gets much more efficient at higher ply levels. The one thing I found is you need larger hash table for deep ply solutions. Without a TT the time to solution on some problems could be months or even years. I can't even imagine running without a TT unless the solution is only a few ply deep.

Are you talking about finding the Mate in 11, or just finding Qh5? Either way, this is not a hard position to solve at all. Even if I turn off EVERYTHING (hash, history, aspiration, SEE, pruning, null move, IIR, etc.), my mediocre Myrddin finds Qh5 in 1.6 seconds at depth 5, although it's a LONG way from finding mate.

If I turn off only hash, it finds Mate in 12 in 19 seconds and Mate in 11 in 67 seconds.

If I enable hash, it finds Mate in 11 in 0.2 seconds.

So there must be several methods of improving the time to find mates that you are not making use of.

So there must be several methods of improving the time to find mates that you are not making use of.

Finding Rh5 is easy, I agree. The evaluation function selects Rh5 as best move, based on check and distance to enemy king.
The checkmate is also fairly easy which is why I used it as a quick example.

You are right, my engine is slow, only averaging about 2 million nodes/second compared to some on this board with 10 million and more. So, if you please reference nodes searched, instead of time, it will give me a better idea of the efficiency of my engine given my older chip.

I will say this, when I joined this board about 2 years ago my engine could do 12,000 nodes per second using VBA in Excel. I had to learn C++ and after some advice from hgm, I'm up to 2,000,000 nps on a 9-year old 2,62 GHz, i7, in a PEXT solution. I'm getting there slowly. I think my engine is error free, but now I need speed.

I'm wondering if those fast solutions are because of a better evaluation function or more efficient programming, or both. Reporting the solution in nodes searched tells me more.

[Henk]

[deleted]

[Chessnut1071]

What is a PEXT solution? I am too stupid to see how one can make use of PEXT for move generation. O wait I better create a new thread.

PEXT parallel bit exchange. Some prefer mailbox, some magic numbers. PEXT solved all my issues with collisions.

[Henk]

What is a PEXT solution? I am too stupid to see how one can make use of PEXT for move generation. O wait I better create a new thread.

PEXT parallel bit exchange. Some prefer mailbox, some magic numbers. PEXT solved all my issues with collisions.

Which collisions? Transposition table I suppose? Sorry I am annoyed I tried to use ChatGPT for generating code for move generation using parallel bit extract but only giving solutions that did not work. Some people having too much faith in AI.

[Chessnut1071]

What is a PEXT solution? I am too stupid to see how one can make use of PEXT for move generation. O wait I better create a new thread.

PEXT parallel bit exchange. Some prefer mailbox, some magic numbers. PEXT solved all my issues with collisions.

Which collisions? Transposition table I suppose? Sorry I am annoyed I tried to use ChatGPT for generating code for move generation using parallel bit extract but only giving solutions that did not work. Some people having too much faith in AI.

I was using magic bit boards and PEXT is an alternative. It speed things up for me and it was easy to code. It appears to be working perfectly up to 22-ply, which is where I'm at due to lack of speed. I need to get to 30-ply.

I'm still at a loss on how you can run deep ply searches without a TT. I get at least 88% reduction in search time and 92% reduction in nodes searched. Some on this board seem to get faster results without a TT, probably from evaluation metrics. My evaluation has less significance that the TT, but still important.

So far I haven't seen anybody report a detailed generic evaluation, it there is such a thing.

[JVMerlino]

I'm wondering if those fast solutions are because of a better evaluation function or more efficient programming, or both. Reporting the solution in nodes searched tells me more.

Myrddin announces Mate in 11 at depth 10, needing 392,133 nodes.

The King, that great mate finder, announces Mate in 11 at depth 4, needing only 16,390 nodes.

[Chessnut1071]

I'm wondering if those fast solutions are because of a better evaluation function or more efficient programming, or both. Reporting the solution in nodes searched tells me more.

Myrddin announces Mate in 11 at depth 10, needing 392,133 nodes.

The King, that great mate finder, announces Mate in 11 at depth 4, needing only 16,390 nodes.

That's my point, they probably didn't need a TT because their evaluation was super accurate, especially The King. The question is, how did they do it?

My eval + TT took 310,019,642 nodes, about 10x Myrddin, using a generic mate solution for all plys, and 200x the nodes for the King. I suspect hgm is right, they probably use iterative deepening. I'm going straight for the 22-ply thread without the ID. Right now, I'm exploring ways to reduce the nodes and if I find out how they did it I will report it.

[hgm]

Umm, the way I process the reported numbers Myrddin is 790x faster than your program, in terms of nodes, and The King about 19,000x. Not 10x and 200x.

Note, however, that finding a mate and proving that it is the fastest mate is not the same thing. It might be for you, using a fixed-depth alpha-beta search, but these other programs of course use all kind of selective extensions and reductions. Otherwise it would of course be impossible to see mate-in-10 at 4 ply. It must have searched at least 19 ply on the branch that leads to the checkmate, otherwise it could not know it is there. But that leaves open the possibility that there would be a faster mate in the branches that were considered unpromising, and were only searched to 4 ply.

About this 'generic evaluation'... Fairy-Max virtually does not have any evaluation at all, other than piece values, a weak attraction to the center for P, N, B and K, and a penalty of half a Pawn for each King move in the branch leading up to the leaf where the evaluation is made. But it does have check extension, null move with R=2, and late-move reduction of 1 ply on all non-captures other than the hash move. So I think you are still on the wrong track by assuming this has anything to do with evaluation.

[Chessnut1071]

Umm, the way I process the reported numbers Myrddin is 790x faster than your program, in terms of nodes, and The King about 19,000x. Not 10x and 200x.

Note, however, that finding a mate and proving that it is the fastest mate is not the same thing. It might be for you, using a fixed-depth alpha-beta search, but these other programs of course use all kind of selective extensions and reductions. Otherwise it would of course be impossible to see mate-in-10 at 4 ply. It must have searched at least 19 ply on the branch that leads to the checkmate, otherwise it could not know it is there. But that leaves open the possibility that there would be a faster mate in the branches that were considered unpromising, and were only searched to 4 ply.

About this 'generic evaluation'... Fairy-Max virtually does not have any evaluation at all, other than piece values, a weak attraction to the center for P, N, B and K, and a penalty of half a Pawn for each King move in the branch leading up to the leaf where the evaluation is made. But it does have check extension, null move with R=2, and late-move reduction of 1 ply on all non-captures other than the hash move. So I think you are still on the wrong track by assuming this has anything to do with evaluation.

Umm, the way I process the reported numbers Myrddin is 790x faster than your program, in terms of nodes, and The King about 19,000x. Not 10x and 200x.


different FEN!