The only difference in this run is I used lockless hashing trick for all entries in hash table and the run was performed using a different set of zobrist randoms.

The only difference in this run is I used lockless hashing trick for all entries in hash table and the run was performed using a different set of zobrist randoms.

That is great news!

I compared data from your first and second run and found the following 7 differences after 3 plies:

I don't know if this implies that there were more than one bit flip in the first run, or if the same flipped bit stayed around in the hash table long enough to affect several sub-trees.

The only difference in this run is I used lockless hashing trick for all entries in hash table and the run was performed using a different set of zobrist randoms.

That is great news!

I compared data from your first and second run and found the following 7 differences after 3 plies:

I don't know if this implies that there were more than one bit flip in the first run, or if the same flipped bit stayed around in the hash table long enough to affect several sub-trees.

Note that I accidentally swapped "run1" and "run2" in the table headers above.

As an extra sanity check I computed perft 11 for the 4 positions corresponding to the above table. The result was:

petero2 wrote:
I don't know if this implies that there were more than one bit flip in the first run, or if the same flipped bit stayed around in the hash table long enough to affect several sub-trees.

Impossible to say. I think there is a good probability of same flipped bit staying in the hash table, but we can't be sure.

Thanks for verifying the mismatching counts!

I am planning to start Perft(15) run, but with my current program it will likely take about 3-4 months and I will need to run it at least twice to catch errors due to random bit flips/cosmic rays.