Debug issue

[evandam]

A while back I decided to change my board representation to try and gain a little speed. I went from a mailbox representation to 0x88. I took care to make sure that moves where generated in the same order and such. My goal was to have the old engine perform exactly with the new engine and then I could feel safe and move on. Well I'm very close, but I just cannot find the difference. I have tried to simplify as much as possible. It seems to take at least searching to a depth of 6 (which make me suspect my 3 repetition draw stuff). Both engine Perft correctly.

I guess my question is what is a good way to debug this sort of issue? We are talking about a differnece of 10 or 12 nodes searched in 6.6million. I'm tempted to say close enough and move on, but I would like identical results so I could perform some of the code intended to speed this thing up and verify that I'm not changing my results. Been working on this problem for weeks and I'm just stumped.

[Sven]

I am not perfectly sure about your exact issue. What I think I understood is that you are observing different node counts for the same search task between old and new engine version, is that right?

Are you using hash tables? If so, with fixed zobrist keys?

Does your engine have the ability to create reproducible output on demand? What I mean is:

switch on "reproducability", then search position X to fixed depth D, repeat N times => always the same result; valid for all (X, D, N)


One further note: 0x88 is "mailbox", too, just with a different scheme to access the squares.

Sven

[evandam]

I am not perfectly sure about your exact issue. What I think I understood is that you are observing different node counts for the same search task between old and new engine version, is that right?

Yes

Are you using hash tables? If so, with fixed zobrist keys?

Hash tables are turned off, with the exception of using them to detect repeated positions and enforcing the 50 move rule. The zobrist keys are fixed.

Does your engine have the ability to create reproducible output on demand? What I mean is:

switch on "reproducability", then search position X to fixed depth D, repeat N times => always the same result; valid for all (X, D, N)

Yes

One further note: 0x88 is "mailbox", too, just with a different scheme to access the squares.

Understood, however the 0x88 has some tricks that you can use that I think will speed things up a bit over my old 10x12 mailbox representation.

[Sven]

Are you using hash tables? If so, with fixed zobrist keys?

Hash tables are turned off, with the exception of using them to detect repeated positions and enforcing the 50 move rule. The zobrist keys are fixed.

- Do old and new engine versions use the same zobrist keys?

- Regarding the exception, do you mean you are using a hash-key based method to detect repetition, or are you really using a hash table for that purpose?

- How does the 50 move rule implementation relate to hash key usage? The usual thing is to maintain a counter per ply which either gets incremented or reset.

- Do you validate incrementally updated hash keys by comparing them (in a debug version) to corresponding hash keys calculated from scratch?

- Do you have a way to dump out the search tree? This is very useful to debug such a kind of problem.

One further note: 0x88 is "mailbox", too, just with a different scheme to access the squares.

Understood, however the 0x88 has some tricks that you can use that I think will speed things up a bit over my old 10x12 mailbox representation.

The speedup may be there but it is not overwhelming. So if your main goal is playing strength, not speedup, then you could think about investing also a considerable amount of time in improving search and evaluation.

Sven

[bob]

A while back I decided to change my board representation to try and gain a little speed. I went from a mailbox representation to 0x88. I took care to make sure that moves where generated in the same order and such. My goal was to have the old engine perform exactly with the new engine and then I could feel safe and move on. Well I'm very close, but I just cannot find the difference. I have tried to simplify as much as possible. It seems to take at least searching to a depth of 6 (which make me suspect my 3 repetition draw stuff). Both engine Perft correctly.

I guess my question is what is a good way to debug this sort of issue? We are talking about a differnece of 10 or 12 nodes searched in 6.6million. I'm tempted to say close enough and move on, but I would like identical results so I could perform some of the code intended to speed this thing up and verify that I'm not changing my results. Been working on this problem for weeks and I'm just stumped.

Brute-force works for me. Dump both trees, move by move. Then use "diff" to see where something is different... I do this every now and then to find the kind of bug you describe...

[bob]

Are you using hash tables? If so, with fixed zobrist keys?

Hash tables are turned off, with the exception of using them to detect repeated positions and enforcing the 50 move rule. The zobrist keys are fixed.

- Do old and new engine versions use the same zobrist keys?

- Regarding the exception, do you mean you are using a hash-key based method to detect repetition, or are you really using a hash table for that purpose?

- How does the 50 move rule implementation relate to hash key usage? The usual thing is to maintain a counter per ply which either gets incremented or reset.

- Do you validate incrementally updated hash keys by comparing them (in a debug version) to corresponding hash keys calculated from scratch?

- Do you have a way to dump out the search tree? This is very useful to debug such a kind of problem.

One further note: 0x88 is "mailbox", too, just with a different scheme to access the squares.

Understood, however the 0x88 has some tricks that you can use that I think will speed things up a bit over my old 10x12 mailbox representation.

The speedup may be there but it is not overwhelming. So if your main goal is playing strength, not speedup, then you could think about investing also a considerable amount of time in improving search and evaluation.

Sven

There are some advantages beyond just recognizing end-of-ray conditions. For example, with an 0x88 sized board, you can determine if two squares are on the same rank/file/diagonal without having to worry about the wrap-around effect where for a diagonal, h4 appears to be followed by a5, and such...

[mcostalba]

Hash tables are turned off, with the exception of using them to detect repeated positions and enforcing the 50 move rule.

Disable 50 move rule, disable 3 repetition draw rule, disable TT access.

What I do in such cases is to disable as much stuff as I can until I get the expected result then I re-enable one by one and when result is unexpected I know where the bug is.

Good Luck

[Kempelen]

I would assure than variables are correctly initialized, both in your old aproach and the new one (it could be a problem in the old). Also make sure all vars are correctly declare, i.e. you can have a 'int vector[32]' with 33 values. I remember also situation where I forgot a comma.
This can be done putting lot of assert here and there. Assert is the best tool for me to catch bug and this kind of issue.