Bug hunting

[Patrice Duhamel]

I'm looking for advices on how to fix and improve my engine.

I think there is something wrong in Cheese, I don't understand if the problem is only the weak evaluation fonction or if there are bugs in other parts of the code.

What should I do to find the origin of the problem ?

If you want to look at the code, it's available here :
https://gitlab.com/cheesechess/cheese

[jtwright]

The first place to start is probably: What are the things you consider to be problems with Cheese? What's giving you the impression there's something wrong?

That will help guide the rest of the debugging process.

[algerbrex]

...
I think there is something wrong in Cheese...
...

Why do you think that exactly? That would be where to start.

[emadsen]

This reminds me of pilot / maintenance humor on the “gripe sheet.”

Pilot: Something loose in cockpit.
Maintenance: Something tightened in cockpit.

[Ras]

What should I do to find the origin of the problem ?

I recommend static code analysis. It seems your build chain is already using compiler warning, that's a good starter. However, CppCheck (free and open source tool) shows a lot of findings. Most of them are likely false positives, but e.g. the redundant if-conditions seem real and may mean that the code doesn't do what you intended. CppCheck is super easy to use. Make sure to show all sorts of findings in the GUI.

The next step can be using Coverity Scan (https://scan.coverity.com) which is free for open source projects, but takes more work to get up and running. However, it provides deeper analysis also across functions.

[Fabio Gobbato]

You can add in your code a lot of assert to find if something is not how you expect or compile the engine with sanitizers on to find some undefined behavior or memory problems. For memory problems is good also valgrind.

[Henk]

Always the same procedure. Go back to a state where it worked. Then add back one feature at a time and test.
So just use simplification.

[abulmo2]

I do not think an engine ranked over 2800 on CCRL is very buggy.
I sent you a merge request for a possible bug, probably with zero impact on the strength of your program though.
You are using YBWC for parallel search. Maybe replacing it with lazy SMP can make your code both simpler and more efficient, and maybe not
I will try to scrutinize your code more thoroughly but I do not expect to find many bugs or possible enhancements.

[Patrice Duhamel]

The first place to start is probably: What are the things you consider to be problems with Cheese? What's giving you the impression there's something wrong?

Yes, I should give more details :

I found moves played during games that I can't reproduce when analyzing the position, and if it's not always blunders some moves are difficult to understand.

Cheese evaluation function is not great, but I was not able to improve eval parameters with Texel's tuning method or other methods, and trying to change parameters at hand is hard.

Now maybe I'm also doing something wrong when running games to test changes in my code, because my results are different compared to CCRL or CEGT.

I recommend static code analysis.

I'm using clang-tidy, and there are few details to look at, but nothing important.
I also have some unit tests, and perft results are good.
I already tried clang address sanitizer, but I don't know how to use memory sanitizer, I had error with the C++ standard library.

I sent you a merge request for a possible bug, probably with zero impact on the strength of your program though.
You are using YBWC for parallel search. Maybe replacing it with lazy SMP can make your code both simpler and more efficient, and maybe not
I will try to scrutinize your code more thoroughly but I do not expect to find many bugs or possible enhancements.

Merci, I think you are right but it will not change the result, I will look at this.
I have both, YBWC by default and optional Lazy SMP, but I'm not sure Lazy SMP works as expected I was only able to test with 4 threads.

[jtwright]

I found moves played during games that I can't reproduce when analyzing the position, and if it's not always blunders some moves are difficult to understand.

To me top possibility here is transposition table. I sometimes find Mantissa plays a move differently in a game than it would if I just started from the position cold (even single threaded), and this is usually due to the TT. Other things that can contribute here is threefold. An FEN string doesn't have history. I've found that occasionally there are positions that would be played/scored differently if the position were set up via `position fen ...` vs. `position startpos moves ...`. So another control could be to load in all previous moves from the pgn if you're trying to replicate a situation from a game. Thirdly, searches with lots of reductions/etc. get pretty unstable, as very small changes to the state can result in different outcomes, even if under *identical* conditions the results will be deterministic. Even using a different set of zobrist keys can cause different behaviors. Or if history tables age instead of clearing, etc. etc. IIRC Cheese plays at a pretty high level and so probably does have a lot of selectivity in its search.

It's possible the inconsistency is caused by a bug but it could be something benign like the above as well. If you're trying to figure out the specific cause you should try to control for as many of these features in your experiments as you can.