How strong should an engine be with various features implemented?

[jmcd]

I've recently gotten into the hobby of chess programming and am gradually improving my engine. My question is how strong should an engine be if you have transposition tables, killer moves, static evaluation exchange, aspiration windows, pv search, etc. A problem I have run into is implementing a feature into my program, and not knowing if it actually worked correctly because I am unaware of the improvement that the feature should result in. I added transposition tables to my engine a month or two ago, but only recently realized that it was not working and made the fix.

Also, what is the best way to test your engine and put a semi reliable number on its skill level? I've simply been running mine against previous iterations of itself and against stockfish at various skill levels in cutechess. At the moment it beats stockfish 11 at skill 7 about 50% of the time under tournament parameters. I've linked my engine source code which describes the features I currently think are working correctly.

https://github.com/jonathanmcdermid/Chess-Engine

[Dann Corbit]

It is really hard to judge those things in terms of Elo.
Consider adding alpha-beta as an improvement over mini-max.
You go from O(mobility*mobility) to O(sqrt(mobility * mobility))
But there is a catch, which is how good is your move ordering?
If your move ordering is terrible you won't see the titanic improvement.
A feature may behave differently depending on whether or not you have IID.
If you added LMR before null move, the benefit will be different from adding it after null move.

Something that the stockfish team did that might be helpful is to put the benefits in terms of Elo for most of the features in search,cpp so that might give you a rough estimate of what to expect. Often the gains are small

[jmcd]

Thanks for replying and taking the time to look through my code Dan. I've already explored stockfish a bit, so I'll make sure to check their comments.

I'm a bit lost on your recommendation on switching to alpha beta though. I was under the impression that alpha beta simply refers to the pruning mechanism used in many minimax functions, which I already have. If I am misinterpreting your recommendation please let me know! I will definitely try changing LMR. I think my move ordering is decent, with hashmoves, pvmoves, killermoves, and winning/losing captures differentiation. I tried adding secondary killermoves but it only slowed the engine down.

[Dann Corbit]

Those were not specific recommendatinos, just general notes.
I did not look at your code, did you provide a link?

[Dann Corbit]

OK, I see you are already using alpha-beta.
I see piece square tables, but your evaluation does not seem to use them. It looks to me like your evaluation is strictly counting the wood, Is that right?

[jmcd]

The piece square tables are used in the evaluation function titled negaEval in bot.cpp. I'm hoping to add some bonuses for pawn structure next but I havent done much research on how to do it effectively.

[jdart]

Null move and LMR are big winners, elo-wise, but that is assuming you have a solid/not-buggy implementation of other features already. I think new programmers are always best advised to focus on correctness first.

[mvanthoor]

I've recently gotten into the hobby of chess programming and am gradually improving my engine. My question is how strong should an engine be if you have transposition tables, killer moves, static evaluation exchange, aspiration windows, pv search, etc. A problem I have run into is implementing a feature into my program, and not knowing if it actually worked correctly because I am unaware of the improvement that the feature should result in. I added transposition tables to my engine a month or two ago, but only recently realized that it was not working and made the fix.

Also, what is the best way to test your engine and put a semi reliable number on its skill level? I've simply been running mine against previous iterations of itself and against stockfish at various skill levels in cutechess. At the moment it beats stockfish 11 at skill 7 about 50% of the time under tournament parameters. I've linked my engine source code which describes the features I currently think are working correctly.

https://github.com/jonathanmcdermid/Chess-Engine

Cool, good luck with your new engine. I'm going to try and follow development. I've been writing Rustic, a chess engine in Rust, and I actually asked the same question as you. After I finish the bare bones version, Rustic will be open-sourced on Github, with the following features:

- Move generator
- A/B-search with iterative deepening
- Simple time management to be able to play timed games ("time left" / "moves to do" = "time per move")
- Simple evaluation (material + PSQT)
- UCI interface, minimal implementation to be able to play under GUI

At that point, I'll start testing the engine, and hopefully come up with a bare-bones engine ELO-rating.

After adding each feature, a new ELO-test shall be run, so I can measure what each feature will add. Obviously, features will stack; I'm not going to remove a feature already added to test each and every feature separately. I still have to decide which feature to add when, but probably the transposition table will be first, as I have written and tested it already for the Perft routine, and it speeds that routine up by almost 85% (!) percent. Also move ordering will be high on the list, as this is very beneficial for a/b-search. Then the Qsearch, killers, etc... one by one.

(One of the best features to add strength through evaluation, seems to be to add detection of passed pawns, and giving a bonus if you have them, and a negative if the opponent has them. Search on madchess.net; it seems for madchess, this one feature added 140 ELO.)

My suggestion would be to strip every feature from the engine, so you have only the bare-bones engine left, and then test it. After that, add each feature back in, testing them one by one.

There's no way to tell how much ELO each feature adds; it's different for every engine, because the speed is different, move generation and ordering is different, etc...

All of my findings will eventually be at https://rustic-chess.org/.

[jmcd]

Null move and LMR are big winners, elo-wise, but that is assuming you have a solid/not-buggy implementation of other features already. I think new programmers are always best advised to focus on correctness first.

LMR was by far the biggest performance change to my engine in terms of search depth, followed by nullmoves. It crushed previous iterations that dont have it. One thing that confuses me about LMR though is that it seems to make the principal variations more inconsistent in what depth they are found. Before LMR, my engine would almost always find the correct move at a position at exactly depth X, but after LMR it will sometimes find it at X or up to X+5.

[jmcd]

My suggestion would be to strip every feature from the engine, so you have only the bare-bones engine left, and then test it. After that, add each feature back in, testing them one by one.

I think this is what I will end up doing soon. Some features I feel I have implemented very cleanly, and some I am not proud of at all. I definitely have a better understanding of all the concepts I've used now than I did when I first wrote them.

I will be sure to follow your development as well!