New engine announcement: Blunder 1.0 (And some questions concerning improvement)

[algerbrex]

Could you give me any estimates of the Elo of my engine I should expect at the following points since as I said, I plan to start completely from the beginning and test each and every feature I'm adding to make sure the Elo is increasing:

• Pure negamax and alpha-beta
• MvvLva
• Iterative deepening
• Piece-square tables
• Quiesence search

These blog posts may be of interest to you. The table at the bottom of each post records the ELO gain I measured for each feature. The features in each table link to a blog post where I explain how I implemented the feature. Usually a mix of expository writing and code snippets.

• MadChess 2.0 ELO Progress (mailbox)
• MadChess 3.0 ELO Progress (bitboards)

Keep in mind, though, chess engine features combine in a non-linear manner. Sometimes the sum is worth more than the parts. Other times the sum is worth less than the parts. Also, if you implement features in a different order than I did, you may measure different gains.

Thank you Eric, those blog posts should be helpful for measuring Elo gain. I'll definitely check them out.

[hgm]

Since you're obviously very knowledgeable about this subject, could you give me any estimates of the Elo of my engine I should expect at the following points since as I said, I plan to start completely from the beginning and test each and every feature I'm adding to make sure the Elo is increasing:

• Pure negamax and alpha-beta
• MvvLva
• Iterative deepening
• Piece-square tables
• Quiesence search

I would say you should be able to get above 1600 Elo with that. Micro-Max 1.6, which doesn't even have all of that, is around 1600, I believe. It doesn't have real PST (just a common centralization table for the light pieces), it doesn't have a real QS (just a recapture extension), and it doesn't have MVV/LVA. So it should be a good test opponent.

There could be a bit of an issue when not using TT, namely whether you have some alternative for benefiting from the iterative deepening. Engines without TT, such as TSCP, do have some provision to make sure the PV from the first iteration is the first branch searched. Micro-Max 1.6 uses Internal Iterative Deepening, which is a standard technique even in engines that do have a TT, to find the move to search first in case the TT did not provide one. But if you have neither of those then there is no way an iteration can know what went on in the previous one, and all your searches are completely uninformed. Micro-Max 1.6, having no TT, must of course then use this in every node. And the best move of every such internal depth iteration is then used as the first move for the next iteration, until it reaches the depth that was requested for that node. This is the only move sorting it has. (Except that there is some implicit 'captures first' ordering, because the IID starts with an iteration at QS level, where all non-captures are skipped.)

[algerbrex]

Since you're obviously very knowledgeable about this subject, could you give me any estimates of the Elo of my engine I should expect at the following points since as I said, I plan to start completely from the beginning and test each and every feature I'm adding to make sure the Elo is increasing:

• Pure negamax and alpha-beta
• MvvLva
• Iterative deepening
• Piece-square tables
• Quiesence search

I would say you should be able to get above 1600 Elo with that. Micro-Max 1.6, which doesn't even have all of that, is around 1600, I believe. It doesn't have real PST (just a common centralization table for the light pieces), it doesn't have a real QS (just a recapture extension), and it doesn't have MVV/LVA. So it should be a good test opponent.

There could be a bit of an issue when not using TT, namely whether you have some alternative for benefiting from the iterative deepening. Engines without TT, such as TSCP, do have some provision to make sure the PV from the first iteration is the first branch searched. Micro-Max 1.6 uses Internal Iterative Deepening, which is a standard technique even in engines that do have a TT, to find the move to search first in case the TT did not provide one. But if you have neither of those then there is no way an iteration can know what went on in the previous one, and all your searches are completely uninformed. Micro-Max 1.6, having no TT, must of course then use this in every node. And the best move of every such internal depth iteration is then used as the first move for the next iteration, until it reaches the depth that was requested for that node. This is the only move sorting it has. (Except that there is some implicit 'captures first' ordering, because the IID starts with an iteration at QS level, where all non-captures are skipped.)

Hmm, I see. Well, I'll download Micro-Max and use that for testing alongside some other ~1600 Elo engines, and I'll make sure to have some way to save the PV between iterative deepening calls. Thanks!

[hgm]

Perhaps you have no cause for worry. Since you do MVV/LVA sorting even an uninformed search might not be extremely slow. You should just keep an eye on depth. If you lose against an engine that reaches much higher depth, it doesn't necessarily indicate you have a bug. But if it consistently loses at the same depth through blunders, there is something wrong. If your engine (after disabling TT) cannot reach the same depth as the opponent, ou can always give it more time.

[algerbrex]

Perhaps you have no cause for worry. Since you do MVV/LVA sorting even an uninformed search might not be extremely slow. You should just keep an eye on depth. If you lose against an engine that reaches much higher depth, it doesn't necessarily indicate you have a bug. But if it consistently loses at the same depth through blunders, there is something wrong. If your engine (after disabling TT) cannot reach the same depth as the opponent, ou can always give it more time.

Hmm, for some reason this I didn't think of this. I'll make sure to give my engine enough time to think on the same level as its opponent so my tests will be more even. Depth definitely might be the issue.

[mvanthoor]

Since you're obviously very knowledgeable about this subject, could you give me any estimates of the Elo of my engine I should expect at the following points since as I said, I plan to start completely from the beginning and test each and every feature I'm adding to make sure the Elo is increasing:

• Pure negamax and alpha-beta
• MvvLva
• Iterative deepening
• Piece-square tables
• Quiesence search

MinimalChess 0.3: 1570 Elo in the CCRL Blitz list.
Rustic Alpha 1: 1675 Elo in the CCRL Blitz list.

The difference between the two engines is purely based on the fact that Rustic is faster.

If you wanted to know what a good set of PST's + QSearch (= Quiescence, which is a horrible word to type all the time) gives you: MinimalChess gained 500 Elo when it added a good set of PST's and QSearch. This was one of the instances where PST + QSearch in the same engine tested to be stronger than the sum of PST's alone + QSearch alone.

The reason why I'm often referring to MinimalChess is that Rustic was never tested without PST's or QSearch Both engines have often tested new features against one another, and used settings/features to get started to see what's at least possible.

[lithander]

The only idea I have right now is that I see Minimal Chess 0.3 had a couple of features I had taken away, most notably iterative deepening and quiescence search. How much do you think those affected the Elo of Minimal Chess at that stage?

The Version History on my Github lists the strength of each version and the main features that got added. To quote from there: "Version 0.3 adds MVV-LVA move ordering, Quiescence Search and replaces material-only evaluation with Piece-Square Tables. With these changes MinimalChess gains about 500 ELO in playing strength over the previous version and achieved 1571 ELO on the CCRL."

The most notable change in terms of gained strength was the transition from material-only evaluation to PSTs. As long as it was thinking that every position with the same pieces is exactly equal (material-only) features like QSearch and MoveOrdering didn't really change a lot in terms of strength.

[algerbrex]

The only idea I have right now is that I see Minimal Chess 0.3 had a couple of features I had taken away, most notably iterative deepening and quiescence search. How much do you think those affected the Elo of Minimal Chess at that stage?

The Version History on my Github lists the strength of each version and the main features that got added. To quote from there: "Version 0.3 adds MVV-LVA move ordering, Quiescence Search and replaces material-only evaluation with Piece-Square Tables. With these changes MinimalChess gains about 500 ELO in playing strength over the previous version and achieved 1571 ELO on the CCRL."

The most notable change in terms of gained strength was the transition from material-only evaluation to PSTs. As long as it was thinking that every position with the same pieces is exactly equal (material-only) features like QSearch and MoveOrdering didn't really change a lot in terms of strength.

Right, thanks, I saw that, but I guess I glossed over it. What I had done though I removed quiescence search and iterative deepening and just had alpha-beta negamax, MvvLva, and PSTs from Rustic Alpha 1. But if I remember correctly from the games I ran yesterday it still had trouble keeping up with MC 0.3. It wasn't necessarily playing horribly from what I could tell though, so I was trying to figure out if quiescence search and iterative deepening being taken away from my engine could explain the gap. And Muller's comment also made me realize I needed to make sure my engine and the version of yours were searching to the same depths, which again led me back to thinking about strength gained by iterative deepening.

[lithander]

I needed to make sure my engine and the version of yours were searching to the same depths, which again led me back to thinking about strength gained by iterative deepening.

At some point I added support for the 'depth' UCI option which you can set as "Plies" in Cute Chess under time control. That way you can give each engine all the time it needs to search to the given depth. Speed differences can be ruled out, that way. But I don't remember if 0.3 already has it...

I just looked at the code and 0.3 does two things I think that give it an edge. It has iterative deepening and so it knows the PV from the last iteration and it will play the move from the PV before captures, if available - if you don't have iterative deepening anymore you'll probably be at a disadvantage here. It also searches every root move after the first with a nullWindow first trying to prove cheaply that move does not improve on the PV. Which is a bit like a poor implementation of PVS I guess.

On the other hand despite the super-simple eval it doesn't manage to search even a million nodes per second. I guess your engine is much faster? So maybe in the end it should even out.