New engine I am working on

[FireDragon761138]

I did Theoria as a project based on Stockfish, so I thought I'd do a companion engine, so of course it gets named Praxis (literally means "deed", but is the root of the word "practice", hence the name). This will be a bot so it won't be involved in any kind of engine vs. engine tournaments for awards. I used Fable to help design the engine, trying to go for a clean room approach. It ended up using alot of ideas from Crafty and Stockfish for the basic search and evaluation, but we added a unique humanization layer that is different from any other engine based on research Fable did on actual human play. It uses contempt, optimism, favoring tricky or complex moves over perfect play.

We calibrated the engine based off Maia weighted tourneys but that wasn't a perfect fit, so we used additional Lichess puzzles, compensating for the known differences between human play on puzzles vs. games. The end result seems to be a strong, fun engine to play against, but it seems to be almost too clever to be believable, particularly at low levels (600-1000). I am wondering if the issue is that the human puzzle vs. game discrepancy wouldn't play out with how negamax engines play chess?

Input is welcome. I'm a bit stuck. Short of putting the bot out there and heavily iterating, I'm not sure what to do. I could ask Fable for more advice, but it burns alot of tokens, and sometimes humans are capable of thinknig completely outside the box and have more experience with working with engines hands on.

[FireDragon761138]

I pivoted to modding MadChess as a research project. I'm using FunSearch and CMA-ES to generate candidates for code and parameters to fit against a psychoacoustic inspired model of human chess perception - my research indicates it's likely that humans don't see the fitting criteria as equally weighted, with some things, like the percentage of major and minor pieces lost, mattering more than abstractions like "centipawn loss". I modified MadChess to have a audio-inspired noise shaper to direct the kinds of errors the engine makes at low Elo, too.

The Maia engine is already pretty high out of the box in terms of humanish metrics, but it's danger coupling is the worst chess engine we've looked at so far. Real low-elo human beings tend to wither under highly dynamic positions, whereas Maia plays very "baked" kind of moves that aren't as context dependent. It also doesn't play sufficiently weak compared to actual human club players. So I'm aiming to make something better that actually plays weakly better than Maia does.

[mysammoore]

Yeah I've been in this neighbourhood with mine, it's a policy-proposes, engine-vetoes setup rather than a weakened search, and the thing I'd pull out of it for your problem is where the strength actually lives. In mine the search never picks a move, a policy head trained on human games ranks the legal moves and the engine only gets to throw candidates out, it plays the first one in the policy's order that doesn't drop more than a tolerance below best play, about 125cp at equal, tighter when slightly worse, wide open when the game's decided. Strength is just depth and how many candidates it's allowed to look at, both scaled off the clock, and that's it, no contempt or optimism on the score side at all. The reason that matters for "too clever" is that contempt and optimism and liking tricky moves are still a full-vision search choosing, so it reads like a strong player being weird on purpose. If the proposer is human the weirdness is already in the right places.

Two things that might actually save you time. Puzzles never worked for me and I don't think they can, a puzzle is a position where the human knows there's something to find, so it measures vision when told to look, and the gap between puzzle rating and game rating is exactly the not looking. In a real game, humans don't "see it's a puzzle," so Maia will never "see it's a puzzle," either, because it's only trained on what humans actually played in real games. And if you calibrated against Maia, Maia-3 conditions on both its own rating and the opponent's, if you didn't set the opponent Elo it was modelling a different player than you thought and the numbers are off because humans also scale their skill vs. their opponent (as much as they can).

The honest caveat is my engine is built for the top end, and the low end is a different challenge. Mine is deterministic, it plays the policy's top passing move and that's believable at 2800 because a strong player's move distribution has a real mode, but at 600-1000 the distribution is flat and any top-1 is going to look too consistent, so you'd need to actually sample. And it turns out even a strong human model needs hard rules about what always gets noticed, mine always plays a mate in five and always takes a free piece and always looks at every check, because the policy alone misses those and it broke believability in edge cases where it would never normally miss something. My guess is the low end is the mirror of that, less about noise and more about rules for what a 700 reliably doesn't see, with a proposer trained at that level and a very loose veto that only catches the outright catastrophes, though at 700 catastrophes are par for the course.