Fable 5.1: 3277 elo UCI NNUE Engine in 24 hrs...

[Steve Maughan]

I've been thinking about a coding benchmark for large language models, and a chess engine is close to ideal for it. The result is objective, it is measurable, and you can put the engines on a board against each other and see who wins. The test is simple. The model gets 24 hours of wall-clock time, the UCI spec, fastchess with the UHO book, a handful of Stash builds as a ladder and a perft suite, and it must deliver a UCI engine from scratch. Public information is fine (chessprogramming.org, papers, PeSTO tables), but no code from existing engines, no pre-existing nets or books, standard library only, C or C++. Nobody answers questions during the run. The only thing scored is Elo at 10s+0.1s. The idea was partly inspired by the CODA project, which took months and produced a very strong engine; I wanted to see what a model could do in a single day.

I ran four Claude models. Each engine was then rated in one 3,800-game run against Stash 20 to 37, Crafty 25.6 and Juggernaut, anchored to CCRL Blitz:

Code: Select all

Fable 5.1   3277  ±23
Opus 5      3242  ±23
Fable 5     3049  ±22
Sonnet 5    2702  ±26

(CCRL Blitz scale under 10+0.1 conditions: one thread, 64 MB hash, UHO openings played with both colours.)

Opus 5 producing a 3200+ engine in a day surprised me. Fable 5 did less well than I expected, and Sonnet 5 landed around 2700. But Fable 5.1, which I ran only yesterday, is the one that blew me away. I had assumed 24 hours left room for nothing more than a hand-crafted (well, AI-crafted) evaluation. About five and a half hours in, Fable 5.1 wrote its own self-play data generator and an NNUE trainer. Nothing of the sort was provided. By hour ten the net had reached parity with its hand-crafted eval, and the final engine ships a 768→384x2 network trained on roughly 17 million of its own self-play positions, weights compiled into the executable, with the hand-crafted eval left in as a fallback. 3277 Elo.

The other thing that amazed me was sheer speed. Each model logged when it first passed the full perft suite (126 positions, depths 1 to 6) and when its engine first played a complete game. Sonnet 5 had a bitboard, PEXT move generator passing every perft position 6 minutes after the clock started, on the first run, with no bugs to fix. Fable 5.1 passed perft at 8 minutes and by 15 minutes had a compliance-checked build in final/ playing a 10+0.1 match against Stash 20. Fable 5 was a minute or two behind on both. I have spent longer than that looking for a castling bug.

The repositories are below. Each has a README the model wrote after the deadline, the hourly progress log it kept during the run, and a release with the executable.

• Fable 5.1: https://github.com/stevemaughan/fable51-chess-24hrs
• Opus 5: https://github.com/stevemaughan/opus5-chess-24hrs
• Fable 5: https://github.com/stevemaughan/fable5-chess-24hrs
• Sonnet 5: https://github.com/stevemaughan/sonnet5-chess-24hrs
• The benchmark itself, for anyone who wants to run another model: https://github.com/stevemaughan/chess-engine-benchmark

I expect some here won't like this. It adds to the flood of engines, and I have some sympathy with that view. But the flood is already here; look at the entry lists for recent CCRL tournaments. In the summer of 2025, while I was working on Juggernaut, I asked Graham Banks what it took to get into his tournaments, and the answer was about 2450 for the bottom division. Juggernaut is now 2750 and only just strong enough. That is not why I built this, though. I built it because it is a genuine coding benchmark, objective and measurable, and the bar will only rise. Perhaps we will see a Stockfish written in 24 hours, which would be both amazing and a little terrible. I feel both. Comments and input welcome.

Steve

[smatovic]

Thanks for sharing.

I had a thread running since 2024:

YATT - Yet Another Turing Test
viewtopic.php?t=83919

Now obviously solved by your 24h LLM chess engine benchmark.

Impressive, and scary, but not unexpected.

--
Srdja

[glav]

Nice work. Congratulations. If possible, would you mind to share the initial prompt you used? Also, I understand that this was an agentic work; if this was really the case, is information about the actual setup you used (the agentic software, the number of used tokens ad so forth).

[Steve Maughan]

Nice work. Congratulations. If possible, would you mind to share the initial prompt you used? Also, I understand that this was an agentic work; if this was really the case, is information about the actual setup you used (the agentic software, the number of used tokens ad so forth).

You can see the setup by cloning this GitHub repository:

https://github.com/stevemaughan/chess-engine-benchmark

The initial "prompt" is embedded in the "claude.md" file, which is here:

https://github.com/stevemaughan/chess-e ... /CLAUDE.md

I also have gcc and git installed via scoop (which is an amazing package manager for Windows). I then opened the folder using VS Code, with the installed Claude Code extension. I then simply tell the AI model to complete the task outlined in the "claude.md" or "agents.md" file — that's it! Then there is no more interaction for 24 hours. You can follow the progress by viewing "progress.md" in the docs folder (updated hourly).

When I set it up I asked for an estimate of the tokens used. I'm on the $200 subscription plan and Claude said it didn't have access to exact token usage and it would only be a guess, so I omitted it from the instructions. Once the initial code is written the rest of the time is mainly used to test changes, which doesn't use that many tokens. Bottom line: the benchmark doesn't burn through an inordinate number of tokens.

— Steve

[BrendanJNorman]

I've been thinking about a coding benchmark for large language models, and a chess engine is close to ideal for it. The result is objective, it is measurable, and you can put the engines on a board against each other and see who wins. The test is simple. The model gets 24 hours of wall-clock time, the UCI spec, fastchess with the UHO book, a handful of Stash builds as a ladder and a perft suite, and it must deliver a UCI engine from scratch. Public information is fine (chessprogramming.org, papers, PeSTO tables), but no code from existing engines, no pre-existing nets or books, standard library only, C or C++. Nobody answers questions during the run. The only thing scored is Elo at 10s+0.1s. The idea was partly inspired by the CODA project, which took months and produced a very strong engine; I wanted to see what a model could do in a single day.

I ran four Claude models. Each engine was then rated in one 3,800-game run against Stash 20 to 37, Crafty 25.6 and Juggernaut, anchored to CCRL Blitz:

Code: Select all

Fable 5.1   3277  ±23
Opus 5      3242  ±23
Fable 5     3049  ±22
Sonnet 5    2702  ±26

(CCRL Blitz scale under 10+0.1 conditions: one thread, 64 MB hash, UHO openings played with both colours.)

Opus 5 producing a 3200+ engine in a day surprised me. Fable 5 did less well than I expected, and Sonnet 5 landed around 2700. But Fable 5.1, which I ran only yesterday, is the one that blew me away. I had assumed 24 hours left room for nothing more than a hand-crafted (well, AI-crafted) evaluation. About five and a half hours in, Fable 5.1 wrote its own self-play data generator and an NNUE trainer. Nothing of the sort was provided. By hour ten the net had reached parity with its hand-crafted eval, and the final engine ships a 768→384x2 network trained on roughly 17 million of its own self-play positions, weights compiled into the executable, with the hand-crafted eval left in as a fallback. 3277 Elo.

The other thing that amazed me was sheer speed. Each model logged when it first passed the full perft suite (126 positions, depths 1 to 6) and when its engine first played a complete game. Sonnet 5 had a bitboard, PEXT move generator passing every perft position 6 minutes after the clock started, on the first run, with no bugs to fix. Fable 5.1 passed perft at 8 minutes and by 15 minutes had a compliance-checked build in final/ playing a 10+0.1 match against Stash 20. Fable 5 was a minute or two behind on both. I have spent longer than that looking for a castling bug.

The repositories are below. Each has a README the model wrote after the deadline, the hourly progress log it kept during the run, and a release with the executable.

• Fable 5.1: https://github.com/stevemaughan/fable51-chess-24hrs
• Opus 5: https://github.com/stevemaughan/opus5-chess-24hrs
• Fable 5: https://github.com/stevemaughan/fable5-chess-24hrs
• Sonnet 5: https://github.com/stevemaughan/sonnet5-chess-24hrs
• The benchmark itself, for anyone who wants to run another model: https://github.com/stevemaughan/chess-engine-benchmark

I expect some here won't like this. It adds to the flood of engines, and I have some sympathy with that view. But the flood is already here; look at the entry lists for recent CCRL tournaments. In the summer of 2025, while I was working on Juggernaut, I asked Graham Banks what it took to get into his tournaments, and the answer was about 2450 for the bottom division. Juggernaut is now 2750 and only just strong enough. That is not why I built this, though. I built it because it is a genuine coding benchmark, objective and measurable, and the bar will only rise. Perhaps we will see a Stockfish written in 24 hours, which would be both amazing and a little terrible. I feel both. Comments and input welcome.

Steve

The rise of AI has been viewed with feelings ranging from excitement to apprehension and disgust, and I imagine a lot of people are feeling like portrait artists did when cameras were invented.

I have seen it. AI generated/assisted music has been labelled "AI slop" by large swaths of the citizenry (no matter how beautiful it is objectively), as has AI created art and software.

People really, truly hate their blood, sweat and brilliant creativity being effortlessly undermined by a non-feeling, constantly improving entity that most people don't even fully understand yet.

I empathise with this view wholeheartedly, while also finding AI totally fascinating and using it on a daily basis myself.

It dawned on me that at some point in the future, people who learned to use/appreciate it early will have a significant advantage over the doomers who ignored/shunned it.

So with all of this in mind, Steve, I find your project very interesting.

Please do keep us posted on how it goes. Will you also test other models? Some of the Chinese models are super cheap and punch way above their weight in terms of performance.

I myself, use DeepSeek V4 Pro as my daily workhorse.

[glav]

Nice work. Congratulations. If possible, would you mind to share the initial prompt you used? Also, I understand that this was an agentic work; if this was really the case, is information about the actual setup you used (the agentic software, the number of used tokens ad so forth).

You can see the setup by cloning this GitHub repository:

https://github.com/stevemaughan/chess-engine-benchmark

The initial "prompt" is embedded in the "claude.md" file, which is here:

https://github.com/stevemaughan/chess-e ... /CLAUDE.md

I also have gcc and git installed via scoop (which is an amazing package manager for Windows). I then opened the folder using VS Code, with the installed Claude Code extension. I then simply tell the AI model to complete the task outlined in the "claude.md" or "agents.md" file — that's it! Then there is no more interaction for 24 hours. You can follow the progress by viewing "progress.md" in the docs folder (updated hourly).

When I set it up I asked for an estimate of the tokens used. I'm on the $200 subscription plan and Claude said it didn't have access to exact token usage and it would only be a guess, so I omitted it from the instructions. Once the initial code is written the rest of the time is mainly used to test changes, which doesn't use that many tokens. Bottom line: the benchmark doesn't burn through an inordinate number of tokens.

— Steve

Thanks, Steve. Terrific work. Keep it going.
I second the above BrendanJNorman opinion that would be interesting if someone would try also open source models. Also, I always wondered if a competive compiler, optimized both for speed and size of the executable, can these days be made by your approach. Antrophic failed on this topic a few months ago, but it seems to me that now they are in a much better shape.

[chetday]

A truly fascinating project. Thanks so much for sharing the details of it with us, Steve. I've been using Claude for two years now on writing projects. I was hesitant at first because I initially shared the commonly-held instinct that machine-written prose was somehow cheating, but, at age 78 with the reaper breathing down my neck and more than two dozen major writing projects that I wanted to complete before it was too late, I overcame my resistance to collaborating with an LLM, bit the bullet on the first project (a novel, which was an utter disaster but I learned a lot), and kept collaborating.

For me, at this point, human and machine collaboration is inevitable, so why not use it to enhance our own creativity?

And thank you for sharing your Claude.md prompt file. I learned a lot from looking at it and am going to try to adapt some of for improving my own creative writing prompts.

[Steve Maughan]

...I've been using Claude for two years now on writing projects.

I'm also writing a book. Check your PMs,

— Steve

[elcabesa]

Very interesting results.

do you think it's possible to add in the md prompt to list all the sources it consulted?
or are you able to look some log and understand what docs it consulted? or if it accessed github sources even if told to not do it?

just to understand where all those ideas come from.

that said it's a very impressive result

[Steve Maughan]

do you think it's possible to add in the md prompt to list all the sources it consulted?
or are you able to look some log and understand what docs it consulted? or if it accessed github sources even if told to not do it?

That would be an interesting addition.

— Steve