How good are searches and evaluations of AI-written chess engines

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

User avatar
phhnguyen
Posts: 1549
Joined: Wed Apr 21, 2010 4:58 am
Location: Australia
Full name: Nguyen Hong Pham

How good are searches and evaluations of AI-written chess engines

Post by phhnguyen »

Steve Maughan has made an amazing job to create some chess engines with AI agents from scratch, in C/C++, within 24 hours for each.

I wonder how good their searches and evaluations. On one hand, we know where to focus to make those engines be stronger. On the other hand, I may pick some good things for my needs, say, create a new Xiangqi chess engine by using a good search function + my own Xiangqi structure and evaluation. (Even though Stockfish has the best components, its license is GPL, not suitable for some tasks).

To compare, I have created some new chess engines by swapping their searches and evaluations, then make a tournament for them all using Banksia. The result is in the below table. A name “Fable 5.2 chess 24hrs sf19eval” mean the engine Fable 5.2 chess 24hrs + Stockfish 19 NNUE evaluation.

For short: the strongest AI engine Fable 5.2 chess 24hrs is behind Stockfish 19 about 110 Elo on the search and 279 Elo on the evaluation.

More details on my blog.

Code: Select all

 #  Engine                             Elo  95 % interval   Score
-----------------------------------------------------------------
 1  Stockfish 19                        +0  (anchor)        651.5
 2  Fable 5.2 chess 24hrs sf19eval    -110  -147 .. -78     583.5
 3  Opus 5.5 chess 24hrs sf19eval     -114  -148 .. -83       581
 4  Fable 5.1 chess 24hrs sf19eval    -159  -194 .. -125      549
 5  Opus 5 chess 24hrs sf19eval       -161  -194 .. -134      547
 6  Sonnet 5.5 chess 24hrs sf19eval   -195  -229 .. -164    521.5
 7  Fable 5 chess 24hrs sf19eval      -219  -256 .. -189      503
 8  Astra 6 chess 24hrs sf19eval      -272  -305 .. -243    460.5
 9  Stockfish 19 fable52eval          -279  -313 .. -249    455.5
10  Haiku 5.5 chess 24hrs sf19eval    -299  -336 .. -267      439
11  Sonnet 5 chess 24hrs sf19eval     -334  -370 .. -302    411.5
12  Fable 5.2 chess 24hrs             -335  -372 .. -305    410.5
13  Opus 5.5 chess 24hrs              -405  -443 .. -374    356.5
14  Sonnet 5.5 chess 24hrs            -454  -492 .. -421    320.5
15  Fable 5.1 chess 24hrs             -612  -657 .. -577    219.5
16  Opus 5 chess 24hrs                -636  -681 .. -604    206.5
17  Astra 6 chess 24hrs               -686  -731 .. -647      181
18  Fable 5 chess 24hrs               -802  -859 .. -758      130
19  Haiku 5.5 chess 24hrs            -1125  -1196 .. -1075   37.5
20  Sonnet 5 chess 24hrs             -1138  -1209 .. -1076     35
https://banksiagui.com
The most features chess GUI, based on opensource Banksia - the chess tournament manager
Uri Blass
Posts: 11273
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: How good are searches and evaluations of AI-written chess engines

Post by Uri Blass »

I think the value of the evaluation may be dependent on the search because the search does not stop at random points.
A simple example:Imagine you have search to fix depth. All the leaves of the tree has the same side to move so a fixed bonus for the side to move is not going to change the search when the same bonus may be productive when you do not stop the search with the same side to move.

Of course no good engine is going to use fixed depth but still different engines have different rules to stop the search and call the evaluation.
User avatar
phhnguyen
Posts: 1549
Joined: Wed Apr 21, 2010 4:58 am
Location: Australia
Full name: Nguyen Hong Pham

Re: How good are searches and evaluations of AI-written chess engines

Post by phhnguyen »

Uri Blass wrote: ↑Sun Nov 22, 2026 7:33 am I think the value of the evaluation may be dependent on the search because the search does not stop at random points.
A simple example:Imagine you have search to fix depth. All the leaves of the tree has the same side to move so a fixed bonus for the side to move is not going to change the search when the same bonus may be productive when you do not stop the search with the same side to move.

Of course no good engine is going to use fixed depth but still different engines have different rules to stop the search and call the evaluation.
Are you worried swapping-search-evaluation engines don’t work properly? Nope!

All chess engines above are modern, they have very similar designs and close score ranges. I think all AI agents learn a lot from Stockfish and top engines and then create somewhat similar. They are different but not apple-orange.

To solve the problem of different score ranges, we just need to scale and/or convert. It is not hard because we have all source code. Just create a few tens thousand of chess positions from beginning to end, run their evaluations and compare outputs, we know how to scale them to the right ones. Of course it is not really optimal. But it works and gives us information we want.
https://banksiagui.com
The most features chess GUI, based on opensource Banksia - the chess tournament manager