GeminiChess, an LLM built engine

[smatovic]

Out of curiosity, did you program already a chess engine by yourself? Or is chess programming a "foreign territory" to you?

--
Srdja

[glav]

Out of curiosity, did you program already a chess engine by yourself? Or is chess programming a "foreign territory" to you?

--
Srdja

I am passionate about chess and programming, but I never programmed an engine. I have done some unrelated work in C, though. These days I am mostly using R and python, but I found out that chatgpt can often do the work for me, especially the basic stuff.

[glav]

I forgot to add the link to the new version:

[Werewolf]

I did exactly this with ChatGPT 5 Pro - also in bitboard, written in C.

Nice. Would you have an executable of your program since I was unable to compile it?

Did debugging take you ages by any chance?

Not really. Although the model submitted several wrong answers (it had allucinations, produced non-compilable code or code that crashed or didn't pass the tests, a couple of times produced even header files with the '.hh' (!) extension, etc.), the driver script was merciless in rejecting these proposals and asking for new ones. It was not too stressful: once started the run, I never had any direct interaction with the LLM, though I was closely watching.

Can you PM me with your email? I'll send you the file directly. However, it is not 100% bug free yet.

[glav]

Apparently, I wasn't the first one to use Gemini for building a chess playing program. Heiko Hotz had already developed one 5 months ago and, moreover, it was also more complete than mine, because it also had a GUI. Nice also the idea to let the program to explain his reasoning. All in all, a nice read. Srdja might wish to update his list.


https://github.com/heiko-hotz/gemini-ch ... t/thoughts

[smatovic]

Apparently, I wasn't the first one to use Gemini for building a chess playing program. Heiko Hotz had already developed one 5 months ago and, moreover, it was also more complete than mine, because it also had a GUI. Nice also the idea to let the program to explain his reasoning. All in all, a nice read. Srdja might wish to update his list.

https://medium.com/google-cloud/build-a ... fc46af7329
https://github.com/heiko-hotz/gemini-ch ... t/thoughts

Thx for posting, but I see no chess engine in the given repository. As far as I got it, you play chess against LLMs with that app.

--
Srdja

[glav]

Apparently, I wasn't the first one to use Gemini for building a chess playing program. Heiko Hotz had already developed one 5 months ago and, moreover, it was also more complete than mine, because it also had a GUI. Nice also the idea to let the program to explain his reasoning. All in all, a nice read. Srdja might wish to update his list.

https://medium.com/google-cloud/build-a ... fc46af7329
https://github.com/heiko-hotz/gemini-ch ... t/thoughts

Thx for posting, but I see no chess engine in the given repository. As far as I got it, you play chess against LLMs with that app.

--
Srdja

Oops! You are right. Too bad: I was already dreaming of a tournament among LLM-built engines.

[jdart]

My experience so far with AIs for coding (with Claude Code and Github Copilot) is that they are pretty good at implementing things, or restructuring, but quite poor at debugging. Claude seems to be better than Copilot: the latter generated code with some pretty elementary C++ coding errors. I haven't tried generating a complete program. One of my learnings from decades in computer chess is that testing is very important. It is amazing how broken a program can be and still play legal moves. Only a good set of tests can demonstrate that everything is functioning properly. I have both unit tests for program components and also of course use game match testing.

[Werewolf]

My experience so far with AIs for coding (with Claude Code and Github Copilot) is that they are pretty good at implementing things, or restructuring, but quite poor at debugging. Claude seems to be better than Copilot: the latter generated code with some pretty elementary C++ coding errors. I haven't tried generating a complete program. One of my learnings from decades in computer chess is that testing is very important. It is amazing how broken a program can be and still play legal moves. Only a good set of tests can demonstrate that everything is functioning properly. I have both unit tests for program components and also of course use game match testing.

ChatGPT 5 Pro is the best here. One trick you can try is playing them off against each other for de-bugging since they seem blind to their own mistakes but critical of other's code.

[glav]

I let Gemini do to some more work on the engine during the weekend. Basically the work was split in two parts, i.e., debug and adding more features. Briefly,
1) *Debug*
The model was prompted to look for bugs into the current program version (ChessGemini_v2). It found a couple of them in the "PVS/LMR re-search calls" and in the "knight outpost evaluation" (I am using verbatim the model words) and proposed some fixes. I mainly accepted the proposal. However, this was not a fully automatized development, since I asked to the driver script to stop at each iteration to let me review and test the amended code. Anyway it wasn't much of pain since less than 10 iterations went into this and most of them were quite reasonable.
2) *New heuristic added*
I prompted Gemini to review the code and name a critical missing feature. It came out with the need to implement "a pawn hash table to store and retrieve pawn evaluation scores and structural information". I then asked the LLM to craft a prompt to be fed with in order to start working on the project. The resulting prompt was very detailed and included some tests to be run during the building. I started it, but this time I let the driver script to handle the whole process completely unsupervised. It took 9 iterations to complete, but it made a reasonable job leading to a measurable improvement in Elo strength. This version should be about 100 Elo points stronger then the previous version. I guess it is now playing around 2000 points.
I enclose the program for download. Any comment is very welcome.

[pgn]Event "?"]
[Site "?"]
[Date "2025.09.30"]
[Round "1"]
[White "GeminiChess_v2.5"]
[Black "GeminiChess_v2"]
[Result "1-0"]
[ECO "B01"]
[GameDuration "00:02:20"]
[GameEndTime "2025-09-30T13:05:09.136 CEST"]
[GameStartTime "2025-09-30T13:02:48.530 CEST"]
[Opening "Scandinavian (centre counter) defense"]
[PlyCount "80"]
[Termination "adjudication"]
[TimeControl "60/60+1"]

1.e4
{+0.23/11 1.6s}
1...d5
{-1.68/10 1.6s}
2.exd5
{+0.49/11 1.6s}
2...Qxd5
{-1.84/10 1.6s}
3.Nf3
{+0.48/11 1.6s}
3...Qe4+
{-1.68/10 1.6s}
4.Be2
{+0.34/12 1.6s}
4...Qg6
{-1.97/10 1.6s}
5.Rg1
{+0.49/10 1.6s}
5...Bd7
{-1.58/9 1.6s}
6.d4
{+0.44/10 1.6s}
6...Nc6
{-1.56/11 1.6s}
7.Bf4
{+0.41/10 1.6s}
7...O-O-O
{-1.38/10 1.6s}
8.Nc3
{+0.06/10 1.6s}
8...Nb4
{-1.22/10 1.6s}
9.Rc1
{+0.17/10 1.6s}
9...Qf5
{-1.35/9 1.6s}
10.Be3
{+0.24/10 1.6s}
10...Nd5
{-1.20/11 1.6s}
11.Ne5
{+0.28/11 1.7s}
11...Nxe3
{-1.03/10 1.7s}
12.fxe3
{+0.16/11 1.7s}
12...Nh6
{-0.99/10 1.7s}
13.Bd3
{+0.32/10 1.7s}
13...Qe6
{-1.05/10 1.7s}
14.Qf3
{+0.38/9 1.7s}
14...Be8
{-0.98/9 1.7s}
15.Qe4
{+0.25/8 1.7s}
15...f5
{-0.91/10 1.7s}
16.Qf3
{+0.25/10 1.7s}
16...g6
{-0.76/10 1.7s}
17.h3
{+0.11/8 1.7s}
17...c5
{-0.55/10 1.7s}
18.Bc4
{+0.19/9 1.7s}
18...Qb6
{-0.35/11 1.7s}
19.Nd5
{+0.19/8 1.7s}
19...Qxb2
{-0.10/11 1.7s}
20.Nd3
{-0.21/10 1.7s}
20...Qa3
{-0.11/10 1.7s}
21.Rb1
{-0.66/9 1.7s}
21...Bc6
{-0.12/9 1.7s}
22.dxc5
{+0.05/8 1.8s}
22...Rd7
{+0.20/9 1.8s}
23.N3b4
{-1.76/9 1.8s}
23...Qa5
{+0.25/10 1.8s}
24.Kf1
{-1.41/9 1.8s}
24...Qxc5
{+0.31/10 1.8s}
25.Bb3
{-1.67/11 1.8s}
25...Bxd5
{+0.31/8 1.8s}
26.Bxd5
{-1.24/11 1.8s}
26...Qb6
{+0.32/10 1.8s}
27.Rb3
{-0.99/10 1.8s}
27...e6
{+0.43/11 1.8s}
28.Nd3
{-1.35/12 1.8s}
28...Qa6
{+0.37/13 1.8s}
29.Rc3+
{-1.33/12 1.8s}
29...Kd8
{+0.28/11 1.8s}
30.Bb3
{-1.42/11 1.8s}
30...Bg7
{+0.35/9 1.8s}
31.Rc4
{-1.12/11 1.9s}
31...Re7
{+0.15/9 1.9s}
32.Ra4
{-0.90/9 1.9s}
32...Qb6
{+0.17/11 1.9s}
33.Rb4
{-0.84/11 1.9s}
33...Qa6
{-0.48/11 1.9s}
34.Bc4
{-0.73/11 1.9s}
34...Qc6
{-1.10/12 1.9s}
35.Qxc6
{-0.17/13 1.9s}
35...bxc6
{-1.06/12 1.9s}
36.Nc5
{-0.15/12 1.9s}
36...Kc8
{-1.10/11 1.9s}
37.Nxe6
{-0.03/10 1.9s}
37...Bf6
{-1.17/11 1.9s}
38.Kf2
{+0.62/11 2.0s}
38...Rhe8
{-1.37/10 2.0s}
39.Rd1
{+3.64/14 2.0s}
39...Rxe6
{-4.74/12 2.0s}
40.Ba6+
{+5.01/18 2.0s}
40...Kc7
{-6.62/18 2.0s, White wins by adjudication}
1-0
[/pgn]