Interesting. I am yet not sure how to interpret Jost Trillers patch experiment. On one side 3 out of 500 sounds like try and error, on the other side the Stockfish code base must be one of the most optimized out there in the wild. But now the question is out of the bag, how long until LLMs surpass human chess programmers?
--
Srdja
ChatGPT usage in computer chess?
Moderator: Ras
-
- Posts: 3330
- Joined: Wed Mar 10, 2010 10:18 pm
- Location: Hamburg, Germany
- Full name: Srdja Matovic
-
- Posts: 268
- Joined: Wed Jun 16, 2021 2:08 am
- Location: Berlin
- Full name: Jost Triller
Re: ChatGPT usage in computer chess?
Yes it is trial and error, but on the other hand, that's what humans do too, just with a better success ratio.
-
- Posts: 3330
- Joined: Wed Mar 10, 2010 10:18 pm
- Location: Hamburg, Germany
- Full name: Srdja Matovic
Re: ChatGPT usage in computer chess?
Well, trial and error is one method, the other is insight and theory.
Beside LLMs based on neural networks, what we can call probabilistic systems, there are deterministic ones, as expert systems. There is formal AI, symbolic AI, world-model AI, but what we might truly seek is an seed AI, or alike.
But I can agree, that in computer chess a lot of development is trial and error. You have an idea, then you implement and test it.
From Esoteric to Transcendental Chess Programming?
forum3/viewtopic.php?f=7&t=76286
--
Srdja
Beside LLMs based on neural networks, what we can call probabilistic systems, there are deterministic ones, as expert systems. There is formal AI, symbolic AI, world-model AI, but what we might truly seek is an seed AI, or alike.
But I can agree, that in computer chess a lot of development is trial and error. You have an idea, then you implement and test it.
From Esoteric to Transcendental Chess Programming?
forum3/viewtopic.php?f=7&t=76286
--
Srdja
-
- Posts: 5719
- Joined: Tue Feb 28, 2012 11:56 pm
Re: ChatGPT usage in computer chess?
Some time ago I asked GPT-o3 to analyze a position (that was the subject of a forum post). It gave a very interesting analysis why one side was going to win easily by following a particular plan. Of course it was totally wrong.royb wrote: ↑Mon Jul 21, 2025 3:32 pm One other thing you can do with LLMs in general that is interesting/humorous - ask it to play a game of chess against you. They eventually begin making illegal moves - sometimes even making pieces appear on the board from nowhere (i.e. not pawn promotions - just a piece suddenly appears from thin air).
It is hilarious for a bit, then rapidly becomes a complete waste of your time.
-
- Posts: 268
- Joined: Wed Jun 16, 2021 2:08 am
- Location: Berlin
- Full name: Jost Triller
Re: ChatGPT usage in computer chess?
Finally a test also passed LTC: https://github.com/official-stockfish/S ... /pull/6210j.t. wrote: ↑Mon Jul 21, 2025 6:38 pmsmatovic wrote: ↑Mon Jul 21, 2025 2:22 pm - Can it contribute to Stockfish?
No, it would need to load a mental model of Stockfish chess engine, then ponder, then come up with ideas, then implement ideas, then test ideas (with ~10K games selfplay) then decide to commit changes or update mental model. <- This pipeline is currently not present, especially testing, but I think doable in general. Ofc, the whole process of mental model and ideas can be done by brute force method, just try every possible permutation of code generation, driven by some NN heuristic.
I'm working a bit on this currently. No LTC passer as of yet unfortunately, however, there have been a few STC passers (of roughly 500 potential LLM patches):
- https://tests.stockfishchess.org/tests/ ... 4f6388c891
- https://tests.stockfishchess.org/tests/ ... 2d74b172ae
- https://tests.stockfishchess.org/tests/ ... 2d74b1559d
Some of the patches I read and the reasoning how they came up with them are quite reasonable.
Of course all the git and OpenBench submitting part in the pipeline is not done by the LLM itself (yet?), that's all implemented in a series of python scripts. The fishtest submitting of successful OpenBench tests is done fully manually by me.
-
- Posts: 3330
- Joined: Wed Mar 10, 2010 10:18 pm
- Location: Hamburg, Germany
- Full name: Srdja Matovic
Re: ChatGPT usage in computer chess?
j.t. wrote: ↑Wed Aug 06, 2025 2:23 pm [...]
Finally a test also passed LTC: https://github.com/official-stockfish/S ... /pull/6210
How long does it take the Stockfish test framework to run 321012 games?LTC: https://tests.stockfishchess.org/tests/ ... dd5e71127a
LLR: 2.95 (-2.94,2.94) <0.50,2.50>
Total: 321012 W: 82891 L: 81995 D: 156126
Ptnml(0-2): 227, 34421, 90285, 35375, 198
--
Srdja
-
- Posts: 10891
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: ChatGPT usage in computer chess?
start time 2025-08-01 02:31:33smatovic wrote: ↑Wed Aug 06, 2025 8:31 pmj.t. wrote: ↑Wed Aug 06, 2025 2:23 pm [...]
Finally a test also passed LTC: https://github.com/official-stockfish/S ... /pull/6210
How long does it take the Stockfish test framework to run 321012 games?LTC: https://tests.stockfishchess.org/tests/ ... dd5e71127a
LLR: 2.95 (-2.94,2.94) <0.50,2.50>
Total: 321012 W: 82891 L: 81995 D: 156126
Ptnml(0-2): 227, 34421, 90285, 35375, 198
--
Srdja
last updated 2025-08-03 05:00:12
Practically it is less than it because they test many patches in the same time so I guess it takes maybe 4 hours.
-
- Posts: 3330
- Joined: Wed Mar 10, 2010 10:18 pm
- Location: Hamburg, Germany
- Full name: Srdja Matovic
Re: ChatGPT usage in computer chess?
This might be the future of chess programming, congratulations Jost 
--
Srdja

--
Srdja