ChatGPT usage in computer chess?

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

smatovic
Posts: 3330
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic

Re: ChatGPT usage in computer chess?

Post by smatovic »

Interesting. I am yet not sure how to interpret Jost Trillers patch experiment. On one side 3 out of 500 sounds like try and error, on the other side the Stockfish code base must be one of the most optimized out there in the wild. But now the question is out of the bag, how long until LLMs surpass human chess programmers?

--
Srdja
User avatar
j.t.
Posts: 268
Joined: Wed Jun 16, 2021 2:08 am
Location: Berlin
Full name: Jost Triller

Re: ChatGPT usage in computer chess?

Post by j.t. »

Yes it is trial and error, but on the other hand, that's what humans do too, just with a better success ratio.
smatovic
Posts: 3330
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic

Re: ChatGPT usage in computer chess?

Post by smatovic »

Well, trial and error is one method, the other is insight and theory.

Beside LLMs based on neural networks, what we can call probabilistic systems, there are deterministic ones, as expert systems. There is formal AI, symbolic AI, world-model AI, but what we might truly seek is an seed AI, or alike.

But I can agree, that in computer chess a lot of development is trial and error. You have an idea, then you implement and test it.

From Esoteric to Transcendental Chess Programming?
forum3/viewtopic.php?f=7&t=76286

--
Srdja
syzygy
Posts: 5719
Joined: Tue Feb 28, 2012 11:56 pm

Re: ChatGPT usage in computer chess?

Post by syzygy »

royb wrote: Mon Jul 21, 2025 3:32 pm One other thing you can do with LLMs in general that is interesting/humorous - ask it to play a game of chess against you. They eventually begin making illegal moves - sometimes even making pieces appear on the board from nowhere (i.e. not pawn promotions - just a piece suddenly appears from thin air).

It is hilarious for a bit, then rapidly becomes a complete waste of your time.
Some time ago I asked GPT-o3 to analyze a position (that was the subject of a forum post). It gave a very interesting analysis why one side was going to win easily by following a particular plan. Of course it was totally wrong.
User avatar
j.t.
Posts: 268
Joined: Wed Jun 16, 2021 2:08 am
Location: Berlin
Full name: Jost Triller

Re: ChatGPT usage in computer chess?

Post by j.t. »

j.t. wrote: Mon Jul 21, 2025 6:38 pm
smatovic wrote: Mon Jul 21, 2025 2:22 pm - Can it contribute to Stockfish?
No, it would need to load a mental model of Stockfish chess engine, then ponder, then come up with ideas, then implement ideas, then test ideas (with ~10K games selfplay) then decide to commit changes or update mental model. <- This pipeline is currently not present, especially testing, but I think doable in general. Ofc, the whole process of mental model and ideas can be done by brute force method, just try every possible permutation of code generation, driven by some NN heuristic.

I'm working a bit on this currently. No LTC passer as of yet unfortunately, however, there have been a few STC passers (of roughly 500 potential LLM patches):
- https://tests.stockfishchess.org/tests/ ... 4f6388c891
- https://tests.stockfishchess.org/tests/ ... 2d74b172ae
- https://tests.stockfishchess.org/tests/ ... 2d74b1559d
Some of the patches I read and the reasoning how they came up with them are quite reasonable.

Of course all the git and OpenBench submitting part in the pipeline is not done by the LLM itself (yet?), that's all implemented in a series of python scripts. The fishtest submitting of successful OpenBench tests is done fully manually by me.
Finally a test also passed LTC: https://github.com/official-stockfish/S ... /pull/6210
smatovic
Posts: 3330
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic

Re: ChatGPT usage in computer chess?

Post by smatovic »

j.t. wrote: Wed Aug 06, 2025 2:23 pm [...]
Finally a test also passed LTC: https://github.com/official-stockfish/S ... /pull/6210
:thumbsup:
LTC: https://tests.stockfishchess.org/tests/ ... dd5e71127a
LLR: 2.95 (-2.94,2.94) <0.50,2.50>
Total: 321012 W: 82891 L: 81995 D: 156126
Ptnml(0-2): 227, 34421, 90285, 35375, 198
How long does it take the Stockfish test framework to run 321012 games?

--
Srdja
Uri Blass
Posts: 10891
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: ChatGPT usage in computer chess?

Post by Uri Blass »

smatovic wrote: Wed Aug 06, 2025 8:31 pm
j.t. wrote: Wed Aug 06, 2025 2:23 pm [...]
Finally a test also passed LTC: https://github.com/official-stockfish/S ... /pull/6210
:thumbsup:
LTC: https://tests.stockfishchess.org/tests/ ... dd5e71127a
LLR: 2.95 (-2.94,2.94) <0.50,2.50>
Total: 321012 W: 82891 L: 81995 D: 156126
Ptnml(0-2): 227, 34421, 90285, 35375, 198
How long does it take the Stockfish test framework to run 321012 games?

--
Srdja
start time 2025-08-01 02:31:33
last updated 2025-08-03 05:00:12

Practically it is less than it because they test many patches in the same time so I guess it takes maybe 4 hours.
smatovic
Posts: 3330
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic

Re: ChatGPT usage in computer chess?

Post by smatovic »

This might be the future of chess programming, congratulations Jost :)

--
Srdja