YATT - Yet Another Turing Test

[jefk]

Third test candidate was posted by Werewolf, August 14, 2025, generated by GPT-5 Pro:

what you don't mention is that he wrote it's still a lot of work, compiling the
sourcecode from gpt-5, giving the errors back to gpt-5 which does debugging
than and compiling again, in the hope that not some other errors were introduced.
And then you still have to test the chess program itself, where also some most likely
some bugs can be found. Maybe the gpt-5 Pro version is a bit better but in
in my -recent- experience where i'm (occasionally ~ 1 h/day) 'working' (playing) with
a small Mscp C program, enhancing it with AI help(*), and using both grok4 and Gpt-5
(free version) it still is quite a tedious process of such debugging cycles.
(*) adding a simple 3 pos repetition check for a draw rule proved already
an immensely difficult task for the machine.

For you test the AI should be able to do all programming-testcompiling-debugging-test-execution
cycles itself; i don't think GPT-5 can do that. In a year maybe one AI can do it (grok5?) .

Considering the 'Chinese room' talks, don't think the AI is that smart, it looks for and
finds pieces of source codes elsewhere, then combines stuff and tries to see if
it's logical code, so it's not building from scratch, with only C knowledge.
But then a human newbie programmer would also work like that looking at Tscp etc.

[smatovic]

Third test candidate was posted by Werewolf, August 14, 2025, generated by GPT-5 Pro:

what you don't mention is that he wrote it's still a lot of work
[...]

Yes, thanks for pointing out, Werewolf mentioned it was an interactive process.

And you are probably right, the upcoming LLM versions will be more and more autonomous, agentic.

--
Srdja

[towforce]

Idk if generative AI can be called creative. What is the difference of an AI generating new images by mixing different inputs into something new and a human creative mind?

Again, different chatbots have strengths in different areas: I asked a few chatbots to draw a picture that was unlike anything that most humans had ever seen. ChatGPT came last: its output looked like common modern art. At the other end of the scale, I was VERY impressed with Gemini's image: had it been painted by a human, I would have called it a "brilliancy".

Someone said once, "Intelligence is how you perform in an given environment.". By defining the environment and certain goals we can measure intelligence.

That looks like something I would have said. The problem is that different chatbots are strong in different areas.

Regarding chess, this thread is looking at it the wrong way: to write the best chess program, first find a way to delineate the deep underlying patterns that we haven't found yet (otherwise the benefit of searching game trees would be MUCH smaller than it actually is).

[smatovic]

[...]
Regarding chess, this thread is looking at it the wrong way: to write the best chess program, first find a way to delineate the deep underlying patterns that we haven't found yet (otherwise the benefit of searching game trees would be MUCH smaller than it actually is).

Well, the test does not dictate the type of engine, if Shannon type A, type B, a merger, or a hypothetical type C, up to you, or the AI.

--
Srdja

[jefk]

here's what a certain Lex Fridman (you know, from the MIT guy with the tech interviews on yt)
wrote (somewhere else)

"Programming with AI is insanely fun. Process is:
1. generate code
2. read & understand code that was generated
3. make small changes "manually" (still with great autocomplete)
4. test & debug
5. make big changes with new prompt
6. go back to step 1
Pure vibe coding skips step 2 & 3. And I think we'll need human expertise & experience for steps 2, 3 (and 4) for quite a while.
But holy shit, I'm learning much faster, being way more productive, and having more fun.
Not sure we're close to "AGI/ASI", but the software engineering world is definitely getting transformed very rapidly. It feels surreal to be experiencing it directly on a daily basis. Of course, there are both scary (jobs, security) & exciting (productivity, access) consequences to this transformation, as with all powerful technology.

interesting perspective, but for me, although it's only a minor part-time activity now,
it's *not* 'insanely fun'. I only wanted to expand an old simple winboard engine i had playing
gambits (loading a .txt book) with some more stuff, but even with the AI it's still way
more tedious than i thought (it would be). Why i'm doing it, well it may have some spinoff,
thinking of -yet another_ simple engine playing human style (with lots of cool gambits)
at some levels, eg home player, club player (low/middle/high), FM, and possibly IM.
That's all. Maybe i should try to find another way of doing it (looking for another program,
with polyglot support, and levels which can be set at such skill levels; i'll see. With conversations
about theoretical physics with some of the bigger AI including literature surveys, i had more
'fun' at least. But that's a bit off topic; although not regarding the pure 'turing test' aspect
(what i found is that dark energy and dark matter probably don't exist; as for Turing
test, can an AI replace theoretical physicists, well no , it cannot).

PS tf wrote

deep underlying patterns that we haven't found yet

still think that, tf ? well i'm not so sure of that, first of all, the chess rules mean that there are
(sequentially, alternate) move options, this is a fact, like it or not. Thus as result of this there is a
chess 'tree' with possible move options, again like it or not. At the end of this tree, you can
evaluate the position. Ofcourse the endgoal of chess is checkmate, but if you're down a
queen, then you're down in material; and that's usually not an advantage (although maybe
not such a big disadvantage, at least not if you're the Leelazero tuned by kaufman ).
With the nnue method the evaluation methods have been hugely improved, in fact
many 'deep underlying patterns' humans haven't found (or cannot understand with our
limited brain) have been found, in rather strange mathematical ways. Like it or not.
Is there much more to improve for such nnue's ? Well sure, there could be, but this
would be just a (probably minor) gradual improvement, not a real new way of approaching
chess, as you suggest. In addition, it would be nice if the Ai's could convey some of it's positional knowledge
into rules understandable for humans, in simple verbal language, and then some of the more
old-school positional rules/guidelines (*) could be improved/renewed.
*) eg Silman with his imbalances or some old russian school nonsense/dogmas etc
regarding some pawn structures)

[towforce]

[TowForce] wrote

deep underlying patterns that we haven't found yet

still think that, tf ? well i'm not so sure of that, first of all, the chess rules mean that there are
(sequentially, alternate) move options, this is a fact, like it or not. Thus as result of this there is a
chess 'tree' with possible move options, again like it or not. At the end of this tree, you can
evaluate the position...

The game tree for chess is exponential. If the complexity of the underlying patterns needed grows logarithmically with the size of the game tree, then this pattern complexity grows linearly as the game tree grows exponentially (the logarithmic growth of exponential growth is linear). This is credible to me: fruit flies display an impressive array of intelligent behaviours with just 125,000 neurons.

...many 'deep underlying patterns' humans haven't found (or cannot understand with our limited brain) have been found, in rather strange mathematical ways.

In the early days of NNUE chess engines, it was very easy to see that, rather than a small number of deep patterns (what you'd want), the work was actually being done by a large number of shallow (simple) patterns.

In addition, it would be nice if the Ai's could convey some of it's positional knowledge into rules understandable for humans, in simple verbal language...

You'd be getting quantity over quality.

[jefk]

You'd be getting quantity over quality.

if it's the raw data, yes, but with some other coding, ideally it could be clustered,
giving best plans/strategies depending on the sort of position, eg. explaining (*)attacks in
terms of piece attacks, pawn storms, or others, possibly combined. So explaining some
plans(**) would be useful From the alfa-zero games i remember a few more concepts
became clear, like a h-pawn moving forward to a short-castled king, etc.
My impression of P5 is that it's especially strong in piece attacks, which
usually -most likely- requires open positions, but then with pawn sacrifices
(or so-called pawn levers) you sometimes can create more open positions.

(*) at more simple levels of play there already is something like that
https://decodechess.com/
but i found it too simple for my level, maybe useful for beginners in (tactical)
terms like explaining simple forks and stuff like that; there recently was a guy on this
forum indicating his intention to develop stuff (AI explanations)like that (but more
complex i presume) and posted a questionaire; which i filled in and added
the comment that it won't be easy (at least not at higher elo levels).

(*) depending on the sort of position after the opening. Explaining
possible plans of attack, or simply heuristic concepts (like for kings indian
the well known opposite flank play, with sometimes difficult positions
for White in the end if the kings position is weak); Botvinnik tried
to make a chess program with such more vague concepts but lateron
the simple alfa-beta search methods (+qs and eval) turned out to be more
effective Also note that for each plan there -theoretically- exists a counterplan.
but if you're too late, didn't see the attack (eg by P5) coming then you're
toast; and for humans it's often psychologically difficult to defend
(eg their king) eg against complicated tactics.

[towforce]

You'd be getting quantity over quality.

...From the alfa-zero games i remember a few more concepts became clear, like a h-pawn moving forward to a short-castled king...

These will be things that humans interpreted from the games, like a hagiographer praising everything that his hero does, rather than a deep pattern in chess that emerged during the NN's training and which wasn't known before - and that's assuming that these strategies generally work!

[jefk]

assuming that these strategies generally work

well key is, getting the -opportunity for- advantage ; this is especially true in
human play and thus hereby the idea ('deep hidden pattern'?) is to induce one
or more mistakes by the opponent (wrong plan, tactical mistake, whatever.

There various ways to achieve this, and it can be depending on the opponent('s style);
if you're better than your opponent in tactics, gambit (or fierce attacking) play makes
sense. If you're a better positional player, knowledge of some slow openings helps,
at least during the initial phase of the game.

PS in human otb chess, players usually think different about such things, whereby
often at higher level it's a subconscious thought process, thus a GM with a certain
style will (also) not be able to convey (the 'deeper' background of) his methods.

PS2 that the h4 pawn works sometimes is clear from some other books than by Sadler,
also by Simon Williams (Harry attack); and then there also is a book about modern
methods with the g4 advance (eg after opposite castling). The less knowledgeable the
opponent is about such methods, the better your chances, and thus the more difficult
such plans/methods are the bigger the chance they work for you. There are many basic
rules/guidelines as 'develop your pieces' in the opening, keep the king safe etc etc.
And then there are pawn structures, with stuff as minority attack, etc. but if both
sides know such stuff, as outsider(*)/amateur you will hardly notice the underlying
thoughts the GM's had while playing; can there be similar more complicated
plans/rules of thumbs, sure, but again if both sides know them it's no big deal
So basically there imo there hardly are any significant 'deeply hidden patterns'
(if both sides know the 'pattern'). Except for search/eval of course...
For GM's the hidden patterns are simply called 'style'. Morphy developed his pieces,
Tal and Aljekhine complicated the position, Carlsen presses on in the endgame etc
Based on patterns which worked for them; that's all (imho). For SF the pattern
is, play strong chess (elo). Thus you see a balanced style with deep search. Not
really deeply hidden patterns unless it would depend on specific positions, maybe.
(*) what's you rating ? you know a lot about computer chess, but real human otb
chess also is imo (maybe most) important. not that i'm such a strong player btw
But i presume eg Uri Blass, or Kaufman, or 'father' etc know what i'm talking about.

[towforce]

...Tal and Aljekhine complicated the position...

This.

There are 2 patterns one would want ones engine to know:

1. Win/draw
2. Complexity

If the position winning, play the shortest forced win sequence.

If the position is drawing or losing, complicate the position as much as possible. For humans, this will confuse them. For engines, it will throttle the depth of the game tree, worsening the standard of play.