Chess, Chatbots And Intelligence: I Make A Big Claim!

[towforce]

For this discussion, I'm going to need a definition of "ultimate chess": not everyone will agree with this, but I'm going to say:

"Makes moves in 1 second and avoids moves which would allow the opponent to win."

This now enables me to write a definition of "ultimate chatbot":

"Responds in 1 second with an answer that's close to being the best possible answer to the prompt."

Obviously that definition is very far from ideal, because it requires a judgement call on what the best possible answer to the prompt would actually be. Maybe, like skating or dancing competitions, judges would have to give marks on a range of different metrics, and different judges would give different scores.

However, if you stick with my definition for a moment, it leads me to two quite shocking conclusions:

1. There is an upper limit to intelligence

2. We're closer to that upper limit than we realise

We're already unbeatable in correspondence chess. Let's say we'll be unbeatable in a 1 second per move time limit in 10 years. What follows are all guesses, of course - but the only thing that matters is that they're approximately the right order of magnitude in order to justify my conclusions:

* The period in which computers and humans working together were better than either alone ("3-Hirn computer chess") was approximately 2005-2014 (very happy to be corrected by people who know better on this guess)

* computers will reach my "ultimate chess" level in 2034

* so the gap between human/machine being best and ultimate chess will be about 20 years

* today, chatbots are more skilled in humans in some ways (for one thing, they know a lot more than any one person), but humans are more skilled in others (humans have more depth of knowledge in their areas of interest) - and the best results are achieved by humans and chatbots working together right now

* so, by extrapolation from chess, we would expect my "ultimate chatbot" to be created in about 20-30 years

Obviously a chatbot is a heavily flawed proxy for intelligence, but it is a sizeable chunk of what intelligence is (ignoring issues around consciousness: nobody thinks their chess computer is conscious - it's intelligent because it choses good chess moves). On this basis, I going to say that my two "outrageous" conclusions might actually have basis in reality.

[hgm]

Is there any intelligence in chatbots at all? I thought these are just search machines that efficiently find existing solutions or answers on the internet, and present those in a colloquial way. While intelligence is the ability to devise original solutions and draw conclusions.

[towforce]

Is there any intelligence in chatbots at all? I thought these are just search machines that efficiently find existing solutions or answers on the internet, and present those in a colloquial way. While intelligence is the ability to devise original solutions and draw conclusions.

Hmmm... by this definition, you'd have to say that most animals, and even some humans, do not display intelligent behaviour.

Also, chatbots may not be able to devise original solutions (or "draw conclusions") in 1 second - but I would expect them to come up with some ways that the problem might possibly be solved. Also, the latest version of ChatGPT, ChatGPT-o1, does have an ability to chain multiple steps of reasoning together to enable it to be able to resolve more difficult prompts.

One of the problems with AI is that there is no universal agreement about what intelligence is - hence my definition of giving a good response to a prompt (link).

A couple of caveats that came to me overnight:

1. Chatbots can only be expected to know what is possible to know at today's level of knowledge

2. They cannot be expected to give answers that require a large amount of computation/calculation in 1 second (though they could be expected to tell you how to do those calculations, and then, with further prompting, start writing some code that would actually do them)

[towforce]

Is there any intelligence in chatbots at all? I thought these are just search machines that efficiently find existing solutions or answers on the internet, and present those in a colloquial way. While intelligence is the ability to devise original solutions and draw conclusions.

One more thought: I don't think there have ever been any truly "original" solutions: I think that all solutions have, in some way, been derivative from previous solutions.

I actually believe that in today's world, copyright and patent protection periods should be shortened from what they have been historically: however, while I believe that this is a very important discussion, it obviously doesn't belong in this thread.

Finally, if a chatbot is asked to write, say, a poem or a story, by some definitions (including legal: in the claim that Dan Brown's book "Da Vinci Code" copied ideas from the earlier book "Holy Blood Holy Grail", the case failed in a British court ("Holy Blood Holy Grail" is a British book) because while the ideas were very obviously copied, no actual text was), it does come up with an "original solution".

[towforce]

Moving this thread to forum 14 was a negative indicator on the quality of the moderation team.

If you see chess as just a fun game (though obviously a popular game), and computer chess as just another hobby, like collecting stamps, then the moderation decision would have been correct.

If, as many members do, you see computer chess as having value in terms of learning about the software, technology and patterns of progress involved in generating intelligent behaviour in machines, then the decision was not correct.

[towforce]

Moving this thread to forum 14 was a negative indicator on the quality of the moderation team.

If you see chess as just a fun game (though obviously a popular game), and computer chess as just another hobby, like collecting stamps, then the moderation decision would have been correct.

If, as many members do, you see computer chess as having value in terms of learning about the software, technology and patterns of progress involved in generating intelligent behaviour in machines, then the decision was not correct.

The mitigating circumstances: I've noticed in other social media, not related to chess, that older people are very vocal in their negativity towards chatbots. Their favourite thing to say is some variation on the theme that they're not useful (which is obviously wrong - even at today's chatbot level).

I completely accept that older people tend to be more hostile to change than younger ones, and all three of the moderators are of retirement age. However, having experienced the almost entire history of computer chess close up, it's disappointing that they would be negative towards a discussion comparing the progress of computer chess with the progress of chatbots: artificial intelligence has always been a big part of why many people are interested in computer chess: in contrast to other areas of intelligent behaviour, chess has many metrics which can be measured to a useful level of accuracy, and the contained nature of chess knowledge has always allowed progress to be made on a regular basis, so of course it's a good place to get some AI insight.

Chatbots have been a part of AI since Alan Turing wrote his well known "Turing Test" (which was actually before any chatbot actually existed!).

[smatovic]

[...]
1. Chatbots can only be expected to know what is possible to know at today's level of knowledge
[...]

This is based on the assumption of supervised learning, meanwhile big tech is running out of human generated data, and meanwhile human data is generative AI polluted, so the next step in line is some kind of reinforcement learning for generative AI (we already had such a transition in Go and Chess). IMO this is the part where funny things can start to happen....when the machine starts to teach itself.

--
Srdja

[towforce]

This is based on the assumption of supervised learning, meanwhile big tech is running out of human generated data, and meanwhile human data is generative AI polluted, so the next step in line is some kind of reinforcement learning for generative AI (we already had such a transition in Go and Chess). IMO this is the part where funny things can start to happen....when the machine starts to teach itself.

I agree. A similar question is whether LLMs have "emergent properties" (link) that act as a force multiplier to their intelligence. For me, the answer is "yes": I believe LLMs to be more intelligent than they would be if they did not have them. They sometimes hallucinate, but they also, IMO, give good answers to questions which are not in their training data more often than they would do if no emergence had happened.

Quickly looking for evidence, I found this section of a wiki article - link. In that section, I found:

* on the positive side, good patterns containing generalised models of the world do appear in LLMs

* on the negative side, it is shown that sometimes, what looks like highly intelligent work, turns out to have been caused by reasoning shortcuts

Regarding "reasoning shortcuts": in the early days of chatbots (starting with Eliza in 1966), people were more impressed than they ought to have been because "...human judges are so ready to give the benefit of the doubt when conversational responses are capable of being interpreted as intelligent" (from link).

A quick chess thought (my opinion): the metrics measured by clauses of code in an HCE (hand coded evaluation) can fairly be described as "reasoning shortcuts", because they run very quickly, and they give some guidance as to the quality of white's position, but they don't genuinely measure that quality in a way that a sophisticated understanding of chess would, and they are known to not be useful in many positions.

[chrisw]

Moving this thread to forum 14 was a negative indicator on the quality of the moderation team.

If you see chess as just a fun game (though obviously a popular game), and computer chess as just another hobby, like collecting stamps, then the moderation decision would have been correct.

If, as many members do, you see computer chess as having value in terms of learning about the software, technology and patterns of progress involved in generating intelligent behaviour in machines, then the decision was not correct.

A total of nine "I, me, myself and I" in the starting post plus the title, takes it right through the limit for narcissism index and indicate it's not really about chess or chatbots but the OP. KIndergarden for flagrant insertion of self, since you ask.

[Ras]

Is there any intelligence in chatbots at all? I thought these are just search machines that efficiently find existing solutions or answers on the internet

Not even that. What they actually are is stochastic parrots. They just predict, depending on the context as captured in large NNs, what next word would probably follow after this word without understanding anything or verifying it. That's also why they "hallucinate". They mimic the form while not having any understanding of the content. There is no "intelligence" in this so-called "AI".