Testing AI engines

[Gabor Szots]

Accepting that AI engines are the future, I have questions from a tester's point of view.

To begin with, I have been living in the assumption that the goal of the rating lists is to compare the skills of the authors. Now with AI generated engines that goal seems obsolete because there is no author in the traditional sense.

So:

1. If I decide to test an AI engine whom should I credit? Claude? Some other AI program?
2. Which AI engine should I test? They are all gathering the same knowledge, as I see it there should be no difference between them.
3. Has copyright and licencing become obsolete?

[Graham Banks]

I've always thought that the goal of the rating lists is to compare the ratings of engines.

Having said that, none of them would exist without the efforts of their authors.

[Frank Quisinsky]

Gabor,

good questions!

The good old days of testing engines, at least with the feeling that they play differently because they were created individually, are over. So, that are the reason for ETOC-G. I need for myself a successful conclusion to my activities in this regard.

What we have are all the good known programmers, working since years hard on here programs. At the moment I build with the ETOC-G results my own favorits for future testing.

Best
Frank


Graham,

that's right, but if a large number of engines play the same chess, one has to wonder what the point is. In cases like these, long-term developments tend to fall by the wayside, and I think that's a real shame. I am thinking a long time about it without success. Means, what can I do? You should know that AI is a major topic at many schools and universities. And chess can be used to launch some great projects. It's absolutely clear that the market is being flooded with AI programs. Many of these programs might be updated 1–5 times, and then you'll never hear from the developers again.

Often I am thinking:
I like to test engines, three years available and a steady trend can be observed. But ideas like that are difficult to implement because there are simply far too many chess programs.

Another way:
- name of the programmer
- country of the programmer
- age of the programmer
- school / univeristiy project yes / no
- interest on a long time development yes / no

And maybe other things for the decission: I add the engine in a rating list or not

But here is the problem:
A lot of that is really none of our business, better interest.
We are not allowed to request such information.

Again, I am thinking the good and old time is over.
Everyone have to build his own favorits for interesting tourneys in the future.

And excalty this one is a good point.
Many available engines are good (not bad) but thats the real life. Not all AI given us is great.
How many companies have to search another way or the companies go out of business. That's normal in today's IT world.

With AI we lose an important point.
The engines playing styles are becoming more similar, there are exceptions.
And exactly the playing styles was the icing on the cake.

Best
Frank

[smatovic]

People call the current phase of AI use in programming the centaur phase, cos human and AI work together, and achieve better/faster results than human or AI alone. Some programmers report 5x to 10x throughput increase in programming with LLMs, so it is not so much about to produce better code or results, but to achieve these faster. Some people argue that LLMs do not produce new ideas, I disagree, there are reports about emergence in ideas and code. Even if not, AI can increase throughput on a level to parse amounts of present ideas/code, and merge them to something new. So if you test now centaur-engines, then you test the human+AI in teamwork, you test the ability of the LLM and you test the ability of the human to utilize this LLM for a specific task in a specific domain.

How long will the centaur-engines phase last? When will we see fully automated LLMs generating top engines on their own*? Idk. But then it is obvious that you test the ability of the AI alone, and further the ability of the team that created that AI. So one day we can imagine an AI-engine vs. AI-engine tournament, where different AI engineering teams compete with each other.

On copyright/licenses, IMO this is work in progress, different topics are in the courts to settle. Is it fair use to grab all source code out there to train LLMs? Can the output of an LLM be protected by copyright? Generally, code you can protect with copyright, technical implementations you can protect with patents, ideas/thoughts/math are free to use.

--
Srdja

PS: *: LLMs do already generate chess engines on their own, but up to now they are not competitive, as mentioned, we are still in the centaur phase: YATT: viewtopic.php?p=965578#p965578

[Frank Quisinsky]

A vision:

Any gentleman who wishes to apply for an IT position must include his own AI-generated chess program with his application. The companies test it, and that is an important factor in their decision.

Really a vision?
I know an entrepreneur, who is an avid chess player, and is thinking about it.

Nothing is impossible!!

[op12no2]

Having said that, none of them would exist without the efforts of their authors.

yet

[Frank Quisinsky]

smatovic:

Does this matter have to be settled in court? That would take years, and even then, it’s unclear whether anything meaningful would come of it. It’s likely that, over the course of the proceedings, x new types of licenses will be introduced, all of which will be obsolete by the very next day. It’s difficult, but I don’t see any other option.

Here’s what’s interesting:
A graphical user interface that allows users to create an AI engine, one that offers many features, such as the ability for users to design the training process themselves. Any computer chess fan who’s interested could then develop their own program, and these programs could subsequently compete against each other in tournaments. There’s no need to form groups for this, I think this is what the future looks like.

Our great programmers, must be faster which such things.
Faster as KI, we always have to stay one step ahead or we'll lose face.

[mar]

People call the current phase of AI use in programming the centaur phase, cos human and AI work together, and achieve better/faster results than human or AI alone. Some programmers report 5x to 10x throughput increase in programming with LLMs, so it is not so much about to produce better code or results, but to achieve these faster.

I recently found a funny title: "who will clean up after the vibe coding party" - which exactly sums it up I think

"AI" is good at producing low quality garbage very quickly, however the 10x productivity myth has been debunked too,
I've read a study where a group of AI-assisted devs were actually 20% slower than the unassisted ones

my take on this is that quality matters more than anything in the long run

while LLMs are great at understanding the language, they're not sentient, not AGI, not intelligent.
a stochastic slop generator with feedback and compressed human knowledge seems cool until you get sober

[Frank Quisinsky]

No doubt about it ...
Myracle is for me the best available chess GUI.

To the programmer of Myracle, if he reads the thread, ... sorry my test follow later, the weather is the reason. I am happy, if my ETOC-G is still running and the organization for my chess computer tourney is over.

Myracle is such a GUI.
All the good known programming technics should be available in a GUI to create an engine. I know that are hundreds. User can select what they like and can start after with a training of the new engine. All should be possible with one GUI.

With the final results:
We're always one step ahead of AI.

Around the year 1997 I had a discussion about it with Dieter Bürßner (Yace) in Gambit-Soft forum. I wrote him, it would be nice if the programmers create differnt modules, users can combining them into a chess-program. His answer was: Not possible!

Nothing is impossible, today that must be the plan for the future.

Best
Frank

[Frank Quisinsky]

Before missunderstandings:

Idea with the moduls:
Based on CHE ... programming language by Dr. Chrilly Donninger (programmer of NIMZO, some other exp. engines). Around that time are Nimzo 3.0 with own GUI available. Some nice ideas by Chrilly, CHE was only one. So I am thinking why not for Winboard engines, means the moduls I am speaking before. Dieter Bürßner answered in such a thread in the very old Gambit-Soft forum. Here I am admin and Gambit-Soft was the first German Chess Forum.