A different way to develop chess engines

[Uri Blass]

people who develop chess engines usually test it in many games.

I thought people may use a different process.

1)Develop some first version.
2)Give it to play games at some fixed time control(the program lose because it do mistakes)
3)find the mistakes when you can use strong engines for it.
4)Add the mistakes to a test suite that the program need to avoid(in the first case when you are in step 4 the test suite is empty and later it walways get bigger).
5)Change the program to avoid all the mistakes in the test suite.
6)repeat to stage 2.

The program also may become weaker after step 5 and you do not test it in games but after enough changes
I expect it to become stronger because if the program avoid many thousands of mistakes(not by specific opening book but by many general rules) then it can almost never go wrong.

Of course it is a lot of work to do step 5 but I wonder if there are programmers who tried to follow this path.

[brianr]

Your approach has indeed been done for several decades.
It works at first, but then it becomes clearer that tuning for a test set is not the same as actual game strength.
Recall the early focus on WAC 300 scores...

[mclane]

I do measure progress by putting new engine in a outplayed tournament.
If so it plays against all other engines and ranks automatically.

I do no matches or test suites. Just take out an arena tournament, put in the. We engine and let it play against all others x games.
It will find the ranking for the selected engines,

[Uri Blass]

Your approach has indeed been done for several decades.
It works at first, but then it becomes clearer that tuning for a test set is not the same as actual game strength.
Recall the early focus on WAC 300 scores...

The idea is that the test suite is not going to be WAC 300 but based on the positions that the engine went wrong when you add more and more position to the test suite when the target is always to be 100% correct let say in 10 seconds per move.

Of course it is a lot of work to get the program 100% correct not by a special opening book for the specific positions but by a better general evaluation or general search rules.

[brianr]

The idea is that the test suite is not going to be WAC 300 but based on the positions that the engine went wrong...

I think you missed my point.

Test suites generally are minimally helpful; it does not matter how many positions there are.
Actual game play against several opponents (in addition to more frequent self-play) is required.

[Uri Blass]

The idea is that the test suite is not going to be WAC 300 but based on the positions that the engine went wrong...

I think you missed my point.

Test suites generally are minimally helpful; it does not matter how many positions there are.
Actual game play against several opponents (in addition to more frequent self-play) is required.

My point is that if you know to play the best move in every position you also win games so finally by optimizing for test suites that is big enough you are going to play well(because you play very good moves in many positions that is close to play the best move in every position).

Of course this is a long process that may take many years when sometimes the engine may be weaker relative to previous version but I expect the engine to get better(probably more slowly relative to testing it in games).

[brianr]

Your approach is theoretically sound, but not practical for human "hand crafted" changes.

It is pretty much what net training is doing with now into the billions of positions, using automated processes of course.

[Desperado]

people who develop chess engines usually test it in many games.

I thought people may use a different process.

1)Develop some first version.
2)Give it to play games at some fixed time control(the program lose because it do mistakes)
3)find the mistakes when you can use strong engines for it.
4)Add the mistakes to a test suite that the program need to avoid(in the first case when you are in step 4 the test suite is empty and later it walways get bigger).
5)Change the program to avoid all the mistakes in the test suite.
6)repeat to stage 2.

The program also may become weaker after step 5 and you do not test it in games but after enough changes
I expect it to become stronger because if the program avoid many thousands of mistakes(not by specific opening book but by many general rules) then it can almost never go wrong.

Of course it is a lot of work to do step 5 but I wonder if there are programmers who tried to follow this path.

Hi,

this approach is in use already, but it only is an additional tool before playing games. It might speed up things.

Well, today the Texel Tuning method is based on computing millions of positions and minimizing the error of the evaluation function,
to change things. Many Programmers don't use human games or a database like CCRL games, but they store games played by the engine create
samples out of it. The fitness function is based on evaluation error, but of course you can combine it with move statistics or whatever you like.

So, creating an EPD-Suite out of games played by the own engine and learn from it in some way is already done.
Finally this progress (if there is one) needs to be confirmed by a meaningful number of played games.

Regards.