How to Test A New Kind of Chess AI?

[mhull]

What would be the policy of the uniform platform testing groups if a new chess "engine" was made available that didn't use external books, nor would it permit any other entity (like a GUI) to make it's opening moves for it but insisted insisted on making all chess moves from its own "mind"?

Could/would the testing groups still benchmark the AI? If so, how would they do it?

[Graham Banks]

What would be the policy of the uniform platform testing groups if a new chess "engine" was made available that didn't use external books, nor would it permit any other entity (like a GUI) to make it's opening moves for it but insisted insisted on making all chess moves from its own "mind"?

Could/would the testing groups still benchmark the AI? If so, how would they do it?

1. It would have to be tagged as not using an opening book.

2. Any duplicated games against the same opponent would be removed from the equation.

3. If it "learnt" from games that it played and that learning couldn't be turned off, I'd suggest that the testing groups couldn't accurately rate it.

Just my initial thoughts.

[mhull]

What would be the policy of the uniform platform testing groups if a new chess "engine" was made available that didn't use external books, nor would it permit any other entity (like a GUI) to make it's opening moves for it but insisted insisted on making all chess moves from its own "mind"?

Could/would the testing groups still benchmark the AI? If so, how would they do it?

1. It would have to be tagged as not using an opening book.

2. Any duplicated games against the same opponent would be removed from the equation.

3. If it "learnt" from games that it played and that learning couldn't be turned off, I'd suggest that the testing groups couldn't accurately rate it.

Just my initial thoughts.

Thank you for your thoughts on this, Graham.

I'm thinking about a chess "engine" that behaves like a true AI such that it thinks for itself in every aspect that a human would think in a game. In that regard it would also think continually on the opponent's time just like natural intelligences would and it would "learn" to the best of its ability in like manner. This would not be able to be turned off. As for determining a performance rating, it would be rated the same way as a natural intelligence, using real games for which all moves are the responsibility of the AI, which means no external program playing some of the moves and no books chosen by a different intelligence than itself.

So, I wonder if there would be any interest in forming a chess AI league where authors would submit projects that:

1) Determine openings by themselves including learning and building their own opening repertoire from scratch (perhaps like Crafty makes a book from a vast PGN file). All AIs would use the same PGN input, and make their own books from it, modifying it on their own by what they learn from games played over time. Or optionally, authors would be allowed to supply their own PGN input file, but after that, the AIs would be on their own.

2) AIs would learn from non-opening game play also.

3) AIs would think on the opponents time like natural intelligences.

4) Optional: Detect and appropriate whatever hardware it can see.

The goal of such a league would be to encourage computer chess development in traditionally weak areas for computers, by treating the chess AI like a natural intelligence instead of like a specialized utility program that can only search but otherwise cannot think for itself.

[Nelson Hernandez]

Seems a bit bass-ackwards. How about going in the totally opposite direction and using a massive opening database which would be used in much the same way as you propose, only thinking about lines rather than just deciding on the next move?

[Aser Huerga]

Could be interesting:

Sixth Cajastur “Mamdani” Prize for Soft Computing awarded to Dr. David Fogel
Dr. Fogel created the first computer program capable of learning to play chess without human intervention, achieving a playing level on par with Grandmaster, the highest ranking for human chess players.

http://www.natural-selection.com/press/ ... 32012.html

[Don]

What would be the policy of the uniform platform testing groups if a new chess "engine" was made available that didn't use external books, nor would it permit any other entity (like a GUI) to make it's opening moves for it but insisted insisted on making all chess moves from its own "mind"?

Could/would the testing groups still benchmark the AI? If so, how would they do it?

1. It would have to be tagged as not using an opening book.

2. Any duplicated games against the same opponent would be removed from the equation.

3. If it "learnt" from games that it played and that learning couldn't be turned off, I'd suggest that the testing groups couldn't accurately rate it.

Just my initial thoughts.

Thank you for your thoughts on this, Graham.

I'm thinking about a chess "engine" that behaves like a true AI such that it thinks for itself in every aspect that a human would think in a game.

A true AI like a human will sometimes memorize opening moves. Also a true AI has the ability to learn from it's mistakes - I would say it's not really a true AI otherwise.

What you would get are engines that will play the same moves over and over again. For example one engine might always play 1. e4 on the first move, another might always play 1. d4. And they will tend the same few opening systems over and over and over.

However I think most modern programs will not play the same games, they will vary sooner or later. I think I could play komodo vs komodo with no book and probably get dozens of different games - but probably I would get many that did not vary for many moves.

Give it a try and see what happens! It might be fun.

In that regard it would also think continually on the opponent's time just like natural intelligences would and it would "learn" to the best of its ability in like manner. This would not be able to be turned off. As for determining a performance rating, it would be rated the same way as a natural intelligence, using real games for which all moves are the responsibility of the AI, which means no external program playing some of the moves and no books chosen by a different intelligence than itself.

So, I wonder if there would be any interest in forming a chess AI league where authors would submit projects that:

1) Determine openings by themselves including learning and building their own opening repertoire from scratch (perhaps like Crafty makes a book from a vast PGN file). All AIs would use the same PGN input, and make their own books from it, modifying it on their own by what they learn from games played over time. Or optionally, authors would be allowed to supply their own PGN input file, but after that, the AIs would be on their own.

2) AIs would learn from non-opening game play also.

3) AIs would think on the opponents time like natural intelligences.

4) Optional: Detect and appropriate whatever hardware it can see.

The goal of such a league would be to encourage computer chess development in traditionally weak areas for computers, by treating the chess AI like a natural intelligence instead of like a specialized utility program that can only search but otherwise cannot think for itself.