CCRL Engine Authors' Tournament (will start October 1st)

[Sven]

ChessGUI expects the engines to look at the full-move number of FENs to know when a new session starts?

That certainly breaks all of my engines!

I always assumed that all serious testing is done by feeding moves, and never by starting from positions. I often don't even bother to equip them with setboard.

Even with setboard it can work. In Jumbo (as in all my previous engines) I ignore the full move number given through setboard, and if I wouldn't ignore it then I would certainly not assume that some of the moves that belong to a time control session have already been played. So if I get a "level 40 X Y" then the number of remaining moves starts at 40 for me no matter what comes in setboard.

Therefore, considering that Graham tested Jumbo successfully under ChessGUI (see his post above), I'm not sure whether we already know all about this problem ...

[D Sceviour]

ChessGUI expects the engines to look at the full-move number of FENs to know when a new session starts?

That certainly breaks all of my engines!

I always assumed that all serious testing is done by feeding moves, and never by starting from positions. I often don't even bother to equip them with setboard.

Yes. There is no requirement in CECP for the move number to be read from the setboard command. The opening line from the Schooner-Gogobello game posted above was discovered in the debug file:

Code: Select all

SideToMove = White.
SendToEng1Time 0000000396849090 : Eng01 (Schooner 1.5.7 64-bit) -> setboard rn1q1rk1/pbpp1pbp/1p3np1/3p4/2P5/P1N2NP1/1P2PPBP/R1BQK2R w KQ - 0 9
.
nChessVariant(0) = 1.

In my opinion, the tournament level should be 32/25 and not 40 moves in 25 minutes repeating for ChessGUI. It should be up to the GUI to adjust this information and not up to the CECP compliant engine. The CCRL testing committee should post a formal decision about this. Clearly, it is affecting many engines.

[hgm]

..., and if I wouldn't ignore it then I would certainly not assume that some of the moves that belong to a time control session have already been played.

What else could it be used for?

So if I get a "level 40 X Y" then the number of remaining moves starts at 40 for me no matter what comes in setboard.

Yet subtracting it from the first session length does seem the proper thing to do. Unlike what Dennis claims the CECP specs say nothing about ignoring the full-move counter.

The point is that with WinBoard/XBoard it would never happen.Because when you start from a set-up position WinBoard would always store that as ply 0 or 1 (with black to move) of its internal game history.Soalthough its FEN generator faithfully writes the move counter, it will be based on the GUI's game history. Pasting the FEN back into the GUI would not (and cannot) restore that entire game history, and would be taken as move 1 no matter what of the new game.

So it is basically just a matter of GUI implementation, not of protocol. The WinBoard implementation happens to be such that it only would ever send FENs with full-move counter = 1. I expect most WB engines are only designed to handle this case.

Therefore, considering that Graham tested Jumbo successfully under ChessGUI (see his post above), I'm not sure whether we already know all about this problem ...

Actually I doubt that Graham starts from positions rather than opening lines. There are many WB engines that cannot load positions.

But there are many WB engines that also fail to count moves entered in force mode as belonging to a classical TC session. In fact that is a true epidemic. They show exactly the described behavior, expecting new time when they don't get it, leaving themselves without any significant time for most of the next session.

[D Sceviour]

On further thought, what Chess GUI is doing with setboard may be illegal under the rules of chess. By ignoring the opening sequence of moves, no information concerning the 3-move draw rule and the 50-move rule are available.

[Graham Banks]

Clearly, it is affecting many engines.

It doesn't actually because I always check whether or not there are any such issues. Schooner is the only engine I've come across for a very long while.
I would not test an engine with such a problem for CCRL for obvious reasons.

I'll leave it to Matthias to talk about what ChessGUI does or doesn't do.
However, all of my testing is done using this GUI and it is fantastic for engine v engine testing.

I use an opening book, not test suites. In this competition, there is a maximum depth of 8 moves from this book.

Engine authors are very quick to let me know if they suspect any issues with their engine. That's why debug files can be handy.

[D Sceviour]

Clearly, it is affecting many engines.

It doesn't actually because I always check whether or not there are any such issues. Schooner is the only engine I've come across for a very long while.
I would not test an engine with such a problem for CCRL for obvious reasons.

I'll leave it to Matthias to talk about what ChessGUI does or doesn't do.
However, all of my testing is done using this GUI and it is fantastic for engine v engine testing.

I use an opening book, not test suites. In this competition, there is a maximum depth of 8 moves from this book.

Engine authors are very quick to let me know if they suspect any issues with their engine. That's why debug files can be handy.

Graham, you do a fantastic job. Everyone is very appreciative of the tournaments you run. However, there are issues arising concerning ChessGUI's use of the setboard command for xboard based engines. Of course, this is no fault of yours.

[Modern Times]

I have been a heavy user of ChessGUI as well, and I agree with Graham I do not recall seeing this problem with any engine. So it seems that it may just be an issue with Schooner under this GUI.

[hgm]

I use an opening book, not test suites. In this competition, there is a maximum depth of 8 moves from this book.

With an opening book there would never be any need to send setboard commands to the engine.

Where does this idea come from that Schooner would have received setboard commands from ChessGUI? Can someone show the log file where this happened?

[D Sceviour]

I use an opening book, not test suites. In this competition, there is a maximum depth of 8 moves from this book.

With an opening book there would never be any need to send setboard commands to the engine.

Where does this idea come from that Schooner would have received setboard commands from ChessGUI? Can someone show the log file where this happened?

http://www.talkchess.com/forum/viewtopi ... 77&t=60990

[hgm]

OK, I see. This is very strange indeed. A GUI should never do that, when starting games from a book. It should always feed the moves of the opening line.

I am sure this would break many engines, not only Schooner. It would certainly break mine, and Sven just confirmed it would also break his. In fact I have severe doubts if there would be any WB engine it would not break.

When starting from a FEN suite in classical TC, the move counters of the FENs should all be at 1, if the GUI does not correct them so.

Results obtained for WB engines with ChessGUI in this book mode and at classical TC should be considered severely corrupted, as it wrecks their time management.