Why not a linkable interface?

[stegemma]

In my genetical tuning, I need to let two version of my engine play together, with different parameters sets. To test the final engine, I would like to make it play against other engines too. Doing this with WinBoard or Arena is enough good but it would be better if I could call engines with a linkable interface or other direct methods. Intercepting stdin/stdout is a pain and very bad portable among different systems. For sample, on Windows is not very difficult to provide a DLL interface to any C/C++/VB software. On Linux/Unix/Mac shared libraries could be used. I could write an open-source core DLL/library to help giving our softwares such kind of interface, if anybody is interested.

Such kind of standard interface would be great for graphical interface programmers, I think.

Of course the "linkable interface" itself can run any engine in a different thread, to avoid any kind of lock.

Another way could be to define a standard socket interface, even for this I could provide a portable solution. of course one can talk about named-pipes, shared files and any other method that I don't know but the fact is that the stdin/stdout usage is a so old interface that maybe we should search for something better... or not?

[hgm]

stdin/stdout interface should be considered vastly superior over direct linking (as Fritz and TMCI do) and socket-based (e.g. DXP or CSA protocol) connections. For one, it doesn't burden the engine author with knowing about sockets or dynamic linking; the most basic printf/scanf is all you have to know. Direct linking furthermore suffers from strong platform dependence.

The goal should be to keep the engine side simple, and let the GUIs, which will be much less numerous and very complex anyway, pay the price for this. Setting up pipes for a forked-off process is not that big a deal, btw; UCI2WB measures less than 700 lines of code in total, and can do it on Windows, Linux and Mac. (In fact the StartEngine() routine only takes 82 lines of this.) The code for this is public and so elementary that it is not copyrightable, so any C/C++ programmer that wants to interface with engines could just copy-paste it.

[Michel]

stdin/stdout interface should be considered vastly superior over direct linking (as Fritz and TMCI do) and socket-based (e.g. DXP or CSA protocol) connections. For one, it doesn't burden the engine author with knowing about sockets or dynamic linking; the most basic printf/scanf is all you have to know. Direct linking furthermore suffers from strong platform dependence.

The goal should be to keep the engine side simple, and let the GUIs, which will be much less numerous and very complex anyway, pay the price for this. Setting up pipes for a forked-off process is not that big a deal, btw; UCI2WB measures less than 700 lines of code in total, and can do it on Windows, Linux and Mac. (In fact the StartEngine() routine only takes 82 lines of this.) The code for this is public and so elementary that it is not copyrightable, so any C/C++ programmer that wants to interface with engines could just copy-paste it.

As usual there are two sides to this.

Using a shared library approach relieves the engine author from having to implement any command line parsing (I mean parsing the UCI or CECP protocol)... Command line parsing is something nobody likes doing.

Making a shared library is trivial (compile with -shared...). So from this point of view it looks like a good deal.

Maybe in the end a command line approach is a bit more flexible, but saying that it is vastly superior is an overstatement.

[mar]

I see a couple of problems with this:

- how would you load two same engine instances using a shared library? (could be handled by the interface but that would require the author to support multiple instances of the engine)

- safety. what happens if the engine crashes (or even worse accidentally owerwrites some part of GUI data), the GUI then goes down as well

- what happens if the library doesn't clean up properly when exiting? (memory leaks)

- CRT (how do you ensure that there won't be any problems if the engine uses another version of CRT than the GUI, assuming both use dynamic CRT) it may work or not, you could also link CRT statically

[hgm]

@Michel:
That reasoning uses properties of the high-level protocol to judge the lower-level connection method. How you connect is an issue that is quite separate from what you send over the connection. stdin/stdout-based connections could also just start every command they send with the number of a routine, followed by a list of parameters that should be passed to that routine. Either as text or in binary. And direct linking could be used to call a routine with a text-string argument that contains a command to be parsed.

I never found the task of parsing the command line a real burdon. It just looks like

Code: Select all

sscanf(inBuf, "%s", command);
if(!strcmp(command, "new")) {} else
if(!strcmp(command, "force")) {} else
if(!strcmp(command, "go")) {} else
if(!strcmp(command, "time")) { t= atoi(inBuf+5); } else
...

This does not seem essentially different from writing

Code: Select all

void DoNew() {}
void DoForce() {}
void DoGo () {}
void DoTime(int t) {}
...

with the same {} parts, as you would need in the DLL case.

[Michel]

- safety. what happens if the engine crashes (or even worse accidentally owerwrites some part of GUI data), the GUI then goes down as well

Yes I hadn't thought of that. The fact that the engine would be able to take down the GUI rules out this approach.

[hgm]

That definitely seems a big disadvantage. But apparently not enough to discourage ChessBase from using that method. Third Millenium Chess Interface would suffer from it too, but fortunately XBoard only uses TCMI engines through an adapter. So it would just crash the adapter.

[Michel]

@Michel:
That reasoning uses properties of the high-level protocol to judge the lower-level connection method. How you connect is an issue that is quite separate from what you send over the connection. stdin/stdout-based connections could also just start every command they send with the number of a routine, followed by a list of parameters that should be passed to that routine. Either as text or in binary. And direct linking could be used to call a routine with a text-string argument that contains a command to be parsed.

I objected to your claim that the command line method was vastly superior. Now you seem to say the two methods are more or less equivalent from the POV of the engine author. This agrees with my opinion.

But I also agree now that from the GUI POV dynamic linking is not a good option for safety reasons (see Martin's post).

[hgm]

The equivalence is for the high-level protocol, which is independent of the conection method, and thus should be logically equivalent. It just depends on what command keywords you have / what entry points you require.

I recall having once made a DLL for windows, and the process was a lot more complex that just adding a -shared option to the compiler. And can you use Windows DLLs under Linux? I know you can use Windows console programs there under wine.

[stegemma]

The equivalence is for the high-level protocol, which is independent of the conection method, and thus should be logically equivalent. It just depends on what command keywords you have / what entry points you require.

I recall having once made a DLL for windows, and the process was a lot more complex that just adding a -shared option to the compiler. And can you use Windows DLLs under Linux? I know you can use Windows console programs there under wine.

The problem is not about complexity but about flexibility. I would like to have an interface between engines usable for programmers, more than for users. It can be done just writing an interface around the chosen technology, that programmers can simply call. To avoid stability issue, we could use the most portable solution (IMHO) that is sockets. This would helps the software to play through local net or internet and a programmer can challenge another one just providing its IP address, directly, without the needs of an interface. A bugged software as were mine, could easily call another one to ask it for the perft of a position (remember my qpertft encapsulation?) or to compare evaluation function results while playing. My genetical algorithm could do matches against any software, to test the goodness of the parameters, in "real time", without the needs of the interface to complete a tournament.

Of course this are my expectations about a modern interface between softwares but to becomes a "standard" a bigger number of programmers should agree.