unbuffered input/ouput

[sje]

The existing world has to be shaken once in a while. Otherwise, we will be stuck with dinosaurs like xboard that have unnecessary and unused features that are decades old, like having to deal with an ancient gnuchess program. That unneeded complexity hurts because everyone who wants to use a dinosaur program has to read through all the documentation just to understand what needs to be ignored.

Also, many difficulties people have with certain chess GUIs are due to the fact the functions have grown far beyond what are necessary for a GUI. Some GUIs have taken on the tasks of program control (forcing all kinds of extra protocol), of handing program/program matches (more protocol) of handing server communication (more protocol), handling opening books, handing tablebases, and who knows what else. This is not the way to do software engineering.

[Sven]

The existing world has to be shaken once in a while. Otherwise, we will be stuck with dinosaurs like xboard that have unnecessary and unused features that are decades old, like having to deal with an ancient gnuchess program. That unneeded complexity hurts because everyone who wants to use a dinosaur program has to read through all the documentation just to understand what needs to be ignored.

Also, many difficulties people have with certain chess GUIs are due to the fact the functions have grown far beyond what are necessary for a GUI. Some GUIs have taken on the tasks of program control (forcing all kinds of extra protocol), of handing program/program matches (more protocol) of handing server communication (more protocol), handling opening books, handing tablebases, and who knows what else. This is not the way to do software engineering.

xboard, WinBoard, gnuchess compatibilty, ... point accepted!

But throwing away ugly solutions is also no way of "software engineering". In an ideal world you can throw away what you consider as ugly, and replace it. But in the real world you have to consider the "customers", too.

Offer a better protocol spec, then we might get a better GUI implementation, and only then we can start to switch the engines to a better standard.

Nevertheless, whether you like it or not: the WinBoard/xboard GUI of today is quite good in terms of usabilty, for both end-users and programmers - good enough to have about 350-400 different WinBoard engines compared to "only" 100-150 UCI engines. (Numbers quickly estimated based on current RWBC list) That is not quite like being old-fashioned, is it?

[I had to say that, at least for Roger But believe me, Roger, I also said it in the belief that it is true.]

Sven

[bob]

The only issue becomes one of messages. If you use send/recv and you expect message boundaries to be preserved as they are when you send and the other end recv's... it won't work correctly if one end uses send and the other end uses read.

There have been many cases where send/recv, applied to a non-socket, can fail. Hence my recommendation to _always_ use read/write, which work for any file descriptor.

Hmmm, so it is in fact easy to use send()/recv()?

I've not been pushing "send()/recv()" if you have been reading my posts. I have been pushing read()/write(). Those work. For _all_ cases.

c) This (a+b) will also work with PolyGlot.

d) This (a+b+c) will also work on our current main platform which is Windows.

Don't know nor care there.

O.k., "WinBoard only" for the moment, if you prefer ...

In xboard, you can run on a local machine (xboard spawns two new processes and serves as intermediary between the two chess engines using pipes. Or you can use the -fh or -sh (or both) and make xboard create sockets, connect them to engines running on remote machines, and there are zero changes to the engine. They have no idea whether they are reading from a socket, a pipe, or two cans connected by a string...

-fh/-sh also for connecting to a local engine? See my comment further down.

I suppose you can use -fh (first host) or-sh (second host) to connect to the local host, which would use a socket rather than the usual pipe... Wouldn't make a bit of difference, performance-wise, or program-wise, since the program doesn't have a clue and just reads stdin and writes stdout.

Can you, or someone else, show example C code that creates and uses a socket such that you can read the data being written by another process into a pipe, and get the same result as if you were reading directly from the standard input file descriptor, or the "stdin" stream?

Xboard does this, as I mentioned. You can use the pipe() system call to create 2 new and open file descriptors. The first pipe call will return 3 and 4 for the descriptor numbers. You can then create a new process that inherits those open descriptors, just as xboard currently does, and then uses dup2() to stuff one of those on top of stdin. When xboard writes to the other end, the chess engine reads from stdin, which is now that open pipe, and it gets the data xboard wrote. If you use the -fh or -sh, xboard instead creates a socket, connects it to a remote machine, and then stuffs that on top of stdin for the chess engine. And again, the chess engine reads from xboard not knowing whether it is a socket, a pipe, or any other other I/O capable "thing".

In my opinion it does not work this way. If an engine wants to read via a socket then it needs a partner who writes into it, and that must be the GUI. I think you cannot read data from a pipe created by your parent process by reading via a socket using recv().

Who mentioned that? A pipe is a pipe, a socket is a socket, and a file is a file. But with read or write, I can't tell them apart. If two programs (A & B) are going to talk to each other, they can use (1) a pipe; (2) a socket; (3) a FIFO (looks like a file, but is a pipe); etc. But both have to talk over the same resource. The GUI can choose whether that resource is a pipe, or a socket, or a FIFO. The GUI then sets stdin / stdout up, just as it does today. And the rest is automatic.

I'm simply telling you how it actually works. A program can be forced into reading from the keyboard (default stdin setting, or from a file, by opening the file, and then stuffing that descriptor on top of stdin using dup2(), or from a pipe, by (as xboard does) creating a pipe and then stuffing the read end of the pipe's descriptor # on top of stdin, or by creating a socket, connecting it, and then stuffing _that_ descriptor on top of stdin. The program reading from stdin has no idea where the data is coming from, Or where it is going to if it is stdout.

The proposal by Steven was that the engine should use recv() to read. But this requires a socket for which there must be some counterpart.

I agree. This would mean that the GUI can not use a pipe if the other end is going to use send(). But if a GUI already allows remote execution, which xboard/winboard certainly do, then the socket mechanism is already there. Should we standardize on sockets? Doesn't matter to me since Crafty won't care one iota. I use read/write, so that the gui is free to use whichever it wants and I don't need to know. If the GUI creates a socket, and uses send() to send me commands, my read() will pick them up perfectly. Or I could switch to use recv() instead of read(). If I were to use send()/recv() (which I wouldn't since read()/write() already work) then I would have to use a socket. But I don't have to do anything to create the thing, I would _still_ just use stdin/stdout, knowing that the GUI had created a socket rather than a pipe...

Given that, it seems you are in fact talking about a possible change on the GUI side. Either the GUI creates pipes to communicate with engines, as it does now (at least WinBoard), or it creates sockets, but not both.

Xboard/winboard do either. If you look at xboard, in xboard.c, you can find the socket creation code in something like openTCP() or something similar... I do not have winboard source, but it certainly supports -fh and -sh, so it does the same thing there.

-fh/-sh mean "-firstHost" and "-secondHost" and can be used to connect to a remote engine, as you say. But here were are talking about a local engine for which you have to specify its path. WinBoard creates a pipe to each local engine.

Doesn't matter which it uses. You can make winboard use a socket. Just use the -fh or -sh option, but for the host name, use the name of your local machine. Now you are using a socket rather than a pipe. yet all works flawlessly. I play on ICC like this 24/7. xboard runs on my office box. It uses the -fh option to connect to a machine named "ferrum" that is the head of the cluster. Ferrum executes a one line shell script that ssh's a command to one of the compute nodes to actually execute Crafty. Neither xboard nor crafty are aware that both are not local on the same machine. Xboard knows to use a socket to connect. It has no idea what happens when it executes the program name supplied, on the remote machine. If it spawns yet another process, nobody knows, nor cares, just so that when Crafty reads from stdin, it gets the stuff sent from xboard, and vice-versa...

To me it seems -fh/-sh options are not designed for the purpose we are talking about here.

They are simply a way to force xboard/winboard to use a socket. Then you can test my claim that the program can't tell the difference between the two, using read()/write(). If you use send()/recv() you can tell because they won't work on a pipe.

And only after that change (which is highly unlikely) an engine can change its way of reading data.

I may be wrong, though.

Sven

you are. See above. That's been one of the cleanest parts of unix forever. That and the fact that _all_ I/O looks like files, even if you want to talk directly to a modem, or a line printer. You open a file...

Still not on topic, though ...

As can be seen from your reply, you confirm that reading from a socket needs another socket as a counterpart. And you are not showing where we get that from. So where have I been wrong?

Sven

xboard/winboard pulls that socket out of its magic hat. Your program does nothing. Once xboard decides to use a socket, rather than a pipe, because the engine is being executed on a remote host rather than a local host, it is done. Now xboard uses read/write to send data to the chess engine, not knowing nor caring whether it is using a socket or pipe, because the usage is the same.

The GUI gets to choose. You can't use a pipe to talk to a remote machine. You can use a socket to talk to local or remote engine, however...

[Sven]

The only issue becomes one of messages. If you use send/recv and you expect message boundaries to be preserved as they are when you send and the other end recv's... it won't work correctly if one end uses send and the other end uses read.

There have been many cases where send/recv, applied to a non-socket, can fail. Hence my recommendation to _always_ use read/write, which work for any file descriptor.

Hmmm, so it is in fact easy to use send()/recv()?

I've not been pushing "send()/recv()" if you have been reading my posts. I have been pushing read()/write(). Those work. For _all_ cases.

You have not been pushing it. But initially you have stated that using it would "not require a thing" Remember this:

<snip snip ...>
Switching from stdin/stdout usage to sockets or other messaging concepts would of course not require a "complete rewrite" of engines, that was not what I meant. It would require
- a new protocol specification,
- new or extended GUIs and tools,
- and some modifications/extensions in engine code.

It doesn't require a thing. A simple example, have you ever used winboard or xboard to run a local chess engine? What about an engine on a remote machine? Unix I/O does not differentiate between pipes, files, sockets, FIFOs or such at the user level. That's handled deep in the kernel. A simple fscanf() or fgets() or read() or recv() will work with any of those, without knowing which. And without requiring any changes at the program level.

You can read STDIN all you want, but you won't know whether it comes from a network socket, a local socket, a pipe, a file, the console, etc. That's the _point_ for the way Unix I/O was developed...

Now regarding these options ... Again, note that I am discussing what would be implied by introducing sockets. I know you are not pushing that. But we need to get the facts right.

To me it seems -fh/-sh options are not designed for the purpose we are talking about here.

They are simply a way to force xboard/winboard to use a socket. Then you can test my claim that the program can't tell the difference between the two, using read()/write(). If you use send()/recv() you can tell because they won't work on a pipe.

xboard/winboard pulls that socket out of its magic hat. Your program does nothing. Once xboard decides to use a socket, rather than a pipe, because the engine is being executed on a remote host rather than a local host, it is done. Now xboard uses read/write to send data to the chess engine, not knowing nor caring whether it is using a socket or pipe, because the usage is the same.

We should clarify our usage of terms first. Let's define:

"local engine" = engine which is under GUI control, i.e. the GUI knows its file system path, can start and stop it, and can communicate to it via a pipe

"remote engine" = engine which is not under GUI control, i.e. the GUI does not know its file system path and can communicate to it via network only (socket)

The key point of the second definition is, a "remote engine" can actually have a local file system path but it is unknown to the GUI. This is what the "-firstHost/-secondHost" options are about: you cannot use it with a local engine. WinBoard/xboard handle local engines by spawning a child process with two pipes for read+write, and remote engines by creating a socket. This means, with -firstHost/-secondHost the user itself has to care himself about starting and stopping the engine which is simply against the "plug and play" principle for the usual "local" use case. It certainly is a change, even if not a big one.

Presumably this is not an issue for you or for me. But the use case becomes a different one. Many WinBoard users, for instance, rely on setting all engine-specific options, including command line options, in the "winboard.ini" config file. But being forced to also care about engine start/stop, which is usually done by WinBoard automatically, goes beyond acceptance.

Considering this, the only thing to restore compatibility would be a new option that combines both features "local engine" and "socket", which does not exist yet to my knowledge. It may even be easy to add that for WinBoard/xboard. But changing an engine to use sockets and send/recv would immediately make that engine depend on that new GUI option, unless you increase the mess by making the socket usage of the engine depend on another command line option . I am pretty sure you'll agree that this would not make much sense, considering the marginal benefit of using sockets and also the "zero elo gain" aspect.


My conclusion about the socket proposal: it is certainly possible somehow but would require much more preparation.


To come back to the very first topic of this thread: using read/write instead of higher level library functions, as you propose, is certainly not "bad" at all. But in my opinion it solves non-existent problems while introducing too much low level stuff.

I have even thought about a different solution for the "254 character limit" issue. To summarize, after some testing it seems that the problem exists
- on Windows only,
- with unbuffered input only,
- and in console mode only (at least I could not reproduce the problem without console mode).

So the question arises, do we really need unbuffered input in console mode? It could be possible to switch on unbuffered input only when detecting that the standard input is connected to a pipe.

Would that make sense? Would that imply any functional restrictions for the console mode? I don't think so, the only tiny issue I could image would be that you could not expect the engine to receive commands that are typed while the search is running, so in WinBoard/xboard you could not use the "move now" ("?") command with an engine that implements classical polling. However, there is a better design based on threads that makes this tiny problem quite irrelevant.

What do you think? The question goes also to the audience ...

Sven

[rbarreira]

If you force the use of send/recv, you get message boundaries. If you don't, then you have to figure out the message boundaries for yourself, as we do today with the "\n" (line feed) character...

You cannot force message boundaries on TCP/IP sockets, since TCP/IP does not support boundaries. It's a byte-stream protocol. Datagram sockets (i.e. UDP) preserve boundaries, but that's a lossy protocol.

I've actually seen some programs which rely on TCP/IP sockets to get a whole line from a single recv(). They crashed once in a while, when the OS or routers felt like fragmenting or merging parts of the stream.

[bob]

The only issue becomes one of messages. If you use send/recv and you expect message boundaries to be preserved as they are when you send and the other end recv's... it won't work correctly if one end uses send and the other end uses read.

There have been many cases where send/recv, applied to a non-socket, can fail. Hence my recommendation to _always_ use read/write, which work for any file descriptor.

Hmmm, so it is in fact easy to use send()/recv()?

I've not been pushing "send()/recv()" if you have been reading my posts. I have been pushing read()/write(). Those work. For _all_ cases.

You have not been pushing it. But initially you have stated that using it would "not require a thing" Remember this:

If you only talk about switching from pipes to sockets, it does not require a single change to the chess engine. And since most GUIs support "remote execution" (xboard always has) the GUI already supports pipes for local processes and sockets for remote processes, so that there is no changes required there either. One can use a socket and read via read(), fscanf(), fgets() fgetc(), etc. With no problems of any kind.

So sockets vs pipes is a non-issue. A pipe is just a "local socket" and they behave the same. Unless you try to use send()/recv(). They won't work on a pipe. But I don't use 'em. Except when doing UDP where I need sendto() and recvfrom(). Otherwise, everything I do is read()/write().

<snip snip ...>
Switching from stdin/stdout usage to sockets or other messaging concepts would of course not require a "complete rewrite" of engines, that was not what I meant. It would require
- a new protocol specification,
- new or extended GUIs and tools,
- and some modifications/extensions in engine code.

It doesn't require a thing. A simple example, have you ever used winboard or xboard to run a local chess engine? What about an engine on a remote machine? Unix I/O does not differentiate between pipes, files, sockets, FIFOs or such at the user level. That's handled deep in the kernel. A simple fscanf() or fgets() or read() or recv() will work with any of those, without knowing which. And without requiring any changes at the program level.

You can read STDIN all you want, but you won't know whether it comes from a network socket, a local socket, a pipe, a file, the console, etc. That's the _point_ for the way Unix I/O was developed...

Now regarding these options ... Again, note that I am discussing what would be implied by introducing sockets. I know you are not pushing that. But we need to get the facts right.

To me it seems -fh/-sh options are not designed for the purpose we are talking about here.

They are simply a way to force xboard/winboard to use a socket. Then you can test my claim that the program can't tell the difference between the two, using read()/write(). If you use send()/recv() you can tell because they won't work on a pipe.

xboard/winboard pulls that socket out of its magic hat. Your program does nothing. Once xboard decides to use a socket, rather than a pipe, because the engine is being executed on a remote host rather than a local host, it is done. Now xboard uses read/write to send data to the chess engine, not knowing nor caring whether it is using a socket or pipe, because the usage is the same.

We should clarify our usage of terms first. Let's define:

"local engine" = engine which is under GUI control, i.e. the GUI knows its file system path, can start and stop it, and can communicate to it via a pipe

"remote engine" = engine which is not under GUI control, i.e. the GUI does not know its file system path and can communicate to it via network only (socket)

The key point of the second definition is, a "remote engine" can actually have a local file system path but it is unknown to the GUI. This is what the "-firstHost/-secondHost" options are about: you cannot use it with a local engine. WinBoard/xboard handle local engines by spawning a child process with two pipes for read+write, and remote engines by creating a socket. This means, with -firstHost/-secondHost the user itself has to care himself about starting and stopping the engine which is simply against the "plug and play" principle for the usual "local" use case. It certainly is a change, even if not a big one.

Sorry, but I don't follow. I don't have to start my remote engine. xboard does it for me. And even the -xreuse option works perfectly where it will stop and re-start the engine for each new game, even though it is on a remote host. I don't follow your ?the user itself has to care himself about starting and stopping..." which doesn't make sense to me...

It is easy to start a process on a remote machine.. Here is a trivial way:

GUI creates a socket to remote machine. Then uses fork() to create a second process. That process then uses dup2() to stuff that socket on top of stdout/stdin so that this process now will be writing to the socket instead of the console.

The GUI second process now does an exec() that executes "ssh remotehost crafty" And anything the gui reads from the socket comes from the remote crafty. Anything the gui writes into the socket goes to the remote crafty. User doesn't have to do a thing. That is not the only way. There is an rexec facility that does the same thing although most prefer the additional security of using ssh...

Presumably this is not an issue for you or for me. But the use case becomes a different one. Many WinBoard users, for instance, rely on setting all engine-specific options, including command line options, in the "winboard.ini" config file. But being forced to also care about engine start/stop, which is usually done by WinBoard automatically, goes beyond acceptance.

Can you explain exactly what you are talking about? I do it as above, whether I use a local engine or a remote one. xboard does it just fine, automatically, either way...

Considering this, the only thing to restore compatibility would be a new option that combines both features "local engine" and "socket", which does not exist yet to my knowledge. It may even be easy to add that for WinBoard/xboard. But changing an engine to use sockets and send/recv would immediately make that engine depend on that new GUI option, unless you increase the mess by making the socket usage of the engine depend on another command line option . I am pretty sure you'll agree that this would not make much sense, considering the marginal benefit of using sockets and also the "zero elo gain" aspect.

ALACADABRA! Now we have a local pipe and remote socket option that are identical. Oh. We have had it for 15+ years that I have been using xboard. Somehow I am not sure you understand how xboard/winboard use remote engines. The user doesn't know. Here's my xboard command to run xboard on my local box, and execute crafty on a remote machine named "ferrum":

xboard -smpCores 8 -defaultHashSize 800 -xexit -popupExitMessage false -icshelper /usr/bin/timestamp -geometry +0+0 -ics -zp -zippyMaxGames 2 -zippyReplayTimeout 3600 -autoflag -icshost 207.99.83.228 -fcp "ssh 138.26.64.202 ./runc" -size petite -xreuse -xanimate
~

It uses timestamp normally, connects to the normal ics host (207.etc), but it executes crafty on remote machine 138.26.64.202 (my office box is 138.26.64.17). I use the name "runc" because I have a directory named "crafty" on that box already. "runc" is a shell script that changes to the right directory, although I could put everything in the home directory there.

How hard is that?

My conclusion about the socket proposal: it is certainly possible somehow but would require much more preparation.

or not...

[/quote]


To come back to the very first topic of this thread: using read/write instead of higher level library functions, as you propose, is certainly not "bad" at all. But in my opinion it solves non-existent problems while introducing too much low level stuff.

I have even thought about a different solution for the "254 character limit" issue. To summarize, after some testing it seems that the problem exists
- on Windows only,
- with unbuffered input only,
- and in console mode only (at least I could not reproduce the problem without console mode).

So the question arises, do we really need unbuffered input in console mode? It could be possible to switch on unbuffered input only when detecting that the standard input is connected to a pipe.

Would that make sense? Would that imply any functional restrictions for the console mode? I don't think so, the only tiny issue I could image would be that you could not expect the engine to receive commands that are typed while the search is running, so in WinBoard/xboard you could not use the "move now" ("?") command with an engine that implements classical polling. However, there is a better design based on threads that makes this tiny problem quite irrelevant.

What do you think? The question goes also to the audience ...

Sven[/quote]

I think there are too many pitfalls to buffered I/O. If we tell everyone to use read/write from the beginning, none of this buffered vs unbuffered discussion comes up, ever, again. Which would be nice. Because I can't count the number of times I have explained why fscanf(), fgets() and such have problems with xboard/winboard and seem to hang up unexpectedly...

[bob]

If you force the use of send/recv, you get message boundaries. If you don't, then you have to figure out the message boundaries for yourself, as we do today with the "\n" (line feed) character...

You cannot force message boundaries on TCP/IP sockets, since TCP/IP does not support boundaries. It's a byte-stream protocol. Datagram sockets (i.e. UDP) preserve boundaries, but that's a lossy protocol.

I've actually seen some programs which rely on TCP/IP sockets to get a whole line from a single recv(). They crashed once in a while, when the OS or routers felt like fragmenting or merging parts of the stream.

I believe you are right, but am not certain. I don't use send()/recv(). Only case for me was sendto()/recvfrom() which is used for UDP...

[Roger Brown]

xboard, WinBoard, gnuchess compatibilty, ... point accepted!

But throwing away ugly solutions is also no way of "software engineering". In an ideal world you can throw away what you consider as ugly, and replace it. But in the real world you have to consider the "customers", too.

Offer a better protocol spec, then we might get a better GUI implementation, and only then we can start to switch the engines to a better standard.

Nevertheless, whether you like it or not: the WinBoard/xboard GUI of today is quite good in terms of usabilty, for both end-users and programmers - good enough to have about 350-400 different WinBoard engines compared to "only" 100-150 UCI engines. (Numbers quickly estimated based on current RWBC list) That is not quite like being old-fashioned, is it?

[I had to say that, at least for Roger But believe me, Roger, I also said it in the belief that it is true.]

Sven

Hello Sven,

Just saw this.



THANKS.

Of course I am sure that authors and developers await new developments.

I quite like your point about us living in this current world of Winboard and UCI.

My novice question is what precisely do the proposals offer the end user?

There is a pragmatic cost....

I am sure that the numbers are shocking to some but Winboard and XBoard have been making strides (user driven) recently. I think stone age is an unkind reference BUT I am not smart enough to understand the limitations described in this thread so I could be wrong.

I guess I am just a happy user....

Later.

[Sven]

xboard, WinBoard, gnuchess compatibilty, ... point accepted!

But throwing away ugly solutions is also no way of "software engineering". In an ideal world you can throw away what you consider as ugly, and replace it. But in the real world you have to consider the "customers", too.

Offer a better protocol spec, then we might get a better GUI implementation, and only then we can start to switch the engines to a better standard.

Nevertheless, whether you like it or not: the WinBoard/xboard GUI of today is quite good in terms of usabilty, for both end-users and programmers - good enough to have about 350-400 different WinBoard engines compared to "only" 100-150 UCI engines. (Numbers quickly estimated based on current RWBC list) That is not quite like being old-fashioned, is it?

[I had to say that, at least for Roger But believe me, Roger, I also said it in the belief that it is true.]

Sven

Hello Sven,

Just saw this.



THANKS.

Of course I am sure that authors and developers await new developments.

I quite like your point about us living in this current world of Winboard and UCI.

My novice question is what precisely do the proposals offer the end user?

There is a pragmatic cost....

I am sure that the numbers are shocking to some but Winboard and XBoard have been making strides (user driven) recently. I think stone age is an unkind reference BUT I am not smart enough to understand the limitations described in this thread so I could be wrong.

I guess I am just a happy user....

Later.

Hi Roger,

the proposals offer nothing to the end user since they either

- don't work (regarding send/recv and socket usage on engine side),

- are no relevant improvement over a working technology (regarding a possible extension of xboard/WinBoard to actually offer socket support for [local or remote] engines to allow for send/recv usage, as opposed to the working technology based on pipes/stdin/stdout)

- don't solve any really existing problems (regarding read/write usage on engine side instead of the common and comfortable fgets or similar higher level library function), or

- are not related to problems occurring in standard chess (regarding the 254 character limit in Windows console mode which practically affects some chess variants only).

Stay tuned if you are interested in reading my explanations on most of the points above, as well as the opinions of the other side which are well respected by me, although I have strong reasons to say they are not accurate.

Sven

[Sven]

[...]

Code: Select all

xboard -smpCores 8 -defaultHashSize 800 -xexit -popupExitMessage false -icshelper /usr/bin/timestamp -geometry +0+0 -ics -zp -zippyMaxGames 2 -zippyReplayTimeout 3600 -autoflag -icshost 207.99.83.228 -fcp "ssh 138.26.64.202 ./runc" -size petite -xreuse -xanimate

[...]
How hard is that?

We have some different cases to consider.

1) Remote engine "disguised" as local engine towards xboard

Your case is without -firstHost option but with -fcp "ssh HOST PROGRAM". (I use the upper case words HOST and PROGRAM like variables from here on, not to be mixed up with some "shouting" style of communication.).

The following happens (I assume you know this already but there are also other readers):

- xboard forks a local child process running the "ssh" command line, and creates two pipes to talk to that "ssh" process, as usual with a local engine. xboard does not know what "ssh" actually does.

- "ssh" connects to the remote HOST using a socket connection. (I omit all the user login/authentication/encryption/... stuff here which is not important for our context.) The local "ssh" process has an own, local socket. There is also a counterpart for it, the remote socket on the HOST which is already there when the ssh daemon is starting up.

- The HOST has a service running ("ssh daemon") that listens to that remote socket, usually on port 22. On an incoming connection the ssh daemon forks a new process, PROGRAM. The ssh daemon manages for PROGRAM to read from standard input the data that are received via the remote socket, and to send back via that socket the data that are written to stdout by PROGRAM. (I omit stderr here for simplicity, it also works in a similar way.) This is achieved with something like pipes.

- PROGRAM can use low-level read/write as well as "higher-level" fgets/getline/printf/fputs/... I/O functions. There should be no restriction to use only read/write for a remotely running engine since stdin/stdout (resp. standard file descriptors 0/1) are always the same.

- The mechanism described above realizes the simulation of a normal stdin/stdout connection between the local client (xboard/WinBoard) and the remote PROGRAM running on the HOST. The existence of a socket connection over the network is 100% transparent for both client and server programs.

- I am not sure whether it would even be possible to use send/recv in the remote PROGRAM (chess engine) since the described method involves to associate the remote socket with the stdin and stdout file descriptors (done by ssh daemon), so the PROGRAM must read from stdin and write to stdout to get the data from/to the communication partner, and cannot read from the socket directly. I have not tried it but my current understanding is NO.

2) "Real" remote engine started via "-firstHost"

Let me add first that I made a wrong statement when writing that xboard/WinBoard would not start the engine in case of using the -firstHost option. I did not look up the description carefully enough, did actually only refer to your statements (but misunderstood), and therefore initially thought it would just create a socket and connect to a remote socket which would have to be provided directly by the remote engine, not involving any RSH/SSH/... service, i.e. just TCP. That would indeed imply to care about starting of the engine separately, but my thought was wrong, sorry for that.

Here the essential mechanism is the same as above, the differences are in the details:

- Command line arguments are "-firstHost HOST" and "-fcp PROGRAM" (same convention as above).

- xboard/WinBoard uses the "rsh" mechanism to manage remote execution of PROGRAM on HOST. It creates a local socket and connects to the existing remote socket on HOST.

- There is again a daemon process running on the HOST, the "rsh daemon", which listens to the remote socket and forks a child process PROGRAM on an incoming connection. While the fine details may be different the basic mechanism is the same as in the "ssh" case above. "ssh" is preferred today over "rsh" for security reasons.

- Everything written above about stdin/stdout(/stderr) and sockets is also valid here. Therefore I believe that also "-firstHost" does not offer a way for a remotely running engine to access the remote socket via send/recv.

3) Local engine

This is the most relevant case for the majority of chess engine users:

- Command line options are "-fcp PROGRAM". The "-firstHost" option is not present, and PROGRAM is either a path on the local file system pointing to the local engine or a command line starting a local engine.

- xboard/WinBoard internally set HOST to "localhost" in this case, and do not create any socket connection. PROGRAM is forked as local child process of xboard/WinBoard, and pipes for read/write are created as usual.

- There is no chance for PROGRAM to use send/recv since there is no socket to use.


Conclusion:

Same as before Explicitly using sockets via send/recv is not available for local engines, and to my knowledge also not available for remotely running engines, unless some extensions are offered by xboard/WinBoard (and other GUIs/tools as wells) in the future. The key point is, even if the "remote" case would turn out to be possible, it is not possible today for the "local" case covering >99% of the users.


Sven