unbuffered input/ouput

[bob]

[...]

Code: Select all

xboard -smpCores 8 -defaultHashSize 800 -xexit -popupExitMessage false -icshelper /usr/bin/timestamp -geometry +0+0 -ics -zp -zippyMaxGames 2 -zippyReplayTimeout 3600 -autoflag -icshost 207.99.83.228 -fcp "ssh 138.26.64.202 ./runc" -size petite -xreuse -xanimate

[...]
How hard is that?

We have some different cases to consider.

1) Remote engine "disguised" as local engine towards xboard

Your case is without -firstHost option but with -fcp "ssh HOST PROGRAM". (I use the upper case words HOST and PROGRAM like variables from here on, not to be mixed up with some "shouting" style of communication.).

The following happens (I assume you know this already but there are also other readers):

- xboard forks a local child process running the "ssh" command line, and creates two pipes to talk to that "ssh" process, as usual with a local engine. xboard does not know what "ssh" actually does.

- "ssh" connects to the remote HOST using a socket connection. (I omit all the user login/authentication/encryption/... stuff here which is not important for our context.) The local "ssh" process has an own, local socket. There is also a counterpart for it, the remote socket on the HOST which is already there when the ssh daemon is starting up.

- The HOST has a service running ("ssh daemon") that listens to that remote socket, usually on port 22. On an incoming connection the ssh daemon forks a new process, PROGRAM. The ssh daemon manages for PROGRAM to read from standard input the data that are received via the remote socket, and to send back via that socket the data that are written to stdout by PROGRAM. (I omit stderr here for simplicity, it also works in a similar way.) This is achieved with something like pipes.

- PROGRAM can use low-level read/write as well as "higher-level" fgets/getline/printf/fputs/... I/O functions. There should be no restriction to use only read/write for a remotely running engine since stdin/stdout (resp. standard file descriptors 0/1) are always the same.

- The mechanism described above realizes the simulation of a normal stdin/stdout connection between the local client (xboard/WinBoard) and the remote PROGRAM running on the HOST. The existence of a socket connection over the network is 100% transparent for both client and server programs.

- I am not sure whether it would even be possible to use send/recv in the remote PROGRAM (chess engine) since the described method involves to associate the remote socket with the stdin and stdout file descriptors (done by ssh daemon), so the PROGRAM must read from stdin and write to stdout to get the data from/to the communication partner, and cannot read from the socket directly. I have not tried it but my current understanding is NO.

2) "Real" remote engine started via "-firstHost"

Let me add first that I made a wrong statement when writing that xboard/WinBoard would not start the engine in case of using the -firstHost option. I did not look up the description carefully enough, did actually only refer to your statements (but misunderstood), and therefore initially thought it would just create a socket and connect to a remote socket which would have to be provided directly by the remote engine, not involving any RSH/SSH/... service, i.e. just TCP. That would indeed imply to care about starting of the engine separately, but my thought was wrong, sorry for that.

Here the essential mechanism is the same as above, the differences are in the details:

- Command line arguments are "-firstHost HOST" and "-fcp PROGRAM" (same convention as above).

- xboard/WinBoard uses the "rsh" mechanism to manage remote execution of PROGRAM on HOST. It creates a local socket and connects to the existing remote socket on HOST.

- There is again a daemon process running on the HOST, the "rsh daemon", which listens to the remote socket and forks a child process PROGRAM on an incoming connection. While the fine details may be different the basic mechanism is the same as in the "ssh" case above. "ssh" is preferred today over "rsh" for security reasons.

- Everything written above about stdin/stdout(/stderr) and sockets is also valid here. Therefore I believe that also "-firstHost" does not offer a way for a remotely running engine to access the remote socket via send/recv.

That remote daemon simply forks a process, dup2's the socket on top of stdin and stdout, and then uses execve() to execute the indicated program. The Daemon is then "out of the loop" as the new process communicates directly to the remote machine (running xboard) over the socket connection. It can use recv()/send() if it chooses, since it is definitely talking over a socket.

3) Local engine

This is the most relevant case for the majority of chess engine users:

- Command line options are "-fcp PROGRAM". The "-firstHost" option is not present, and PROGRAM is either a path on the local file system pointing to the local engine or a command line starting a local engine.

- xboard/WinBoard internally set HOST to "localhost" in this case, and do not create any socket connection. PROGRAM is forked as local child process of xboard/WinBoard, and pipes for read/write are created as usual.

- There is no chance for PROGRAM to use send/recv since there is no socket to use.

I have already discussed that issue. But it is resolvable, trivially. A change to xboard so that _all_ connections are via socket would work perfectly. For example, if _you_ use -fh localhost, you get a socket connection to a local process. Your program won't know anything is different, except that with the -fh option, it could use recv() (which I am not a proponent of) or it could use fgets() or any other normal input form. Without the -fh, you can use every input option except recv().

Conclusion:

Same as before Explicitly using sockets via send/recv is not available for local engines, and to my knowledge also not available for remotely running engines, unless some extensions are offered by xboard/WinBoard (and other GUIs/tools as wells) in the future. The key point is, even if the "remote" case would turn out to be possible, it is not possible today for the "local" case covering >99% of the users.


Sven

Again, your comments about remote engines is wrong. The remote engine is using a socket when it does _anything_ with stdin/stdout, since they have been replaced by the network socket descriptor...

The local/remote (pipe/socket) issue is _not_ a program issue. It is a GUI issue. If the GUI chooses, it can _always_ use a socket, in which case any valid I/O procedure is acceptable. If the GUI uses a pipe for local execution, then send()/recv() will not work. It's all up to the GUI. I would not mandate send/recv myself, I would mandate read/write which work for _ALL_ cases...

[bob]

xboard, WinBoard, gnuchess compatibilty, ... point accepted!

But throwing away ugly solutions is also no way of "software engineering". In an ideal world you can throw away what you consider as ugly, and replace it. But in the real world you have to consider the "customers", too.

Offer a better protocol spec, then we might get a better GUI implementation, and only then we can start to switch the engines to a better standard.

Nevertheless, whether you like it or not: the WinBoard/xboard GUI of today is quite good in terms of usabilty, for both end-users and programmers - good enough to have about 350-400 different WinBoard engines compared to "only" 100-150 UCI engines. (Numbers quickly estimated based on current RWBC list) That is not quite like being old-fashioned, is it?

[I had to say that, at least for Roger But believe me, Roger, I also said it in the belief that it is true.]

Sven

Hello Sven,

Just saw this.



THANKS.

Of course I am sure that authors and developers await new developments.

I quite like your point about us living in this current world of Winboard and UCI.

My novice question is what precisely do the proposals offer the end user?

There is a pragmatic cost....

I am sure that the numbers are shocking to some but Winboard and XBoard have been making strides (user driven) recently. I think stone age is an unkind reference BUT I am not smart enough to understand the limitations described in this thread so I could be wrong.

I guess I am just a happy user....

Later.

Hi Roger,

the proposals offer nothing to the end user since they either

- don't work (regarding send/recv and socket usage on engine side),

- are no relevant improvement over a working technology (regarding a possible extension of xboard/WinBoard to actually offer socket support for [local or remote] engines to allow for send/recv usage, as opposed to the working technology based on pipes/stdin/stdout)

- don't solve any really existing problems (regarding read/write usage on engine side instead of the common and comfortable fgets or similar higher level library function), or

- are not related to problems occurring in standard chess (regarding the 254 character limit in Windows console mode which practically affects some chess variants only).

Stay tuned if you are interested in reading my explanations on most of the points above, as well as the opinions of the other side which are well respected by me, although I have strong reasons to say they are not accurate.

Sven

For the record, the 254 byte limit is a problem in Crafty. There are commands that let you change evaluation parameters. Some of these are arrays, and you enter all values on a single line, such as

pers 92 val1 val2 val3 ... valn

where n can be at least 64. with a space between values, and with 3 digit values (or two plus - sign) you run over 256 instantly...

[hgm]

Yep. So far only crafty & tscp passed the test. Tscp doesn't do polling though.

To get back to the original problem:

I am not sure what exactly the problem is. You talk about typing to a program that is running in a console. But later on, the talk is about GUIs sending long messages.

These are two completely different things. With a GUI you read from a pipe, and there should not be a limit to the length of a line. It is just a stream of bytes, and newlines have no special significance.

When typing to a console, you have to deal with the keyboard driver, which usually does attach special significance to linefeed, might have its own buffer for local line editing, etc. Setting the input buffer of a program to zero is not the same thing as setting the 'terminal' to 'raw' mode. You can do it even when 'raw mode' has no meaning at all, such as on a file or a pipe.

[Daniel Shawul]

I am not sure what exactly the problem is. You talk about typing to a program that is running in a console. But later on, the talk is about GUIs sending long messages.

I think GUI talk came later but originally it is only console & unbuffered mode .
Also the problem occurs under windows DOS, no problem in linux shell.

These are two completely different things. With a GUI you read from a pipe, and there should not be a limit to the length of a line. It is just a stream of bytes, and newlines have no special significance.

When typing to a console, you have to deal with the keyboard driver, which usually does attach special significance to linefeed, might have its own buffer for local line editing, etc. Setting the input buffer of a program to zero is not the same thing as setting the 'terminal' to 'raw' mode. You can do it even when 'raw mode' has no meaning at all, such as on a file or a pipe.

Well if there is no problem when the connection is through a pipe , as you say, then
then there is no need to go to the trouble of using read(). fgets() much simpler as it
stops at a lf. My doubt with fgets was the statement given in the protocol with the use of
library functions in unbuffered mode. If there is no problem in pipe mode, then that is good
news for fgets users

A second way to fix the problem might be to ask your I/O library not to buffer on input.
It should then be safe to poll the underlying file descriptor as described above.
With C, you can try calling setbuf(stdin, NULL). However, I have never tried this.
Also, there could be problems if you use scanf(), at least with certain patterns,
because scanf() sometimes needs to read one extra character and "push it back" into the buffer;
hence, there is a one-character pushback buffer even if you asked for stdio to be unbuffered. With C++,
you can try cin.rdbuf()->setbuf(NULL, 0), but again, I have never tried this.

My solution was to use buffered input in console mode, but I was not
sure if the problem will manifest in GUI mode. I can't test it since winboard can't handle big size boards.
BTw I tried to compile a bigger sized winboard by changing the dimensions in common.h but it didn't work..

[hgm]

I gues what does not work is parsing moves with double-digit rank numbers? I have a patch that does this (and is also able to handle them internally). Problem is that it is in another branch than the alien branch. I still have to rebase the latter on the latest developments of the main line. I have been too busy the past weeks with the tournament manager patch. But that is in the main branch now (hgmaster at hgm.nubati, and soon in savannah master as well), and it does already use the double-digit move parser. (I did a complete re-write of the parser for that. No more 'flex' nonsese to generate it automatically. It parses double-digit, as well as moves in Shogi coordinates without any ambiguities, and I am working on having it parse traditional Xiangqi notation as well.)

About the problem:
I made the following program:

Code: Select all

/**************************************************
********************************************************
********************************************************
********************************************************
********************************************************
********************************************************
********************************************************
********************************************************
************************************************/

#include <stdio.h>

main&#40;)
&#123;
  char buf&#91;1024&#93;, *p;
  setbuf&#40;stdin, NULL&#41;;
  p = buf;
  while&#40;(*p++ = getchar&#40;)) != '\n');
  *p = 0;
  printf&#40;"input = '%s'\n", buf&#41;;
&#125;

(The comment is really a single line, but the forum software here takes that seriously, so I had to break it to keep this thread readable. )

I compiled it with gcc -mno-cygwin under Cygwin.

It behaves as you describe when I run it from the console, both the DOS box and the Cygwin console: After three lines plus a bit, you cannot type anymore. (But you can backspace and then retype.)

But when I let it read from a file or pipe, there is no problem. E.g. in the Cygwin console, with a pipe:

Code: Select all

$ cat buftest.c | ./buftest
input = '/**********************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************/
'

I guess the explanation is this: when you don't allow the program to buffer you are entirely dependent on the system's keyboard-driver buffer, and in Wndows it s only 256 bytes large. If you try to type more, there is just no space to put what you type, and it s refused. In 1974 Unix used to 'wrap' the keyboard buffer, so you wouldn't notice anything, but the program would just get the last 256 characters you typed, as these overwrote the rest.

Why does it work with buffered input? Apparently the library routines for reading are implemented in such a way that, when they see the input is a terminal, they switch that terminal to 'raw mode', letting it send each character immediately, so it can be placed in their own buffer (which is bigger), and they implement their own local editing functions on it. When you specify unbuffered, they obviously cannot di the latter, and you will perceive the limitations of the OS. These are tight, which is of course exactly why the library tries to work around them if you alow it to.

[Daniel Shawul]

I gues what does not work is parsing moves with double-digit rank numbers? I have a patch that does this (and is also able to handle them internally). Problem is that it is in another branch than the alien branch. I still have to rebase the latter on the latest developments of the main line. I have been too busy the past weeks with the tournament manager patch. But that is in the main branch now (hgmaster at hgm.nubati, and soon in savannah master as well), and it does already use the double-digit move parser. (I did a complete re-write of the parser for that. No more 'flex' nonsese to generate it automatically. It parses double-digit, as well as moves in Shogi coordinates without any ambiguities, and I am working on having it parse traditional Xiangqi notation as well.)

Great work. I looked at the winboard sources but it was too big for me to familiarize myself with. I will test all added features when it is ready.

I guess the explanation is this: when you don't allow the program to buffer you are entirely dependent on the system's keyboard-driver buffer, and in Wndows it s only 256 bytes large. If you try to type more, there is just no space to put what you type, and it s refused. In 1974 Unix used to 'wrap' the keyboard buffer, so you wouldn't notice anything, but the program would just get the last 256 characters you typed, as these overwrote the rest.

Why does it work with buffered input? Apparently the library routines for reading are implemented in such a way that, when they see the input is a terminal, they switch that terminal to 'raw mode', letting it send each character immediately, so it can be placed in their own buffer (which is bigger), and they implement their own local editing functions on it. When you specify unbuffered, they obviously cannot di the latter, and you will perceive the limitations of the OS. These are tight, which is of course exactly why the library tries to work around them if you alow it to.

I have repeated the test without cygwin on dos, and it gives similar results with input redirection. Problem solved.

Code: Select all

C:\Documents and Settings\dabdi001.CEE-3624-AB52.005\Desktop\Test\test\Debug>tes
t.exe <test.cpp
input = '/**********************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
**********************************/
'

[Sven]

That remote daemon simply forks a process, dup2's the socket on top of stdin and stdout, and then uses execve() to execute the indicated program. The Daemon is then "out of the loop" as the new process communicates directly to the remote machine (running xboard) over the socket connection. It can use recv()/send() if it chooses, since it is definitely talking over a socket.

Then how do you obtain the value of the socket parameter to pass to recv()? It is a local variable of the daemon process, but how to access it from the implementation of the forked child process? Is there a library function for that task?

And how would reading from that socket, if it were possible, influence the standard input? If you read some bytes from that socket, have you "read" these also from standard input? And what if you continue to read from standard input, would that also act like an "implicit recv()"?

I would regard it as a major design error if you had the choice to read the same input data stream in two different ways, where one is like an alias to the other.

It would be optimal if someone would in fact try to implement it.

3) Local engine

[...]

- There is no chance for PROGRAM to use send/recv since there is no socket to use.

I have already discussed that issue. But it is resolvable, trivially. A change to xboard so that _all_ connections are via socket would work perfectly. For example, if _you_ use -fh localhost, you get a socket connection to a local process. Your program won't know anything is different, except that with the -fh option, it could use recv() (which I am not a proponent of) or it could use fgets() or any other normal input form. Without the -fh, you can use every input option except recv().

Whether that works or not does mostly depend on the final outcome of the send/recv issue above. Nevertheless, even if we assume for a short moment that remote engines can use send/recv already today, I would not use the word "trivial" for your proposed change. It would require new code in xboard/WinBoard (the current socket related code is deeply twisted with the "rsh" mechanism and, at another place, with the "ics" related implementation), and it would require lots of testing to find out the real quirks of that change.

As to "-fh localhost", currently this seems to do nothing related to sockets since "localhost" is already the default value for the internal HOST variable (its name is different of course), and if HOST == "localhost" then the normal StartChildProcess is called which does no remote things. (I only looked into the code but did not test it.)

Btw, I am pretty sure that we are discussing about a feature (send/recv) that almost nobody would ever use, not even yourself, and that most probably will also never be implemented in xboard/WinBoard by HGM.

Sven

[bob]

That remote daemon simply forks a process, dup2's the socket on top of stdin and stdout, and then uses execve() to execute the indicated program. The Daemon is then "out of the loop" as the new process communicates directly to the remote machine (running xboard) over the socket connection. It can use recv()/send() if it chooses, since it is definitely talking over a socket.

Then how do you obtain the value of the socket parameter to pass to recv()? It is a local variable of the daemon process, but how to access it from the implementation of the forked child process? Is there a library function for that task?

What "socket parameter" are you talking about? The first argument? zero (descriptor zero, which is stdin).

"recv(0, &buffer, &size, 0)"

99% of recv() calls don't pass anything but 0 (NULL) for the 4th argument.

And how would reading from that socket, if it were possible, influence the standard input? If you read some bytes from that socket, have you "read" these also from standard input? And what if you continue to read from standard input, would that also act like an "implicit recv()"?

What we have here is a total failure to communicate. So once more.

I create a socket, which is going to be descriptor 3 if we have not opened anything else. I now have 4 open descriptors, 0 (stdin), 1 (stdout), 2 (stderr) and 3 (open/connected socket).

I then do a "dup2(3, 0)"

At that instant, "stdin" is now a socket that is opened. A normal scanf() will suck in data from that socket. A normal fgets() will suck in a line from that socket. a normal fgetc() will suck in one character from that socket. And, of course, a recv() will suck in data from that same socket. And they would all swear they are reading from stdin, which is normally connected to the keyboard. yet all would be reading from the socket.

So in this scheme, whatever you use takes data from the socket. You choose the I/O system call you want to use. Any will work equally well. I choose to use read(). I now don't get hung up on blocking until a new line is found (fgets()) or such. Of course, a single read can read in a big chunk of data, and I have to separate the data into different commands, depending on how I write the code...

I would regard it as a major design error if you had the choice to read the same input data stream in two different ways, where one is like an alias to the other.

That's the point, and the beauty of unix. You can never read the same data twice. But anyone that would actually mix fgets(), read(), recv() and fscanf() in the same program will get _exactly_ what they deserve, which is a headache. But pick one, and it will work perfectly.

It would be optimal if someone would in fact try to implement it.

Find a simple client/server program (they can be found on the net. I have some samples from my network programming class if you can't. Take my example with read/write and change it to send/recv and watch it work perfectly. Then change it to use fgets()/fputs() and watch it work perfectly. Just don't mix them so that you do a read and then a recv(). That's asking for buffering issues and missing some data...

3) Local engine

[...]

- There is no chance for PROGRAM to use send/recv since there is no socket to use.

I have no idea what that means... stdin _IS_ a socket in the above discussion. That is the only choice the program has to read from. Fortunately, any normal way of reading data will work on that socket perfectly, even though the program doesn't know it is a socket...

I have already discussed that issue. But it is resolvable, trivially. A change to xboard so that _all_ connections are via socket would work perfectly. For example, if _you_ use -fh localhost, you get a socket connection to a local process. Your program won't know anything is different, except that with the -fh option, it could use recv() (which I am not a proponent of) or it could use fgets() or any other normal input form. Without the -fh, you can use every input option except recv().

Whether that works or not does mostly depend on the final outcome of the send/recv issue above. Nevertheless, even if we assume for a short moment that remote engines can use send/recv already today, I would not use the word "trivial" for your proposed change. It would require new code in xboard/WinBoard (the current socket related code is deeply twisted with the "rsh" mechanism and, at another place, with the "ics" related implementation), and it would require lots of testing to find out the real quirks of that change.

Doesn't require a change at all to test. rsh works on a local machine as well as it works on a remote machine. If you are on node "zorro", just add -fh zorro, or -fh localhost and suddenly you are using a socket rather than a pipe, and you won't be able to tell the difference inside an xboard/winboard compatible engine...

As to "-fh localhost", currently this seems to do nothing related to sockets since "localhost" is already the default value for the internal HOST variable (its name is different of course), and if HOST == "localhost" then the normal StartChildProcess is called which does no remote things. (I only looked into the code but did not test it.)

Might be that Tim or HGM optimized that, which makes sense. But just plug in your node's local host name, and it should use a socket. Although I suppose that could be optimized away into a pipe although I would not have done so myself (and in my referee program I do not. localhost uses pipes, anything else uses a remote socket in my cluster testing referee program).

Btw, I am pretty sure that we are discussing about a feature (send/recv) that almost nobody would ever use, not even yourself, and that most probably will also never be implemented in xboard/WinBoard by HGM.

Sven

I have been using read()/write() since Tim and I first dicussed the buffering program. If you want to ask the question "is there any input to read?" the most convenient way to do this is to use select(). But that has a flaw, because that is an operating system test that asks "does the system have any data buffered up that can be read?" If you use scanf, gets, etc, you introduce a _second_ buffer in the library code. The library code then does a read(), sucks the data into its buffer, and gives you the first line (for gets). But more data is left. Yet select will say "no data" because now the system buffer is empty (the library buffer has data, but select() doesn't know about that).

So, buffering is problematic if you want to do things that way. If you use a second I/O thread, as we did in Cray Blitz, then there is no problem, just use fgets as long as you want, and let it block when there is no more lines of data. I didn't do my I/O that way, because back in the early days on the PC, a second thread was not possible, and polling was the only choice. And if you want to poll, you have to watch the buffering issues.

[hgm]

Note that input buffering in itself is not a bad thing, in an engine or GUI. Problems arise because we usually use incompatible routines to read input, and to check for it. If we use a call that asks the OS if it has input for us, it might say no, but we might still have input in the application's buffer. What we would have needed was a routine that first checks if there is anything in our own buffer, and only if there isn't, asks if the OS has any input for us.

Forcing unbuffered input is just a kludge to enable us to use the wrong input check, at the cost of lowered efficiency.

In practice, WB protocol gives trouble with the time/otim/MOVE commands. These are sent so quickly after another, that the engine gets them all at once when it is doing buffered input, and then the engine reads away the time command, but otim & MOVE go into the buffer. When you start pondering then, and use a call that asks the OS if there is input to terminate it, you hang. Because the input was already there, invisible to you, and there is never going to arrive something else. Some of my engines solve this by not pondering after receiving time and otim. That solves the problem in practice. But a proper check for input (i.e. buffer + OS) would be a safer, fundamental solution to the problem.

[hgm]

Might be that Tim or HGM optimized that, which makes sense.

Indeed, the string "localhost" is the default setting for -fh and -sh, and used internally as a signal that pipes should be used rather than sockets.
The code (by Tim):

Code: Select all

    if (strcmp(cps->host, "localhost") == 0) {
        err = StartChildProcess(cps->program, cps->dir, &cps->pr);
    } else if (*appData.remoteShell == NULLCHAR) {
        err = OpenRcmd(cps->host, appData.remoteUser, cps->program, &cps->pr);
    } else {
        if (*appData.remoteUser == NULLCHAR) {
          snprintf(buf, sizeof(buf), "%s %s %s", appData.remoteShell, cps->host,
                    cps->program);
        } else {
          snprintf(buf, sizeof(buf), "%s %s -l %s %s", appData.remoteShell,
                    cps->host, appData.remoteUser, cps->program);
        }
        err = StartChildProcess(buf, "", &cps->pr);
    }

OpenRcmd is a platform-dependent routine that opens a socket connection. StartChildProcess creates a local process connected to us by pipes. When a remoteShell is given, opening the socket connection is delegated to that, and we conect to that locally running program through pipes.