unbuffered input/ouput

[sje]

What are some of the pertinent benefits of using recv/send?

First, since recv/send can be operated in message mode, there's no need to be concerned about embedding a message terminating character (like a newline) in the message. A single character, non-newline message can be reliably sent (e.g., a "." for a "move now" message).

Second, there's no need to be concerned about fixed buffer lengths that prevent long messages from being sent atomically. Therefore, the 256 character line length limit problem mentioned at the start of this thread goes away. If for some reason a "setboard" command takes a thousand bytes, there's still no worry about it not getting to its destination in one piece.

Third, one can still use recv/send just like read/write to minimize code change. All that's needed is the inclusion of the socket/bind/etc calls for initialization, and these can be copied from long tested, working code examples.

[Sven]

<long quotation skipped ...>

You are writing a lot of true sentences, however they are not much related to the exact topic we are now discussing. Steven proposed to use recv/send instead of read/write in a chess engine. How is that related to your statements? How is that *not* a change of the engine implementation? How is that *not* a change of the protocol specification (WB/UCI)? Exactly *which* software do you think would go unchanged, and which not?

Sven

The _original_ topic was discussing using sockets vs pipes. In Unix, a program does not have to do _anything_ differently to use either. As far as the protocol, how, exactly, does the winboard / UCI protocol dictate how you read/write? I can do write(), you can do send(), and the GUI is not going to be able to tell the difference. If you want to _change_ the protocol, so that one _has_ to use send/recv so that you preserve message boundaries, that's ok, but nothing makes the GUI use recv even if I use send. It _may_ use recv if it wants, or it can use read()...

I don't think the WB/XB protocol needs anything more defined than the strings and what they mean. Each command is supposed to be terminated by a l/f character. What more does one need?

The WinBoard GUI, at least, creates two pipes per local engine, one for sending commands to the engine and another to receive replies from the engine. Instead of a native WinBoard engine you may use PolyGlot to connect to a UCI engine, where PolyGlot in turn creates another two pipes to the engine.

So you say the following:

a) An engine X can replace its current "setbuf(stdin, 0); ... fgets(..., stdin); ... if (input_available(...)) ..." mimics, or getline(), or read(), ..., by code that reads via a socket using recv(), and the same GUI commands written by the GUI into its "writing pipe" will be received by the engine, without any change in the GUI, and without any loss of functionality.

b) An engine X can replace its current "setbuf(stdout, 0); ... printf(...); ..." mimics, or fputs(..., stdout), or write(), ..., by code that writes to a socket using send(), and the GUI will still receive the same engine commands via its "reading pipe", without any change in the GUI, and without any loss of functionality.

c) This (a+b) will also work with PolyGlot.

d) This (a+b+c) will also work on our current main platform which is Windows.

e) This (a+b+d) will work with UCI GUIs the same way as it works with WinBoard GUIs.

If it were that simple, then why did nobody try it up to now? Why is it not mentioned in the WB/UCI protocol specs?

Can you, or someone else, show example C code that creates and uses a socket such that you can read the data being written by another process into a pipe, and get the same result as if you were reading directly from the standard input file descriptor, or the "stdin" stream?


In my opinion it does not work this way. If an engine wants to read via a socket then it needs a partner who writes into it, and that must be the GUI. I think you cannot read data from a pipe created by your parent process by reading via a socket using recv().

Given that, it seems you are in fact talking about a possible change on the GUI side. Either the GUI creates pipes to communicate with engines, as it does now (at least WinBoard), or it creates sockets, but not both. And only after that change (which is highly unlikely) an engine can change its way of reading data.

I may be wrong, though.

Sven

[sje]

Gosh, it's so simple to do:

http://www.linuxhowtos.org/C_C++/socket.htm

[bob]

Gosh, it's so simple to do:

http://www.linuxhowtos.org/C_C++/socket.htm

There are issues with sockets. Port number for example. Running two simultaneous games on same host is problematic and now you have to pass each engine a port # to use. Pipes don't have such issues. Sockets, once open, don't have any issues either. But gettting them "open" can be interesting if you want to run (say) 16 games at a time on a 16-core box, which requires 32 engines. And 32 ports...

[Sven]

Gosh, it's so simple to do:

http://www.linuxhowtos.org/C_C++/socket.htm

You made the proposal So I think it's not up to me to work out the details. I asked Bob, and implicitly also you, to show how it works. Actually I know enough about sockets to know that you have to have a server and a client. But the link you posted actually confirms what I had in mind before, just look at the first section.

So to recur, here is the programming task:

Parent process A creates a child process B and a pipe between A and B such that A can write into the pipe and B read from it. A is unchangeable for you. B currently implements its communication part by simply reading from the standard input stream with fgets() (replace it by getline(), read(), or whatever if you like, just plain standard input reading). Now change B such that it will use recv() instead of fgets() to read data sent by A. Once again: without changing A, that means, A does *not* have a socket. The final requirement is, B shall work as before, "plug and play".

Replace A by "Winboard GUI" and B by "WinBoard engine", or A by "PolyGlot" and B by "UCI engine", to get back to our real life.

I'm sorry but I fail to see a solution offered by the document you have linked to above. I think there is none, unless you also change the GUI to offer a socket to connect to.

Sven

[bob]

Gosh, it's so simple to do:

http://www.linuxhowtos.org/C_C++/socket.htm

You made the proposal So I think it's not up to me to work out the details. I asked Bob, and implicitly also you, to show how it works. Actually I know enough about sockets to know that you have to have a server and a client. But the link you posted actually confirms what I had in mind before, just look at the first section.

So to recur, here is the programming task:

Parent process A creates a child process B and a pipe between A and B such that A can write into the pipe and B read from it. A is unchangeable for you. B currently implements its communication part by simply reading from the standard input stream with fgets() (replace it by getline(), read(), or whatever if you like, just plain standard input reading). Now change B such that it will use recv() instead of fgets() to read data sent by A. Once again: without changing A, that means, A does *not* have a socket. The final requirement is, B shall work as before, "plug and play".

Replace A by "Winboard GUI" and B by "WinBoard engine", or A by "PolyGlot" and B by "UCI engine", to get back to our real life.

I'm sorry but I fail to see a solution offered by the document you have linked to above. I think there is none, unless you also change the GUI to offer a socket to connect to.

Sven

Here's the issue. I can create any sort of open file descriptor I want, whether it is for a socket, a pipe, a FIFO (if you know what a unix FIFO is), a tcp/ip socket, a datagram (udp) socket, etc. If this is the first new thing I open, it will be descriptor #3 (0=stdin, 1=stdout, 2=stderr, always, unless you close them). I can then dup2(3,0). I then fork() a new process, exec() your chess engine, and no matter what you do to stdin, you are reading directly from whatever it was I opened, because I crammed that on top of stdin, losing the original stdin connection forever. Now, whether you use read, recv, fgets, fgetc, or whatever, you are going to read from whatever I opened.

If you force the use of send/recv, you get message boundaries. If you don't, then you have to figure out the message boundaries for yourself, as we do today with the "\n" (line feed) character...

There are still quirks with recv(). It does not guarantee to return a complete message, only what has been received at the point where the recv() is done. Which means some effort to recognize a complete message is required, such as a EOM character you choose, etc... Or fixed-length messages. Etc...

[bob]

<long quotation skipped ...>

You are writing a lot of true sentences, however they are not much related to the exact topic we are now discussing. Steven proposed to use recv/send instead of read/write in a chess engine. How is that related to your statements? How is that *not* a change of the engine implementation? How is that *not* a change of the protocol specification (WB/UCI)? Exactly *which* software do you think would go unchanged, and which not?

Sven

The _original_ topic was discussing using sockets vs pipes. In Unix, a program does not have to do _anything_ differently to use either. As far as the protocol, how, exactly, does the winboard / UCI protocol dictate how you read/write? I can do write(), you can do send(), and the GUI is not going to be able to tell the difference. If you want to _change_ the protocol, so that one _has_ to use send/recv so that you preserve message boundaries, that's ok, but nothing makes the GUI use recv even if I use send. It _may_ use recv if it wants, or it can use read()...

I don't think the WB/XB protocol needs anything more defined than the strings and what they mean. Each command is supposed to be terminated by a l/f character. What more does one need?

The WinBoard GUI, at least, creates two pipes per local engine, one for sending commands to the engine and another to receive replies from the engine. Instead of a native WinBoard engine you may use PolyGlot to connect to a UCI engine, where PolyGlot in turn creates another two pipes to the engine.

So you say the following:

a) An engine X can replace its current "setbuf(stdin, 0); ... fgets(..., stdin); ... if (input_available(...)) ..." mimics, or getline(), or read(), ..., by code that reads via a socket using recv(), and the same GUI commands written by the GUI into its "writing pipe" will be received by the engine, without any change in the GUI, and without any loss of functionality.

Correct. I can not answer for windows, however. Only for Unix. For unix, my students generally use read/write, send/rev, sendto/recvfrom, or whatever they want to send/receive data. Ditto for servers like ICC that have to talk to my program. xboard uses fread/fwrite, or at least it did the last time I looked. I have seen ICC clients that use send/recv, read/write (same as fread/fwrite basically).

The only issue becomes one of messages. If you use send/recv and you expect message boundaries to be preserved as they are when you send and the other end recv's... it won't work correctly if one end uses send and the other end uses read.

b) An engine X can replace its current "setbuf(stdout, 0); ... printf(...); ..." mimics, or fputs(..., stdout), or write(), ..., by code that writes to a socket using send(), and the GUI will still receive the same engine commands via its "reading pipe", without any change in the GUI, and without any loss of functionality.

Correct, up to a point. There have been many cases where send/recv, applied to a non-socket, can fail. Hence my recommendation to _always_ use read/write, which work for any file descriptor.

c) This (a+b) will also work with PolyGlot.

d) This (a+b+c) will also work on our current main platform which is Windows.

Don't know nor care there.

e) This (a+b+d) will work with UCI GUIs the same way as it works with WinBoard GUIs.

I don't see why not. But I am not familiar with how UCI has evolved. But it clearly works with read/write since anything works with that.

If it were that simple, then why did nobody try it up to now? Why is it not mentioned in the WB/UCI protocol specs?

Try what? The point is that you don't eve _know_ already. In xboard, you can run on a local machine (xboard spawns two new processes and serves as intermediary between the two chess engines using pipes. Or you can use the -fh or -sh (or both) and make xboard create sockets, connect them to engines running on remote machines, and there are zero changes to the engine. They have no idea whether they are reading from a socket, a pipe, or two cans connected by a string...

Which is how it should be...

Can you, or someone else, show example C code that creates and uses a socket such that you can read the data being written by another process into a pipe, and get the same result as if you were reading directly from the standard input file descriptor, or the "stdin" stream?

Xboard does this, as I mentioned. You can use the pipe() system call to create 2 new and open file descriptors. The first pipe call will return 3 and 4 for the descriptor numbers. You can then create a new process that inherits those open descriptors, just as xboard currently does, and then uses dup2() to stuff one of those on top of stdin. When xboard writes to the other end, the chess engine reads from stdin, which is now that open pipe, and it gets the data xboard wrote. If you use the -fh or -sh, xboard instead creates a socket, connects it to a remote machine, and then stuffs that on top of stdin for the chess engine. And again, the chess engine reads from xboard not knowing whether it is a socket, a pipe, or any other other I/O capable "thing".

In my opinion it does not work this way. If an engine wants to read via a socket then it needs a partner who writes into it, and that must be the GUI. I think you cannot read data from a pipe created by your parent process by reading via a socket using recv().

I'm simply telling you how it actually works. A program can be forced into reading from the keyboard (default stdin setting, or from a file, by opening the file, and then stuffing that descriptor on top of stdin using dup2(), or from a pipe, by (as xboard does) creating a pipe and then stuffing the read end of the pipe's descriptor # on top of stdin, or by creating a socket, connecting it, and then stuffing _that_ descriptor on top of stdin. The program reading from stdin has no idea where the data is coming from, Or where it is going to if it is stdout.

Given that, it seems you are in fact talking about a possible change on the GUI side. Either the GUI creates pipes to communicate with engines, as it does now (at least WinBoard), or it creates sockets, but not both.

Xboard/winboard do either. If you look at xboard, in xboard.c, you can find the socket creation code in something like openTCP() or something similar... I do not have winboard source, but it certainly supports -fh and -sh, so it does the same thing there.

And only after that change (which is highly unlikely) an engine can change its way of reading data.

I may be wrong, though.

Sven

you are. See above. That's been one of the cleanest parts of unix forever. That and the fact that _all_ I/O looks like files, even if you want to talk directly to a modem, or a line printer. You open a file...

[Sven]

The only issue becomes one of messages. If you use send/recv and you expect message boundaries to be preserved as they are when you send and the other end recv's... it won't work correctly if one end uses send and the other end uses read.

There have been many cases where send/recv, applied to a non-socket, can fail. Hence my recommendation to _always_ use read/write, which work for any file descriptor.

Hmmm, so it is in fact easy to use send()/recv()?

c) This (a+b) will also work with PolyGlot.

d) This (a+b+c) will also work on our current main platform which is Windows.

Don't know nor care there.

O.k., "WinBoard only" for the moment, if you prefer ...

In xboard, you can run on a local machine (xboard spawns two new processes and serves as intermediary between the two chess engines using pipes. Or you can use the -fh or -sh (or both) and make xboard create sockets, connect them to engines running on remote machines, and there are zero changes to the engine. They have no idea whether they are reading from a socket, a pipe, or two cans connected by a string...

-fh/-sh also for connecting to a local engine? See my comment further down.

Can you, or someone else, show example C code that creates and uses a socket such that you can read the data being written by another process into a pipe, and get the same result as if you were reading directly from the standard input file descriptor, or the "stdin" stream?

Xboard does this, as I mentioned. You can use the pipe() system call to create 2 new and open file descriptors. The first pipe call will return 3 and 4 for the descriptor numbers. You can then create a new process that inherits those open descriptors, just as xboard currently does, and then uses dup2() to stuff one of those on top of stdin. When xboard writes to the other end, the chess engine reads from stdin, which is now that open pipe, and it gets the data xboard wrote. If you use the -fh or -sh, xboard instead creates a socket, connects it to a remote machine, and then stuffs that on top of stdin for the chess engine. And again, the chess engine reads from xboard not knowing whether it is a socket, a pipe, or any other other I/O capable "thing".

In my opinion it does not work this way. If an engine wants to read via a socket then it needs a partner who writes into it, and that must be the GUI. I think you cannot read data from a pipe created by your parent process by reading via a socket using recv().

I'm simply telling you how it actually works. A program can be forced into reading from the keyboard (default stdin setting, or from a file, by opening the file, and then stuffing that descriptor on top of stdin using dup2(), or from a pipe, by (as xboard does) creating a pipe and then stuffing the read end of the pipe's descriptor # on top of stdin, or by creating a socket, connecting it, and then stuffing _that_ descriptor on top of stdin. The program reading from stdin has no idea where the data is coming from, Or where it is going to if it is stdout.

The proposal by Steven was that the engine should use recv() to read. But this requires a socket for which there must be some counterpart.

Given that, it seems you are in fact talking about a possible change on the GUI side. Either the GUI creates pipes to communicate with engines, as it does now (at least WinBoard), or it creates sockets, but not both.

Xboard/winboard do either. If you look at xboard, in xboard.c, you can find the socket creation code in something like openTCP() or something similar... I do not have winboard source, but it certainly supports -fh and -sh, so it does the same thing there.

-fh/-sh mean "-firstHost" and "-secondHost" and can be used to connect to a remote engine, as you say. But here were are talking about a local engine for which you have to specify its path. WinBoard creates a pipe to each local engine.

To me it seems -fh/-sh options are not designed for the purpose we are talking about here.

And only after that change (which is highly unlikely) an engine can change its way of reading data.

I may be wrong, though.

Sven

you are. See above. That's been one of the cleanest parts of unix forever. That and the fact that _all_ I/O looks like files, even if you want to talk directly to a modem, or a line printer. You open a file...

Still not on topic, though ...

As can be seen from your reply, you confirm that reading from a socket needs another socket as a counterpart. And you are not showing where we get that from. So where have I been wrong?

Sven

[sje]

A socket connects only to another socket. If a chess GUI doesn't have socket support, then the support needs to be added. The change is simple if the GUI has been structured properly.

[Sven]

A socket connects only to another socket. If a chess GUI doesn't have socket support, then the support needs to be added. The change is simple if the GUI has been structured properly.

That is the point I am trying to make for two days now. We have a real world, not a dream world. The GUIs are there, and people (and engine programmers) are using them as they are. We are not in the middle of nowhere, there is an existing world of programs around us.

And even if you change a couple of free GUIs, tournament managers, referees, engine adapters, you name it - you will simply notice that commercial chess GUIs will not make the change you propose. And if they don't then each engine that uses a socket for reading will no longer work with a traditionally working commercial GUI.

Neither WinBoard protocol spec nor UCI spec contain the requirement that a GUI implementing the protocol must offer socket support for local engines.

You have also confirmed my view that it is not such an "easy task" to make such a change, and that it can't be done independently at just one place.

I just noticed you changed the subject line into "It's socket to socket; no one has claimed otherwise". Hmmmmmm...... no one, really?

Sven