unbuffered input/ouput

[sje]

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

All that is needed is a replacement of the I/O calls. If this means that a complete re-write is needed for a particular program, then that program was already broken in the first place.

[bob]

Read solves a significant problem. It eliminates two levels of buffering, making it harder to determine if there is input available since the OS is not aware of the library buffer contents, and vice versa.

You are using an extra buffer in your code which we don't need.

But that is the _only_ buffer I have to query to see if there is any more input to process. Otherwise you have the O/S buffer, and the library buffer.

Do what you want. If you want unbuffered I/O, it makes more sense to use unbuffered I/O, to me...

Using fgets() gets you exactly one command. Besides that you introduced additonal work to complete partially read command, remove intermediate line feeds etc... So the buffer and other work you saved from the library function fgets(), you have it in Read().

I perfectly understand that fgets() is only failing in windows. My point is, that read() is _not_ failing anywhere. How is that not relevant?

You can't use this new problem on windows for your argument because that is not really necessary for a chess engine at all. So fgets() is _not_ failing anywhere AFAIK. There is zero problem with fgets() so far, unless you know of other systems where it failed.

I wrote part of it. Yes it evolved to add set the library buffer size to zero. Initially it recommended read(). For good reason...

read() has other advantages. It doesn't need a l/f to terminate a line. It simply leaves the parsing up to the author. I fail to see how using read() is so onerous. My students have to do it in network programming, one never uses fgets() type calls there for obvious reasons. I've used it in my chess program, my testing referee, it is not an onerous programming task...

That is not an advantage but infact a big disadvantage. The winboard protcol has '\n' as the linefeed. So fgets() happens to be a blessing because of that.
You may need read() for network programing and other stuff where you want complete freedom as to the formating but in this case is completely unnecessary. I have implemented three protocols winboard, uci, and go text protocol (gtp) and all use the same thing. I would say the standard is '\n' for GUI engine communication. read() is more appropriate for binary data and other stuff but not for a text based GUI protocol.

It also forces you to read a bunch of commands at one time instead of just one and process that ? If there is f.i a function readN() which does exactly like read() but stops at '\n', would you still use read() ?
read() is justified only if you have a reason for that.

If there was a readN() that read a single line, _unbuffered_ I would certainly use that which is simpler. Otherwise I use that which is known to work everywhere, which is read(). fgets and friends have always had a quirk here and there due to the buffering issues...

[bob]

While read()/write() are good, recv()/send() are even better. There are many more options plus all the benefits of socket connections.

Unix pipes are a good idea, but it's not such a good idea to force them to do something much beyond their original design.

http://www.linuxmanpages.com/man2/recv.2.php

http://www.linuxmanpages.com/man2/send.2.php

I fully agree that using sockets in general would be superior to using pipes and stdio. However, could you please elaborate on how this could help in solving the problem discussed here, given the existing chess engine protocols WinBoard and/or UCI?

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

Am I missing something?

Sven

I don't even understand the point. read/write works with _anything_. Sockets. Pipes. Files. network sockets. FIFOs. You name it. The protocol is not influenced by what is used. The initiating program can create pipes, sockets, FIFOs, etc. and then use dup2() to force the initiated program to read from one of those when it references STDIN (descriptor 0).

[bob]

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

All that is needed is a replacement of the I/O calls. If this means that a complete re-write is needed for a particular program, then that program was already broken in the first place.

I'm missing something. Why does a program care whether it is reading a pipe, socket, FIFO or file under Unix? The whole idea is counter to the design of Unix I/O.

[sje]

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

All that is needed is a replacement of the I/O calls. If this means that a complete re-write is needed for a particular program, then that program was already broken in the first place.

I'm missing something. Why does a program care whether it is reading a pipe, socket, FIFO or file under Unix? The whole idea is counter to the design of Unix I/O.

More than just reading is possible with recv/send. With the latter, it's easier to set up message-based communication vs the character-based read/write transfers. With recv/send, out-of-band communication is supported, and that makes things like sending signals possible -- very handy for managing a client on a remote system. The man pages further describe other benefits.

[Sven]

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

All that is needed is a replacement of the I/O calls. If this means that a complete re-write is needed for a particular program, then that program was already broken in the first place.

I'm missing something. Why does a program care whether it is reading a pipe, socket, FIFO or file under Unix? The whole idea is counter to the design of Unix I/O.

More than just reading is possible with recv/send. With the latter, it's easier to set up message-based communication vs the character-based read/write transfers. With recv/send, out-of-band communication is supported, and that makes things like sending signals possible -- very handy for managing a client on a remote system. The man pages further describe other benefits.

I do not quite understand what both of you are talking about. Both WinBoard and UCI protocols are explicitly designed such that engines read from standard input and write to standard output. Nothing about sockets so far, and no way to change that easily. At least I assume we are talking about current chess engine programming standards here, not about general programming stuff, UNIX I/O concepts, or future chess programming designs.

Switching from stdin/stdout usage to sockets or other messaging concepts would of course not require a "complete rewrite" of engines, that was not what I meant. It would require
- a new protocol specification,
- new or extended GUIs and tools,
- and some modifications/extensions in engine code.

Today I see no need for that type of changes. While everything you both have written may certainly be correct, I think it is not an issue at the moment for chess programming.

Sven

[bob]

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

All that is needed is a replacement of the I/O calls. If this means that a complete re-write is needed for a particular program, then that program was already broken in the first place.

I'm missing something. Why does a program care whether it is reading a pipe, socket, FIFO or file under Unix? The whole idea is counter to the design of Unix I/O.

More than just reading is possible with recv/send. With the latter, it's easier to set up message-based communication vs the character-based read/write transfers. With recv/send, out-of-band communication is supported, and that makes things like sending signals possible -- very handy for managing a client on a remote system. The man pages further describe other benefits.

OOB may or may not work. But certainly is not appropriate for all uses of file descriptors, which is why they are specifically targeted at network sockets.

[bob]

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

All that is needed is a replacement of the I/O calls. If this means that a complete re-write is needed for a particular program, then that program was already broken in the first place.

I'm missing something. Why does a program care whether it is reading a pipe, socket, FIFO or file under Unix? The whole idea is counter to the design of Unix I/O.

More than just reading is possible with recv/send. With the latter, it's easier to set up message-based communication vs the character-based read/write transfers. With recv/send, out-of-band communication is supported, and that makes things like sending signals possible -- very handy for managing a client on a remote system. The man pages further describe other benefits.

I do not quite understand what both of you are talking about. Both WinBoard and UCI protocols are explicitly designed such that engines read from standard input and write to standard output. Nothing about sockets so far, and no way to change that easily. At least I assume we are talking about current chess engine programming standards here, not about general programming stuff, UNIX I/O concepts, or future chess programming designs.

Switching from stdin/stdout usage to sockets or other messaging concepts would of course not require a "complete rewrite" of engines, that was not what I meant. It would require
- a new protocol specification,
- new or extended GUIs and tools,
- and some modifications/extensions in engine code.

It doesn't require a thing. A simple example, have you ever used winboard or xboard to run a local chess engine? What about an engine on a remote machine? Unix I/O does not differentiate between pipes, files, sockets, FIFOs or such at the user level. That's handled deep in the kernel. A simple fscanf() or fgets() or read() or recv() will work with any of those, without knowing which. And without requiring any changes at the program level.

You can read STDIN all you want, but you won't know whether it comes from a network socket, a local socket, a pipe, a file, the console, etc. That's the _point_ for the way Unix I/O was developed...

Today I see no need for that type of changes. While everything you both have written may certainly be correct, I think it is not an issue at the moment for chess programming.

Sven

[Sven]

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

All that is needed is a replacement of the I/O calls. If this means that a complete re-write is needed for a particular program, then that program was already broken in the first place.

I'm missing something. Why does a program care whether it is reading a pipe, socket, FIFO or file under Unix? The whole idea is counter to the design of Unix I/O.

More than just reading is possible with recv/send. With the latter, it's easier to set up message-based communication vs the character-based read/write transfers. With recv/send, out-of-band communication is supported, and that makes things like sending signals possible -- very handy for managing a client on a remote system. The man pages further describe other benefits.

I do not quite understand what both of you are talking about. Both WinBoard and UCI protocols are explicitly designed such that engines read from standard input and write to standard output. Nothing about sockets so far, and no way to change that easily. At least I assume we are talking about current chess engine programming standards here, not about general programming stuff, UNIX I/O concepts, or future chess programming designs.

Switching from stdin/stdout usage to sockets or other messaging concepts would of course not require a "complete rewrite" of engines, that was not what I meant. It would require
- a new protocol specification,
- new or extended GUIs and tools,
- and some modifications/extensions in engine code.

It doesn't require a thing. A simple example, have you ever used winboard or xboard to run a local chess engine? What about an engine on a remote machine? Unix I/O does not differentiate between pipes, files, sockets, FIFOs or such at the user level. That's handled deep in the kernel. A simple fscanf() or fgets() or read() or recv() will work with any of those, without knowing which. And without requiring any changes at the program level.

You can read STDIN all you want, but you won't know whether it comes from a network socket, a local socket, a pipe, a file, the console, etc. That's the _point_ for the way Unix I/O was developed...

Today I see no need for that type of changes. While everything you both have written may certainly be correct, I think it is not an issue at the moment for chess programming.

Sven

You are writing a lot of true sentences, however they are not much related to the exact topic we are now discussing. Steven proposed to use recv/send instead of read/write in a chess engine. How is that related to your statements? How is that *not* a change of the engine implementation? How is that *not* a change of the protocol specification (WB/UCI)? Exactly *which* software do you think would go unchanged, and which not?

Sven

[bob]

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

All that is needed is a replacement of the I/O calls. If this means that a complete re-write is needed for a particular program, then that program was already broken in the first place.

I'm missing something. Why does a program care whether it is reading a pipe, socket, FIFO or file under Unix? The whole idea is counter to the design of Unix I/O.

More than just reading is possible with recv/send. With the latter, it's easier to set up message-based communication vs the character-based read/write transfers. With recv/send, out-of-band communication is supported, and that makes things like sending signals possible -- very handy for managing a client on a remote system. The man pages further describe other benefits.

I do not quite understand what both of you are talking about. Both WinBoard and UCI protocols are explicitly designed such that engines read from standard input and write to standard output. Nothing about sockets so far, and no way to change that easily. At least I assume we are talking about current chess engine programming standards here, not about general programming stuff, UNIX I/O concepts, or future chess programming designs.

Switching from stdin/stdout usage to sockets or other messaging concepts would of course not require a "complete rewrite" of engines, that was not what I meant. It would require
- a new protocol specification,
- new or extended GUIs and tools,
- and some modifications/extensions in engine code.

It doesn't require a thing. A simple example, have you ever used winboard or xboard to run a local chess engine? What about an engine on a remote machine? Unix I/O does not differentiate between pipes, files, sockets, FIFOs or such at the user level. That's handled deep in the kernel. A simple fscanf() or fgets() or read() or recv() will work with any of those, without knowing which. And without requiring any changes at the program level.

You can read STDIN all you want, but you won't know whether it comes from a network socket, a local socket, a pipe, a file, the console, etc. That's the _point_ for the way Unix I/O was developed...

Today I see no need for that type of changes. While everything you both have written may certainly be correct, I think it is not an issue at the moment for chess programming.

Sven

You are writing a lot of true sentences, however they are not much related to the exact topic we are now discussing. Steven proposed to use recv/send instead of read/write in a chess engine. How is that related to your statements? How is that *not* a change of the engine implementation? How is that *not* a change of the protocol specification (WB/UCI)? Exactly *which* software do you think would go unchanged, and which not?

Sven

The _original_ topic was discussing using sockets vs pipes. In Unix, a program does not have to do _anything_ differently to use either. As far as the protocol, how, exactly, does the winboard / UCI protocol dictate how you read/write? I can do write(), you can do send(), and the GUI is not going to be able to tell the difference. If you want to _change_ the protocol, so that one _has_ to use send/recv so that you preserve message boundaries, that's ok, but nothing makes the GUI use recv even if I use send. It _may_ use recv if it wants, or it can use read()...

I don't think the WB/XB protocol needs anything more defined than the strings and what they mean. Each command is supposed to be terminated by a l/f character. What more does one need?