unbuffered input/ouput

[Daniel Shawul]

Also the read() function reads everything from the stream unlike fgets() which stops at the end of a new line. So you get only one command with fgets(). However you have to use yet another buffer, clear out all the intermediate carriage returns and other stuff etc... It can read only part of the command , and while you are processing what you read you realize you need more parameters and do another read to get the whole command.
It is a lot more worse than I thought, and I now remember why I used it only in console mode... The amount of code spent on crafty Read() and ReadInput() says a lot about the inconvience of the method.

[bob]

If it doesn't work, why does it matter why? Move on to something that does, and always has. This is called "beating your head against a brick wall". It is not a very effective way of developing code.

Wrong again. It hasn't stopped me from moving on playing on a 19x19 board. Infact I had already something in place when I first asked the question. If it doesn't interest you that I pursue this, _you_ move on.
Would be a lot more easier if you concentrate on the question.

*strchr(buffer, '\n') = NULL;

will do the trick. If you are not sure a n/l character is present,

if ((p = strchr(buffer, '\n')) *p = NULL;

This is not enough for winboard gui.

read(fileno(stdin),buffer,MAX_STR);
if ((p = strchr(buffer, '\n')) *p = NULL;

can not replace a simple fgets() yet. So you see the inconviniece has already grown...

boo hoo. There is a solution. Tim gives it in the engine interface document. I implemented my input exactly that way. I have exactly zero problems on any platform I use. And I use every platform known to man. Why would you not want to do things the right and easy way, and be done with it once and for all? Sometimes convenience is not what this is about. It is about functionality first.

[bob]

Also the read() function reads everything from the stream unlike fgets() which stops at the end of a new line. So you get only one command with fgets(). However you have to use yet another buffer, clear out all the intermediate carriage returns and other stuff etc... It can read only part of the command , and while you are processing what you read you realize you need more parameters and do another read to get the whole command.
It is a lot more worse than I thought, and I now remember why I used it only in console mode... The amount of code spent on crafty Read() and ReadInput() says a lot about the inconvience of the method.

Again, this is about functionality. You have to write what works, not what is easy and what ought to work. It is not hard to read in lines of data. It is not a hundred lines of code in Crafty, and it can be called from anywhere after it is written. Problem solved, once and for all.

What's so hard about that???

[Daniel Shawul]

As to functionality, what other problems are there in using fgets(). Tim does indeed mention scanf() could be damaged with some patterns in some cases. There is really no problem with fgets() for most engines as no sane chess GUI will send a 254 character long command. Do you know of a problem except the one I brought up where fgets() is functionally worse than read() ?

[bob]

As to functionality, what other problems are there in using fgets(). Tim does indeed mention scanf() could be damaged with some patterns in some cases. There is really no problem with fgets() for most engines as no sane chess GUI will send a 254 character long command. Do you know of a problem except the one I brought up where fgets() is functionally worse than read() ?

fgets() uses two levels of buffering, one in the operating system, one in the file system. It is hard to reliably ask the question "Is there any more input pending" because who are you asking? A system call will ask the system, but the system won't know about the stuff fgets() in the C library has already read and buffered. If you use a library call, it won't know about stuff buffered in the system.

read() has none of those issues. One buffer in the O/S where you can check for input accurately, without giving up buffering on output to make things faster/smoother...

That's why this is the recommended approach in engines-intf.html with xboard...

[Sven]

As to functionality, what other problems are there in using fgets(). Tim does indeed mention scanf() could be damaged with some patterns in some cases. There is really no problem with fgets() for most engines as no sane chess GUI will send a 254 character long command. Do you know of a problem except the one I brought up where fgets() is functionally worse than read() ?

fgets() uses two levels of buffering, one in the operating system, one in the file system. It is hard to reliably ask the question "Is there any more input pending" because who are you asking? A system call will ask the system, but the system won't know about the stuff fgets() in the C library has already read and buffered. If you use a library call, it won't know about stuff buffered in the system.

read() has none of those issues. One buffer in the O/S where you can check for input accurately, without giving up buffering on output to make things faster/smoother...

That's why this is the recommended approach in engines-intf.html with xboard...

Bob, with this post you have completely missed the point IMO. The reported problem does occur *only* with unbuffered input, so all your notes regarding buffers are exactly 100% irrelevant in the given case.

Furthermore, you are also missing the fact that the reported problem occurs on Windows only, and there you cannot talk about "system calls" really, in the way you did. While under UNIX/Linux read() is in fact a system call, it isn't under Windows. MSVC provides a read() (or _read() ?) library function that internally calls some WinAPI functions, one of these being ReadFile(), at least in the "unbuffered input" case.

It may be correct that using read() as in Crafty *can* solve the reported problem related to long FEN strings and other input lines >254 chars. But this problem does *not* exist for standard chess, so read() solves exactly nothing which practically exists as a problem relevant for most of us.

And as noted by Daniel, read() introduces a lot of useless low-level complexity. Please consider that other engines are working perfectly for standard chess with fgets() or getline().

It is also definitely wrong to state that the read() method were the method recommended by Tim Mann in "engine-intf.html". Please read again carefully.

Sven

[bob]

As to functionality, what other problems are there in using fgets(). Tim does indeed mention scanf() could be damaged with some patterns in some cases. There is really no problem with fgets() for most engines as no sane chess GUI will send a 254 character long command. Do you know of a problem except the one I brought up where fgets() is functionally worse than read() ?

fgets() uses two levels of buffering, one in the operating system, one in the file system. It is hard to reliably ask the question "Is there any more input pending" because who are you asking? A system call will ask the system, but the system won't know about the stuff fgets() in the C library has already read and buffered. If you use a library call, it won't know about stuff buffered in the system.

read() has none of those issues. One buffer in the O/S where you can check for input accurately, without giving up buffering on output to make things faster/smoother...

That's why this is the recommended approach in engines-intf.html with xboard...

Bob, with this post you have completely missed the point IMO. The reported problem does occur *only* with unbuffered input, so all your notes regarding buffers are exactly 100% irrelevant in the given case.

Not so fast. When you set the buffer size to zero, which one are you setting?

Furthermore, you are also missing the fact that the reported problem occurs on Windows only, and there you cannot talk about "system calls" really, in the way you did.

Windows came from DECs VMS group. It is a perfectly normal-looking operating system and certainly has system calls. So I am not sure what you are trying to say in the above...

While under UNIX/Linux read() is in fact a system call, it isn't under Windows. MSVC provides a read() (or _read() ?) library function that internally calls some WinAPI functions, one of these being ReadFile(), at least in the "unbuffered input" case.

It may be correct that using read() as in Crafty *can* solve the reported problem related to long FEN strings and other input lines >254 chars. But this problem does *not* exist for standard chess, so read() solves exactly nothing which practically exists as a problem relevant for most of us.

Read solves a significant problem. It eliminates two levels of buffering, making it harder to determine if there is input available since the OS is not aware of the library buffer contents, and vice versa. I perfectly understand that fgets() is only failing in windows. My point is, that read() is _not_ failing anywhere. How is that not relevant?

And as noted by Daniel, read() introduces a lot of useless low-level complexity. Please consider that other engines are working perfectly for standard chess with fgets() or getline().

It is also definitely wrong to state that the read() method were the method recommended by Tim Mann in "engine-intf.html". Please read again carefully.

Sven

I wrote part of it. Yes it evolved to add set the library buffer size to zero. Initially it recommended read(). For good reason...

read() has other advantages. It doesn't need a l/f to terminate a line. It simply leaves the parsing up to the author. I fail to see how using read() is so onerous. My students have to do it in network programming, one never uses fgets() type calls there for obvious reasons. I've used it in my chess program, my testing referee, it is not an onerous programming task...

[Daniel Shawul]

Read solves a significant problem. It eliminates two levels of buffering, making it harder to determine if there is input available since the OS is not aware of the library buffer contents, and vice versa.

You are using an extra buffer in your code which we don't need.
Using fgets() gets you exactly one command. Besides that you introduced additonal work to complete partially read command, remove intermediate line feeds etc... So the buffer and other work you saved from the library function fgets(), you have it in Read().

I perfectly understand that fgets() is only failing in windows. My point is, that read() is _not_ failing anywhere. How is that not relevant?

You can't use this new problem on windows for your argument because that is not really necessary for a chess engine at all. So fgets() is _not_ failing anywhere AFAIK. There is zero problem with fgets() so far, unless you know of other systems where it failed.

I wrote part of it. Yes it evolved to add set the library buffer size to zero. Initially it recommended read(). For good reason...

read() has other advantages. It doesn't need a l/f to terminate a line. It simply leaves the parsing up to the author. I fail to see how using read() is so onerous. My students have to do it in network programming, one never uses fgets() type calls there for obvious reasons. I've used it in my chess program, my testing referee, it is not an onerous programming task...

That is not an advantage but infact a big disadvantage. The winboard protcol has '\n' as the linefeed. So fgets() happens to be a blessing because of that.
You may need read() for network programing and other stuff where you want complete freedom as to the formating but in this case is completely unnecessary. I have implemented three protocols winboard, uci, and go text protocol (gtp) and all use the same thing. I would say the standard is '\n' for GUI engine communication. read() is more appropriate for binary data and other stuff but not for a text based GUI protocol.

It also forces you to read a bunch of commands at one time instead of just one and process that ? If there is f.i a function readN() which does exactly like read() but stops at '\n', would you still use read() ?
read() is justified only if you have a reason for that.

[sje]

While read()/write() are good, recv()/send() are even better. There are many more options plus all the benefits of socket connections.

Unix pipes are a good idea, but it's not such a good idea to force them to do something much beyond their original design.

http://www.linuxmanpages.com/man2/recv.2.php

http://www.linuxmanpages.com/man2/send.2.php

[Sven]

While read()/write() are good, recv()/send() are even better. There are many more options plus all the benefits of socket connections.

Unix pipes are a good idea, but it's not such a good idea to force them to do something much beyond their original design.

http://www.linuxmanpages.com/man2/recv.2.php

http://www.linuxmanpages.com/man2/send.2.php

I fully agree that using sockets in general would be superior to using pipes and stdio. However, could you please elaborate on how this could help in solving the problem discussed here, given the existing chess engine protocols WinBoard and/or UCI?

It seems that defining a new protocol which is based on communication via sockets could be a possible way to go. But all engines would have to be rewritten, and the same applies to all chess GUIs and other tools.

Am I missing something?

Sven