Reborn Compile DevC++ but misses GUI commands,works by hand

[Daniel White]

But I think the important part is to switch off INPUT buffering, e.g. by

Code: Select all

setvbuf(stdin, NUL, _IONBF, 0);

at program startup, since otherwise the engine will not receive GUI commands immediately (as it should) but only when the input buffer is filled.

Uh?

None of my engines or adapters has ever done anything like that. (I did not even know this was possible.) So I don't think what you say can be true, for it would certainly prevent them from working.

My experience is that when you issue a read call and there is input, you will get the input. Both from console and pipes.

There is another problem that puzzles me, though. It seems that sometimes the methods to check for pending input (e.g. to decide if you should abort pondering) fail to see the input. In particular this happens when you resume pondering after receiving a time or otim command. You will then never become aware that the opponent has moved.

The time, otim and usermove are all sent by the GUI at the same time (though in different write calls). So what seems to happen is that as soon as you start reading in response to the arrival of 'time', all pending input (including the 'usermove') is read and put into a buffer in the engine. And that the PeekNamedPipe command for checking if there is input only looks at the pipe, and not if there still is stuff left over in the buffer from a previous pipe read.

Perhaps switching of stdin buffering in the way you mention would prevent this. I currently prevent this by skipping ponder code after reception of time and otim. But the preferred way of solving it would be to check if the buffer is empty. Not buffering really makes for very inefficient I/O code.

As I'm sure you are aware there is also the option of having a separate thread which simply blocks waiting for input. If relevant input is received during pondering a flag is updated and the search thread returns to a waiting function (or continues if ponderhit).

[hgm]

Sure, I am aware of that. But I don't like to use threads when I can avoid it; it introduces a lot of platform dependence in the code, you need to link with extra libraries, etc. So I only use threads in my adapters, where it is sort of unavoidable, because you never know from which side (engine or GUI) the next input will come.

[Sven]

But I think the important part is to switch off INPUT buffering, e.g. by

Code: Select all

setvbuf(stdin, NUL, _IONBF, 0);

at program startup, since otherwise the engine will not receive GUI commands immediately (as it should) but only when the input buffer is filled.

Uh?

None of my engines or adapters has ever done anything like that. (I did not even know this was possible.) So I don't think what you say can be true, for it would certainly prevent them from working.

My experience is that when you issue a read call and there is input, you will get the input. Both from console and pipes.

There is another problem that puzzles me, though. It seems that sometimes the methods to check for pending input (e.g. to decide if you should abort pondering) fail to see the input. In particular this happens when you resume pondering after receiving a time or otim command. You will then never become aware that the opponent has moved.

The time, otim and usermove are all sent by the GUI at the same time (though in different write calls). So what seems to happen is that as soon as you start reading in response to the arrival of 'time', all pending input (including the 'usermove') is read and put into a buffer in the engine. And that the PeekNamedPipe command for checking if there is input only looks at the pipe, and not if there still is stuff left over in the buffer from a previous pipe read.

Perhaps switching of stdin buffering in the way you mention would prevent this. I currently prevent this by skipping ponder code after reception of time and otim. But the preferred way of solving it would be to check if the buffer is empty. Not buffering really makes for very inefficient I/O code.

Well, the fact that you never used it in any of your engines does not prove that it does not work, it only proves that you don't know whether it works

I found a couple of engines on my harddisk which are actually using setbuf(stdin, NULL) or setvbuf(stdin, NULL, _IONBF, 0), both WinBoard and UCI engines. Here you are:

Cheng
Dabbaba
Daydreamer
Fruit
Glaurung
Strelka
Sungorus
Surprise ( )
KnockOut ( )
DoubleCheck
Stockfish
Toga

The Chess Engine Communication Protocol which is something you might know says this (here I assume the text was written by Tim Mann):

The most direct way to fix this problem is to use unbuffered operating system calls to read (and poll) the underlying file descriptor directly. [...]

A second way to fix the problem might be to ask your I/O library not to buffer on input. It should then be safe to poll the underlying file descriptor as described above. With C, you can try calling setbuf(stdin, NULL). However, I have never tried this. Also, there could be problems if you use scanf(), at least with certain patterns, because scanf() sometimes needs to read one extra character and "push it back" into the buffer; hence, there is a one-character pushback buffer even if you asked for stdio to be unbuffered. With C++, you can try cin.rdbuf()->setbuf(NULL, 0), but again, I have never tried this.

A third way to fix the problem is to check whether there are characters in the buffer whenever you poll. [...]

A fourth way to fix the problem is to use a separate thread to read from stdin. [...]

That protocol spec was initially written a long while ago. Several credible programmers (see above) have tried switching off input buffering since then, and proven that it works. I can add that it is not only a working solution for the "polling for input during search" problem (which is the starting point for the four proposals of Tim Mann shown above) but also, at least with some compilers on some systems, for the simplistic "one-thread-one-loop-using-fgets()" approach of engines lacking the polling feature. Here you CAN be lost already when using buffered input. But don't focus on that since most engines will add the polling feature at some point in time, and there you go to choose one of those four ways.

When using read() instead of a line-oriented method like fgets(), you are right of course, but as you say there are certain other problems to be solved then (they are in fact solved but not "trivial" for everyone). I remember quite a long thread about that a while ago. I suggest to accept, though, that there are indeed also engines starting with a more simplistic I/O concept, not with the sophisticated (and most probably better) "read()" approach, and it is not a big effort to switch to high-tech solutions in a later stage.

Your comment regarding "inefficiency" is wrong in my opinion since a GUI never sends a lot of data towards the engine. Only the other direction (output from engine to GUI) can become a performance problem, which is another topic not to be discussed here (there are several working solutions for it).

Sven

[diep]

Sven, if you manage to convince HGM, you get the special award: "managed to convince a professor award".

That's a precious award, gets awarded more seldom than Nobel Prizes

[Sven]

Sven, if you manage to convince HGM, you get the special award: "managed to convince a professor award".

That's a precious award, gets awarded more seldom than Nobel Prizes

Sounds attractive to me. I'll do what I can

Would you convert to bitboarding in case I succeed?

Sven

[hgm]

Well, the fact that you never used it in any of your engines does not prove that it does not work, it only proves that you don't know whether it works

Note that I was not claiming it would not work. Just that it was not needed, and that your argument for why it would be needed seems falsified by the fact that my engines work without doing this:

In all Windows or Linux systems I encountered so far, an arbitrarily short line sent through a pipe will always immediately be picked up by the receiving program. There is no 'waiting until the input buffer fills up'. (There is on output, of course.) It could be that there is buffering until the reception of a linefeed, but WB protocol is constructed so that this would never cause a problem. (All commands end in LF.) This is using getchar() for input.

I found a couple of engines on my harddisk which are actually using setbuf(stdin, NULL) or setvbuf(stdin, NULL, _IONBF, 0), both WinBoard and UCI engines. Here you are:

Cheng
Dabbaba
Daydreamer
Fruit
Glaurung
Strelka
Sungorus
Surprise ( )
KnockOut ( )
DoubleCheck
Stockfish
Toga

Well, that they use it does not imply they need it. Have you tried running them with those calls deleted? (And, like I said, they might need it to solve the other problem, of PeekNamedPipe nort working reliably, which I solved in another way.)

Your comment regarding "inefficiency" is wrong in my opinion since a GUI never sends a lot of data towards the engine. Only the other direction (output from engine to GUI) can become a performance problem, which is another topic not to be discussed here (there are several working solutions for it).

Well, this would have to be measured. In general I try to minimize calling the system, as you never know what horrible things it will do. It might decide to swap your process. It seems that without buffering you would have to use system calls to read one character at the time.

Checking of the buffer seems the logical solution for this problem. I was disgusted to discover that PeekNamedPipe did not do this automatically, and that no higher-level routine that does is recommended to replace it.

[Pablo Vazquez]

Well, the fact that you never used it in any of your engines does not prove that it does not work, it only proves that you don't know whether it works

Note that I was not claiming it would not work. Just that it was not needed, and that your argument for why it would be needed seems falsified by the fact that my engines work without doing this:

In all Windows or Linux systems I encountered so far, an arbitrarily short line sent through a pipe will always immediately be picked up by the receiving program. There is no 'waiting until the input buffer fills up'. (There is on output, of course.) It could be that there is buffering until the reception of a linefeed, but WB protocol is constructed so that this would never cause a problem. (All commands end in LF.) This is using getchar() for input.

I found a couple of engines on my harddisk which are actually using setbuf(stdin, NULL) or setvbuf(stdin, NULL, _IONBF, 0), both WinBoard and UCI engines. Here you are:

Cheng
Dabbaba
Daydreamer
Fruit
Glaurung
Strelka
Sungorus
Surprise ( )
KnockOut ( )
DoubleCheck
Stockfish
Toga

Well, that they use it does not imply they need it. Have you tried running them with those calls deleted? (And, like I said, they might need it to solve the other problem, of PeekNamedPipe nort working reliably, which I solved in another way.)

Your comment regarding "inefficiency" is wrong in my opinion since a GUI never sends a lot of data towards the engine. Only the other direction (output from engine to GUI) can become a performance problem, which is another topic not to be discussed here (there are several working solutions for it).

Well, this would have to be measured. In general I try to minimize calling the system, as you never know what horrible things it will do. It might decide to swap your process. It seems that without buffering you would have to use system calls to read one character at the time.

Checking of the buffer seems the logical solution for this problem. I was disgusted to discover that PeekNamedPipe did not do this automatically, and that no higher-level routine that does is recommended to replace it.

The problem with checking the buffer is portability. The stdin->_cnt thing is beyond ugly, and I remember it didn't compile with some windows compilers, not to mention linux. Input buffering could certainly be left on, and there wouldn't be much of a difference, except in extreme cases like sending go + stop or go ponder + ponderhit at the same time or very quickly, so that when the engine reads the go command, the stop or ponderhit goes to the buffer, select() misses it and the engine doesn't stop the search. I don't know if there would be more problems in wb though.

[diep]

Sven, if you manage to convince HGM, you get the special award: "managed to convince a professor award".

That's a precious award, gets awarded more seldom than Nobel Prizes

Sounds attractive to me. I'll do what I can

Would you convert to bitboarding in case I succeed?

Sven

My incremental datastructure is way faster in 32 bits than 64 bits. Only a few things are faster in 64 bits, for example zobrist hashkey.

That's not much but it's something.

Fact is that bitboards are way way slower for Diep as i use the information i generate in evaluation a lot. If one has to regenerate that with bitboards, you slow down bigtime.

I've already shown examples of that.

As a matter of fact my current hashtables use 32 bits code.

Maybe i'll convert that to 64 bits if you convince the professor

That is, in 32 bits Diep is 7-11% faster than in 64 bits at core2 using the intel c++ compiler.

So i don't really see why i would do that much effort for 64 bits yet

You can issue more instructions in 32 bits than in 64 bits for a processor, so 32 bits is always faster for Diep of course.

And at the current time controls everyone tests over here i don't really see why a hashtable of more than 4GB would be useful to anyone


That said i am busy converting wherever possible 'signed int' to 'unsigned int', so that future 64 bits attempts aren't that much slower like they are now.

7% slowdown moving from 32 bits to 64 bits really is too much.

[Sven]

Sven, if you manage to convince HGM, you get the special award: "managed to convince a professor award".

That's a precious award, gets awarded more seldom than Nobel Prizes

Sounds attractive to me. I'll do what I can

Would you convert to bitboarding in case I succeed?

Sven

My incremental datastructure is way faster in 32 bits than 64 bits. Only a few things are faster in 64 bits, for example zobrist hashkey.

That's not much but it's something.

Fact is that bitboards are way way slower for Diep as i use the information i generate in evaluation a lot. If one has to regenerate that with bitboards, you slow down bigtime.

I've already shown examples of that.

As a matter of fact my current hashtables use 32 bits code.

Maybe i'll convert that to 64 bits if you convince the professor

That is, in 32 bits Diep is 7-11% faster than in 64 bits at core2 using the intel c++ compiler.

So i don't really see why i would do that much effort for 64 bits yet

You can issue more instructions in 32 bits than in 64 bits for a processor, so 32 bits is always faster for Diep of course.

And at the current time controls everyone tests over here i don't really see why a hashtable of more than 4GB would be useful to anyone


That said i am busy converting wherever possible 'signed int' to 'unsigned int', so that future 64 bits attempts aren't that much slower like they are now.

7% slowdown moving from 32 bits to 64 bits really is too much.

Mailbox with 64 bits will indeed most probably be slower than mailbox with 32 bits. But that does not say anything about the comparison between "mailbox/32 bits" and "bitboards/64 bits", considering the tremendous speedup you would get in the evaluation function from using bitboards there. Also your attack table could be dumped with bitboards.

Most of the information you generate during evaluation, and reuse later on during search, will be available a lot cheaper with bitboards.

I think we have already left the original thread topic ...

Sven