Losing on time

[mvk]

Therefore only the "same process group" approach would help for the SIGSTOP approach.

That would not work with my program in multi-cpu mode: There I create my own process group such that I can use signals internally, without messing up whoever launched me. And that trick then goes both ways: a signal sent to the adaptor wouldn't propagate to my program and its child processes.

[Sven]

The engine might not buffer, but the OS will. A pipe is not something inside the engine. It is a memory buffer (or even a disk file) in the OS, similar to sectors in the disk cache. The user program does not have any control over this, neither the program that writes, nor the program that reads. setbuf or fflush only determine when the user program will use the write() system call to hand the data to the OS (namely immediately), but not what the OS does with that data once it passed though the write().

Sounds right in principle, but then how do you think xboard interacts with an engine to avoid delays when receiving engine moves (or other commands)? You don't want to say that the GUI might happen to process commands from the engine only after the corresponding pipe buffer is full?

Does xboard use real pipes or pseudo-ttys on Linux? I would assume it uses real pipes, but in the latter case we would already have the explanation why it works (for pseudo-ttys line buffering is used) ...

I think the explanation why xboard is not affected by pipe buffering is in "backend.c", function "StartChessProgram()". There you have a call to AddInputSource() with parameter "lineByLine" set to TRUE, so I'd say that this code manages somehow to always receive engine input line by line. Would you agree?

It works because in case of "lineByLine" xboard uses read() which reads immediately what is available in the pipe, in this case up to the next line ending.

So to return to the original point regarding SIGSTOP: you are right, the pipe buffer will fill up with commands sent by the engine that the adapter does not read since it has received a SIGSTOP. 64k pipe buffer size (Linux) are a lot, engines usually do not produce that much output in a short time frame. So the direction engine -> adapter would not cause an implicit blocking of the engine. What about the other direction, adapter -> engine? The adapter is stopped so it does not write anything into the pipe from which the engine tries to read commands. So the engine processes all lines that made it into the pipe, then it sees no further input is available. What happens now would depend on the input processing strategy of the engine: it may or may not do a blocking read; usually it won't. So this direction will probably not cause implicit blocking as well.

Therefore only the "same process group" approach would help for the SIGSTOP approach.

HGM 1 - Sven 0
or (to obtain a better ELO for me)
HGM 99 - Sven 98

For the record, read reads EVERYTHING up to either (a) max byte count given in the read or (b) EVERYTHING available in the pipe/socket/file/etc... It doesn't stop at a linefeed.

Yes, xboard uses read() to read as much characters as available *and* as it has space available in its own buffer, then splits the input into lines and processes each line separately, then it moves any characters that are still left unprocessed (i.e., incomplete line parts) to the beginning of that buffer. That's what I meant but my wording was not precise enough. The important part for the point under discussion, though, was that read() lets xboard receive immediately what is available in the pipe, while buffered input would not.

[Rebel]

Sure HGM, you define your rules, it's your software after all, no problem. However in the end it's the programmer who decides which GUI's he is going to support.

And it will be the user who decides if he wants to use the buggy engines that such a programmer produces...

If an engine behaves exactly as intended by the programmer there aren't any bugs.

I like a rich featured engine and if it so now and then loses a crazy bullet game on time (but never on serious time controls) that's a small sacrifice.

Tell me, which of the engines supports:

1. Finds any game(s) by position in a multi million PGN database within 1 centi second and produces an HTML?

2. Which engine creates a large (extended) opening book from human and/or computer games and adds considerable elo on top of the regular opening book?

3. Which engine produces statistics like these ?

I could go on.

Obviously we have conflcting perceptions of what an engine is, for me it's more than just producing mainlines and moves. However that doesn't give you the right to call someone else toil buggy. Especially not if the engine exactly performs as the programmer intended.

[bob]

Therefore only the "same process group" approach would help for the SIGSTOP approach.

That would not work with my program in multi-cpu mode: There I create my own process group such that I can use signals internally, without messing up whoever launched me. And that trick then goes both ways: a signal sent to the adaptor wouldn't propagate to my program and its child processes.

Do you not have problems with signal delivery latency? On a multi-core Linux box I am not sure they are acted on instantly. You can set the signal bit in the signal vector, but that specific process usually reacts to it only when something happens (it does an I/O, its quantum runs out, an interrupt preempts that process on that core, etc.)

I have never liked the instability in terms of latency, nor the fact that if two different threads send the same thread a signal at the same time, the target thread only gets one signal delivered.

[hgm]

Obviously we have conflcting perceptions of what an engine is, for me it's more than just producing mainlines and moves. However that doesn't give you the right to call someone else toil buggy. Especially not if the engine exactly performs as the programmer intended.

Well, freedom of speech actually means I do have the right to say that.

But what I think of your engine should not bother you at all: I don't use it anyway. I am just informing you how I think the average user would perceive your program.

But you are right, of course: if the engine behaves exactly as the programmer intended, it is not a bug but a feature. I would call software that is intended to not work as the standard prescribes 'sabotaged'.

The reasons you give seem mostly a smokescreen, however. Finding positions in a multi-million PGN database (how do you do that, BTW? must the database be indexed first?) hardly seems something you would do during a game, and thus seems a bit of a weak excuse for forfeiting those games on time.

Don't get me wrong: these are very nice and useful features. But it seems to me they should be promoted as compensation for other bad behavior, rather than as a reason for it. If the problem can be solved by letting the GUI allow the engine to go a certain fixed time interval over the time limit, it can always be solved in the engine by subtracting that same time interval from the time limit it receives from the GUI. If you argue that the amount of time needed for this is system dependent, and must be configured by the user, you can provide an engine option for the user to set it. Apparently you want to force him to set an option anyway, and to the user it will not be any more cumbersome when it is an engine option. On the contrary, by making it an engine option the user will be immediately reminded that your engine needs this particular option, which he might not realize if it is just a general GUI option.

There seems to be little reason for facilitating excess time use in engines through the GUI. Users who think time control should not be strictly enforced, they can always switch Auto Flag off, and engines will be allowed to exceed time as much as they want.

[bob]

Sure HGM, you define your rules, it's your software after all, no problem. However in the end it's the programmer who decides which GUI's he is going to support.

And it will be the user who decides if he wants to use the buggy engines that such a programmer produces...

If an engine behaves exactly as intended by the programmer there aren't any bugs.

I like a rich featured engine and if it so now and then loses a crazy bullet game on time (but never on serious time controls) that's a small sacrifice.

Tell me, which of the engines supports:

1. Finds any game(s) by position in a multi million PGN database within 1 centi second and produces an HTML?

2. Which engine creates a large (extended) opening book from human and/or computer games and adds considerable elo on top of the regular opening book?

3. Which engine produces statistics like these ?

I could go on.

Obviously we have conflcting perceptions of what an engine is, for me it's more than just producing mainlines and moves. However that doesn't give you the right to call someone else toil buggy. Especially not if the engine exactly performs as the programmer intended.

I think you miss HIS point. Would someone REALLY want his program to slightly overstep the time control limit on each and every move? Doesn't seem reasonable. This reminds me of the old doubles game argument you used to jump on whenever it came up. Remember my response? Rather than complaining about the SSDF allowing doubles when YOU thought they should cull them and not count 'em, I told you you ought to fix your program so it would not repeat lost games multiple times. I took my own advice and spent a good bit of time adding "book learning" to Crafty and I never had to worry about playing the same opening over and over and losing every game as a result...

Rather than wanting a GUI that lets you overstep the time limit, why not fix the code so that it doesn't do that? Solves far more problems than it causes...

[Evert]

In Sjef (my match-playing program for chess variants based on Sjaak) I have an option to ignore the flag and always give a certain minimum time to the engine. The idea here was to transition from a normal time control to a fixed-time-per-move time control once time is close to running out. The reason I did this was to improve the quality of play at ultra short time controls, so I could play more games (for better statistics) without the results being dominated by horrible moves played at move 40.

I don't think it actually worked all that well and I don't use the option anymore (in fact, I don't use Sjef all that often either - for some reason that I have yet to figure out the draw rate in self-play matches is much higher than if I run the match under XBoard, which is weird). In fact, using Fischer time-controls is probably a better idea for a similar overall effect.

[Rebel]

I think you miss HIS point. Would someone REALLY want his program to slightly overstep the time control limit on each and every move? Doesn't seem reasonable. This reminds me of the old doubles game argument you used to jump on whenever it came up. Remember my response? Rather than complaining about the SSDF allowing doubles when YOU thought they should cull them and not count 'em, I told you you ought to fix your program so it would not repeat lost games multiple times. I took my own advice and spent a good bit of time adding "book learning" to Crafty and I never had to worry about playing the same opening over and over and losing every game as a result...

Funny that you bring up this old cow as both CCRL and CEGT don't allow opening books nor learning.

[mvk]

Funny that you bring up this old cow as both CCRL and CEGT don't allow opening books nor learning.

I believe those policies are just as bad as allowing time overruns, or disabling pondering, or not giving access to endgame tables. Such policies eventually stifle development of the eco system.

[Rebel]

Funny that you bring up this old cow as both CCRL and CEGT don't allow opening books nor learning.

I believe those policies are just as bad as allowing time overruns, or disabling pondering, or not giving access to endgame tables. Such policies eventually stifle development of the eco system.

I am in agreement. On one hand I can understand the wish for a pure engine strength rating list as practiced in the last 10+ years. On the other hand the rating list folks could consider a complete rating list where opening books, PB and learning is allowed as it closely emulates programmer tournaments like CCT, CSVN and (uff hehe) ICGA.

It would trigger the programmers creativity to explore a new area of competition.