Writing to a Text File (Thread Safe)

Discussion of chess software programming and technical issues.

Moderator: Ras

syzygy
Posts: 5714
Joined: Tue Feb 28, 2012 11:56 pm

Re: You are all wrong

Post by syzygy »

sje wrote:You are all wrong. (...)
So show me where I recommended to not use locking.

Maybe here:
It might still be wise to use locks and not depend on printf() being atomic if you want to run on Windows.
Hmmm, guess not.

The point I am making is a very small point. The point is that POSIX requires fprintf() to be atomic, and that (non-)atomicity of lower level write()s has nothing to do with that.

That's it.

And of course there is no way around relying on the correctness of system software, unless you write your own OS, your own compiler, your own libraries, etc. And even if you do that, by your logic you would also be forced to design and produce your own chips.

Nice thing about Linux and glibc is that everybody can check for himself that fprintf() is locking internally. I did check this, btw.
User avatar
sje
Posts: 4675
Joined: Mon Mar 13, 2006 7:43 pm

fprintf()

Post by sje »

I guess the question here is why anyone is using fprintf() and its pals in any code written this century. The last C program I wrote was a tablebase generator back in 1994; it's been C++ (and Lisp and Pascal) ever since. The C language formatted I/O routines are all antiques dating back forty years ago as copies of Fortran routines which were in use since the 1960s. These routines have no type safety and have no way of checking arguments against the format string. The fscanf() routines are just as bad.

C++ has the string and stringstream classes which greatly simplify character data manipulation and remove the need for any sprintf()/sscanf() foolishness. C++ strings have auto allocation and auto reference counting, so no worry about buffer overflow or memory leakage.

The C++ I/O classes are far superior to the old C routines in every way. For any object, a coder can write custom methods for stream operators to read and write complex objects just as if they were single scalar values. But even without custom stream operators, I/O is still easier to write and easier to read. And this is important for those looking at open source programs for guidance.

In particular, I'm thinking of Bob's Crafty which has been around for nineteen years. It is a strong program and its source has been seen and used by many, sometimes without acknowledgement. I've suggested that Bob re-do Crafty in C++ as it would then be much easier for others to learn from it. This would take some effort, but I believe the result would be justify the work and the program would be just as strong and even easier to improve. Now I'm sure that Bob is a busy guy, so he may not have the time for this. But that's okay as maybe someone else could pick up the gauntlet.
syzygy
Posts: 5714
Joined: Tue Feb 28, 2012 11:56 pm

Re: fprintf()

Post by syzygy »

sje wrote:I guess the question here is why anyone is using fprintf() and its pals in any code written this century. (...)
Oh boy....
mvk
Posts: 589
Joined: Tue Jun 04, 2013 10:15 pm

Re: fprintf()

Post by mvk »

sje wrote:I guess the question here is why anyone is using fprintf() and its pals in any code written this century.
Several reasons exist. Linus uses it because it keeps a certain type of developer at a safe distance from his projects:
Linus wrote:"Quite frankly, even if the choice of C were to do *nothing* but keep the C++ programmers out, that in itself would be a huge reason to use C."
sje wrote:In particular, I'm thinking of Bob's Crafty which has been around for nineteen years. It is a strong program and its source has been seen and used by many, sometimes without acknowledgement. I've suggested that Bob re-do Crafty in C++ as it would then be much easier for others to learn from it. This would take some effort, but I believe the result would be justify the work and the program would be just as strong and even easier to improve. Now I'm sure that Bob is a busy guy, so he may not have the time for this. But that's okay as maybe someone else could pick up the gauntlet.
Agreed, Crafty code is hard to read due to the mixing of abstraction levels all over the place where C++ notation will help (but at least his variable names are understandable and the comments are just great). For your objective, one could start with Stockfish, remove lines until it plays 312 elo weaker and you're there.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: fprintf()

Post by bob »

mvk wrote:
sje wrote:I guess the question here is why anyone is using fprintf() and its pals in any code written this century.
Several reasons exist. Linus uses it because it keeps a certain type of developer at a safe distance from his projects:
Linus wrote:"Quite frankly, even if the choice of C were to do *nothing* but keep the C++ programmers out, that in itself would be a huge reason to use C."
sje wrote:In particular, I'm thinking of Bob's Crafty which has been around for nineteen years. It is a strong program and its source has been seen and used by many, sometimes without acknowledgement. I've suggested that Bob re-do Crafty in C++ as it would then be much easier for others to learn from it. This would take some effort, but I believe the result would be justify the work and the program would be just as strong and even easier to improve. Now I'm sure that Bob is a busy guy, so he may not have the time for this. But that's okay as maybe someone else could pick up the gauntlet.
Agreed, Crafty code is hard to read due to the mixing of abstraction levels all over the place where C++ notation will help (but at least his variable names are understandable and the comments are just great). For your objective, one could start with Stockfish, remove lines until it plays 312 elo weaker and you're there.
Here's a simple question. Do you think Fruit 2.1 is easier to read than Crafty? I don't? Only takes a dozen procedure calls to generate moves, which is quite hard to follow.

To each his own, I suppose. But duplicated code (multiple search functions, ala robo and friends) and search don't particularly strike me as "understandable."
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: You are all wrong

Post by bob »

syzygy wrote:
sje wrote:You are all wrong. (...)
So show me where I recommended to not use locking.

Maybe here:
It might still be wise to use locks and not depend on printf() being atomic if you want to run on Windows.
Hmmm, guess not.

The point I am making is a very small point. The point is that POSIX requires fprintf() to be atomic, and that (non-)atomicity of lower level write()s has nothing to do with that.

That's it.

And of course there is no way around relying on the correctness of system software, unless you write your own OS, your own compiler, your own libraries, etc. And even if you do that, by your logic you would also be forced to design and produce your own chips.

Nice thing about Linux and glibc is that everybody can check for himself that fprintf() is locking internally. I did check this, btw.
Unfortunately, as I mentioned, it locks, but not enough to guarantee no data corruption...
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: fprintf()

Post by bob »

sje wrote:I guess the question here is why anyone is using fprintf() and its pals in any code written this century. The last C program I wrote was a tablebase generator back in 1994; it's been C++ (and Lisp and Pascal) ever since. The C language formatted I/O routines are all antiques dating back forty years ago as copies of Fortran routines which were in use since the 1960s. These routines have no type safety and have no way of checking arguments against the format string. The fscanf() routines are just as bad.

C++ has the string and stringstream classes which greatly simplify character data manipulation and remove the need for any sprintf()/sscanf() foolishness. C++ strings have auto allocation and auto reference counting, so no worry about buffer overflow or memory leakage.

The C++ I/O classes are far superior to the old C routines in every way. For any object, a coder can write custom methods for stream operators to read and write complex objects just as if they were single scalar values. But even without custom stream operators, I/O is still easier to write and easier to read. And this is important for those looking at open source programs for guidance.

In particular, I'm thinking of Bob's Crafty which has been around for nineteen years. It is a strong program and its source has been seen and used by many, sometimes without acknowledgement. I've suggested that Bob re-do Crafty in C++ as it would then be much easier for others to learn from it. This would take some effort, but I believe the result would be justify the work and the program would be just as strong and even easier to improve. Now I'm sure that Bob is a busy guy, so he may not have the time for this. But that's okay as maybe someone else could pick up the gauntlet.
I use fprintf() because it is the easiest way to format output in standard C. And it works with stdout as well as files and everything else...
mvk
Posts: 589
Joined: Tue Jun 04, 2013 10:15 pm

Re: fprintf()

Post by mvk »

bob wrote:Here's a simple question. Do you think Fruit 2.1 is easier to read than Crafty? I don't? Only takes a dozen procedure calls to generate moves, which is quite hard to follow.
Yes, I consider Fruit 2.1 a breeze to read and easier to follow. But not as easy as Stockfish.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: fprintf()

Post by bob »

mvk wrote:
bob wrote:Here's a simple question. Do you think Fruit 2.1 is easier to read than Crafty? I don't? Only takes a dozen procedure calls to generate moves, which is quite hard to follow.
Yes, I consider Fruit 2.1 a breeze to read and easier to follow. But not as easy as Stockfish.
Different strokes for different folks I suppose. I've spent too many years looking at operating system, compiler and such code.
User avatar
sje
Posts: 4675
Joined: Mon Mar 13, 2006 7:43 pm

C++ string and stringstream classes

Post by sje »

In an earlier version of Symbolic, I did not use the C++ string classes because they were a bit buggy at the time. So I fell back to the old ways of fixed length character buffers, some static, some auto, and some on the heap. And so I always had to be careful of buffer overflow and memory leaks.

Now that the C++ string classes have been purged of nasty bugs for several years, I have used them without major problems and this makes the code much easier to read.

I have noticed one aspect of using these classes which can affect performance. Symbolic has a multi-threaded class which generates unique chess positions by building a binary ordered tree and each element of the tree contains a FEN string for the position of interest. There can be millions of tree nodes for a deep enumeration. When the FEN string is built, it is done one character at a time and this means that the underlying string class has to do dynamic allocation and that means accessing the heap. In a multi-threaded environment, the heap allocator has a mutex to guard against chaos. Since the FEN construction was a major part of building the tree, I saw a big slowdown because of the locking with four threads simultaneously bashing away at the allocator.

I fixed the problem by writing an EncodeToArray() method for the Position class which avoided the string classes. This is useful only for the unique position tree building operation, but not for anything else.