A note for C programmers

[Henk]

I remember rewriting name resolving and I changed the order of the symbols or something I can't remember. Customer got angry for it expected the symbols in a different order. Though the specification did not say anything about the order.

So this is real life.

[bob]

Wylie Garvin's comment which you quote is from a discussion of integer overflow not strcpy.

We've bounced back and forth between a couple of different issues in this thread; what they all have in common is the "this is undefined behavior" and "compiler is allowed to make demons fly out of your nose if you do that" angle.



FWIW, I read through all of the comments on the original bugzilla report in Fedora (638477) and the upstream glibc bug report (12518) (This was memcpy saga). I'll try to briefly summarize it here. I'm not a Linux user and not too familiar with the bug-resolving processes of these projects, and the bugs had some political overtones, so I probably missed some nuances.

In September 2010 some Fedora 14 users started reporting that sound in Flash and at least one other mp3-playing app (gstreamer?) was garbled. Investigation showed that the unsupported 64-bit binary Flash plugin from Adobe contained memcpy calls with overlapping source and dest range, i.e. undefined behavior. It showed up because some Intel engineers had contributed new code to glibc to optimize memcpy and memmove to copy backwards in some cases, making them faster for "small" x86 chips (Atom, etc.) In some cases the new versions were 2x to 3x faster. Glibc 2.13 shipped with this new code, which uses some runtime info (e.g. CPU feature bits, but also other transient info such as the alignment of the source and dest addresses) to choose the fastest way of copying any particular block. However, it wasn't a major version bump so existing binary apps that were linking with the glibc shared library now started showing odd behaviors they hadn't shown before. The bugs are in those binary apps, not glibc (the apps invoke undefined behavior by passing overlapping buffers to memcpy). Nonetheless, Linus argued that breaking binary compatibility in the shared library was a bad thing to do.

During the end of 2010 up to perhaps february 2011, some drama played out. Irate users complained, Fedora suggested they direct the complaints to Adobe and the other authors of the broken programs. Fedora said "this is not our problem, those programs are buggy, complain to their authors." Linus argued that the change hurt users, and made a sort-of-convincing argument that the memcpy implementation could be changed to be identical to memmove without decreasing its performance in any notable way. The Fedora maintainers argued that working around bugs in 3rd-party proprietary apps was not something they cared about. Then it was argued that a lot of other apps, even nice open-source apps that were part of the core Fedora package, might be broken in silent ways by this memcpy change. And even that there might be security vulnerabilities existing purely because of this undefined behavior case. Some counter-arguments were also made (such as this one by Siarhei Siamashka). The main Fedora response became "complain to glibc then, it was their change, whatever they decide to do upstream is fine with us". In February 2011, Linus reported it as a bug against glibc and offered them a patch to turn their memcpy into something equivalent to memmove. Adding the test-and-abort stuff was also discussed. The glibc guys then implemented a patch and resolved their bug as "fixed". I'm not totally clear what their patch did, it sounds like it arranged things so that when existing binaries linked to the shared object, their memcpy symbol was redirected to memmove. But new binaries compiled from source, get memcpy when they ask for it (the new mega-featured one).

Anyway, all of this played out more than 2 years ago. Its surprising to see the same kind of thing happening again with strcpy.

Stupid things get repeated more often than smart things. Perhaps that specific glibc maintainer moved to Apple for all we know. There's little point to breaking lots of existing hardware, just to discourage future undefined behavior usage. The memcpy() was actually a bit more reasonable, because there was a performance gain on some Intel CPUs. Changing to make something faster is generally acceptable, although one would assume speed was already adequate and since the change broke existing code it had an overall negative effect.

[rbarreira]

Since one of the ongoing quibbles here is about the definition of "undefined behavior", here is the usual definition. One that has been used since the 1950's in fact. "Behavior that can not be predicted." This has ALWAYS been an issue of unexpected results caused, NOT by the compiler intentionally doing something off-the-wall, but by the code the compiler produces, which might not always do the expected thing.

You are the only one quibbling with the Standard's definition of UB. Everyone else is on the same page here. Don't pull in whatever colloquial speech you are using into a precise and exact definition used by every compiler writer.
UB is about the freedom of a compiler to translate your C code to machine instructions.

The standards does not adequately define "undefined behavior" To wit:

3.4.3
1 undefined behavior
behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements
NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).
EXAMPLE An example of undefined behavior is the behavior on integer overflow.

So it can ignore the situation, to behaving in a documented way (both of which make sense) to terminating the compilation with a message. No mention of demons flying out one's nose or anything else.

So how about we get back to planet earth.

On planet earth, you didn't even read properly what you quoted:

"ignoring the situation completely with unpredictable results" OR "behaving during translation or program execution in a documented manner"

So yes, demons flying out of one's nose are a possible case of undefined behavior.

[simon]

Why do you say that one compiler "fails" and another one "works". The program invokes undefined behavior. The language standard defines no semantics for it at all. All of those compilers are working as intended.

It would be nice to see Bob acknowledging this. I certainly hope he's not telling his students that there's no such thing as undefined behavior.

I myself (in very rare cases) do rely on undefined behavior in C when it comes to implementing lockless algorithms, but I'm fully aware of the risk I'm taking and I wouldn't blame the compiler if things went south at some point. I also wouldn't use that in software that needs to be reliable and portable...

Note I am NOT "blaming the compiler". I am most definitely "blaming the library maintainer". Exactly as Torvalds with the memcpy() debacle.

Changing something to fix a bug is good. Changing something to make it faster is good. Changing something just to break code is NOT good, and this is exactly what Apple did (and only Apple to date).

I'm not a proponent of undefined behavior, but I do consider myself quite capable of using it when it is safe, as in lockless hashing.

Nothing the compiler can do to break that.

Wylie Garvin's comment which you quote is from a discussion of integer overflow not strcpy.

The THREAD is about strcpy(). There are multiple flavors of "undefined behavior" mentioned herein. From overlapping operands in strcpy, to integer overflow, to race conditions. My comments apply to all equally.

Do you dislike compilers which exploit undefined behaviour in the case of integer overflow to improve optimisation?

[bob]

I remember rewriting name resolving and I changed the order of the symbols or something I can't remember. Customer got angry for it expected the symbols in a different order. Though the specification did not say anything about the order.

Software engineering 101. But the issue is, once they discover internal behavior, they begin to plan on it. I didn't intentionally go for undefined behavior in strcpy(). It happened. And it also happened to work. Until Mavericks, anyway.

I fixed it immediately, but the bug was harder to find than normal because I rarely see true compiler/library issues, and immediately thought "I changed something in the last version that somehow broke PGN parsing." A lot of looking and testing later, I discovered it was not so.

BTW this generally marks the difference between theoretical and practical. The theoretical guy says "In theory I can make demons fly out of his nose if he does this, so maybe I'll just format his hard drive instead, since undefined does mean undefined." The practical guy says "I can't predict what the result will be in this case, but I can at least do exactly what the programmer specified and he has to live with the result if it does not match his expectations..."

With the deranged reasoning here, if one does a BSF instruction and the source is all zero bits, the destination register is undefined. Should the program crash? Because you can always use a "jz" to determine that there were no one bits set and eax has no useful value in it (actually, it is unchanged from its contents prior to executing the bsf).

I'm happy to say that many are "reasonable" including Intel.

[wgarvin]

LLVM Project Blog: What Every Programmer Should Know About Undefined Behavior

It turns out that C is not a "high level assembler" like many experienced C programmers (particularly folks with a low-level focus) like to think, and that C++ and Objective-C have directly inherited plenty of issues from it.

...

Both LLVM IR and the C programming language have the concept of "undefined behavior". Undefined behavior is a broad topic with a lot of nuances. The best introduction I've found to it is a post on John Regehr's Blog. The short version of this excellent article is that many seemingly reasonable things in C actually have undefined behavior, and this is a common source of bugs in programs. Beyond that, any undefined behavior in C gives license to the implementation (the compiler and runtime) to produce code that formats your hard drive, does completely unexpected things, or worse. Again, I would highly recommend reading John's article.

Since one of the ongoing quibbles here is about the definition of "undefined behavior", here is the usual definition. One that has been used since the 1950's in fact. "Behavior that can not be predicted." This has ALWAYS been an issue of unexpected results caused, NOT by the compiler intentionally doing something off-the-wall, but by the code the compiler produces, which might not always do the expected thing.

I don't think anybody is quibbling about it except for you. Everyone debating you appears to accept what the standard says; that literally anything might happen, and you have no recourse. You keep insisting there should be some logic or reason to what happens, in case X. Which is silly, because there is no meaningful difference between the compiler intentionally doing something off-the-wall and unintentionally causing you an unexpected result. Undefined behavior is UNDEFINED. You have to avoid it, or the semantics of your program are UNDEFINED. Period, full stop.

C89 defined undefined behavior like this:

* Undefined behavior --- behavior, upon use of a nonportable or erroneous program construct, of erroneous data, or of indeterminately-valued objects, for which the Standard imposes no requirements. Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).

A working draft of C11 still defines it in the same way:

3.4.3
1
undefined behavior
behavior, upon use of a nonportable or erroneous program construct or of erroneous data,
for which this International Standard imposes no requirements
2
NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable
results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).
3
EXAMPLE An example of undefined behavior is the behavior on integer overflow.

So the language stance on this has not changed at all in over 20 years. Either your understanding of it is dangerously deficient for someone who teaches programming in C to others, or else you really do understand it and just continue to argue out of stubornness. I suspect its the latter, and I apologize if I offend you by saying so, but do you not realize how foolish that looks?

[bob]

Why do you say that one compiler "fails" and another one "works". The program invokes undefined behavior. The language standard defines no semantics for it at all. All of those compilers are working as intended.

It would be nice to see Bob acknowledging this. I certainly hope he's not telling his students that there's no such thing as undefined behavior.

I myself (in very rare cases) do rely on undefined behavior in C when it comes to implementing lockless algorithms, but I'm fully aware of the risk I'm taking and I wouldn't blame the compiler if things went south at some point. I also wouldn't use that in software that needs to be reliable and portable...

Note I am NOT "blaming the compiler". I am most definitely "blaming the library maintainer". Exactly as Torvalds with the memcpy() debacle.

Changing something to fix a bug is good. Changing something to make it faster is good. Changing something just to break code is NOT good, and this is exactly what Apple did (and only Apple to date).

I'm not a proponent of undefined behavior, but I do consider myself quite capable of using it when it is safe, as in lockless hashing.

Nothing the compiler can do to break that.

Wylie Garvin's comment which you quote is from a discussion of integer overflow not strcpy.

The THREAD is about strcpy(). There are multiple flavors of "undefined behavior" mentioned herein. From overlapping operands in strcpy, to integer overflow, to race conditions. My comments apply to all equally.

Do you dislike compilers which exploit undefined behaviour in the case of integer overflow to improve optimisation?

I dislike compilers that would completely remove the calculations, yes. Because the HARDWARE is perfectly capable of handling the overflow and producing the result any good asm textbook gives. I don't consider that "an improved optimization". I'd be interested to know what hardware doesn't deal with integer overflow correctly. 99.9%+ are intel compatible and they certainly handle it. As does Cray, Alpha, MIPS, SPARC, Itanium, old univacs, CDCs, I can't think of a single machine I have programmed on that might have a problem, unless you go back to something like the IBM 1620 which was a decimal machine rather than binary. And even then overflow was handled as correctly as it can be, or at least predictably since you lose a digit.

[Rein Halbersma]

The standards does not adequately define "undefined behavior" To wit:

3.4.3
1 undefined behavior
behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements
NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).
EXAMPLE An example of undefined behavior is the behavior on integer overflow.

So it can ignore the situation, to behaving in a documented way (both of which make sense) to terminating the compilation with a message. No mention of demons flying out one's nose or anything else.

In an ISO Standard, notes are not normative or exhaustive, but only for clarification. The "imposes no requirements" is the normative text. Since it is an open set, "demons flying out one's nose" would be allowed should said nose be attached to the machine in question.

[bob]

Since one of the ongoing quibbles here is about the definition of "undefined behavior", here is the usual definition. One that has been used since the 1950's in fact. "Behavior that can not be predicted." This has ALWAYS been an issue of unexpected results caused, NOT by the compiler intentionally doing something off-the-wall, but by the code the compiler produces, which might not always do the expected thing.

You are the only one quibbling with the Standard's definition of UB. Everyone else is on the same page here. Don't pull in whatever colloquial speech you are using into a precise and exact definition used by every compiler writer.
UB is about the freedom of a compiler to translate your C code to machine instructions.

The standards does not adequately define "undefined behavior" To wit:

3.4.3
1 undefined behavior
behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements
NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).
EXAMPLE An example of undefined behavior is the behavior on integer overflow.

So it can ignore the situation, to behaving in a documented way (both of which make sense) to terminating the compilation with a message. No mention of demons flying out one's nose or anything else.

So how about we get back to planet earth.

On planet earth, you didn't even read properly what you quoted:

"ignoring the situation completely with unpredictable results" OR "behaving during translation or program execution in a documented manner"

So yes, demons flying out of one's nose are a possible case of undefined behavior.

Yes I did. And read my first sentence. That is not "adequately defined." And assuming it means demons is nonsensical. It should mean exactly what it implies. The operation will be done as given, the results are unpredictable depending on the architecture of the underlying machine.

For example, what happens when you compute x / 0.0 on Intel? Depends on whether you have allowed divide by zero and enabled infinity. On a Vas? program exception. Good example of undefined. Can't say which without knowing details. Once I know the CPU and in the case of intel, the floating point control word, I can tell you exactly what will happen, whether you like it or not is a different issue, but one you can handle within your program.

To claim that it can do something completely unrelated is utter nonsense.

[bob]

If you want to be sure that strcpy always copies from left to right better create a scrcpy2 yourself that does copy from left to right. You only need a global substitution to change strcpy in strcpy2.

The worst thing is to debug and find out that strcpy had been the cause of the bug.

I remember I had to implement an application in a so called TU(Lisp) language while they were changing the compiler/interpreter almost every day. They used me for testing their compiler. I did not like that at all.

By the way bugs in the garbage collector, I think, are the worst.

One doesn't need to use a library at all, but it is a large effort project to eliminate such. If things switched from working to crashing and back regularly, no one would use a library in the first place.