malloc out of memory

[marcelk]

so it's very simple, just check the errno and that's it.

If you do it like that you get false errors: malloc doesn't clear errno in case of success. Errno is used to communicate the reason of failure, not the failure itself.

Just check for NULL returned out of malloc.

(you can of course clear errno yourself before the function call, but that is not idiomatic)

PS: In case of realloc, beware to store the return value first in a different location than the input parameter. Because in case of failure, the old region is still allocated. If you do "p = realloc(p, ...)" you have a memory leak in case of failure.

[lucasart]

Here is Symbolic's code for determining physical RAM availability for OpenBSD (Apple Mac OS/X) and for Linux:

Code: Select all

typedef unsigned int ui;
typedef unsigned long long ui64;

ui64 PhysicalBytes(void)
{
  ui64 count;

#if HostOsApple
  {
    int mib[2];
    size_t countlen = sizeof(count);

    mib[0] = CTL_HW;
    mib[1] = HW_MEMSIZE;
    sysctl(mib, 2, &count, &countlen, NULL, 0);
  };
#endif

#if HostOsLinux
  {
    const ui pagecount = (ui) sysconf(_SC_PHYS_PAGES);
    const ui pagesize = (ui) sysconf(_SC_PAGE_SIZE);

    count = ((ui64) pagecount) * ((ui64) pagesize);
  };
#endif

  return count;
}

Thanks! Is there no POSIX standard way of doing this ? So that I need only code the POSIX case for all POSIX systems the same way ?

[lucasart]

so it's very simple, just check the errno and that's it.

If you do it like that you get false errors: malloc doesn't clear errno in case of success. Errno is used to communicate the reason of failure, not the failure itself.

Just check for NULL returned out of malloc.

(you can of course clear errno yourself before the function call, but that is not idiomatic)

PS: In case of realloc, beware to store the return value first in a different location than the input parameter. Because in case of failure, the old region is still allocated. If you do "p = realloc(p, ...)" you have a memory leak in case of failure.

Thanks! Although that's really the correct way to proceed according to the ISO C standard, in reality a failed malloc or realloc crashes Windows and causes Linux to unleash the OOM killer, killing "randomly" chosen processes to save memory. So really, there's no need to handle the case of a failed memory allocation

[Sven]

so it's very simple, just check the errno and that's it.

If you do it like that you get false errors: malloc doesn't clear errno in case of success. Errno is used to communicate the reason of failure, not the failure itself.

Just check for NULL returned out of malloc.

(you can of course clear errno yourself before the function call, but that is not idiomatic)

PS: In case of realloc, beware to store the return value first in a different location than the input parameter. Because in case of failure, the old region is still allocated. If you do "p = realloc(p, ...)" you have a memory leak in case of failure.

Thanks! Although that's really the correct way to proceed according to the ISO C standard, in reality a failed malloc or realloc crashes Windows and causes Linux to unleash the OOM killer, killing "randomly" chosen processes to save memory. So really, there's no need to handle the case of a failed memory allocation

No, a failed malloc() does not necessarily crash the system. That depends on the available amount of virtual memory. Under Linux there is usually plenty of that available. The point is, as soon as virtual memory is really being used above the available amount of RAM (i.e., you write into memory areas that are actually mapped to swap space on disk) the system is slowing down a lot.

The "OOM killer" of Linux will only jump in if also the swap space is exhausted. That may be the case if no swap space, or only a tiny amount of it, has been configured, or if some applications have allocated a lot of memory without ever freeing it. In a normally working system you should never observe the "OOM killer" scenario.

I am not quite sure about the exact behaviour of Windows in case of virtual memory being exhausted. Most probably Windows will not handle that case more smoothly than Linux. But at least there is a virtual memory concept that works somehow, using something like a swap file, so malloc() and related allocation functions are able to allocate more memory than you have unused RAM available.

For a chess engine, though, you will not like to come into a situation where the OS has to access swap space. Therefore an approach like Steven's to find out how much RAM is really available for the program is highly recommended.

Conclusions:

1) If you use C++ with exceptions (usually not in a chess program) then use the "new" operator and deal with exceptions at an appropriate point. You can safely assume that the return value of "new" is non-zero (otherwise exception handling control flow does not guide you to the next statement behind the call of "new").

2) If you use C++ without exceptions then use "new", too, and check the pointer returned by "new" against 0. To allocate arrays use new[] accordingly. To dynamically resize allocated memory areas use either delete[] and new[] or realloc().

3) If you use plain C then you have to use malloc() or realloc(). Always check its return value against NULL. Don't care about "errno", especially don't look at "errno" without having checked for occurrence of an error itself. This is common UNIX philosophy: "errno" indicates the reason of an error if there is any, but is undefined in the default case that there isn't any (otherwise all system calls would have to set errno=0 in the beginning each time which is a waste of CPU time).

4) In any case, define a proper reaction on a failed memory allocation. Apart from not using the returned pointer (since it is 0), you need to define what that "out of memory" case really means for you. If the amount of memory to be allocated was defined by some external parameter, like a hash table size specified by the user, then use a smaller value. Of course early rejecting the user's request based on knowing how much memory is available is even better, but conditions may become worse at runtime so you still need to check every time. If the memory allocation is crucial for your program to continue its work and trying to allocate a smaller amount of memory is not an option then abort the program after giving appropriate feedback.

Chess programs will not need "new" or malloc() too often, I use it only to allocate my hash table.

Merry christmas,
Sven

[sje]

1) If you use C++ with exceptions (usually not in a chess program) then use the "new" operator and deal with exceptions at an appropriate point. You can safely assume that the return value of "new" is non-zero (otherwise exception handling control flow does not guide you to the next statement behind the call of "new").

2) If you use C++ without exceptions then use "new", too, and check the pointer returned by "new" against 0. To allocate arrays use new[] accordingly. To dynamically resize allocated memory areas use either delete[] and new[] or realloc().

In C++, only the special nothrow version of new() can return a zero value. The plain new() should never return a zero, either with or without exception handling.

Details: http://www.cplusplus.com/reference/std/ ... tor%20new/

[Sven]

1) If you use C++ with exceptions (usually not in a chess program) then use the "new" operator and deal with exceptions at an appropriate point. You can safely assume that the return value of "new" is non-zero (otherwise exception handling control flow does not guide you to the next statement behind the call of "new").

2) If you use C++ without exceptions then use "new", too, and check the pointer returned by "new" against 0. To allocate arrays use new[] accordingly. To dynamically resize allocated memory areas use either delete[] and new[] or realloc().

In C++, only the special nothrow version of new() can return a zero value. The plain new() should never return a zero, either with or without exception handling.

Details: http://www.cplusplus.com/reference/std/ ... tor%20new/

But if you compile without exception handling, don't you get the nothrow version of operator new offered by default, since the other versions are not available?

It seems, however, that at least under MSVC++ it is not easy (or even impossible) to have both "throw" and "nothrow" versions available in the same executable.

Sven

[lucasart]

Thank you all for your useful comments. Here is my code for allocating (or resizing) the hash table. I put lots of comments, because all this was not obvious for me

Code: Select all

void tt_alloc(uint64_t size)
/* allocates TT.count elements, where TT.count is the largest power of two such that the total size
 * sizeof(hash_t) * TT.count is less or equal to size bytes.
 * Note: This function can be re-called at will. It will internally do the right thing, either:
 * - nothing
 * - realloc and zero out any exceeding portion only (existing hash is preserved)
 * - alloc for the first time, and zero out the buffer
 */
{
	const uint64_t old_size = TT.count * sizeof(hash_t);
	
	// calculate the number of hash slots to allocate
	for &#40;TT.count = 1; TT.count * sizeof&#40;hash_t&#41; <= size; TT.count *= 2&#41;;
	TT.count /= 2;
	
	// adjust size to a correct power of two
	size = sizeof&#40;hash_t&#41; * TT.count;
	
	// TT.buf already allocated to the correct size&#58; nothing to do
	if &#40;size == old_size&#41;
		return;
	
	/* use realloc to preserve the first min&#40;size,old_size&#41; bytes
	 * Note&#58; If TT.buf is NULL then realloc&#40;) will call malloc&#40;), so no need to treat this case
	 * separately. If TT.buf is moved, realloc will call free&#40;old TT.buf&#41;. */
	TT.buf = realloc&#40;TT.buf, size&#41;;
	if (!TT.buf&#41;
		/* Whether or not this is useful is an open question. A failed realloc&#40;) should mean that the
		 * RAM and swap space have both been exceeded, so it's unlikely that we'll have a stable
		 * operating system at this point anyway */
		my_fatal&#40;"tt_alloc&#40;)&#58; Could not realloc&#40;) %"PRIu64"bytes!\n", size&#41;;

	// when growing the buffer, the exceeding portion must be cleared
	if &#40;size > old_size&#41;
		memset&#40;TT.buf + old_size, 0, size - old_size&#41;;
&#125;

Let me know if you see anything that doesn't make sense.

[sje]

1) If you use C++ with exceptions (usually not in a chess program) then use the "new" operator and deal with exceptions at an appropriate point. You can safely assume that the return value of "new" is non-zero (otherwise exception handling control flow does not guide you to the next statement behind the call of "new").

2) If you use C++ without exceptions then use "new", too, and check the pointer returned by "new" against 0. To allocate arrays use new[] accordingly. To dynamically resize allocated memory areas use either delete[] and new[] or realloc().

In C++, only the special nothrow version of new() can return a zero value. The plain new() should never return a zero, either with or without exception handling.

Details: http://www.cplusplus.com/reference/std/ ... tor%20new/

But if you compile without exception handling, don't you get the nothrow version of operator new offered by default, since the other versions are not available?

It seems, however, that at least under MSVC++ it is not easy (or even impossible) to have both "throw" and "nothrow" versions available in the same executable.

It would be dangerous to assume that the plain new() would do anything other than what the standard says regardless of the presence of exception handling.

As for MSVC, I haven't used it in years for a number of good (to me, at least) reasons.

I agree that exception handling should be not be employed in most user applications. Rather, it should be reserved for real time applications which run with little or no user supervision (e.g., embedded fail-safe boxes for aircraft and spacecraft).

[lucasart]

As for MSVC, I haven't used it in years for a number of good (to me, at least) reasons.

Me too! it really sucks because it's not even ISO C99 compliant.

I agree that exception handling should be not be employed in most user applications. Rather, it should be reserved for real time applications which run with little or no user supervision (e.g., embedded fail-safe boxes for aircraft and spacecraft).

Although it could be argued that such programs should never be written in an error-prone language like C++. CAML for example, or even Python are certainly better choices for programs whose failure could put people's lives at risk. Personally, I wouldn't like to sit in a plane knowing that the aircraft system was coded in C++, and compiled with MSVC running on Micro$oft Windows...

[sje]

Thank you all for your useful comments. Here is my code for allocating (or resizing) the hash table. I put lots of comments, because all this was not obvious for me

Let me know if you see anything that doesn't make sense.

I have found it handy to use the log2 of the entry count when dealing with transposition table allocation and indexing issues. Separate inline functions with the log2 entry count as a formal parameter are used for entry count determination, address mask determination, total table size determination, etc. Other routines use a log2 entry count formal for allocation. I also have four different transposition table types plus the perft table, so the log2 entry count functions are used in several places.