Malloc

[Rebel]

Just out of curiosity....

On a Windows 7 PC (64-bit) with 4Gb RAM my compiler (digitalmars) returns a maximum for malloc of 1.5Gb free memory. Dev-C++ returns a maximum of 2Gb.

What's the trick to get more ?

[mar]

That's weird. Are you compiling a 64-bit application?
malloc accepts size_t which should be 64-bit for x64 builds.
EDIT: if you are compiling 32-bit code, windows has a per process limit of 2G virtual address space. Either way you can only address up to 4G in 32-bit mode anyway.

[wgarvin]

That's weird. Are you compiling a 64-bit application?
malloc accepts size_t which should be 64-bit for x64 builds.
EDIT: if you are compiling 32-bit code, windows has a per process limit of 2G virtual address space. Either way you can only address up to 4G in 32-bit mode anyway.

Try linking your 32-bit app with the /LARGEADDRESSAWARE flag. That sets a flag in the PE header promising to Windows that you won't crash if it gives you more than 2 GB.

With that flag set, I think your 32-bit process will have almost the full 4 GB of virtual space to play with if you run it on a 64-bit version of Windows.

If you run it on a 32-bit version of Windows (XP or newer), it will still only have 2 GB, unless you booted Windows in the hacky /3GB mode (generally not recommended).

[mar]

Try linking your 32-bit app with the /LARGEADDRESSAWARE flag. That sets a flag in the PE header promising to Windows that you won't crash if it gives you more than 2 GB.

With that flag set, I think your 32-bit process will have almost the full 4 GB of virtual space to play with if you run it on a 64-bit version of Windows.

If you run it on a 32-bit version of Windows (XP or newer), it will still only have 2 GB, unless you booted Windows in the hacky /3GB mode (generally not recommended).

Thanks for the tips, but I would prefer to build a 64-bit application in that particular case
OT: Since I know you are a game programmer, I'd like to use the opportunity to ask about how consoles handle this. PS3/360 are 32-bit, right? Are PC versions 64-bit by default now?
A very large project can hit the 2G limit easily, not speaking about fragmentation when one would allocate very large blocks dynamically.

[Rebel]

That's weird. Are you compiling a 64-bit application?

Either a 32 or 64 bit compile limits to 2Gb.

Some example code:

Code: Select all

#define CHT_ENTRIES		165000000			// 165.xxx.xxx (max Dev-C++ compiler)

struct hash
  { long key1;
    long key2;
    long key3;
  } cht[CHT_ENTRIES];

Increasing 165 to 170 million exceeds to 2Gb (12*170M) and the executable refuses to run.

Via malloc:

Code: Select all

struct hash
  { long key1;
    long key2;
    long key3; };
struct hash *cht;

unsigned int ms;
char *p_memory;

for (ms = 0x7f000000; ms >= 2000; ms -= 100000)		// seek till 12 Gb
 { p_memory = malloc(ms); if (p_memory) break; }
printf("Free memory = %d Mb\n\n",ms>>20);

cht=(struct hash *)p_memory;       // connect to structure

Also limits the virtual memory to 2Gb either 32 or 64 bit compile.

[mar]

Either a 32 or 64 bit compile limits to 2Gb.

Strange Ed,
I just tried to malloc 4 gigs (64-bit build, 8G physical ram) and works just fine.
Are you 100% sure thay your compiler generated 64-bit executable?
You should see 32-bit apps in task manager marked with *32.
Or even better, you can try something like

Code: Select all

assert( sizeof(void *) >= 8 );

to make sure you run a in 64-bit mode.

[wgarvin]

OT: Since I know you are a game programmer, I'd like to use the opportunity to ask about how consoles handle this. PS3/360 are 32-bit, right? Are PC versions 64-bit by default now?
A very large project can hit the 2G limit easily, not speaking about fragmentation when one would allocate very large blocks dynamically.

Yes, the PS3 and 360 are both 32-bit machines. The 4 GB address space is plenty for them, since they both only have 512 MB of RAM and paging to disk is either not supported or at least, something console games never want to do. In practice, if you need to allocate memory and there isn't any RAM left (or your heap is too fragmented, for example) then you just crash. Console games can often run with very low amounts of free memory remaining, sometimes as little as a couple of megabytes, and still not crash. Sometimes the game will have some "last resort" functions that it invokes when free memory gets really low, that try to scavenge some memory by freeing high-res textures, or something like that. But you can't afford to have memory leaks in a console game because that will always be fatal. I remember a particularly bad one near the end of development on one game I worked on, that was caused by a compiler bug that only showed up with high optimization settings and whole-program optimizations enabled. It caused a piece of correct code to be miscompiled in a way that still worked, but constantly leaked small amounts of memory. Within a couple of hours, a few tens of megabytes had been leaked and that was enough that levels which previously fit reliably in memory would no longer fit, and the game would crash. But only on the final retail build -- other builds used for testing, did not get mis-compiled, and could run for 24 hours with no problems.

I don't play PC games much myself, and I haven't paid much attention to what other developers are shipping (32-bit or 64-bit, etc). For this generation I preferred to just buy and play console games. But for AAA games I imagine 64-bit is preferred now, because many gamers already have more than 4 GB of memory. Players with 4 GB or less are also likely to be running ancient Windows XP or something, like my desktop machine at home XP doesn't support the newer versions of DirectX, and back when it was still popular, when Windows Vista was still new-ish, a lot of developers continued to target DX9 and maybe add a DX11 mode with some extra features or something. But by now, I imagine a lot of AAA developers are focusing primarily on DX11 and building 64-bit apps. My development PC at work is a 64-bit Windows 7 machine, and it has more RAM in it than all of the PCs I own combined.

[Rebel]

Either a 32 or 64 bit compile limits to 2Gb.

Strange Ed,
I just tried to malloc 4 gigs (64-bit build, 8G physical ram) and works just fine.
Are you 100% sure thay your compiler generated 64-bit executable?
You should see 32-bit apps in task manager marked with *32.
Or even better, you can try something like

Code: Select all

assert( sizeof(void *) >= 8 );

to make sure you run a in 64-bit mode.

It's 64 bit, no *32. Perhaps I need to plug-in an extra 4Gb before the compiler grants me more than 2Gb, it's a Windows PC after all

[Joost Buijs]

Just out of curiosity....

On a Windows 7 PC (64-bit) with 4Gb RAM my compiler (digitalmars) returns a maximum for malloc of 1.5Gb free memory. Dev-C++ returns a maximum of 2Gb.

What's the trick to get more ?

According to the website of Digital Mars their latest C++ compiler does not support x64 builds under Windows yet.

I think that the MinGW compiler that comes default with Dev C++ does not support x64 builds either, although it is possible to use a different version of GCC under Dev C++.

Probably this is why you are bound to a 2gB. limit.

[mar]

It's 64 bit, no *32. Perhaps I need to plug-in an extra 4Gb before the compiler grants me more than 2Gb, it's a Windows PC after all

Well the system should be able to give you more than you have physical memory (if your swap file is large enough),
so the call should actually succeed and even be fast because it won't start
swapping until you access the memory and the system starts geting page faults.