Nalimov and memory for indexes (are you aware?)

[Gian-Carlo Pascutto]

Yes, via ZwCreateProcess in NTDLL.

See for example.

http://sourceforge.jp/cvs/view/pards/pa ... iew=markup

However, as for calling using undocumented internal kernel functions from a userspace program "being supported": I don't think so!

[bob]

Yes, via ZwCreateProcess in NTDLL.

See for example.

http://sourceforge.jp/cvs/view/pards/pa ... iew=markup

However, as for calling using undocumented internal kernel functions from a userspace program "being supported": I don't think so!

In any case, the problem exists, although not with the indices (if you have a true copy-on-write O/S). But the cache certainly gets replicated N times, since each time a page of the cache is modified, it gets duplicated, if you use processes rather than threads. Threads are the way to go using egtb.cpp as currently written.

[Zach Wegner]

Ok. According to the chessprogramming Wiki, you, Vincent and Vasik are the lone holdouts, then.

Well, ZCT still uses processes, but I don't. So looks like it's down to two people.

[michiguel]

Well I have to agree that there is something different with R3 and Nalimov memory usage. I noticed that when I put EGTB's on a USB drive it takes ages for the engine to load and unload. For some reason if EGTB's are on HDD then this is not a problem. Once loaded things are OK. I don't see this behaviour with other engines and to be honest I have never figured this one out.

Each of the threads load their own indexes? Then it is 20 MiB x cores?

Miguel

Each *process*, see my comment above.

Oops, now I get it. Mmmhhh... I have to think if Gaviota TBs are MP friendly or not... I think they should be if everything is initialized before forking.

Don't bet on it if you use windows. Someone pointed out that windows doesn't do copy-on-write as unix does... which would produce duplicates of everything when you fork().

If the initialization (before fork) involves a malloc and it keeps a static pointer to it, after forking, the pointer gets copied but the memory allocated by malloc is not. I assume that this is correct even in Windows. Is that right?. If that is the case, Gaviota TBs "may" be multiprocessing friendly. I still have to think carefully about the whole thing to be sure.

My engine has been always multi-threaded and I never thought about supporting MP mechanisms, but if a simple modification could do it, I will.

Miguel

Miguel

There's a well-known bug where SMP Rybka doesn't use tablebases correctly, and it's closely related: none of the Nalimov stuff is shared and Vasik forgot to pass a parameter from the master process to the slaves.

I'm glad I got rid of sh*t like that in DS 3.0

[Gian-Carlo Pascutto]

If the initialization (before fork)[...] I assume that this is correct even in Windows.

As was already pointed out, fork() does not exist on Windows.

Not just the function, the entire concept doesn't exist.

[michiguel]

If the initialization (before fork)[...] I assume that this is correct even in Windows.

As was already pointed out, fork() does not exist on Windows.

Not just the function, the entire concept doesn't exist.

Ok, I was not really accurate (launching processes is the correct concept?). I am still interested in knowing if this is correct (rephrasing it):

If the initialization involves a malloc with a static pointer to it, after processes are launched the pointer gets copied but the memory allocated by malloc is not. I assume that this is correct even in Windows. Is that right?. Or you have to start the program with all the processes already running?

Miguel

[lmader]

Ok, I was not really accurate (launching processes is the correct concept?). I am still interested in knowing if this is correct (rephrasing it):

If the initialization involves a malloc with a static pointer to it, after processes are launched the pointer gets copied but the memory allocated by malloc is not. I assume that this is correct even in Windows. Is that right?. Or you have to start the program with all the processes already running?

Miguel

I don't quite understand the question.

But to clarify, in Linux when you fork(), the child process gets a copy of all the memory in the Parent. But it starts with a virtual copy, which means that only the page tables (the OS's virtual memory mappings) for the memory in the parent process are copied into the child process. It doesn't actually reallocate the memory pointed to in the page tables until it is written to (copy-on-write). Windows doesn't support this type of process creation (but supports copy-on-write everywhere).

In your question, it sounded like you might be asking if the child process gets a static pointer to the actual memory in the parent process. This doesn't happen and wouldn't work - a process cannot "see" another processes memory through an actual pointer. This is true in Linux and Windows.

Does that help?

[michiguel]

Ok, I was not really accurate (launching processes is the correct concept?). I am still interested in knowing if this is correct (rephrasing it):

If the initialization involves a malloc with a static pointer to it, after processes are launched the pointer gets copied but the memory allocated by malloc is not. I assume that this is correct even in Windows. Is that right?. Or you have to start the program with all the processes already running?

Miguel

I don't quite understand the question.

But to clarify, in Linux when you fork(), the child process gets a copy of all the memory in the Parent. But it starts with a virtual copy, which means that only the page tables (the OS's virtual memory mappings) for the memory in the parent process are copied into the child process. It doesn't actually reallocate the memory pointed to in the page tables until it is written to (copy-on-write). Windows doesn't support this type of process creation (but supports copy-on-write everywhere).

In your question, it sounded like you might be asking if the child process gets a static pointer to the actual memory in the parent process. This doesn't happen and wouldn't work - a process cannot "see" another processes memory through an actual pointer. This is true in Linux and Windows.

Does that help?

You are saying that everything that has been allocated with malloc() is copied when processes are launched? that was my original question simplified.

Miguel

[lmader]

You are saying that everything that has been allocated with malloc() is copied when processes are launched? that was my original question simplified.

Miguel

Ahhh, I see. Then the answer is:

Yes for Linux (including the malloc'd memory)

No for Windows. (no memory, not even malloc'd)

Windows programs can create new child processes (of course) but Windows doesn't have (an exposed API) for creating a child process with a copy of any of the memory from the parent (except for inheriting things like environment variables, security descriptors, etc).

[Gian-Carlo Pascutto]

Windows programs can create new child processes (of course) but Windows doesn't have (an exposed API) for creating a child process with a copy of any of the memory from the parent (except for inheriting things like environment variables, security descriptors, etc).

Not only that, but the process starts normally from the main() and not from the point where the process creation happened in the child.

This is what I meant with: the entire *concept* of fork() just doesn't exist. It's a bit like you only have system() instead (but one that runs in parallel).