Clustering support in the OS kernel

[glorfindel]

I have stumbled upon the website of DragonFly BSD, an operating system whose "ultimate goal is to provide native clustering support in the kernel. This involves the creation of a sophisticated cache management framework for filesystem namespaces, file spaces, and VM spaces, which allows heavily interactive programs to run across multiple machines with cache coherency fully guaranteed in all respects. This also involves being able to chop up resources, including the cpu by way of a controlled VM context, for safe assignment to unsecured third-party clusters over the internet (though the security of such clusters itself might be in doubt, the first and most important thing is for systems donating resources to not be made vulnerable through their donation)".

Given the (somewhat) recent talk here that some programmers are interested in making their programs run on many many processors, basically, my question is if this OS can give something new to chess programmers, making it easier for them to write such a program. Or perhaps other OSs already have the support needed through the use of other software, so that "native support in the kernel" is not really needed?

Edit: link: http://www.dragonflybsd.org/

[Gian-Carlo Pascutto]

Other OS's already have the support needed, which is why several programs already have cluster versions.

They just require more work and thought from the programmer. The result will be that the program is more efficient that one that has to rely on the OS alone. The OS does not know the innards of the chessprogram, but the chessprogrammer does.

So, operating system support will make this easier, but not necessarily better, and quite probably, worse.

[sje]

This involves the creation of a sophisticated cache management framework for filesystem namespaces, file spaces, and VM spaces, which allows heavily interactive programs to run across multiple machines with cache coherency fully guaranteed in all respects.

Most chess programs couldn't care less about filesystem namespaces, file spaces, and VM spaces. File I/O is wholly unimportant during calculation. Heavy reliance on slow VM paging is death to a calculator.

What is really needed here is low level hardware support -- like a low latency, core to core network with semaphores that sidesteps memory entirely. This will be a long time in coming as most applications work just fine with existing support from pthreads and the like.

[bob]

I have stumbled upon the website of DragonFly BSD, an operating system whose "ultimate goal is to provide native clustering support in the kernel. This involves the creation of a sophisticated cache management framework for filesystem namespaces, file spaces, and VM spaces, which allows heavily interactive programs to run across multiple machines with cache coherency fully guaranteed in all respects. This also involves being able to chop up resources, including the cpu by way of a controlled VM context, for safe assignment to unsecured third-party clusters over the internet (though the security of such clusters itself might be in doubt, the first and most important thing is for systems donating resources to not be made vulnerable through their donation)".

Given the (somewhat) recent talk here that some programmers are interested in making their programs run on many many processors, basically, my question is if this OS can give something new to chess programmers, making it easier for them to write such a program. Or perhaps other OSs already have the support needed through the use of other software, so that "native support in the kernel" is not really needed?

Edit: link: http://www.dragonflybsd.org/

Message is still the problem. Sandia has done a lightweight kernel for their big IBM cluster. The idea being to introduce as little system overhead as possible for maximum cluster efficiency. But you _still_ have to deal with message passing traffic which is one huge bottleneck.