Simplest way to implement quick and dirty lazy smp

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
syzygy
Posts: 4364
Joined: Tue Feb 28, 2012 10:56 pm

Re: Simplest way to implement quick and dirty lazy smp

Post by syzygy » Sun Jan 06, 2019 12:30 pm

lucasart wrote:
Sun Jan 06, 2019 12:44 am
I fail to see how this is easier.
Easier than rewriting the whole engine to make it thread safe...

User avatar
xr_a_y
Posts: 315
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Simplest way to implement quick and dirty lazy smp

Post by xr_a_y » Sun Jan 06, 2019 1:41 pm

syzygy wrote:
Sun Jan 06, 2019 12:30 pm
lucasart wrote:
Sun Jan 06, 2019 12:44 am
I fail to see how this is easier.
Easier than rewriting the whole engine to make it thread safe...
Appart from looking for static (or global) variables and making some table member of the "thread context", there is not much to make it thread safe I guess.

User avatar
hgm
Posts: 22817
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Simplest way to implement quick and dirty lazy smp

Post by hgm » Sun Jan 06, 2019 7:28 pm

JVMerlino wrote:
Sat Jan 05, 2019 9:23 pm
The only significant difference between your implementation and mine is that I pass the address for the hash tables as commandline parameters to the slaves. How do you pass commands down the chain if not via pipes?
Oh, I understood that you had one master process that would have all other processes as child, and communicate directly to them. Sorry if I was wrong. And yes, I do pass along the commands through pipes. I just used the same code as that WinBoard and adapters use to fork off an engine process, and set up the pipes to it.
lucasart wrote:
Sun Jan 06, 2019 12:44 am
I fail to see how this is easier. Honestly, managing subprocesses, and communications via pipes, is a lot more complicated in my experience (especially on POSIX systems, with the fork logic being quite mind bending). And it's hideous, from a design/coding standpoint. Simply start threads in an id_loop() and join when done. That's all there is to it. The rest is details, which you can improve on as you go along.
It would not be easier for an engine that you design from scratch, of course. But if you already have a single-threaded engine, using many global variables, it would require a lot of modifications to even run two searches in parallel. As it was I only required to copy-paste existing code for starting an engine process, and some minor modifications to the command-interpreter loop (re-ordering the command matching so that all those that do not have to be passed along are matched first, and the rest only after the input has been passed along, change the code that handles the hash-table allocation to use shared memory, and have the execution of a few specific commands pass something special). Much less error prone. I did the entire conversion in the plane while flying towards the tournament.

User avatar
lucasart
Posts: 3023
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: Simplest way to implement quick and dirty lazy smp

Post by lucasart » Mon Jan 07, 2019 12:20 pm

hgm wrote:
Sun Jan 06, 2019 7:28 pm
lucasart wrote:
Sun Jan 06, 2019 12:44 am
I fail to see how this is easier. Honestly, managing subprocesses, and communications via pipes, is a lot more complicated in my experience (especially on POSIX systems, with the fork logic being quite mind bending). And it's hideous, from a design/coding standpoint. Simply start threads in an id_loop() and join when done. That's all there is to it. The rest is details, which you can improve on as you go along.
It would not be easier for an engine that you design from scratch, of course. But if you already have a single-threaded engine, using many global variables, it would require a lot of modifications to even run two searches in parallel. As it was I only required to copy-paste existing code for starting an engine process, and some minor modifications to the command-interpreter loop (re-ordering the command matching so that all those that do not have to be passed along are matched first, and the rest only after the input has been passed along, change the code that handles the hash-table allocation to use shared memory, and have the execution of a few specific commands pass something special). Much less error prone. I did the entire conversion in the plane while flying towards the tournament.
Putting lipstick on a pig is never a good long term solution in chess engine development :lol:

If global variables are the problem, and you are looking for a cheap and easy solution, then just put "thread_local" in front of your globals, and voila. Forget about performance on Windows though (Linux implements a zero cost model for TLS, but Windows implementation is crap as always).

In fact, that's how I started with Demolito. Eventually I made it available for windows, and that's how I discovered the cost of TLS on Windows makes it unusable in a chess engine.

Eventually you refactor the code to get rid of most of the globals (ie. "thread_local"), and finally you put the remaining ones in per thread struct, and you have a clean code base.

A good starting point is to question all these global variables you have. How many of them are both read and written during search? Probably very few.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

User avatar
hgm
Posts: 22817
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Simplest way to implement quick and dirty lazy smp

Post by hgm » Mon Jan 07, 2019 12:41 pm

Long term solutions are of little use when the available time is short...

Post Reply