Aside from myself, are there any BOINC http://boinc.berkeley.edu/ participants here? Could a distributed perft project be a good candidate for a BOINC implementation? At one time there was a Chess960 project, but that seems to be moribund at present.
A perft BOINC client application would likely not be able to rely upon huge mass storage availability which is needed for some algorithms. Same comment for huge RAM requirements. But for lots and lots of small sized, in-memory processing BOINC might get us faster to the really big sums than any other method.
Again, distributed perft
Moderators: hgm, Dann Corbit, Harvey Williamson
-
Ajedrecista
- Posts: 1952
- Joined: Wed Jul 13, 2011 9:04 pm
- Location: Madrid, Spain.
Re: Again, distributed perft.
Hello Steven:
Probably (very unlikely) I will join (I am not a BOINC member: it is the first time I read about it, so thanks for the link), but of course many more people are needed. In the rare case I join, my contribution will be less than modest.
It would be sad that this distributed project, if started, is later abandoned (just thinking in the initiative of Komodo).
Regards from Spain.
Ajedrecista.
First of all: you are talking about Perft(14), right? I think that people need more details, such as how many plies will have each draft: six plies? Seven plies? More things: which perft counter/s will be used? Yours (Symbolic), gperftd (by Paul Byrne), ...? Sorry if I sound harsh, but in fact it is the opposite! Could you post more details, please? Thanks in advance.sje wrote:Aside from myself, are there any BOINC http://boinc.berkeley.edu/ participants here? Could a distributed perft project be a good candidate for a BOINC implementation? At one time there was a Chess960 project, but that seems to be moribund at present.
A perft BOINC client application would likely not be able to rely upon huge mass storage availability which is needed for some algorithms. Same comment for huge RAM requirements. But for lots and lots of small sized, in-memory processing BOINC might get us faster to the really big sums than any other method.
Probably (very unlikely) I will join (I am not a BOINC member: it is the first time I read about it, so thanks for the link), but of course many more people are needed. In the rare case I join, my contribution will be less than modest.
It would be sad that this distributed project, if started, is later abandoned (just thinking in the initiative of Komodo).Regards from Spain.
Ajedrecista.
-
sje
- Posts: 4675
- Joined: Mon Mar 13, 2006 7:43 pm
Re: Again, distributed perft.
The foremost motivation for using BOINC is that very little technical expertise on the part of the end users is required, either for installation or for regular operation.
Current BOINC power:
Active: 252,406 volunteers, 421,848 computers.
24-hour average: 6.715 PetaFLOPS.
Current BOINC power:
Active: 252,406 volunteers, 421,848 computers.
24-hour average: 6.715 PetaFLOPS.
-
sje
- Posts: 4675
- Joined: Mon Mar 13, 2006 7:43 pm
Attributes of a well-behaved BOINC client
Some attributes of a well-behaved BOINC client application:
1. A client uses only a modest amount of disk space. If many gigabytes are needed, then several problems occur. It will be hard to find users who will be both able and willing to contribute if asked for too much.
2. A client uses only a modest amount of memory. See #1; also, a client needs to be able to efficiently swap in and out of virtual memory frequently to make way for higher priority user applications.
3. A client has to be able to periodically save its state, preferably once a minute or so. One reason for this is to avoid losing work across system reboots which are unpredictable to the client.
So neither the Byrne approach (with huge disk requirements) nor the Edwards approach (with huge memory requirements) is going to work well. But I have an idea which is a compromise of sorts.
Let the client be a simple bitboard program with the usual move generation, execution, and retraction routines. The client will also have an efficient bulk move counter routine to handle the last ply enumeration. (All of this can be found in my CookieCat program, written in Free Pascal.)
Add to the above a position preprocessor which builds a small three ply tree for the given root position. The tree is adorned with hashes, counts, and links so that transpositions of the form (move-0 move-1 move-2) to (move-2 move-1 move-0) are detected to lessen total counting effort. With a mean count of 32 moves per position, the tree entry count will be around 32,768. This tree represents most of the current state of the client; the remaining part will be the current move path from the root. The actual size will vary, but the total should be well under a megabyte and so can be easily written to disk as a checkpoint every minute or so.
The three ply tree can also used for a simple multithreaded approach by dealing the ply three perft(N-3) one per available processor core. (This increases the save state size slightly by requiring a move path for each enumeration thread.)
1. A client uses only a modest amount of disk space. If many gigabytes are needed, then several problems occur. It will be hard to find users who will be both able and willing to contribute if asked for too much.
2. A client uses only a modest amount of memory. See #1; also, a client needs to be able to efficiently swap in and out of virtual memory frequently to make way for higher priority user applications.
3. A client has to be able to periodically save its state, preferably once a minute or so. One reason for this is to avoid losing work across system reboots which are unpredictable to the client.
So neither the Byrne approach (with huge disk requirements) nor the Edwards approach (with huge memory requirements) is going to work well. But I have an idea which is a compromise of sorts.
Let the client be a simple bitboard program with the usual move generation, execution, and retraction routines. The client will also have an efficient bulk move counter routine to handle the last ply enumeration. (All of this can be found in my CookieCat program, written in Free Pascal.)
Add to the above a position preprocessor which builds a small three ply tree for the given root position. The tree is adorned with hashes, counts, and links so that transpositions of the form (move-0 move-1 move-2) to (move-2 move-1 move-0) are detected to lessen total counting effort. With a mean count of 32 moves per position, the tree entry count will be around 32,768. This tree represents most of the current state of the client; the remaining part will be the current move path from the root. The actual size will vary, but the total should be well under a megabyte and so can be easily written to disk as a checkpoint every minute or so.
The three ply tree can also used for a simple multithreaded approach by dealing the ply three perft(N-3) one per available processor core. (This increases the save state size slightly by requiring a move path for each enumeration thread.)
-
sje
- Posts: 4675
- Joined: Mon Mar 13, 2006 7:43 pm
Re: Attributes of a well-behaved BOINC client
Regarding client multithreading, it appears that the preferred BOINC approach is to have a single worker thread for nearly all of the calculation effort. This is done to have the common BOINC client controller application to map work units to cores in a general way. There are some good reasons for this, the main one being to prevent one client of a given kind to greedily appropriating all the cores on a host at the expense of other clients of a different kind. So there is a small loss in efficiency for a gain in simplicity.
Some BOINC clients provide a screen saver capability. This can eat a noticeable chunk of processor time, but it can also attract users with a hankering for eye candy. In the case of a perft client, the obvious choice for a screen saver design is one which includes a board display and the current move path from the work unit's root position. If more than worker thread is running, then perhaps there would be a separate board and path for each such thread. When activated, the screen saver display might be updated at a fixed rate of once per second, or perhaps at some user selectable frequency. Note that screen saver code needs to somewhat customized for different host operating systems (Linux, Mac OS/X, and others).
It's a good thing that there are many talented programmers in the chess world who can help with a perft screen saver, as I myself am nearly totally talent-free in regard to graphics coding.
Some BOINC clients provide a screen saver capability. This can eat a noticeable chunk of processor time, but it can also attract users with a hankering for eye candy. In the case of a perft client, the obvious choice for a screen saver design is one which includes a board display and the current move path from the work unit's root position. If more than worker thread is running, then perhaps there would be a separate board and path for each such thread. When activated, the screen saver display might be updated at a fixed rate of once per second, or perhaps at some user selectable frequency. Note that screen saver code needs to somewhat customized for different host operating systems (Linux, Mac OS/X, and others).
It's a good thing that there are many talented programmers in the chess world who can help with a perft screen saver, as I myself am nearly totally talent-free in regard to graphics coding.
-
sje
- Posts: 4675
- Joined: Mon Mar 13, 2006 7:43 pm
Re: Again, distributed perft
Looking at this further, it appears that the biggest obstacle with getting the project started is getting a capable (and expensive) server machine plus costly bandwidth. This really isn't something which you could run on a home computer with only consumer level connectivity.
Added to that is the need for administrators and a regularly monitored web site for project participants to review their progress.
Added to that is the need for administrators and a regularly monitored web site for project participants to review their progress.
-
gladius
- Posts: 568
- Joined: Tue Dec 12, 2006 10:10 am
- Full name: Gary Linscott
Re: Again, distributed perft
The fishtest distributed project is using the Amazon ec2 micro instance to host everything. You can sign up and get ec2 micro for a year for free, and after that, it would be about $10-15 or so a month.sje wrote:Looking at this further, it appears that the biggest obstacle with getting the project started is getting a capable (and expensive) server machine plus costly bandwidth. This really isn't something which you could run on a home computer with only consumer level connectivity.
Added to that is the need for administrators and a regularly monitored web site for project participants to review their progress.
It is a limited machine, with one core and ~700mb of ram, but more than enough to handle the webpage/database for fishtest at least.
-
sje
- Posts: 4675
- Joined: Mon Mar 13, 2006 7:43 pm
BOINCification
Some of the big BOINC projects require a dozen or so machines, all of them expensive. For a small project like Perft@home, at least two machines of the Mac mini server class (~ US$1,000) would be needed, and maybe more for redundancy. Then there are the co-location fees plus the need to have two or three dedicated volunteers or grad student slaves for various administration purposes.
BOINC projects which tried to succeed with only one guy on a tight budget haven't fared very well.
The moderately sized MilkyWay@home project has about a dozen guys for science and administration. There are maybe a hundred thousand or so client host machines, each reporting in every hour or so. And moving stars around the sky is not much more complicated than moving pieces around a chessboard.
BOINC projects which tried to succeed with only one guy on a tight budget haven't fared very well.
The moderately sized MilkyWay@home project has about a dozen guys for science and administration. There are maybe a hundred thousand or so client host machines, each reporting in every hour or so. And moving stars around the sky is not much more complicated than moving pieces around a chessboard.
-
gladius
- Posts: 568
- Joined: Tue Dec 12, 2006 10:10 am
- Full name: Gary Linscott
Re: BOINCification
If you end up having thousands of volunteer machines, that's great, but until then, I'd go with a fewer grad students, and a lot less hardwaresje wrote:Some of the big BOINC projects require a dozen or so machines, all of them expensive. For a small project like Perft@home, at least two machines of the Mac mini server class (~ US$1,000) would be needed, and maybe more for redundancy. Then there are the co-location fees plus the need to have two or three dedicated volunteers or grad student slaves for various administration purposes.
BOINC projects which tried to succeed with only one guy on a tight budget haven't fared very well.
The moderately sized MilkyWay@home project has about a dozen guys for science and administration. There are maybe a hundred thousand or so client host machines, each reporting in every hour or so. And moving stars around the sky is not much more complicated than moving pieces around a chessboard.
. Certainly no need for a colo!If you really want a dedicated box, just start with some off the shelf linux box for $300. The free EC2 offer for a year is pretty great though.
-
sje
- Posts: 4675
- Joined: Mon Mar 13, 2006 7:43 pm
Co-location server requirements
The history of distributed computing, and SETI@home in particular, shows that server requirements have been repeatedly and severely underestimated.
With the above caveat in mind, my estimate for a Perft@home BOINC server would require at least two Mac mini servers (2.3 GHz quad core Intel i7 CPU, 16 GB RAM, 2 x 1 TB drive each), and 1 TB monthly bandwidth. One server runs the work unit I/O, the scheduler, the database, and the web server while the other is for development, backup, and hot-swap redundancy. This would cost about US$2,500 up front and maybe about US$100 per month.
Example co-location vendors:
http://www.macminicolo.net/
http://www.macstadium.com/
http://www.macminivault.com/
http://www.macminicloud.net/
The competition keeps the prices reasonable.
With the above caveat in mind, my estimate for a Perft@home BOINC server would require at least two Mac mini servers (2.3 GHz quad core Intel i7 CPU, 16 GB RAM, 2 x 1 TB drive each), and 1 TB monthly bandwidth. One server runs the work unit I/O, the scheduler, the database, and the web server while the other is for development, backup, and hot-swap redundancy. This would cost about US$2,500 up front and maybe about US$100 per month.
Example co-location vendors:
http://www.macminicolo.net/
http://www.macstadium.com/
http://www.macminivault.com/
http://www.macminicloud.net/
The competition keeps the prices reasonable.