Questions on SMP search

Discussion of chess software programming and technical issues.

Moderator: Ras

User avatar
marcelk
Posts: 348
Joined: Sat Feb 27, 2010 12:21 am

Re: Questions on SMP search

Post by marcelk »

Gian-Carlo Pascutto wrote:
Laskos wrote: I don't think you have both qualifications and merit to attack in this manner Robert.
What qualifications or merit do you think he needs? He wrote his own program. He can see the Houdini output. That makes him overqualified, if anything.
By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.
I find it hard to appreciate the "skills" of Mr. Houdart when he himself makes every possible effort to pussyfoot away from the question how much of the performance is his own doing.

If Mr. Houdart had posted that he implemented an efficient parallelization onto Robbolito, there would be no controversy. But that's not what he is doing. On the contrary, these discussions turn into flamewars because he keeps implying Houdini isn't an Ippolit derivative, despite clear evidence to the contrary. So who's the troll exactly?
Mr. Houdart's ego should be satisfied with seeing wikipedia credit him for Houdini which must come in handy at family parties and job interviews. Indeed the distractions spark from him trying to weasel out of answering the obvious direct question with all evidence so nicely presented here. Not once but repeatedly. He is intelligent: only implying and suggesting he wrote all of it and avoiding a direct statement except on his webpage. That puts him almost on par with some people who have entered such programs in OTB tournaments. We'll see more of these "Houdarts" come and go. It is sad because the exact same contribution could also be received in a extremely positive way, if not for the dishonesty that comes bundled with it.

In the meantime it is telling that apparently there is still so much elo improvement to be squeezed out of current designs, and that the old-timers are as bad (or worse) at finding it as newcomers. In that way these clones keep me motivated to keep going on for at least another year or two.

I have to say that I have a lot more respect for the anonymous authors of the ippolit series. Unlike the other cloners they demonstrate that they know what they're doing in every detail of their engine. Their code looks like a decompiled pile of rubbish, but there are so many differences with Rybka that I can be made to believe that it is a decompilation of their own code, just distributed like that for the purpose of obfuscation. (Bring me back to reality if you wish).

And of course a big hurray for Ben-Hur Carlos for putting up with all of this deviation in his nice SMP thread while making his engine the way we old people do: 100% from the ground up without secretly copying other's people's work. Our game in Leiden last year was very enjoyable and his progress is steady, fast and intimidating... :)
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Questions on SMP search

Post by Laskos »

Albert Silver wrote:
Laskos wrote:
Albert Silver wrote:
Laskos wrote:
In short, Houdini seems to gain 7+/-3 (95% confidence) Elo points more than Crafty from doubling the number of cores. Rybka is even worse than Crafty in this regard.

Kai
What was the total Elo gain for each?
I didn't play 8 core vs 1 core, only 1 core vs 1 core and 8 core vs 8 core, so, I don't know. As the averaged time per solution goes, I forgot the absolute numbers, something like ~6.5 for Houdini and ~6 for Crafty on 8 vs 1 core, but I am not sure.

Kai
I have a problem with this testing, unless you tested them not against each other, but against opponents within ~100 Elo of each other. The problem is that the difference between ratings of Houdini and Crafty is so enormous (between 450-500 Elo in single CPU), that you are talking about scores in the 95%-5% range, and variations would be in the order of a fraction of a percent up to one percent.

Consider that the percentage difference for 0-100 Elo is 0-13% (roughly) while the percentage difference between 400-500 Elo is 92%-96%.
Yes, I had problems, I tested them directly, ~400 Elo points (~91% score) difference between Crafty 23.x and Houdini 1.5. Still managed to get an asymmetric +/- (+14 -8 or something like that) 95% conf. error margins after thousands of games. The main point was that the improvement of Houdini was beyond 95% error margins larger than improvement in Crafty when going from 1 core to 8. In fact, >99% confidence.

Kai
Last edited by Laskos on Wed Apr 27, 2011 8:42 pm, edited 1 time in total.
G.B. Harms
Posts: 42
Joined: Sat Aug 08, 2009 2:18 pm
Location: Almere

Re: Questions on SMP search

Post by G.B. Harms »

Laskos wrote:
G.B. Harms wrote:
Laskos wrote:
Albert Silver wrote:
Laskos wrote:And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's
Data?
Don't have right now, but I did the following:
NPS and time to depth are useless.

1. I put them on 8 core machine at my dept. to play short TC games, and measured the strength. I compared this strength difference to 1 core difference, it was roughly by 20 +/- 10 (95% confidence) Elo points larger for Houdini 1.5 in 8 core mode (I repeat, relative difference 8-1 core, not the absolute one).

2. I selected 40 positions from WM test, solvable by both Houdini and Crafty on one core in less than a minute. Then performed 8 core test. Took the geometric average of the results (time used for each solution), and Houdini again came relatively better in 8 core mode (again, relatively 8-1 core Houdini to 8-1 core Crafty).

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

Kai
My guess is that you really cannot compare the qualities of SMP implementations that way. Because the code bases of Crafty and Robbolito are completely different.
Yes, in this sense the intrinsic quality cannot be compared by any means, and all the discussion in this thread was pointless.

Kai
Yes, I agree. And discussions will remain pointless until the question whether or not Houdini started out as a -verbatim copy- of some Robbolito version remains unanswered.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Questions on SMP search

Post by bob »

Laskos wrote:
bob wrote:
I am personally convinced, 100%, that you are simply a fraud. Yes, you managed to improve Robo* in a fairly significant way. That doesn't change a thing, however....
Bob, can you keep that sort of wording to Rolf? With all kinds of trolls you engage in lengthy "intellectual" endeavors, both you and Rolfes polluting the forum and hijacking the threads. I don't think you have both qualifications and merit to attack in this manner Robert. And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's, with all 30 years of your "majestic" parallelization effort from Cray Blitz. Without talking of 3-400 Elo points gap...

Kai
So Houdini's "speedup" is better than Crafty's? I'd like to see some actual data showing that...
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Questions on SMP search

Post by bob »

Milos wrote:
Albert Silver wrote:
Laskos wrote:And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's
Data?
Well for any meaningful comparison it's also necessary for Bob to at least once present some data newer than those 30 years old from Cray Blitz...
please grow up, and learn to read. I have presented SMP data many times. Including making a complete set of 1-cpu, 2-cpu, 4-cpu and 8-cpu log files available for a known set of positions, searched on the same hardware, using different numbers of CPUs. Others have taken that data (Martin Fierz was one that did it for several such logs) and he posted the speedup numbers, which were a bit better than my linear approximation predicted.

I haven't reported CB speedup numbers since 1995 when it was retired, so perhaps you need to do a little research before posting???
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Questions on SMP search

Post by bob »

Laskos wrote:
Albert Silver wrote:
Laskos wrote:And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's
Data?
Don't have right now, but I did the following:
NPS and time to depth are useless.

In other words, you have no data. Time to depth is the _only_ way to measure SMP performance. and I do mean the _ONLY_ way.


1. I put them on 8 core machine at my dept. to play short TC games, and measured the strength. I compared this strength difference to 1 core difference, it was roughly by 20 +/- 10 (95% confidence) Elo points larger for Houdini 1.5 in 8 core mode (I repeat, relative difference 8-1 core, not the absolute one).

2. I selected 40 positions from WM test, solvable by both Houdini and Crafty on one core in less than a minute. Then performed 8 core test. Took the geometric average of the results (time used for each solution), and Houdini again came relatively better in 8 core mode (again, relatively 8-1 core Houdini to 8-1 core Crafty).

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

Kai
I agree. He is an excellent cut & paste person... Both of those other people you mentioned are, in fact...
Gian-Carlo Pascutto
Posts: 1260
Joined: Sat Dec 13, 2008 7:00 pm

Re: Questions on SMP search

Post by Gian-Carlo Pascutto »

bob wrote: In other words, you have no data. Time to depth is the _only_ way to measure SMP performance. and I do mean the _ONLY_ way.
Depends on the implementation. With things like LMR it matters how you hand out moves at a splitpoint, and even how exactly you lock there.

ELO gain from more cores is fine too. But comparing that between different engines is tricky, you'd need to establish a baseline for thinking 2x, 4x, ... as long first.

For example, if you use LMR over-aggressively, you might get "superlinear" ELO speedups depending on how strictly you check the "late" condition when you deal out moves at a splitpoint.

So basically, I don't really believe in comparing SMP between two engines. And for my own engine, I prefer ELO gain testing. Even if I don't get the exact speedup, I can still pick the one that did best and know I have the strongest engine, which is what matters.
Last edited by Gian-Carlo Pascutto on Wed Apr 27, 2011 9:04 pm, edited 1 time in total.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Questions on SMP search

Post by Laskos »

bob wrote:
Laskos wrote:
Albert Silver wrote:
Laskos wrote:And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's
Data?
Don't have right now, but I did the following:
NPS and time to depth are useless.

In other words, you have no data. Time to depth is the _only_ way to measure SMP performance. and I do mean the _ONLY_ way.

A little silly from you. You must know that the ELO gain and time to solution are MUCH more reliable. Depth "8 core" is not the same as Depth "1 core" (maybe in Crafty it's the same, that would be pretty unusual).

Kai
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Questions on SMP search

Post by bob »

Gian-Carlo Pascutto wrote:
bob wrote: In other words, you have no data. Time to depth is the _only_ way to measure SMP performance. and I do mean the _ONLY_ way.
ELO gain from more cores is fine too. But comparing that between different engines is tricky, you'd need to establish a baseline for thinking 2x, 4x, ... as long first.
And you don't need to compare two engines with a really significant gap between them. Nor for just a small number of positions. Etc... And even if I get a Elo gain or loss, that does not translate to the traditional SMP performance measurement everyone expects, namely "how much faster is 4 cores than 1?" and such...

This entire problem has simply spiralled completely out of control over recent years. And the "perps" still think that anyone with any experience at all is going to take their overnight success as legitimate, no questions asked. It does seem that the majority are willing to take that leap of faith. I'm not.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Questions on SMP search

Post by bob »

Laskos wrote:
bob wrote:
Laskos wrote:
Albert Silver wrote:
Laskos wrote:And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's
Data?
Don't have right now, but I did the following:
NPS and time to depth are useless.

In other words, you have no data. Time to depth is the _only_ way to measure SMP performance. and I do mean the _ONLY_ way.

A little silly from you. You must know that the ELO gain and time to solution are MUCH more reliable. Depth "8 core" is not the same as Depth "1 core" (maybe in Crafty it's the same, that would be pretty unusual).

Kai
You do realize that for _most_ positions, time to depth and time to solution are _identical_? Only those odd positions where a program changes its mind due to pure non-determinism will be different. Choose enough positions and those won't be a problem. If there is a clear best move, a program should find that move at the same depth, same PV, same score, most of the time. Hence a lot of runs to wash out the sampling error.