Questions on SMP search

[Albert Silver]

And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's

Data?

Don't have right now, but I did the following:
NPS and time to depth are useless.

1. I put them on 8 core machine at my dept. to play short TC games, and measured the strength. I compared this strength difference to 1 core difference, it was roughly by 20 +/- 10 (95% confidence) Elo points larger for Houdini 1.5 in 8 core mode (I repeat, relative difference 8-1 core, not the absolute one).

2. I selected 40 positions from WM test, solvable by both Houdini and Crafty on one core in less than a minute. Then performed 8 core test. Took the geometric average of the results (time used for each solution), and Houdini again came relatively better in 8 core mode (again, relatively 8-1 core Houdini to 8-1 core Crafty).

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

Kai

I think the test has no validity. The randomness of finding or not positions in SMP makes this sort of thing too unreliable. Just as you can get wildly varied results in 100-game matches with identical conditions, the same is true of solving positions in SMP.

I think the only way would be to test for strength via matches, and compare the Elo results with one core and then multiple ones.

[Laskos]

And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's

Data?

Don't have right now, but I did the following:
NPS and time to depth are useless.

1. I put them on 8 core machine at my dept. to play short TC games, and measured the strength. I compared this strength difference to 1 core difference, it was roughly by 20 +/- 10 (95% confidence) Elo points larger for Houdini 1.5 in 8 core mode (I repeat, relative difference 8-1 core, not the absolute one).

2. I selected 40 positions from WM test, solvable by both Houdini and Crafty on one core in less than a minute. Then performed 8 core test. Took the geometric average of the results (time used for each solution), and Houdini again came relatively better in 8 core mode (again, relatively 8-1 core Houdini to 8-1 core Crafty).

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

Kai

I think the test has no validity. The randomness of finding or not positions in SMP makes this sort of thing too unreliable. Just as you can get wildly varied results in 100-game matches with identical conditions, the same is true of solving positions in SMP.

I think the only way would be to test for strength via matches, and compare the Elo results with one core and then multiple ones.

Did you read my post?

I did do the match-up in thousands of games, and Houdini came better. Even the time to solution test is pretty valid, as there were 40 positions to solve. I took the geometric average for time used because time runs exponentially in regard to finding solutions.

In short, Houdini seems to gain 7+/-3 (95% confidence) Elo points more than Crafty from doubling the number of cores. Rybka is even worse than Crafty in this regard.

Kai

[Albert Silver]

And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's

Data?

Don't have right now, but I did the following:
NPS and time to depth are useless.

1. I put them on 8 core machine at my dept. to play short TC games, and measured the strength. I compared this strength difference to 1 core difference, it was roughly by 20 +/- 10 (95% confidence) Elo points larger for Houdini 1.5 in 8 core mode (I repeat, relative difference 8-1 core, not the absolute one).

2. I selected 40 positions from WM test, solvable by both Houdini and Crafty on one core in less than a minute. Then performed 8 core test. Took the geometric average of the results (time used for each solution), and Houdini again came relatively better in 8 core mode (again, relatively 8-1 core Houdini to 8-1 core Crafty).

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

Kai

I think the test has no validity. The randomness of finding or not positions in SMP makes this sort of thing too unreliable. Just as you can get wildly varied results in 100-game matches with identical conditions, the same is true of solving positions in SMP.

I think the only way would be to test for strength via matches, and compare the Elo results with one core and then multiple ones.

Did you read my post?

I did do the match-up in thousands of games, and Houdini came better. Even the time to solution test is pretty valid, as there were 40 positions to solve. I took the geometric average for time used because time runs exponentially in regard to finding solutions.

In short, Houdini seems to gain 7+/-3 (95% confidence) Elo points more than Crafty from doubling the number of cores. Rybka is even worse than Crafty in this regard.

Kai

What was the total Elo gain for each?

[Laskos]

In short, Houdini seems to gain 7+/-3 (95% confidence) Elo points more than Crafty from doubling the number of cores. Rybka is even worse than Crafty in this regard.

Kai

What was the total Elo gain for each?

I didn't play 8 core vs 1 core, only 1 core vs 1 core and 8 core vs 8 core, so, I don't know. As the averaged time per solution goes, I forgot the absolute numbers, something like ~6.5 for Houdini and ~6 for Crafty on 8 vs 1 core, but I am not sure.

Kai

[Gian-Carlo Pascutto]

I don't think you have both qualifications and merit to attack in this manner Robert.

What qualifications or merit do you think he needs? He wrote his own program. He can see the Houdini output. That makes him overqualified, if anything.

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

I find it hard to appreciate the "skills" of Mr. Houdart when he himself makes every possible effort to pussyfoot away from the question how much of the performance is his own doing.

If Mr. Houdart had posted that he implemented an efficient parallelization onto Robbolito, there would be no controversy. But that's not what he is doing. On the contrary, these discussions turn into flamewars because he keeps implying Houdini isn't an Ippolit derivative, despite clear evidence to the contrary. So who's the troll exactly?

[G.B. Harms]

And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's

Data?

Don't have right now, but I did the following:
NPS and time to depth are useless.

1. I put them on 8 core machine at my dept. to play short TC games, and measured the strength. I compared this strength difference to 1 core difference, it was roughly by 20 +/- 10 (95% confidence) Elo points larger for Houdini 1.5 in 8 core mode (I repeat, relative difference 8-1 core, not the absolute one).

2. I selected 40 positions from WM test, solvable by both Houdini and Crafty on one core in less than a minute. Then performed 8 core test. Took the geometric average of the results (time used for each solution), and Houdini again came relatively better in 8 core mode (again, relatively 8-1 core Houdini to 8-1 core Crafty).

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

Kai

My guess is that you really cannot compare the qualities of SMP implementations that way. Because the code bases of Crafty and Robbolito are completely different.

[Laskos]

I don't think you have both qualifications and merit to attack in this manner Robert.

What qualifications or merit do you think he needs? He wrote his own program. He can see the Houdini output. That makes him overqualified, if anything.

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

I find it hard to appreciate the "skills" of Mr. Houdart when he himself makes every possible effort to pussyfoot away from the question how much of the performance is his own doing.

If Mr. Houdart had posted that he implemented an efficient parallelization onto Robbolito, there would be no controversy. But that's not what he is doing. On the contrary, these discussions turn into flamewars because he keeps implying Houdini isn't an Ippolit derivative, despite clear evidence to the contrary. So who's the troll exactly?

I will not willingly digress into this. We all know certain things, like that Bob has his original engine built almost from scratch. Maybe we have to accept that times have changed. Besides Rybka and Ippolit affairs, look at Adam Hair's list in the main forum. Among top 10 engines I doubt that more than 2-3 are nearly as original as Crafty. The time of "purists" seems to be at its end. I do not know why some folks will try to muddy R. Houdart and not all those 7-8 out of top 10 engine authors. Some of them are commercial, which at least R. Houdart is not. Some of them are closer to an open source engine in output than Houdini is.

Now, if you guys are so brave in muddying R.Houdart, please start trash-talk commercial Naum 4.2. Ask Adam or Miguel for which reasons.

I am not even sure what to say about Fritz 11/12. Can somebody perform a Sim test with them?

Kai

[Laskos]

And on topic, Houdini 1.5 parallelization _practically_ works better than Crafty's

Data?

Don't have right now, but I did the following:
NPS and time to depth are useless.

1. I put them on 8 core machine at my dept. to play short TC games, and measured the strength. I compared this strength difference to 1 core difference, it was roughly by 20 +/- 10 (95% confidence) Elo points larger for Houdini 1.5 in 8 core mode (I repeat, relative difference 8-1 core, not the absolute one).

2. I selected 40 positions from WM test, solvable by both Houdini and Crafty on one core in less than a minute. Then performed 8 core test. Took the geometric average of the results (time used for each solution), and Houdini again came relatively better in 8 core mode (again, relatively 8-1 core Houdini to 8-1 core Crafty).

By the way, Rybka scales worse than both, after 6 years of another "majestic" efforts. I think some people have to appreciate the skills of R. Houdart.

Kai

My guess is that you really cannot compare the qualities of SMP implementations that way. Because the code bases of Crafty and Robbolito are completely different.

Yes, in this sense the intrinsic quality cannot be compared by any means, and all the discussion in this thread was pointless.

Kai

[Gian-Carlo Pascutto]

I will not willingly digress into this. We all know certain things, like that Bob has his original engine built almost from scratch. Maybe we have to accept that times have changed.
...
The time of "purists" seems to be at its end.

If that is your vision, fine by me, but just be upfront about it. I don't have a problem with Stockfish and I most certainly don't have a problem with collaborative development.

But I have a problem with people who do everything to avoid telling the truth.

[Albert Silver]

In short, Houdini seems to gain 7+/-3 (95% confidence) Elo points more than Crafty from doubling the number of cores. Rybka is even worse than Crafty in this regard.

Kai

What was the total Elo gain for each?

I didn't play 8 core vs 1 core, only 1 core vs 1 core and 8 core vs 8 core, so, I don't know. As the averaged time per solution goes, I forgot the absolute numbers, something like ~6.5 for Houdini and ~6 for Crafty on 8 vs 1 core, but I am not sure.

Kai

I have a problem with this testing, unless you tested them not against each other, but against opponents within ~100 Elo of each other. The problem is that the difference between ratings of Houdini and Crafty is so enormous (between 450-500 Elo in single CPU), that you are talking about scores in the 95%-5% range, and variations would be in the order of a fraction of a percent up to one percent.

Consider that the percentage difference for 0-100 Elo is 0-13% (roughly) while the percentage difference between 400-500 Elo is 92%-96%.