Peculiarity of Komodo 5.1MP

[bob]

I think we should agree that SMP efficiency in chess is measured in Elos, not depths.

For an end user, yes that would be the definition. But Bob may call that "SMP Effectiveness" or something like that, which he views as different from his narrower technical definition of SMP efficiency.

That would be perfectly acceptable since there is no such term used in existing parallel performance discussions. Just not "SMP efficiency".

[bob]

I think we should agree that SMP efficiency in chess is measured in Elos, not depths.

For an end user, yes that would be the definition. But Bob may call that "SMP Effectiveness" or something like that, which he views as different from his narrower technical definition of SMP efficiency.

Bob's definition may not be practical for real world applications but it is a valid concept. In computer science you often study some specific thing in isolation to understand its theoretical properties. Still, I don't believe that is what was happening here.

It certainly wasn't. Otherwise he could (and should) have simply stated that Kai was using an incorrect technical term but really meant <whatever term Bob would have liked better>. He could have been constructive. Instead he chose the path of "you are all wrong".

Note that for about the first half of the thread he insisted on "time-to-depth is the only way to measure smp efficiency" and even "time-to-depth is the only way to measure the COMPLETE smp implementation" rather than "time-to-depth is the technical definition of smp efficiency even though it may not capture elo gain at all" which he has turned to now to escape what he sees as "defeat".

This is his style - in fact not just his but most arguments on this forum go that way. Two people take a stand on some issue and then you will notice that both sides try to "close the box", which is term I made up which means they will try to re-define the other persons position in terms they can argue against better. Try to bend it and shape it their way.

It sometimes works if one of the parties is skilled enough because they will do it very gradually just by little changes in the wording of the other party and sometime the other party will accept the re-definition of their stance without realizing it. But the other side of this is that when one of the parties sees they are losing they will re-frame THEIR OWN statements. You have to be perceptive sometimes because that can be legitimate - especially if you were sloppy about what you said and it doesn't reflect what you actually meant. However it's difficult to convince someone that you meant something other than what you said even if it's true. But Bob is especially skilled at re-framing his position, you will see him swear nothing has changed as he gradually re-defines his statement in a way that you cannot pin him down. For him it comes in the forms of "revelations", he was talking about that the whole time and evidently just never bothered to say it. In reality we know better.

The thing that really give him away in this case is simply the fact that he started out being extremely critical. Once that happened nothing he said afterwards really mattered.

If you want to discuss like that, fine. Let's be REALLY critical. You have yet to post ANYTHING of any technical merit here. I pointed out two problems with the initial post.

1. It shows NOTHING about parallel performance. ABSOLUTELY nothing. Because it omits the critical ingredient, TIME. ALL that one can conclude from the original data is that your search looks at more stuff when using parallel search than it does with a serial search. That says nothing about whether the parallel search gains or loses Elo, because one can't tell how much that "wider search" slows things down.

2. The term "SMP efficiency" was mis-used and I pointed that out. Nothing more, nothing less.

If you want to resort to "he just does this because..." feel free. Won't win any debates with that approach however. And you can do so much better, if you want to.

I've not changed my original discussion points anywhere in this discussion. However, it definitely has grown tiring, so you can have the last word. Hopefully something useful as opposed to something sarcastic?

[michiguel]

So using "SMP efficiency" was the wrong term to use in the first post in this thread.

So this whole regrettable discussion turns around your personal definition of "SMP efficiency" that you have decided to impose upon all of us.

Any reasonable person understood perfectly well how the term was used in the first post.

This is typical of Bob's style when he is backed into a corner, he finds some technicality and digs in. It would be far easier for him to just to just yield a little ground and he could do this and still maintain his dignity but he will keep going until he cannot back down.

Don, did you not go to graduate school? And study parallel computing among other things?

Please provide ANY citation you want where "SMP efficiency" is defined any way other than what I quoted. Ever heard of Amdahl's law? Know what it is based on? That SAME definition.

Not all problems are governed by Amdhal's law (fixed task, time variable). Some are time constant, task variable (Guftafson's law)
http://en.wikipedia.org/wiki/Gustafson%27s_Law

There, with Guftafson's problems, the speedup is not measured by time to reach a constant task ("time to depth") but how much extra work (EDIT: or quality!!) you can fit into a constant time.

Chess engines when become parallel do not necessarily follow Amdhal's law since they do not do the exact same thing in MP mode, and they do not travel the exact same tree (and it could be engine dependent as it was demonstrated in this thread). In addition, same depth, does not imply same quality of work.

The goal on most problems is solve them quicker, but not in chess. The goal is to play stronger, in which playing quicker is one of the parameters that control strength, but not the only one. At least, not necessarily the only one. So, to be pedantic, since it looks what this thread became about, SMP efficiency is a concept that cannot (necessarily) be used in the "traditional" way here. We apply Guftafson's, Amdahl's, or neither?

Miguel

Jeez, why so sanctimonious here? The term has a specific meaning. It was used in the first post in this thread. I pointed out it was wrong. It is STILL wrong. I'm not in any sort of corner at all, just you guys that want to completely misuse a standard CIS term...

[Cumnor]

So using "SMP efficiency" was the wrong term to use in the first post in this thread.

So this whole regrettable discussion turns around your personal definition of "SMP efficiency" that you have decided to impose upon all of us.

Any reasonable person understood perfectly well how the term was used in the first post.

This is typical of Bob's style when he is backed into a corner, he finds some technicality and digs in. It would be far easier for him to just to just yield a little ground and he could do this and still maintain his dignity but he will keep going until he cannot back down.

Don, did you not go to graduate school? And study parallel computing among other things?

Please provide ANY citation you want where "SMP efficiency" is defined any way other than what I quoted. Ever heard of Amdahl's law? Know what it is based on? That SAME definition.

Jeez, why so sanctimonious here? The term has a specific meaning. It was used in the first post in this thread. I pointed out it was wrong. It is STILL wrong. I'm not in any sort of corner at all, just you guys that want to completely misuse a standard CIS term...

Actually I was just reading about Amdahl's law http://www.clustermonkey.net/Parallel-P ... r-law.html

[Laskos]

I think we should agree that SMP efficiency in chess is measured in Elos, not depths.

For an end user, yes that would be the definition. But Bob may call that "SMP Effectiveness" or something like that, which he views as different from his narrower technical definition of SMP efficiency.

I think for everybody, Bob included, was clear what was meant. Going from 1 to N cores, how large is the factor time(1cpu)/time(Ncpu) at equal _strength_. Bob started this "misunderstanding" game after 60-70 posts in this thread, and tha shows his character, no more.

The fact is time-to-depth is not a measure of the effectiveness of SMP implementation (as defined above), because parallelly one doesn't search the same tree as sequentially.

[Joerg Oster]

Actually I was just reading about Amdahl's law http://www.clustermonkey.net/Parallel-P ... r-law.html

Funny and interesting read. Thanks for sharing!

[Laskos]

I am back to my desktop i7, 4 physical cores. I computed the COMPLETE SMP implementation of Komodo 5.1MP as following:
30 positions repeated 5 times to _fixed_ depth 21 on 1 core and 4 cores. Total time is given below:

Code: Select all

Komodo 5.1MP 4 cores 108:50 minutes
Komodo 5.1MP 1 core  183:05 minutes

ratio: 1.68

I tested Houdini 3 too:

Code: Select all

Houdini 3 4 cores 58:09 minutes
Houdini 3 1 core  177:28 minutes

ratio: 3.05

These are time-to-depth ratios from 1 to 4 cores. As we see, time-to-depth is bullocks if the Elo gain in Komodo and Houdini is comparable going from 1 to 4 cores.

[Laskos]

I tested at 5''+0.1'' the effectiveness of Komodo MP implementation to that of Houdini, from 1 to 4 threads

Code: Select all

    Program                            Score     %     Elo

  1 Houdini 3  1 thread           : 640.5/1000  64.0   3050
  2 Komodo 5.1 1 thread           : 359.5/1000  35.9   2950

Houdini +100 points


    Program                            Score     %     Elo 

  1 Houdini 3  4 threads          : 642.5/1000  64.2   3051
  2 Komodo 5.1 4 threads          : 357.5/1000  35.8   2949

Houdini +102 points

The effectiveness is very comparable, within error margins.

[Houdini]

I tested at 5''+0.1'' the effectiveness of Komodo MP implementation to that of Houdini, from 1 to 4 threads

Code: Select all

    Program                            Score     %     Elo

  1 Houdini 3  1 thread           : 640.5/1000  64.0   3050
  2 Komodo 5.1 1 thread           : 359.5/1000  35.9   2950

Houdini +100 points


    Program                            Score     %     Elo 

  1 Houdini 3  4 threads          : 642.5/1000  64.2   3051
  2 Komodo 5.1 4 threads          : 357.5/1000  35.8   2949

Houdini +102 points

The effectiveness is very comparable, within error margins.

Kai, thanks, very interesting results.

To isolate the impact of the SMP implementation from any TC scaling effects, you might consider comparing (5"+0.1" with 4 CPU) to (15"+0.3" with 1 CPU) - or whatever TC that produces approximately the same average depth for Houdini.

The fact that Houdini maintains its 100 point lead while you're effectively tripling the TC by 3 suggests that Houdini's SMP implementation is marginally more effective (if results were confirmed with a larger number of games).

[Laskos]

I tested at 5''+0.1'' the effectiveness of Komodo MP implementation to that of Houdini, from 1 to 4 threads

Code: Select all

    Program                            Score     %     Elo

  1 Houdini 3  1 thread           : 640.5/1000  64.0   3050
  2 Komodo 5.1 1 thread           : 359.5/1000  35.9   2950

Houdini +100 points


    Program                            Score     %     Elo 

  1 Houdini 3  4 threads          : 642.5/1000  64.2   3051
  2 Komodo 5.1 4 threads          : 357.5/1000  35.8   2949

Houdini +102 points

The effectiveness is very comparable, within error margins.

Kai, thanks, very interesting results.

To isolate the impact of the SMP implementation from any TC scaling effects, you might consider comparing (5"+0.1" with 4 CPU) to (15"+0.3" with 1 CPU) - or whatever TC that produces approximately the same average depth for Houdini.

The fact that Houdini maintains its 100 point lead while you're effectively tripling the TC by 3 suggests that Houdini's SMP implementation is marginally more effective (if results were confirmed with a larger number of games).

Yes, I agree, I wanted to say that, but didn't to not complicate things. The interesting point is that the factor of ~3 is almost the same for Houdini and Komodo, although time-to-depth is 1.8 higher in Komodo. This 1.8 comes from widening.

I will do the match 3x time for single core, and I, as you, expect a marginally better SMP implementation of Houdini, but error margins are still large for just 1,000 games.