44 elo swing depending on hardware!

[lkaufman]

meaning one test per thread rather than per core

Do you mean you use hyper-threading ?

Yes.

But now I have some new information that radically changes the conclusions. The above data was all run on LittleBlitzer as one on one matches using the Gauntlet option. Last night I did a run where I had Komodo (same version as will play in stage 3 of tcec), Houdini 3, and Stockfish oct 13 play round robins on LittleBlitzer at 30" +.3" on 16 core and 20" + .2" on quad (since quad is significantly faster per core), no overprovisioning (4 games at a time on quad, 16 on 16 core). This time the nodes per second, depths reached, and results were almost the same on the two machines; in fact the NPS ratio of Houdini to Komodo was slightly higher (1.38 vs 1.35) on the 16 core! So whatever the problem was, it was apparently tied to something about the LittleBlitzer software. I'll need to run some further tests to determine whether the problem relates to Gauntlet mode, or to the use of just two opponents. Probably it has something to do with the accounting for lag, which I suppose is much more of a problem on a two processor machine than on a single processor machine.

[shrapnel]

I ran a match on a third generation I7 quad between Komodo (latest dev. version) and Houdini 3 at bullet speed (30 seconds plus a quarter of a second increment), overprovisioned (meaning one test per thread rather than per core). Just as the light-speed list shows, at these speeds there is a sizable gap; Houdini won by 34 elo after 2600 games. No surprise.
Then I ran the same match on my sixteen core machine, with the time limit increased by a third to make the average depth reached about the same. Much to my surprise, Komodo won the match by ten elo after 2300 games! This is a swing of 44 elo points. Incredible!
The relative NPS of the two programs explalns the result. On the quad the ratio (Houdini NPS divided by Komodo) was 1.53. On the 16 core the same ratio was 1.25. This huge difference can account for something like 44 elo points at this fast a level.
But the question is WHY is there such a huge difference in the ratio between the two programs?
I ran the same test with Komodo against Stockfish but got only a negligible difference in results or in NPS ratio. Overprovisioning doesn't seem to be a big factor; when I run the same tests limited to the number of cores I get smaller NPS ratios on both machines, still much higher on the quad.
I also tried timing the opening position, 20 ply search on a single core. Here the ratios both grew: on the quad it was 1.68, on the 16 core it was 1.64.
Can anyone explain these huge disparities in NPS ratios?

I have already reported this yesterday on the Rybka forum.
http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=27808
It is my observation that the people criticizing Komodo 6 were generally using average or below-average Hardware, while the people praising it were generally using modern, fast hardware.
I play online Engine-Engine Matches on www.come2play.com.
To my surprise, I found that I was beating people using Houdini 3, if I was using Komodo 6....the very same opponents drew with me if I was using Houdini 3, instead ! This happened not once, but several times !
Finally, as I reported there, I came to the conclusion that Komodo 6 scales EXTREMELY WELL with increasing strength of Hardware used, MUCH MORE than the Komodo Engine authors themselves expected !
The more the kN/s, the more the strength of the Engine , MUCH MORE so than if one was using H 3 !
Others may not agree with me, but this is my GUT feeling, and I KNOW I'm right about this !

[lkaufman]

Can anyone explain these huge disparities in NPS ratios?

I have already reported this yesterday on the Rybka forum.
http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=27808
It is my observation that the people criticizing Komodo 6 were generally using average or below-average Hardware, while the people praising it were generally using modern, fast hardware.
I play online Engine-Engine Matches on www.come2play.com.
To my surprise, I found that I was beating people using Houdini 3, if I was using Komodo 6....the very same opponents drew with me if I was using Houdini 3, instead ! This happened not once, but several times !
Finally, as I reported there, I came to the conclusion that Komodo 6 scales EXTREMELY WELL with increasing strength of Hardware used, MUCH MORE than the Komodo Engine authors themselves expected !
The more the kN/s, the more the strength of the Engine , MUCH MORE so than if one was using H 3 !
Others may not agree with me, but this is my GUT feeling, and I KNOW I'm right about this !

In the past, when I talked about Komodo scaling well against Houdini, I was talking about the effect of giving more time on the same hardware, or using faster hardware that is otherwise identical. Now we are talking about a different effect, the advantage of using newer hardware given equivalent search depth on average. This isn't really "scaling" but it just seems that better, newer processors tend to help Komodo more than Houdini, for unknown reasons. In your case, another factor may be how many threads do you run MP? If you are comparing results using say 16 threads MP with those using 4 (for example) at the same time limit, you are really talking about the effect of at least doubling the time limit.
Anyway, regarding the topic, I established that when only two engines are playing in LittleBlitzer, regardless of mode (Gauntlet or RR), there is this huge difference between machines. When I run three engines in gauntlet mode, the disparity shrinks but remains serious. But when I run three engines in RR mode, the problem virtually disappears. So it's clearly a GUI issue, not a true one. Can anyone explain the above facts?

[shrapnel]

In your case, another factor may be how many threads do you run MP? If you are comparing results using say 16 threads MP with those using 4 (for example) at the same time limit, you are really talking about the effect of at least doubling the time limit.

Hi Larry
I think you miss my point. Even if I use say, 16 threads MP against those using 4, I'm unable to beat them while using Houdini 3, but am easily able to do so, if I use Komodo 6 !
This is because Houdini 3 plays almost equally strong whether one is using a quad-core or a 6 core or a 8 core, in my experience.
Now this may be a good thing or bad thing, depending on your point of view.
But in my point of view, this is a bad thing, as it doesn't seem to make much sense in investing in more modern and powerful Hardware if any idiot with a core2duo laptop can hold you to a draw, if he just happens to be using H3 !
But, with Komodo 6 the opposite happens ( hence my delight with Komodo 6 ) !!
Investment in good, fast hardware really starts to pay off if one is using Komodo 6, which is not the case with H3 !
Whether this is just a fluke, or whether you guys planned it this way, I, for one, am very happy with the results !

[lkaufman]

In your case, another factor may be how many threads do you run MP? If you are comparing results using say 16 threads MP with those using 4 (for example) at the same time limit, you are really talking about the effect of at least doubling the time limit.

Hi Larry
I think you miss my point. Even if I use say, 16 threads MP against those using 4, I'm unable to beat them while using Houdini 3, but am easily able to do so, if I use Komodo 6 !
This is because Houdini 3 plays almost equally strong whether one is using a quad-core or a 6 core or a 8 core, in my experience.
Now this may be a good thing or bad thing, depending on your point of view.
But in my point of view, this is a bad thing, as it doesn't seem to make much sense in investing in more modern and powerful Hardware if any idiot with a core2duo laptop can hold you to a draw, if he just happens to be using H3 !
But, with Komodo 6 the opposite happens ( hence my delight with Komodo 6 ) !!
Investment in good, fast hardware really starts to pay off if one is using Komodo 6, which is not the case with H3 !
Whether this is just a fluke, or whether you guys planned it this way, I, for one, am very happy with the results !

I understand your point. There are three different factors at work here:
1. Searching deeper seems to help Komodo more than Houdini.
2. Newer processors seem to help Komodo search deeper in a given time compared to Houdini.
3. Using more cores for MP may help Komodo more than Houdini (this is not so clear).

So what you are saying is that these three factors combined seem to be a big deal, when comparing a new 16 core machine to an old 4 core machine, both running MP. How big a deal is hard to quantify.

[Dr.Wael Deeb]

In your case, another factor may be how many threads do you run MP? If you are comparing results using say 16 threads MP with those using 4 (for example) at the same time limit, you are really talking about the effect of at least doubling the time limit.

Hi Larry
I think you miss my point. Even if I use say, 16 threads MP against those using 4, I'm unable to beat them while using Houdini 3, but am easily able to do so, if I use Komodo 6 !
This is because Houdini 3 plays almost equally strong whether one is using a quad-core or a 6 core or a 8 core, in my experience.
Now this may be a good thing or bad thing, depending on your point of view.
But in my point of view, this is a bad thing, as it doesn't seem to make much sense in investing in more modern and powerful Hardware if any idiot with a core2duo laptop can hold you to a draw, if he just happens to be using H3 !
But, with Komodo 6 the opposite happens ( hence my delight with Komodo 6 ) !!
Investment in good, fast hardware really starts to pay off if one is using Komodo 6, which is not the case with H3 !
Whether this is just a fluke, or whether you guys planned it this way, I, for one, am very happy with the results !

This aspect is the future of computer chess in my opinion....

Having over the edge hardware should provide you with a remarkble boost strength wise....
Dr.D

[Modern Times]

Investment in good, fast hardware really starts to pay off if one is using Komodo 6, which is not the case with H3 !

Komodo 6 benefits more I think - and that is why I think it will win TCEC this time around.

[pohl4711]

meaning one test per thread rather than per core

Do you mean you use hyper-threading ?

Yes.

But now I have some new information that radically changes the conclusions. The above data was all run on LittleBlitzer as one on one matches using the Gauntlet option. Last night I did a run where I had Komodo (same version as will play in stage 3 of tcec), Houdini 3, and Stockfish oct 13 play round robins on LittleBlitzer at 30" +.3" on 16 core and 20" + .2" on quad (since quad is significantly faster per core), no overprovisioning (4 games at a time on quad, 16 on 16 core). This time the nodes per second, depths reached, and results were almost the same on the two machines; in fact the NPS ratio of Houdini to Komodo was slightly higher (1.38 vs 1.35) on the 16 core! So whatever the problem was, it was apparently tied to something about the LittleBlitzer software. I'll need to run some further tests to determine whether the problem relates to Gauntlet mode, or to the use of just two opponents. Probably it has something to do with the accounting for lag, which I suppose is much more of a problem on a two processor machine than on a single processor machine.

Hi Larry,

I still believe, that if there is a problem with the LittleBlitzerGUI, than it is the fact, that there are 2 CPUs in one machine. So I can only repeat myself and say: Turn off Hyperthreading and run 2 instances of the LBG (2 * 8 games on your 16 core, 2 * 10 games on your 20 core) and see what happens...

Best - Stefan

[shrapnel]

This aspect is the future of computer chess in my opinion....

Having over the edge hardware should provide you with a remarkble boost strength wise....
Dr.D

I hope you are right !
It would be very disappointing if Houdini 4 plays with almost equal strength on all computers, like H 3 does at present !

[lkaufman]

meaning one test per thread rather than per core

Do you mean you use hyper-threading ?

Yes.

But now I have some new information that radically changes the conclusions. The above data was all run on LittleBlitzer as one on one matches using the Gauntlet option. Last night I did a run where I had Komodo (same version as will play in stage 3 of tcec), Houdini 3, and Stockfish oct 13 play round robins on LittleBlitzer at 30" +.3" on 16 core and 20" + .2" on quad (since quad is significantly faster per core), no overprovisioning (4 games at a time on quad, 16 on 16 core). This time the nodes per second, depths reached, and results were almost the same on the two machines; in fact the NPS ratio of Houdini to Komodo was slightly higher (1.38 vs 1.35) on the 16 core! So whatever the problem was, it was apparently tied to something about the LittleBlitzer software. I'll need to run some further tests to determine whether the problem relates to Gauntlet mode, or to the use of just two opponents. Probably it has something to do with the accounting for lag, which I suppose is much more of a problem on a two processor machine than on a single processor machine.

Hi Larry,

I still believe, that if there is a problem with the LittleBlitzerGUI, than it is the fact, that there are 2 CPUs in one machine. So I can only repeat myself and say: Turn off Hyperthreading and run 2 instances of the LBG (2 * 8 games on your 16 core, 2 * 10 games on your 20 core) and see what happens...

Best - Stefan

I may have to try that, although I have no experience with turning off Hyperthreading. Does turning it off increase, decrease, or leave roughly unchanged the nodes per second (assuming you run the same number of threads as cores in each case)?
How do the testing groups handle this? Do they leave Ht on or turn it off?