Stockfish 020114 - Houdini 4 x64A Testing 39 of 100 played.

[shrapnel]

My PGN have all depth, time, and move info in them. All you have to do is put them into a fritz GUI.

Fritz GUI is piece of crap (I don't posses that piece of crap and don't intend to use it in my life) and its PGNs don't contain any depth information.
Only totally illiterate person could claim that after looking into them.

There are many things other then a fritz GUI you don't posses.

Nice one ! This fight is getting interesting !

[shrapnel]

Hi Mark
Could you test Komodo TCECr against Houdini or Stockfish in LTC control games ?
I'm getting surprisingly good results with Komodo TCECr in my online LTC games !
Regards

[Rebel]

So it still appears to me that Houdini 4 is stronger on one core and SF Dec. 30 on four cores, regardless of hyperthreading, sleeping threads, etc. Milos observed that SF takes much more time than Houdini, but I think this is simplly algorithmic; SF is much more aggressive about time use than any other top engine, which does help its middlegame play but leaves it short of time for the endgame. Maybe this tendency is exaggerated in MP play for some reason.

If you just go through pgns of Mark you will notice that despite weird TC (2'+12'') H4 TM is totally crazy i.e. it ends every game (winning or losing) with 5+ minutes on the clock. This is impossible to be normal behavior (algorithmic) and it doesn't behave like this at all (with the same TC) on my 6-core AMD machine and different GUI. I have neither Fritz nor Intel i7 to try so I can't debug the reason (even though I suspect it's GUI and if provided UCI logs, it would be easy to find where is the error), but something is obviously wrong.

You bet, but that doesn't automatically mean the blame is on the GUI or tester. Consider a statistic with a tool of mine that counts all move times found in a PGN and calculates an average time per move. If I take a recent PGN from CEGT I get:

http://www.top-5000.nl/tc3.htm

Surely H4 looks not good and these games are all played at 40/8m.

[bnculp]

Code: Select all

System    Hyperthread       Engine     Threads     ELO 

 i7-3720QM   Yes           SF-311213       8       +7 
 i7-3720QM   Yes           Houdini-4       4       -7 

 i7-3720QM   Yes           SF-311213       8       +7 
 i7-3720QM   Yes           SF-311213       4       -7 

 i7-3720QM   Yes           Houdini-4       8       -14 
 i7-3720QM   Yes           Houdini-4       4       +14 

 i7-2600k    No            SF-311213       4       -16 
 i7-2600k    No            Houdini-4       4       +16 

 i7-2600k    No            SF-311213       1       -53
 i7-2600k    No            Houdini-4       1       +53

 i7-2600k    Yes           SF-311213       1       -63
 i7-2600k    Yes           Houdini-4       1       +63

 i7-2600k    Yes           SF-311213       2       -19
 i7-2600k    Yes           Houdini-4       2       +19

Match conditions :

GUI - Cutechess
Games - 1000
Time control - 15 sec + .05 sec
Hash - 128mb
Houdini contempt - 0
Openings - 8moves_v3.pgn , repeat with colors reversed
EGTBs - none

==>> Update - 2 more tests added
Its seems that Stockfish likes hyperthreading and more threads. Houdini is still king at single thread.

[mwyoung]

Code: Select all

System    Hyperthread       Engine     Threads     ELO 

 i7-3720QM   Yes           SF-311213       8       +7 
 i7-3720QM   Yes           Houdini-4       4       -7 

 i7-3720QM   Yes           SF-311213       8       +7 
 i7-3720QM   Yes           SF-311213       4       -7 

 i7-3720QM   Yes           Houdini-4       8       -14 
 i7-3720QM   Yes           Houdini-4       4       +14 

 i7-2600k    No            SF-311213       4       -16 
 i7-2600k    No            Houdini-4       4       +16 

 i7-2600k    No            SF-311213       1       -53
 i7-2600k    No            Houdini-4       1       +53

 i7-2600k    Yes           SF-311213       1       -63
 i7-2600k    Yes           Houdini-4       1       +63

 i7-2600k    Yes           SF-311213       2       -19
 i7-2600k    Yes           Houdini-4       2       +19

Match conditions :

GUI - Cutechess
Games - 1000
Time control - 15 sec + .05 sec
Hash - 128mb
Houdini contempt - 0
Openings - 8moves_v3.pgn , repeat with colors reversed
EGTBs - none

==>> Update - 2 more tests added
Its seems that Stockfish likes hyperthreading and more threads. Houdini is still king at single thread.

My testing concurs with your results.

[shrapnel]

Code: Select all

System    Hyperthread       Engine     Threads     ELO 

 i7-3720QM   Yes           SF-311213       8       +7 
 i7-3720QM   Yes           Houdini-4       4       -7 

 i7-3720QM   Yes           SF-311213       8       +7 
 i7-3720QM   Yes           SF-311213       4       -7 

 i7-3720QM   Yes           Houdini-4       8       -14 
 i7-3720QM   Yes           Houdini-4       4       +14 

 i7-2600k    No            SF-311213       4       -16 
 i7-2600k    No            Houdini-4       4       +16 

 i7-2600k    No            SF-311213       1       -53
 i7-2600k    No            Houdini-4       1       +53

 i7-2600k    Yes           SF-311213       1       -63
 i7-2600k    Yes           Houdini-4       1       +63

 i7-2600k    Yes           SF-311213       2       -19
 i7-2600k    Yes           Houdini-4       2       +19

Match conditions :

GUI - Cutechess
Games - 1000
Time control - 15 sec + .05 sec
Hash - 128mb
Houdini contempt - 0
Openings - 8moves_v3.pgn , repeat with colors reversed
EGTBs - none

==>> Update - 2 more tests added
Its seems that Stockfish likes hyperthreading and more threads. Houdini is still king at single thread.

My testing concurs with your results.

Sorry, but in this case, you are both wrong. I see that in your HT enabled System, you have allotted 8 threads to SF and 4 to Houdini 4, on the assumption that HT is bad for Houdini 4.
Now, it may be correct that HT-enabled and using 8 threads is bad for Houdini4, I don't dispute THAT, but by enabling HT AND allotting only 4 threads to Houdini 4, you are actually giving only around 50 % of the PCs resources to Houdini 4, causing a marked fall in Houdini's ELO.
The correct way is to disable HT in the BIOS itself !
But then of course, you wouldn't be able run Stockfish with all 8 threads .
So sorry, the only correct way to test Houdini 4 with HT off against Stockfish with HT on, is to run the Test on two identical but different Systems, while also making sure to turn HT Off in the BIOS itself on the PC/Laptop running Houdini 4.
The kN/s got with HT ON and using 4 Threads is markedly different from the kN/s got with HT OFF and using 4 threads !
Let me explain with an example from my own 6-Core i7 3930 k.
First, this is the Fritz Benchmark with HT ON, but using only the 6 real(so-called) cores:-

Then, with HT OFF in BIOS itself, thereby PROPERLY using the 6 Cores :-

It can be seen, in the above case, there is almost a 3000kN/s difference having HT ON and OFF, even for the same number of Cores/Threads !!!
So, I should think its obvious , that you are actually HURTING Houdini 4 by leaving HT ON !
If your Laptop doesn't allow you turn off HT in BIOS, bad luck, but I'm sorry to say, that ANY Test with HT ON, if Houdini is using only real cores and the other Engine is allowed to use ALL threads virtual and real, is badly BIASED against Houdini 4 and is NOT a fair Test.
So, in conclusion, you will have to find some way to turn off HT in BIOS itself AND run the Tests on 2 identical Systems, if you insist on leaving hyperthreading on for Stockfish.
Thank You for your time.

[arjuntemurnikar]

Code: Select all

System    Hyperthread       Engine     Threads     ELO 

 i7-3720QM   Yes           SF-311213       8       +7 
 i7-3720QM   Yes           Houdini-4       4       -7 

 i7-3720QM   Yes           SF-311213       8       +7 
 i7-3720QM   Yes           SF-311213       4       -7 

 i7-3720QM   Yes           Houdini-4       8       -14 
 i7-3720QM   Yes           Houdini-4       4       +14 

 i7-2600k    No            SF-311213       4       -16 
 i7-2600k    No            Houdini-4       4       +16 

 i7-2600k    No            SF-311213       1       -53
 i7-2600k    No            Houdini-4       1       +53

 i7-2600k    Yes           SF-311213       1       -63
 i7-2600k    Yes           Houdini-4       1       +63

 i7-2600k    Yes           SF-311213       2       -19
 i7-2600k    Yes           Houdini-4       2       +19

Match conditions :

GUI - Cutechess
Games - 1000
Time control - 15 sec + .05 sec
Hash - 128mb
Houdini contempt - 0
Openings - 8moves_v3.pgn , repeat with colors reversed
EGTBs - none

==>> Update - 2 more tests added
Its seems that Stockfish likes hyperthreading and more threads. Houdini is still king at single thread.

My testing concurs with your results.

Sorry, but in this case, you are both wrong. I see that in your HT enabled System, you have allotted 8 threads to SF and 4 to Houdini 4, on the assumption that HT is bad for Houdini 4.
Now, it may be correct that HT-enabled and using 8 threads is bad for Houdini4, I don't dispute THAT, but by enabling HT AND allotting only 4 threads to Houdini 4, you are actually giving only around 50 % of the PCs resources to Houdini 4, causing a marked fall in Houdini's ELO.
The correct way is to disable HT in the BIOS itself !
But then of course, you wouldn't be able run Stockfish with all 8 threads .
So sorry, the only correct way to test Houdini 4 with HT off against Stockfish with HT on, is to run the Test on two identical but different Systems, while also making sure to turn HT Off in the BIOS itself on the PC/Laptop running Houdini 4.
The kN/s got with HT ON and using 4 Threads is markedly different from the kN/s got with HT OFF and using 4 threads !
Let me explain with an example from my own 6-Core i7 3930 k.
First, this is the Fritz Benchmark with HT ON, but using only the 6 real(so-called) cores:-

Then, with HT OFF in BIOS itself, thereby PROPERLY using the 6 Cores :-

It can be seen, in the above case, there is almost a 3000kN/s difference having HT ON and OFF, even for the same number of Cores/Threads !!!
So, I should think its obvious , that you are actually HURTING Houdini 4 by leaving HT ON !
If your Laptop doesn't allow you turn off HT in BIOS, bad luck, but I'm sorry to say, that ANY Test with HT ON, if Houdini is using only real cores and the other Engine is allowed to use ALL threads virtual and real, is badly BIASED against Houdini 4 and is NOT a fair Test.
So, in conclusion, you will have to find some way to turn off HT in BIOS itself AND run the Tests on 2 identical Systems, if you insist on leaving hyperthreading on for Stockfish.
Thank You for your time.

BINGO! Mystery solved.

Btw, can you show us your fritz benchmark with HT on and 12 threads, just for the hell of it?

[bnculp]

Now, it may be correct that HT-enabled and using 8 threads is bad for Houdini4, I don't dispute THAT, but by enabling HT AND allotting only 4 threads to Houdini 4, you are actually giving only around 50 % of the PCs resources to Houdini 4, causing a marked fall in Houdini's ELO

I understand that it is easy to come to this false conclusion. If 4 threads is only allowing Houdini 50% of the CPU then why did Houdini 4 threads beat Houdini 8 threads ? Think about it ! I will run another test with Houdini 8 threads vs Stockfish 8 threads.

If your Laptop doesn't allow you turn off HT in BIOS, bad luck, but I'm sorry to say, that ANY Test with HT ON, if Houdini is using only real cores and the other Engine is allowed to use ALL threads virtual and real, is badly BIASED against Houdini 4 and is NOT a fair Test.
So, in conclusion, you will have to find some way to turn off HT in BIOS itself AND run the Tests on 2 identical Systems, if you insist on leaving hyperthreading on for Stockfish

So what would constitute a fair test on a HT enabled system ? 8 threads vs 8 threads ? 4 threads vs 4 threads ? It almost seems you are saying that any test on an HT system is BIASED against Houdini. Think about what that says about Houdini

[bnculp]

"The architecture of Houdini (and of chess engines in general) is not very well suited for hyper-threading; using more threads than physical cores will usually degrade the performance of the engine. Although the hyper-threads often produce a slightly higher node speed, the increased inefficiency of the parallel alpha-beta search more than offsets the speed gain obtained with the additional hyper-threads. To give a practical example, it's more efficient to use 4 threads running at 2,000 kN/s each than 8 threads running at 1,100 kN/s each, although the latter situation produces a higher total node speed. For this reason it's best to set the number of threads not higher than the number of physical cores of your hardware."

The above is from the Houdini 4 Pro users guide. Note the example where a higher total node speed is NOT necessarily a better setup.

[shrapnel]

I understand that it is easy to come to this false conclusion. If 4 threads is only allowing Houdini 50% of the CPU then why did Houdini 4 threads beat Houdini 8 threads ? Think about it ! I will run another test with Houdini 8 threads vs Stockfish 8 threads.

So what would constitute a fair test on a HT enabled system ? 8 threads vs 8 threads ? 4 threads vs 4 threads ? It almost seems you are saying that any test on an HT system is BIASED against Houdini. Think about what that says about Houdini

I thought I explained everything properly, but let me further clarify.
My answer in 2 parts :-
(a) Houdini 4 threads beats Houdini with 8 threads BECAUSE of "the increased inefficiency of the parallel alpha-beta search more than
offsets the speed gain obtained with the additional hyper-threads".
(b) Yes, Barry, I'm sorry, but I'm afraid if you insist on enabling ALL threads for Stockfish, the Test will indeed be biased against Houdini whether Houdini uses 4 threads or 8 !! Let me elucidate further.
If Houdini uses 4 threads on a 4-Core HT-enabled System, it will not be getting full resources of the PC ; while if it uses all 8 threads, the 'increased inefficiency of parallel alpha-beta search' factor kicks in !
Really, its a case of Heads, Houdini loses and Tails, Stockfish wins !
I can't be clearer than that !
So, as I mentioned earlier, the solution is to run the Tests using TWO identical Systems, taking care that HT is disabled in the BIOS itself in the System running Houdini, OR else, if its impossible to disable HT in BIOS, run both Stockfish and Houdini with 4 threads each on a System like yours !
Hope I've made things clearer to excellent and dedicated Testers like you and Mark Young !
Regards