Stockfish 020114 - Houdini 4 x64A Testing 39 of 100 played.

[lkaufman]

I'm rerunning my test using Fritz 11, but so far the results are similar; SF leading by 31 elo after 74 games. So the GUI doesn't seem to be the issue.

I am not having any issues in my test with the gui, even when I tested at 1m+0s. But I am running a very clean setup, my CPU usage is 0 to 2% before testing. I have nothing running in the background other then normal windows system usage.

So what is going on... are the results legit in our test that show stockfish beating Houdini at these time controls. Are we missing something... I am stumped at this point. If somehow we are giving Stockfish a unfair advantage in our setup. You are testing Stockfish the standard way 4 cpu on a 4 core system. I know I am testing HT, but the results is the same. Meaning stockfish is winning by more then just noise factors. In you add up all the results. Or not?

Maybe someone can see a problem in the games I posted. I left all move and time data in the PGN. So it could be put straight into a Fritz GUI for evaluation.

Well, if you just posted pgns earlier it would be easier to detect. Even though your pgn's are mostly hopeless (no depth or nps info) judging by the time (and despite ridiculous TC), SF is using 20-30% more time than H4 (this is a drastic difference), or simply said H4 is never using its time which is obviously GUI problem.
So my message to you don't use Fritz or similar crap (which is known for not following UCI standard properly) for serious engine testing.

Which GUI or GUIs do you recommend?

[Milos]

Well, if you just posted pgns earlier it would be easier to detect. Even though your pgn's are mostly hopeless (no depth or nps info) judging by the time (and despite ridiculous TC), SF is using 20-30% more time than H4 (this is a drastic difference), or simply said H4 is never using its time which is obviously GUI problem.
So my message to you don't use Fritz or similar crap (which is known for not following UCI standard properly) for serious engine testing.

Which GUI or GUIs do you recommend?[/quote]
For LTC engine testing Arena, for STC cutechess-cli or Winboard (all free, cutechess-cli and Winboard also GNU).

[lkaufman]

I'm rerunning my test using Fritz 11, but so far the results are similar; SF leading by 31 elo after 74 games. So the GUI doesn't seem to be the issue.

I am not having any issues in my test with the gui, even when I tested at 1m+0s. But I am running a very clean setup, my CPU usage is 0 to 2% before testing. I have nothing running in the background other then normal windows system usage.

So what is going on... are the results legit in our test that show stockfish beating Houdini at these time controls. Are we missing something... I am stumped at this point. If somehow we are giving Stockfish a unfair advantage in our setup. You are testing Stockfish the standard way 4 cpu on a 4 core system. I know I am testing HT, but the results is the same. Meaning stockfish is winning by more then just noise factors. In you add up all the results. Or not?

Maybe someone can see a problem in the games I posted. I left all move and time data in the PGN. So it could be put straight into a Fritz GUI for evaluation.

SF is still +14 elo after 279 bullet games (4t each on quad) using Fritz, so it does appear that I am confirming your findings. However this particular machine is not so quiet, seems to have a lot of anti-virus stuff; I'll have to do something about that. But I don't know any reason why this would favor one engine over the other, it just adds a bit more randomness to the result I think. So it still appears to me that Houdini 4 is stronger on one core and SF Dec. 30 on four cores, regardless of hyperthreading, sleeping threads, etc. Milos observed that SF takes much more time than Houdini, but I think this is simplly algorithmic; SF is much more aggressive about time use than any other top engine, which does help its middlegame play but leaves it short of time for the endgame. Maybe this tendency is exaggerated in MP play for some reason.

[Milos]

So it still appears to me that Houdini 4 is stronger on one core and SF Dec. 30 on four cores, regardless of hyperthreading, sleeping threads, etc. Milos observed that SF takes much more time than Houdini, but I think this is simplly algorithmic; SF is much more aggressive about time use than any other top engine, which does help its middlegame play but leaves it short of time for the endgame. Maybe this tendency is exaggerated in MP play for some reason.

If you just go through pgns of Mark you will notice that despite weird TC (2'+12'') H4 TM is totally crazy i.e. it ends every game (winning or losing) with 5+ minutes on the clock. This is impossible to be normal behavior (algorithmic) and it doesn't behave like this at all (with the same TC) on my 6-core AMD machine and different GUI. I have neither Fritz nor Intel i7 to try so I can't debug the reason (even though I suspect it's GUI and if provided UCI logs, it would be easy to find where is the error), but something is obviously wrong.

[syzygy]

Code: Select all

Blitz, Blitz 2m+12s  0

                                      
1   Stockfish 020114 64 SSE4.2   +62  +22/=43/-9 58.78%   43.5/74
2   Houdini 4 Pro x64A           -62  +9/=43/-22 41.22%   30.5/74

Here are some of the games that are wins and losses for both programs in my current test, I can not find any issues for Houdini in this test. It is getting its typical NPS on my system and search depth. It just gets badly out searched by Stockfish. Let me know if you see any issues in the games. Tested with 8CPU HT Sleeping threads on for Stockfish. For Houdini defaults settings, 4CPU no HT. Contempt = 0.

I'll skip on going through all the moves, but if H4 is achieving its normal nps and search depth and there are no strange time losses, then things look OK. I suspect the 124 Elo gap to shrink with more games played, though

[mwyoung]

So it still appears to me that Houdini 4 is stronger on one core and SF Dec. 30 on four cores, regardless of hyperthreading, sleeping threads, etc. Milos observed that SF takes much more time than Houdini, but I think this is simplly algorithmic; SF is much more aggressive about time use than any other top engine, which does help its middlegame play but leaves it short of time for the endgame. Maybe this tendency is exaggerated in MP play for some reason.

If you just go through pgns of Mark you will notice that despite weird TC (2'+12'') H4 TM is totally crazy i.e. it ends every game (winning or losing) with 5+ minutes on the clock. This is impossible to be normal behavior (algorithmic) and it doesn't behave like this at all (with the same TC) on my 6-core AMD machine and different GUI. I have neither Fritz nor Intel i7 to try so I can't debug the reason (even though I suspect it's GUI and if provided UCI logs, it would be easy to find where is the error), but something is obviously wrong.

My PGN have all depth, time, and move info in them. All you have to do is put them into a fritz GUI.

I am not sure what your problem is, or why you don't understand even some very basic concepts.

For the record there is nothing wrong with the PGN or Houdini 4's time.

The time control is 2+12, Houdini 4 is using table base. So it is gaining lots of time at the end of the game because the table base moves instantly and that gains Houdini 12s every time it moves like this in a long Table Base endgame.

In a Fritz GUI if you have the clock window open as you play over the game, you can see this as it was played in my test games.

[mwyoung]

Code: Select all

Blitz, Blitz 2m+12s  0

                                      
1   Stockfish 020114 64 SSE4.2   +62  +22/=43/-9 58.78%   43.5/74
2   Houdini 4 Pro x64A           -62  +9/=43/-22 41.22%   30.5/74

Here are some of the games that are wins and losses for both programs in my current test, I can not find any issues for Houdini in this test. It is getting its typical NPS on my system and search depth. It just gets badly out searched by Stockfish. Let me know if you see any issues in the games. Tested with 8CPU HT Sleeping threads on for Stockfish. For Houdini defaults settings, 4CPU no HT. Contempt = 0.

I'll skip on going through all the moves, but if H4 is achieving its normal nps and search depth and there are no strange time losses, then things look OK. I suspect the 124 Elo gap to shrink with more games played, though

Thank you for looking at the games for me. That was my conclusion also.

For the record, the elo gap is not 124 elo. It is only +62 elo for Stockfish.

[mwyoung]

I'm rerunning my test using Fritz 11, but so far the results are similar; SF leading by 31 elo after 74 games. So the GUI doesn't seem to be the issue.

I am not having any issues in my test with the gui, even when I tested at 1m+0s. But I am running a very clean setup, my CPU usage is 0 to 2% before testing. I have nothing running in the background other then normal windows system usage.

So what is going on... are the results legit in our test that show stockfish beating Houdini at these time controls. Are we missing something... I am stumped at this point. If somehow we are giving Stockfish a unfair advantage in our setup. You are testing Stockfish the standard way 4 cpu on a 4 core system. I know I am testing HT, but the results is the same. Meaning stockfish is winning by more then just noise factors. In you add up all the results. Or not?


Always check milos claims before sighting milos. The reason houdini has time at the end of the game is very simple. I test to checkmate. Houdini is using table bases. The time control is 2m 12s. When houdini reaches a table base position it moves instantly gaining 12s every move. In a mate in 20. Or 30 that adds a lot of time at the end of the game.






Maybe someone can see a problem in the games I posted. I left all move and time data in the PGN. So it could be put straight into a Fritz GUI for evaluation.

SF is still +14 elo after 279 bullet games (4t each on quad) using Fritz, so it does appear that I am confirming your findings. However this particular machine is not so quiet, seems to have a lot of anti-virus stuff; I'll have to do something about that. But I don't know any reason why this would favor one engine over the other, it just adds a bit more randomness to the result I think. So it still appears to me that Houdini 4 is stronger on one core and SF Dec. 30 on four cores, regardless of hyperthreading, sleeping threads, etc. Milos observed that SF takes much more time than Houdini, but I think this is simplly algorithmic; SF is much more aggressive about time use than any other top engine, which does help its middlegame play but leaves it short of time for the endgame. Maybe this tendency is exaggerated in MP play for some reason.

[Dr.Wael Deeb]

So it still appears to me that Houdini 4 is stronger on one core and SF Dec. 30 on four cores, regardless of hyperthreading, sleeping threads, etc. Milos observed that SF takes much more time than Houdini, but I think this is simplly algorithmic; SF is much more aggressive about time use than any other top engine, which does help its middlegame play but leaves it short of time for the endgame. Maybe this tendency is exaggerated in MP play for some reason.

If you just go through pgns of Mark you will notice that despite weird TC (2'+12'') H4 TM is totally crazy i.e. it ends every game (winning or losing) with 5+ minutes on the clock. This is impossible to be normal behavior (algorithmic) and it doesn't behave like this at all (with the same TC) on my 6-core AMD machine and different GUI. I have neither Fritz nor Intel i7 to try so I can't debug the reason (even though I suspect it's GUI and if provided UCI logs, it would be easy to find where is the error), but something is obviously wrong.

My PGN have all depth, time, and move info in them. All you have to do is put them into a fritz GUI.

I am not sure what your problem is, or why you don't understand even some very basic concepts.

For the record there is nothing wrong with the PGN or Houdini 4's time.

The time control is 2+12, Houdini 4 is using table base. So it is gaining lots of time at the end of the game because the table base moves instantly and that gains Houdini 12s every time it moves like this in a long Table Base endgame.

In a Fritz GUI if you have the clock window open as you play over the game, you can see this as it was played in my test games.

A piece of advice Mark:

Manipulate one of your testing frameworks so that Houdini wins a direct match against the latest Stockfish version....

This time you'll suffer less attacks regards,
Dr.D

[mwyoung]

So it still appears to me that Houdini 4 is stronger on one core and SF Dec. 30 on four cores, regardless of hyperthreading, sleeping threads, etc. Milos observed that SF takes much more time than Houdini, but I think this is simplly algorithmic; SF is much more aggressive about time use than any other top engine, which does help its middlegame play but leaves it short of time for the endgame. Maybe this tendency is exaggerated in MP play for some reason.

If you just go through pgns of Mark you will notice that despite weird TC (2'+12'') H4 TM is totally crazy i.e. it ends every game (winning or losing) with 5+ minutes on the clock. This is impossible to be normal behavior (algorithmic) and it doesn't behave like this at all (with the same TC) on my 6-core AMD machine and different GUI. I have neither Fritz nor Intel i7 to try so I can't debug the reason (even though I suspect it's GUI and if provided UCI logs, it would be easy to find where is the error), but something is obviously wrong.

My PGN have all depth, time, and move info in them. All you have to do is put them into a fritz GUI.

I am not sure what your problem is, or why you don't understand even some very basic concepts.

For the record there is nothing wrong with the PGN or Houdini 4's time.

The time control is 2+12, Houdini 4 is using table base. So it is gaining lots of time at the end of the game because the table base moves instantly and that gains Houdini 12s every time it moves like this in a long Table Base endgame.

In a Fritz GUI if you have the clock window open as you play over the game, you can see this as it was played in my test games.

A piece of advice Mark:

Manipulate one of your testing frameworks so that Houdini wins a direct match against the latest Stockfish version....

This time you'll suffer less attacks regards,
Dr.D

I could have stockfish show total contempt for houdini. Contempt=999. It is nice to have my results confirmed here and on ccrl. As predicted from my testing , stockfish holds the top spot.