Houdini 3

[Lion]

So on a 16 cores, at 3min blitz you may already be stronger ?

[diep]

Basically I would like to know if Komodo (and Stockfish) are weak at bullet chess because we are doing something wrong, or whether we are stronger at slower chess because Ippo/Ivanhoe is doing something wrong (or both). Comments anyone?

komodo and stockfish are intelligent evaluation programs.
ippo/ivanhoe or rybka etc. are stupid search programs.

the intelligent programs are weak at bullet, and better at slow chess.

Look i don't want to get into the discussion too much here, but obviously all the names you quote here are total beancounters. Rybka has 2 things more in evaluation than ippo. That's a peanuts difference.

It's of course search, compiler and implementation and way how things get timed in this case which as these (super)bullet time controls determines the difference in case engines have such similar evaluation functions.

are you talking with larry or with me vincent.

i see a major difference in evaluation-behaviour between stockfish /komodo in opposite to the rybka-clone-programs .

what do you mean with bean counter ?? that komodo / stockfish
are bean counters ? or do you talk about the rybka-clones ?

Stockfish is quite different indeed, yet it has an ultra simple evaluation function. You can have a look in the source yourself.

This is the biggest superbeancounter of everything. Even combining hard razoring with futility seems to work for Stockfish, that's how simple its evaluation is. It's well tuned - sure, also it's more agressive tuned.

Yet please don't write as if this is a huge evaluation function, it's ultra tiny.

The differences between all the ippo's/komodo's/rybka's/houdini's/fire engines is basically tuning. The evaluation functions are very very similar. In the end it all is derived from rybka, which in turn again stole a few tables from Fruit.

The difference between ippo & clones versus rybka 3.0 for example seems to be rybka getting some sort of bonus for 2 connected passers which ippo doesn't. Now i didn't disassemble rybka so i'm not 100% sure how this pattern is inside rybka, yet in games i saw the difference clearly there.

This influenced back then lots of games, sure. Especially if difference is tiny, then a single pattern makes a HUGE difference. Yet from source code viewpoint seen it's nearly the same engines. Evaluation function again is really tiny; sure some huge tables get used and they're automatic tuned and/or precalculated using automatic tuned values, but still evaluation function is supertiny.

Just razoring last few plies doesn't work of course if your evaluation function is a tad bigger and say knows a tad more about mobility. You see how razoring doesn't work for a whole range of engines.

Yet realize this gives a HUGE SEARCH DEPTH WIN for all those engines.

The first guy in this century who revived it was Omid David Tabibi, around 2000-start 2001 it was he described to me his razoring he did do with Falcon.

This can only work for beancounters of course, which just count material and have PSQ tables - as then you simply don't have the system time to be more sophisticated, to quote Frans Morsch.

[Werewolf]

Which engine has the biggest / most sophisticated evaluation function?

Is there any advantage to having a big one?

[mclane]

i would say Chess System Tal has it.

[Sedat Canbaz]

Stockfish is quite different indeed, yet it has an ultra simple evaluation function. You can have a look in the source yourself.

This is the biggest superbeancounter of everything. Even combining hard razoring with futility seems to work for Stockfish, that's how simple its evaluation is. It's well tuned - sure, also it's more agressive tuned.

Yet please don't write as if this is a huge evaluation function, it's ultra tiny.

The differences between all the ippo's/komodo's/rybka's/houdini's/fire engines is basically tuning. The evaluation functions are very very similar. In the end it all is derived from rybka, which in turn again stole a few tables from Fruit.

The difference between ippo & clones versus rybka 3.0 for example seems to be rybka getting some sort of bonus for 2 connected passers which ippo doesn't. Now i didn't disassemble rybka so i'm not 100% sure how this pattern is inside rybka, yet in games i saw the difference clearly there.

This influenced back then lots of games, sure. Especially if difference is tiny, then a single pattern makes a HUGE difference. Yet from source code viewpoint seen it's nearly the same engines. Evaluation function again is really tiny; sure some huge tables get used and they're automatic tuned and/or precalculated using automatic tuned values, but still evaluation function is supertiny.

Just razoring last few plies doesn't work of course if your evaluation function is a tad bigger and say knows a tad more about mobility. You see how razoring doesn't work for a whole range of engines.

Yet realize this gives a HUGE SEARCH DEPTH WIN for all those engines.

The first guy in this century who revived it was Omid David Tabibi, around 2000-start 2001 it was he described to me his razoring he did do with Falcon.

This can only work for beancounters of course, which just count material and have PSQ tables - as then you simply don't have the system time to be more sophisticated, to quote Frans Morsch.

Agreed...

Just i'd like to add:one of the biggest mistakes is that we noticed Rybka is disqualified and banned

This is a really big injustice over Vasik Rajlich

For example,if i follow the rules of ICGA/WCCC,then i should cancel and stop organizing SCCT Book Tournaments
And its not so hard to find the answer:simply,because all opening books are made automatically by importing (coping) games

Can anybody inform me please:
-Which opening book is made 100 % by own ideas ?!

Btw,since 2012,SCCT's new rules:
•Some engines which are debated to be close derivatives of others are allowed for testing
*Due to mostly of the engines are each other relative (its hard to find 100 % original work)


Best Regards,
Sedat

[geots]

[quote="mclane & Stockfish.

Maybe my explanation is even more simple but true: Rybka, Ippo & Co are simply better. The weaker engines just benefit from the bigger draw rates at big depths/long time controls.

This might be a factor for Houdini, but certainly Ivanhoe (Ippo), Rybka, and all other Ippo-related programs are not stronger than Komodo and not measurably stronger than SF except at bullet chess, so this cannot be the explanation here.

There seem to be two theories to explain the observed scaling behavior:
1. Komodo (and perhaps also SF) are more intelligent but slower, and that this tradeoff usually (but not always) favors the fast programs in blitz and the smart programs at long time controls.
2. For whatever reason, the search in Komodo (and perhaps SF) scales better than the search in the Rybka/Ippo family.

Both could be true. If the second is true, can anyone suggest WHY SF might scale better in search than Ivanhoe et al?

I don't think ''intelligence'' is the right word. There are two different approaches to the search in my view.

Ippo family engines have a search for which they find tactical moves in a very short time.
I think it is a question of search (and evaluation). Like humans. Some humans search mainly for tactical moves.
I'm not a programmer but i think it depends by the evaluation that an engine gives to each move that it analyses.
In this way it could happen that all the moves that don't give a ''break'' in the evaluation are discarded or analysed less time (and perhaps they are good positional moves).
If this would be true, it is logical that these engines find more often and more quickly tactical moves.

On the other hand Komodo plays mainly positionally.
Komodo often manouevres his pieces and pawns for many moves without playing any tactical move.

Probably it depends by the fact that Komodo gives a special evaluation on some moves that lead to some positional patterns.

I mean, probably during the search, Komodo seems to make the opposite of the engines of the ippo family.

Then Komodo analyzes more the ''positional moves'' (probably because it consider them better than other moves with its evaluation) and gives less time or discards some tactical moves.

In fact, sometimes Komodo overlooks some tactical moves, and it depends by the search or sometimes by the evaluation.

If all i wrote would be true, it is logical that in very fast time control, ippo family engines excel.

With more time to think, a positional engine like Komodo has 2 advantages:

1. What Komodo overlooks or evaluates bad in a few seconds can be seen well with more time available.

2.The weight of positional play of Komodo increases a lot and becomes more important than its (relative) weakness in tactics.

Generally speaking i think that the strenght of Komodo is that it plays nice positional and logical moves. It seems that it uses the search just to verify that everything is ok.

On the other hand the strenght of ippo family engines is the approach of the search (i think built mainly for tactical moves) and the evaluation that allows them to find sometimes tactical moves apparently hidden.

In long time control, especially in very long time control, tactics ability is almost useless because the opponent (if he has a good search and evaluation) has all the time to see it.

So the thing that has more importance is the positional/strategical sensibility.


Best Regards

I am coming to the conclusion that the relative weakness of both Komodo and Stockfish at bullet speeds vs. slower tests (which is obvious if you compare any bullet list with say the IPON list) is not mainly a question of slowness due to more or slower evaluation. I say this because I noticed that Rybka 4.1 shares the same behavior as Ivanhoe and all the other top programs (except Komodo and Stockfish) in that it is also relatively stronger at bullet speeds than at normal speeds. If we make the assumption that Rybka 4.1 is at least not substantially "dumber" than Rybka 3, I can say that the evaluation I did for Rybka 3 was also rather "smart" and "slow", rather like Komodo, yet Rybka 4.1 shares the behavior of Ippo and family. So I can only conclude that the behavior is due to some aspect of the Ippo and Rybka search that is not present in either Komodo or Stockfish. Since we have at least tried almost every search idea in Ippo (and rejected many of them), I can't guess what that aspect could be. If we can solve this mystery perhaps we can make Komodo as competitive with Houdini at bullet and blitz chess as it already appears to be at longer time controls.

Interesting that you mention the Ippos, Rybka, Houdini- and, I would have to imagine you include Critter. The reason I imagine you use Critter is because you have no choice, really. What amazes me about this pile of engines you have gathered- you don't include Strelka. You can make a great argument for leaving it out- but the argument becomes futile and silly when you leave all the others in. You want to have it both ways. Sorry. Strelka is not MP- but it uses 32bit and 64bit. You use Komodo with either- which noting its lack of strength in 32-bit would have to be 64. We shall both monitor a 50 game match with Komodo on 1cpu vs. Strelka 5.1. You pick the time limit of your choice. In a 50 game match you can start with white- and I will give you the gift of a win in game 1 with no moves. It won't amount to a dot in the universe. Any time control of your choice- I am sorry- Strelka 5.1 will go thru Komodo like Sherman thru Georgia. But that is not being cruel- no other engines can play with him either- outside of Houdini and Critter. And the last match, Critter got his butt kicked. Look, when I say any time limits of your choice- that is up to and thru 40/40. No slower- simply because it is a waste of my time. As it will change nothing, and at 63 time becomes a valuable commodity.


Best regards Larry- it's certainly nothing personal,

george


Please no bars and graphs and formuli that are supposed to show where one program intersects with another. There has already been too much talk and not enough "walk". Let's get the match on.

If you are talking about Komodo 4, you are probably right that Strelka 5.1 would be favored in a match at 40/40, since I understand it to be just a slightly weaker relative of Houdini 1.5. But although I have never tested Strelka, I would be willing to make a substantial wager that the current Komodo would win a hundred game match at that time control on an i7 or similar 64 bit SSE4 machine, if we can find a neutral trustworthy party to hold the stakes and run the match. I say this based on the assumption that Strelka 5.1 is at least marginally weaker than Houdini (1.5 or 2.0, I don't care); if this is not true please enlighten me. I realize that this offer is a bit unfair to you, as you don't have any Komodo newer than 4.0.

Larry, a 50 game match has been run. Too much junk scattered about on my new box to run 64bit this minute- so I turned to one of the 2 best testers in existence because I wanted no mistakes. You could bring up the margin of error theory, as I am sure you will. I would counter that more than likely Komodo got lucky with a few lines Strelka may not play well.

Misha Popovich ran a 50 game match starting last night, with Strelka 5.1 64 bit ag. Komodo 4 64bit SSE. The time control was 15 min. per game. Strelka won the match 14 - 8, with 28 draws. I contend Komodo was damn lucky to stay that close. I don't care to discuss it any further here. Don has my email address. Email me and I will send you the crosstable and the PGNs. But, please do the same to Robert Flesher and get the pgns from the suppposed 100 game (15' per game) match that he ran with Komodo SSE 64bit v Strelka 5.1 64bit in which, as he said, Komodo "smoked" Strelka. I contend it never happened. I will trade you these pgns for the ones from Flesher's match. I am waiting to hear from you.


george

[Robert Flesher]

[quote="mclane & Stockfish.

Maybe my explanation is even more simple but true: Rybka, Ippo & Co are simply better. The weaker engines just benefit from the bigger draw rates at big depths/long time controls.

This might be a factor for Houdini, but certainly Ivanhoe (Ippo), Rybka, and all other Ippo-related programs are not stronger than Komodo and not measurably stronger than SF except at bullet chess, so this cannot be the explanation here.

There seem to be two theories to explain the observed scaling behavior:
1. Komodo (and perhaps also SF) are more intelligent but slower, and that this tradeoff usually (but not always) favors the fast programs in blitz and the smart programs at long time controls.
2. For whatever reason, the search in Komodo (and perhaps SF) scales better than the search in the Rybka/Ippo family.

Both could be true. If the second is true, can anyone suggest WHY SF might scale better in search than Ivanhoe et al?

I don't think ''intelligence'' is the right word. There are two different approaches to the search in my view.

Ippo family engines have a search for which they find tactical moves in a very short time.
I think it is a question of search (and evaluation). Like humans. Some humans search mainly for tactical moves.
I'm not a programmer but i think it depends by the evaluation that an engine gives to each move that it analyses.
In this way it could happen that all the moves that don't give a ''break'' in the evaluation are discarded or analysed less time (and perhaps they are good positional moves).
If this would be true, it is logical that these engines find more often and more quickly tactical moves.

On the other hand Komodo plays mainly positionally.
Komodo often manouevres his pieces and pawns for many moves without playing any tactical move.

Probably it depends by the fact that Komodo gives a special evaluation on some moves that lead to some positional patterns.

I mean, probably during the search, Komodo seems to make the opposite of the engines of the ippo family.

Then Komodo analyzes more the ''positional moves'' (probably because it consider them better than other moves with its evaluation) and gives less time or discards some tactical moves.

In fact, sometimes Komodo overlooks some tactical moves, and it depends by the search or sometimes by the evaluation.

If all i wrote would be true, it is logical that in very fast time control, ippo family engines excel.

With more time to think, a positional engine like Komodo has 2 advantages:

1. What Komodo overlooks or evaluates bad in a few seconds can be seen well with more time available.

2.The weight of positional play of Komodo increases a lot and becomes more important than its (relative) weakness in tactics.

Generally speaking i think that the strenght of Komodo is that it plays nice positional and logical moves. It seems that it uses the search just to verify that everything is ok.

On the other hand the strenght of ippo family engines is the approach of the search (i think built mainly for tactical moves) and the evaluation that allows them to find sometimes tactical moves apparently hidden.

In long time control, especially in very long time control, tactics ability is almost useless because the opponent (if he has a good search and evaluation) has all the time to see it.

So the thing that has more importance is the positional/strategical sensibility.


Best Regards

I am coming to the conclusion that the relative weakness of both Komodo and Stockfish at bullet speeds vs. slower tests (which is obvious if you compare any bullet list with say the IPON list) is not mainly a question of slowness due to more or slower evaluation. I say this because I noticed that Rybka 4.1 shares the same behavior as Ivanhoe and all the other top programs (except Komodo and Stockfish) in that it is also relatively stronger at bullet speeds than at normal speeds. If we make the assumption that Rybka 4.1 is at least not substantially "dumber" than Rybka 3, I can say that the evaluation I did for Rybka 3 was also rather "smart" and "slow", rather like Komodo, yet Rybka 4.1 shares the behavior of Ippo and family. So I can only conclude that the behavior is due to some aspect of the Ippo and Rybka search that is not present in either Komodo or Stockfish. Since we have at least tried almost every search idea in Ippo (and rejected many of them), I can't guess what that aspect could be. If we can solve this mystery perhaps we can make Komodo as competitive with Houdini at bullet and blitz chess as it already appears to be at longer time controls.

Interesting that you mention the Ippos, Rybka, Houdini- and, I would have to imagine you include Critter. The reason I imagine you use Critter is because you have no choice, really. What amazes me about this pile of engines you have gathered- you don't include Strelka. You can make a great argument for leaving it out- but the argument becomes futile and silly when you leave all the others in. You want to have it both ways. Sorry. Strelka is not MP- but it uses 32bit and 64bit. You use Komodo with either- which noting its lack of strength in 32-bit would have to be 64. We shall both monitor a 50 game match with Komodo on 1cpu vs. Strelka 5.1. You pick the time limit of your choice. In a 50 game match you can start with white- and I will give you the gift of a win in game 1 with no moves. It won't amount to a dot in the universe. Any time control of your choice- I am sorry- Strelka 5.1 will go thru Komodo like Sherman thru Georgia. But that is not being cruel- no other engines can play with him either- outside of Houdini and Critter. And the last match, Critter got his butt kicked. Look, when I say any time limits of your choice- that is up to and thru 40/40. No slower- simply because it is a waste of my time. As it will change nothing, and at 63 time becomes a valuable commodity.


Best regards Larry- it's certainly nothing personal,

george


Please no bars and graphs and formuli that are supposed to show where one program intersects with another. There has already been too much talk and not enough "walk". Let's get the match on.

If you are talking about Komodo 4, you are probably right that Strelka 5.1 would be favored in a match at 40/40, since I understand it to be just a slightly weaker relative of Houdini 1.5. But although I have never tested Strelka, I would be willing to make a substantial wager that the current Komodo would win a hundred game match at that time control on an i7 or similar 64 bit SSE4 machine, if we can find a neutral trustworthy party to hold the stakes and run the match. I say this based on the assumption that Strelka 5.1 is at least marginally weaker than Houdini (1.5 or 2.0, I don't care); if this is not true please enlighten me. I realize that this offer is a bit unfair to you, as you don't have any Komodo newer than 4.0.

Larry, a 50 game match has been run. Too much junk scattered about on my new box to run 64bit this minute- so I turned to one of the 2 best testers in existence because I wanted no mistakes. You could bring up the margin of error theory, as I am sure you will. I would counter that more than likely Komodo got lucky with a few lines Strelka may not play well.

Misha Popovich ran a 50 game match starting last night, with Strelka 5.1 64 bit ag. Komodo 4 64bit SSE. The time control was 15 min. per game. Strelka won the match 14 - 8, with 28 draws. I contend Komodo was damn lucky to stay that close. I don't care to discuss it any further here. Don has my email address. Email me and I will send you the crosstable and the PGNs. But, please do the same to Robert Flesher and get the pgns from the suppposed 100 game (15' per game) match that he ran with Komodo SSE 64bit v Strelka 5.1 64bit in which, as he said, Komodo "smoked" Strelka. I contend it never happened. I will trade you these pgns for the ones from Flesher's match. I am waiting to hear from you.


george

I guess when I questioned you regarding your illogical statements of me beta testing Vitruvius and it having nothing to do with the previous conversation, it really stung you. It seems when anyone tries to question or debate anything with you, you become harsh and attack the charactyer of the person. Gerorge, I can tell you this, you don't know me. So why you you question if I played a match? This is asinine behaviour beyond belief. Disappointed, I am!

[lkaufman]

[
Larry, a 50 game match has been run. Too much junk scattered about on my new box to run 64bit this minute- so I turned to one of the 2 best testers in existence because I wanted no mistakes. You could bring up the margin of error theory, as I am sure you will. I would counter that more than likely Komodo got lucky with a few lines Strelka may not play well.

Misha Popovich ran a 50 game match starting last night, with Strelka 5.1 64 bit ag. Komodo 4 64bit SSE. The time control was 15 min. per game. Strelka won the match 14 - 8, with 28 draws. I contend Komodo was damn lucky to stay that close. I don't care to discuss it any further here. Don has my email address. Email me and I will send you the crosstable and the PGNs. But, please do the same to Robert Flesher and get the pgns from the suppposed 100 game (15' per game) match that he ran with Komodo SSE 64bit v Strelka 5.1 64bit in which, as he said, Komodo "smoked" Strelka. I contend it never happened. I will trade you these pgns for the ones from Flesher's match. I am waiting to hear from you.

george

I have no reason to disbelieve either the result of Mr. Popovich or of Mr. Flesher. If the two engines are about equally matched under these conditions then it is perfectly normal for one to win by a few games in one match and the other to win by a few games in the other. But anyway I don't claim that Komodo 4 is an even match for either Houdini or a Houdini near-clone (if that is what Strelka is) at game/15'. For Komodo it makes all the difference to have a decent increment, let's say at least ten seconds. I think you will find that, whatever the results at straight game/15', the result with a ten second increment added will be much better for Komodo than without the increment. I don't yet understand why this is so, but it seems that as long as both engines have at least ten seconds for each move, the big advantage of Houdini/Strelka fades away.

[Robert Flesher]

[
Larry, a 50 game match has been run. Too much junk scattered about on my new box to run 64bit this minute- so I turned to one of the 2 best testers in existence because I wanted no mistakes. You could bring up the margin of error theory, as I am sure you will. I would counter that more than likely Komodo got lucky with a few lines Strelka may not play well.

Misha Popovich ran a 50 game match starting last night, with Strelka 5.1 64 bit ag. Komodo 4 64bit SSE. The time control was 15 min. per game. Strelka won the match 14 - 8, with 28 draws. I contend Komodo was damn lucky to stay that close. I don't care to discuss it any further here. Don has my email address. Email me and I will send you the crosstable and the PGNs. But, please do the same to Robert Flesher and get the pgns from the suppposed 100 game (15' per game) match that he ran with Komodo SSE 64bit v Strelka 5.1 64bit in which, as he said, Komodo "smoked" Strelka. I contend it never happened. I will trade you these pgns for the ones from Flesher's match. I am waiting to hear from you.

george

I have no reason to disbelieve either the result of Mr. Popovich or of Mr. Flesher. If the two engines are about equally matched under these conditions then it is perfectly normal for one to win by a few games in one match and the other to win by a few games in the other. But anyway I don't claim that Komodo 4 is an even match for either Houdini or a Houdini near-clone (if that is what Strelka is) at game/15'. For Komodo it makes all the difference to have a decent increment, let's say at least ten seconds. I think you will find that, whatever the results at straight game/15', the result with a ten second increment added will be much better for Komodo than without the increment. I don't yet understand why this is so, but it seems that as long as both engines have at least ten seconds for each move, the big advantage of Houdini/Strelka fades away.

I have not tried the incremental time setting with any engine matches. This is will be next on my list when I run a Komodo match. It will be interesting to determine why this makes a difference.

Regarding the different results of testers, results will vary when engines are close in strength and only balance out in favour of one engine after many hundreds if not thousands of games. Of course you and Don know this more than most.

In my test of only 100 games I think the margin for error is something like 60 elo. ( I read this somewhere, that is why I use it). Rendering the result almost pointless. Unless you play over every game for the chess. (this is why I play these matches). The reason I pointed my results out to you was to attempt to goad you into a match with the new Komodo. Apparently George missed this subtlety, even though I explained it to him in the very next post.

[M ANSARI]

I this entire debate about scaling of Komodo vs. Houdini is pointless, rather than handicap all engines to play on one core so that they can compete with Komodo, it makes a lot of sense to make Komodo MP to compete with other engines. Time is a constant in computers, and this constant gets exponentially improved with each new generation of hardware. But I will be first to admit that some engines perform dramatically better at very short time controls than others, and yes the Ippolit derivatives are extremely strong at very fast blitz times. I don't know why that is so, but I always had the belief that in the case of Rybka, it has a lot of overhead in initializing start of processes (as it is a process based engine). These overheads are in mili seconds, but they can quickly become prominent when time runs down and games are very fast 1 0 or 2 0.