hgm wrote:That would have been a noble cause, if your 'education' was based on actual knowledge you had and other lacked.
But in fact, what you are presenting here like it is gospel, is pure speculation on your behalf, based on nothing but prejudice.
And that statement is based on ignorance, so I guess at some level we must be "equal" here. Searching to a fixed number of nodes distorts the results. I have already explained why. Programs are not uniform in their NPS, so limiting them to a fixed number of nodes helps their search depth in some positoins because they are searching slower than normal and your extra nodes are like extra time. Same problem when the program goes fast, now it is penalized.
The first point here is that any changes that influence program speed will give invalid results, because fixed node searches eliminates the NPS from affecting the game at all. And many changes are all about speed.
The second point is that when playing an opponent whose search speed varies significantly over the different phases of the game, the program will seem to play stronger in positions where it is slower, and vice versa, because of the time bias fixed nodes introduces.
As one example, suppose you are playing against a program that has the 3x speed variance Ferret has (used to have when it was active). A change to your program that causes it to trade material a little more aggressively will appear to be a good change, because your winning rate goes up somewhat. But not because of your change, but because you are forcing Ferret to play in the part of the game where it is normally a lot faster, but now you are time-handicapping it.
I don't see why that is so hard to understand. And it is _not_ based on "prejudice". It is based on understanding how the tree search progresses and how a fixed number of nodes can affect the search of two different engines in two different ways, or perhaps not even affect one at all, which is even worse.
In reality you haven't the slightest idea if the Ferret that is playng by node count is actually stronger or weaker than the Ferret playing by time. Or if evaluating the Elo gain in engine X would produce a different value when evaluating that same change in a gauntlet against a number of engines playing by time as it would when evaluating it in a gauntlet aganst the same bunch playing by nodes. You have not tested any of those things.
Must you continue to post such absolute rubbish? Do you _really_ think that if I slow a program down by a factor of 2 that it will _not_ play weaker? If so, you need to find another hobby. You are so far out in left field here, you are completely out of the stadium and out in the parking lot.
For the record, I have run tests like this. And I noticed that the results were not matching the fixed-time-limit games very accurately. And I studied the problem to figure out why. Have _you_ ever done that? I have. And this is not guesswork.
Show us fact, and we might believe you. (If such fact passes usual scientific scruteny, that is...) But engine authors are in fact very ill advised to listen to your gut reaction "this is not the way I have been doing for the past 49 years, so it must be wrong". The latter is just wrecking a fruitful discussion by interjecting dis-info.
Notice I did not say "this is the way I have been doing it for the last 49 years." I simply said that this is not the best way to test and gave a concrete (and easily understandable) reason for anyone interested enough to think about it)
I do not believe it is reasonable to test in an environment where I can make changes to my program that let it exploit a weakness in the test methodology. Full-width searches are amazing in the way they can steer the game toward favorable situations when you are looking at wins vs losses.
If you play against an opponent that speeds up in the endgame, then playing with fixed node searches will make any change that reaches an endgame position more quickly look like it is good. Not because the program is playing better, but because it is suddenly searching "much faster" virtually. Because it has steered the game ttoward positions where the opponent slows down significantly.
By the same token, if you play against an opponent that slows down in the endgame (Crafty is an example here) then any changes that avoid trading will look good because you now force Crafty to search smaller trees than it normally would, giving your program a time advantage.
neither of those cases have anything at all to do with making your program play better, just making it play toward positions where fixed node searches are more favorable to you than your opponent.
Let's take the first case. You tune your eval to the programs that speed up in the endgame and think that because of your unknowing speed advantage, you are getting better. And then in real games, you steer the games toward the endgame where suddenly you discover that your opponents know more about pawns and such than you do and without that time handicap, you lose more than you did before.
Again, you need to test like you intend to play. Since we don't have fixed depth or fixed node count tournaments, it is simply more efficient to test and tune like you are going to really play.
No prejudice. No "dis-info". Just logical reasoning.
