Gian-Carlo Pascutto wrote:
You can achieve the same by making the program twice faster in NPS without any difference to pruning (and incidentally, RobboLito is much faster than most of those programs), so your conclusion doesn't follow from the data at all.
Twice faster is impossible IMHO. I have spent long hours at profiler to squeeze even the last drop out of SF and I know the pure speed increase we had through the versions starting from 1.3 is on average much smaller and from 1.4 to 1.5.1 when I've done a super optimization hard work I was able to get perhaps up to 8-10% of more speed, no more.
Today, if someone could squeeze another 5% out of SF (pure speed, no other tricks) is a profiling genius
I think that is a mistake to optimize for speed.
If somebody optimize the program for speed it may be harder to improve it by other ways without bugs.
I think that it is better to care more about evaluation and search improvements.
Even if somebody can work hard and get 10% speed improvement from stockfish then it is better to get more elo at the same time by other things.
Uri Blass wrote:
I think that is a mistake to optimize for speed.
If somebody optimize the program for speed it may be harder to improve it by other ways without bugs.
I think that it is better to care more about evaluation and search improvements.
Even if somebody can work hard and get 10% speed improvement from stockfish then it is better to get more elo at the same time by other things.
Uri
These are 2 different things. It is important for me to know that there are no unnecessarily slow paths inside SF, for me has been very interesting to profile and optimize for speed.
Regarding your statement: "If somebody optimize the program for speed it may be harder to improve it by other ways"
Here the keyword is "maybe"
I think we have optimized SF in a way to preserve code readability and I can even add that, although _could_ seem counter intuitive, the best optimizations were also code cleanups. We have avoided hacks and ugly tricks because also if they seem an optimization "on the book", when you look at the light of a profiler (as you should always do if you want to seriously optimize and not hand waving optimize as is very common especially by the beginner) you see that the added benefit is always near to zero. Instead code cleanup that reduce clutter and add simplicity turns out to produce also a faster code.
Gian-Carlo Pascutto wrote:
You can achieve the same by making the program twice faster in NPS without any difference to pruning (and incidentally, RobboLito is much faster than most of those programs), so your conclusion doesn't follow from the data at all.
Twice faster is impossible IMHO. I have spent long hours at profiler to squeeze even the last drop out of SF and I know the pure speed increase we had through the versions starting from 1.3 is on average much smaller and from 1.4 to 1.5.1 when I've done a super optimization hard work I was able to get perhaps up to 8-10% of more speed, no more.
Today, if someone could squeeze another 5% out of SF (pure speed, no other tricks) is a profiling genius
"
I agree, it is impossible to imagine that Robbolito is more than twice as fast as Stockfish in "pure speed", yet it runs thru the plies as if it were. So I guess you agree it must be pruning much more than Stockfish, even though Stockfish does prune quite heavily already. Yet there is no sign of Robbolito paying any price for this pruning, based on fixed-depth matchups. So what in your opinion is the reason for this huge discrepancy? If you could cure it Stockfish would catch Robbolito easily. I talk about Robbolito and not Rybka because Robbolito is open-source so you are not just guessing.
lkaufman wrote:because those last three plies were pruned much more heavily than in other programs prior to Rybka 1.
I complained about this before, and I'll do so again: you have no way of knowing how Shredder, Junior, Fritz, Hiarcs and Sjeng work unless you've broken into our offices, so please don't claim you do, or I'll call the cops
Even now, it is apparent that Rybka and her derivatives must be pruning more than other programs, because Robbolito (which reports proper depths) outsearches all unrelated programs by a ply or more in general.
You can achieve the same by making the program twice faster in NPS without any difference to pruning (and incidentally, RobboLito is much faster than most of those programs), so your conclusion doesn't follow from the data at all.
"
I can tell a lot about programs from simple tactical tests. I know nothing of Sjeng and not too much about some of the others, but Fritz at least seems to be a pretty traditional program (with modern enhancements). Anyway I based my conclusion on the assumption that it is not possible for another program to be radically "faster" in NPS (measured in some ideal way) than Fritz or Stockfish. Moreover Fritz 12 reports considerably higher NPS than Robbolito. So do you believe that Robbolito is understating NPS in some sense and really does twice as many as say Fritz 12, or do you agree with me that more and better pruning must be the explanation of Robbolito's large rating lead over these programs? Or do you have some other theory?
lkaufman wrote:
Yet there is no sign of Robbolito paying any price for this pruning
Pruning is just one term in the equation that balances pruning and accuracy, the other term is depth. A lot of people don't have clear that you can prune a lot, really a lot, but if you do this in a progressive way (as all the engines do), pruning more at swallow depths and near the leafs, you could end up with an engine that at the same depth prunes more then another, but because it searches say 2 plies deeper on average at the end it is like to prune less because the same position will be reached by the faster engine say at depth 3, while the slower engine reaches the same position at depth 1 or even in qsearch so that the same position is evaluated with more accuracy by the engine that prunes more !
Regarding Robbo, I don't think there is a silver bullet, but it is a sum of different components that gives a big pruning preserving accuracy....well...we have some more detailed ideas about this actually, but I won't tell you if you don't convince your buddy to open source Komodo
Don is hoping to work full-time with Komodo so we plan to start charging at least for a multiprocessor version once it's ready, so we can't very well go open-source. However our eval is fully configurable by the parameter file, which are seeded with our actual values just prior to the name change, so no big secrets there, though some details are original. In general I can say that our eval is closer to Rybka than to other programs while our search has more in common with Stockfish. Probably we would be much stronger if it were the other way around, but the eval must be more like Rybka because I worked on both Rybka and Komodo eval. Anyway I'm willing to be reasonably open about what we do except for keeping secret any ideas we believe to be original.
Regarding Robbolito, I can say it acts as if it were pretty much like other programs but running something like three times as fast. Fixed depth testing makes it seem to be no more selective than other top programs, roughly speaking, but that is surely not so, as we agree this is impossible. So I guess my question is this: if you know how Robbolito can achieve this speed at no apparent cost, why can't you duplicate this in Stockfish?
Yes I know you are willing to go commercial, I was joking
Regarding Robbo I can say that its evaluation is lighter then SF, and evaluation weights a lot !
Just as an example if we could halve the weight of evaluation, i.e. make it to go at double speed I think we could gain almost a ply, perhaps even a bit more, keeping the same search.
That is not the case with Komodo; our eval is rather small and cheap, though pretty good and improving daily. The big speedup from Rybka to Robbolito was indeed achieved partly (or mostly) by simplifying the eval, leaving out terms whose cost roughly offset their benefit. So if you used only very cheap, simple eval in stockfish do you think you could come close to Robo-speed? If so you would then presumably only lose to Robo because you need riskier pruning to achieve that speed, is that right? So in that case I ask: why couldn't you achieve comparable speed to Robo if you used simple eval without lower search quality; is it because A:you don't know how they do it? or B: You know how but for technical reasons can't implement it in Stockfish or C: some other reason? In our case the answer is A.
lkaufman wrote:That is not the case with Komodo; our eval is rather small and cheap, though pretty good and improving daily. The big speedup from Rybka to Robbolito was indeed achieved partly (or mostly) by simplifying the eval, leaving out terms whose cost roughly offset their benefit. So if you used only very cheap, simple eval in stockfish do you think you could come close to Robo-speed? If so you would then presumably only lose to Robo because you need riskier pruning to achieve that speed, is that right? So in that case I ask: why couldn't you achieve comparable speed to Robo if you used simple eval without lower search quality; is it because A:you don't know how they do it? or B: You know how but for technical reasons can't implement it in Stockfish or C: some other reason? In our case the answer is A.
If you are really interested in speed, you should spend a few minutes downloading and playing around with some weak(comparatively) but fast programs. For example, if I remember right, Romichess and Olithink are both efficient programs with light evals. If you download them and play around with them a litle it will give you a feel for what is realistic (and what is not) in terms of NPS. I don't think Stockfish is designed with speed in mind, so its probably not as informative to experiment with.
I don't believe the issue is raw speed. Actually Komodo is pretty fast going by NPS (if these figures are really comparable among engines); only Fritz comes to mind as clearly faster. The issue is how Robbolito manages to reach higher search depths than other engines while still remaining competitive at the same depth of search. How do they have their cake and eat it too?
