lkaufman wrote:
are the I5 machines and the AMD machines you mention fairly close in speed for typical chess engines?
Hi Larry,
yes they are!
There is no need for a time adjustment.
Werners i7 is somewhat faster but will be used rather seldomly as Werner will only help from time to time with some single matches. In this case both engines will benefit from his better hardware of course.
Thanks, I agree that 6 or 7% difference is negligible if it applies to both engines being tested. I do want to mention though that some recent tests have shown that the differences between engines on different hardware vary by more than I believed. Relative performance of one engine vs. another can vary as much as 20% depending on hardware! In particular, it seems that Komodo benefits more from running on the third (or fourth) generation I7 machines, while performing relatively poorly on AMD. I think this is the primary reason why our ratings on the rating lists often seemed to fall short of our expectations, which were based on Intel machines only. There are also big differences between Stockfish and Houdini in terms of which machines favor which engine. It is now clear that this hardware factor, rather than other issues like books, hash size, etc. is the main cause of disparate results among different testers.
JB
Komodo has finally caught or passed Houdini 3 at levels close to your new test level on my Intel machines. But I expect that if we released a new version now it would still end up some 10 to 20 elo behind Houdini on your lists just due to the use of AMD. This is not a complaint, I understand that many people use AMD and your use of both AMD and INTEL is quite fair. But on the best desktop models on sale now with i7, our results should be noticeably better.
But maybe (i do not know) others like Gull, Critter, Chrion ...
also benefits from a new i7 like Komodo-Beta does ?!
Best wishes,
G.S.
(CEGT team)
With Adam we have seen a similar phenomenon. That is, a big variability of Gaviota elo in different type of computers, that could easily be 10-20 elo (depending of the opponents, since some of them show this behavior too). Not to mention that the scalability of SMP is different comps is, even among AMDs! Gaviota scales much better in my quad AMD than in a hexa AMD (using 4 cores of course).
On a single core, my observation was that Komodo did better on AMD. But as usual the error margins mean that these observations were inconclusive. You have played huge numbers of games, so you will have a far better picture than me.
But maybe (i do not know) others like Gull, Critter, Chrion ...
also benefits from a new i7 like Komodo-Beta does ?!
Best wishes,
G.S.
(CEGT team)
I only test against close rivals, meaning Houdini and Stockfish, so I don't know either, although I would expect that all the Ippo - derived engines would be similar in this respect. We'll just have to show a double-digit lead over Houdini 3 on my computers for me to be confident in calling Komodo the number 1 engine. We're not there yet, but...
Modern Times wrote:On a single core, my observation was that Komodo did better on AMD. But as usual the error margins mean that these observations were inconclusive. You have played huge numbers of games, so you will have a far better picture than me.
It also depends on how new your Intel was/is. The older ones may not be much better than AMD for Komodo, but the ones from this year (i.e. "third generation" or beyond) are very Komodo-friendly.
lkaufman wrote:
I only test against close rivals, meaning Houdini and Stockfish, so I don't know either, although I would expect that all the Ippo - derived engines would be similar in this respect.
[...snip...]
Hi Larry !
Way too much games/matches against (nearly) the same opponents is not our way!
Therefore its good that we have a wide range of different engines within our lists!
There are some very strange lists around with curious time-controls like:
a few seconds for the whole game plus deciseconds for each move, against
IPPO/ROBBO/LITTO and all that s..t.
I'm good, thanks.
I agree that you should include many opponents, because you are comparing many different engines. Since we are only concerned with engines within about 50 elo of us (either way), it makes more sense for us to limit our opponents to these engines for our own testing, as that reduces the error bars. Regarding the strange other lists, testing at ultra fast levels like game in ten seconds or so is only valid when comparing different versions of the same engine; it is quite useless for comparing Stockfish to Houdini to Komodo for example, as too much depends on setup time at the start and other such factors. Testing at levels around game in one minute is mostly free of this bias, but as it doesn't reflect the way engines are actually used it is not too useful for most people.
Bottom line: with your new list I think you are doing things very well!
Larry
ThatsIt wrote:
lkaufman wrote:
I only test against close rivals, meaning Houdini and Stockfish, so I don't know either, although I would expect that all the Ippo - derived engines would be similar in this respect.
[...snip...]
Hi Larry !
Way too much games/matches against (nearly) the same opponents is not our way!
Therefore its good that we have a wide range of different engines within our lists!
There are some very strange lists around with curious time-controls like:
a few seconds for the whole game plus deciseconds for each move, against
IPPO/ROBBO/LITTO and all that s..t.
I'm good, thanks.
