IWB wrote:Don wrote:Ingo,
The question I have is which compile do you actually test and do you allow a program author to supply you with a version specialized to run well on your hardware?
I have been told by testers before that they only allow a "release" binary for their tests - otherwise I would submit a special version for each tester (after finding out what they run on) and even get them to help me determine which binary is fastest for their test. The problem with that is that you add to the confusion - every tester is actually running a different compile. It would give you an advantage that most programs are not taking advantage of.
Larry and I once discussed getting an AMD and releasing a different version for each hardware setup, for example an AMD version, a Sandy Bridge compile, a "Haswell" compile and so on.
If we did that would you test the appropriate one? I think for some testers the answer is no. How do you stand?
It is also possible to make a universal binary similar to what you can do with Mac, but it's a lot more complicated. I started doing that with the automatic SSE detection because some testers were using the wrong binary saying that they would only run the lowest common denominator and they didn't want to have to pick and choose which to test but wanted a single official binary. Where do you stand on this?
I understand the latest Intel processors may have some advanced instructions that could benefit chess if the compilers learn to take advantage of them or if the authors are willing to write assembly code. I don't know the exact details but we get into a situation where a chess program cannot take advantage of the latest hardware without providing multiple compiles.
Don
I see your point and can only answer that the "compile" has to be somehow official and not to specialized on HW.
Right now I use a compile from the site were the Stockfish team took its compile as well. I just used a different one which is not
penelaizing my hardware. If I would have run with the completly "official" releases I would have to run with the 32bit compile which is 20 to 25% slower (guess what would happen i nthe fora!? Actually I and every other rating list should have done it, I bet the next compiles of th eStockfish eam would be choosen differently!). On the other hand the compile I used is identicaly fast on any new intel as the "official" compile. So actually I did no harm to anyone.
As I refused to run a test with a 25% slower compile I would refuse a "special" compile which runs 25% faster on my hardware than on anyone elses. It simply make no sence. (Of course I would use that faster compile for private purposes)
It comes down to some common sence. As long as the difference is not too big it is fine. I could have finished with the normal x64 compile as my setup would not be able to measure that 3% difference.
Bye
Ingo
It wouldn't be fair to run a 32 bit Stockfish and nobody would be happy with this solution and I would be on Stockfish's side if you had done this.
On the other hand it also doesn't seem fair to have a list of compiles that you choose from where you pick the best one for your hardware. Why would that not be fair? Because I am not allowed to do that. I tried to do that and was rebuked for it. Our plan was to provide an official version in the distribution for each processor family, including multiple ones for older Intel and newer Intel if we could make it work - and YOU told me that only 1 official version could be used on your test.
So I'm more than a little annoyed that so much effort is spend on finding the one perfect compile of stockfish that you can use - that the rules changed for Stockfish and that it's even a question which one you should use. Even if there is a link to multiple compiles on their website that seems like a workaround - for example if someone produces a special Haswell compile that only worked well on Haswell then are you saying that all they have to do is add that to their growing list of binaries on that site and that it's all ok?
To be completely clear, I think there SHOULD be multiple compiles for multiple platforms and that this SHOULD be allowed - so I'm not saying Stockfish users should not enjoy this advantage and I advocate this kind of free choice - but what I am saying is that if it's ok for them it should be ok for any of us and not JUST them.
As I said before this is an issue that is not going to go away because I expect future processors to have more and more capabilities that might be utilized in chess programs just as popcount has been. So I strongly suggest that all the major testers agree upon some rules here and this not become an ad-hoc per-program decision.
I propose a liberal policy that lets a program author include in his release as many binaries as the author wishes to produce - but that for official tests this is confined to what is available at release time. For casual users there is no problem with allowing compiles to be added to a growing list if that can benefit the end-users. But for testing it is equivalent to having a new release every time a new compile is produced and I don't think anyone wants that.
What is NOT acceptable is for a program author to work with each tester to make a specialized compile just for the test - which is not what I'm saying is happening here but I could see this happening in the future if there is not some rules.
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
)
