Komodo 2.03 SSE42 available

[Don]

Komodo 2.03 for high performance computers is now avaiable from the komodochess web site.

It is fast, PGO compiled to use the Advanced Bit Manipulation instructions provided with AMD and i7 chips.

It's a bit of a misnomer to call this the "sse4.2" version as Komodo does not use any sse4.2 instructions. It only uses the fast 64 bit popcount instruction which is NOT part of the sse4.2 instruction set but is often treated as if it is. It just happens to be included in Intel chips that also support the sse4.2 instruction set and it's part of the ABM instructions introduced originally by AMD.

This binary works well on AMD and Intel i7 and for those machines is significantly faster than the Jim Ablett versions which do not support ABM.

Dann Corbit worked hard on providing this for me and his help is greatly appreciated.

Jim Ablett has also been a huge help and his binaries are better by far than anything I can produce for non-see4 machines.

[Albert Silver]

Komodo 2.03 for high performance computers is now avaiable from the komodochess web site.

It is fast, PGO compiled to use the Advanced Bit Manipulation instructions provided with AMD and i7 chips.

It's a bit of a misnomer to call this the "sse4.2" version as Komodo does not use any sse4.2 instructions. It only uses the fast 64 bit popcount instruction which is NOT part of the sse4.2 instruction set but is often treated as if it is. It just happens to be included in Intel chips that also support the sse4.2 instruction set and it's part of the ABM instructions introduced originally by AMD.

This binary works well on AMD and Intel i7 and for those machines is significantly faster than the Jim Ablett versions which do not support ABM.

Dann Corbit worked hard on providing this for me and his help is greatly appreciated.

Jim Ablett has also been a huge help and his binaries are better by far than anything I can produce for non-see4 machines.

So it should be just fine on i5 processors as well.

[gaard]

Should the analysis between these versions, even with 'deterministic' turned on, differ?

[Dann Corbit]

Should the analysis between these versions, even with 'deterministic' turned on, differ?

It would not surprise me if there were differences. Despite optimization obeying the 'as-if' rule, re-ordering of instructions or which branch is taken first could easily affect (for instance) the shape of a tree that is being searched.

Also, Komodo uses floating point for evaluation and so (for instance) an interrupt can shave off the high order bits used for intermediate calculations so that even the same version can produce slightly different results at times.

[Don]

Should the analysis between these versions, even with 'deterministic' turned on, differ?

As it turns out that is a more complicated question that it should be

Komodo has a hidden self-test but we were not getting consistent behavior from platform to platform and sometimes even from different compilers or options.

So the short answer is NO. However the same binary should produce consistent results within itself.

We found there was a minor bug that affected the consistency (which relied on implementation defined behavior of compilers) which has been fixed, but did not make this version.

The self test can be run in a command shell like this:

komodo.exe -test

It may crash however - because the test insists on perfect reproducibility (it is a unit test of sorts similar to perft) but don't panic if it does. Very extensive testing proved that there is no strength difference that can be measured between these versions. The nature of the bug is such that it does not cause any catastrophic failures or anything serous. In fact the primary difference is specifically that different margins are used at one specific depth of search in a routine that has only a small impact on the playing strength anyway. So technically some versions of the program are "different" than others but not in a way you can measure strength-wise.

Aart Bik actually identified this bug while trying to get the self-test to work on the Android platform. It involved left shifting a value by a negative amount. I don't believe C defines how that should behave.

So in principle, you should be able to run komodo 2.03 on any platform while setting deterministic to true and get the same score, nodecount and PV - but in practice it's not quite working right. It works now in our current branch of development and will in the next release.

So while I'm in the mood to confess, people have noticed that Komodo will get into bishop and (wrong) rook pawn vs King and think the bishop is winning. They assumed we had not implemented that yet, but in fact it's been in the program for a long time. It just doesn't work! That has been fixed too, but did not make it into the 2.03 release.

[Eelco de Groot]

Should the analysis between these versions, even with 'deterministic' turned on, differ?

It would not surprise me if there were differences. Despite optimization obeying the 'as-if' rule, re-ordering of instructions or which branch is taken first could easily affect (for instance) the shape of a tree that is being searched.

Also, Komodo uses floating point for evaluation and so (for instance) an interrupt can shave off the high order bits used for intermediate calculations so that even the same version can produce slightly different results at times.

Excuse the interrupt Dann, but does that sound very strange. An interrupt can cause the result of a floating point calculation to be rounded off or terminated early Would that not invalidate our computers for any serious mathematical, statistical or otherwise scientific application. Or are these interrupts designed into Komodo only? Could the intermediate results not be stored and resumed correctly after the interrupt? How long has this been going on and can we blame Bill Gates

Nondeterministic behaviour with Stockfish -different behaviour compared to a stock compile I mean- was I recall the result of a wrong optimization behaviour in the compilation, but I would not know if Komodo is less deterministic. Is that not problematic for bughunting for instance, I suppose that is why there is a switch?

Eelco

[Don]

Should the analysis between these versions, even with 'deterministic' turned on, differ?

It would not surprise me if there were differences. Despite optimization obeying the 'as-if' rule, re-ordering of instructions or which branch is taken first could easily affect (for instance) the shape of a tree that is being searched.

Also, Komodo uses floating point for evaluation and so (for instance) an interrupt can shave off the high order bits used for intermediate calculations so that even the same version can produce slightly different results at times.

Excuse the interrupt Dann, but does that sound very strange. An interrupt can cause the result of a floating point calculation to be rounded off or terminated early Would that not invalidate our computers for any serious mathematical, statistical or otherwise scientific application. Or are these interrupts designed into Komodo only? Could the intermediate results not be stored and resumed correctly after the interrupt? How long has this been going on and can we blame Bill Gates

Nondeterministic behaviour with Stockfish -different behaviour compared to a stock compile I mean- was I recall the result of a wrong optimization behaviour in the compilation, but I would not know if Komodo is less deterministic. Is that not problematic for bughunting for instance, I suppose that is why there is a switch?

Eelco

Dann is saying that optimizations are not always correct, not that it SHOULD produce non-deterministic behavior but that in practice it DOES sometimes. Even the manuals for compilers will label some optimization as "risky" because they make certain assumptions.

Most of the floating point has been removed from Komodo to eliminate any possible source of non-determinism. I am a big believer in unit testing and I cannot live with the fact that Komodo is not deterministic. However I doubt the floating point operations are the cause but there could be a portability benefit to removing the floating point stuff. And now Komodo can be compiled without the math library

[Dann Corbit]

Should the analysis between these versions, even with 'deterministic' turned on, differ?

It would not surprise me if there were differences. Despite optimization obeying the 'as-if' rule, re-ordering of instructions or which branch is taken first could easily affect (for instance) the shape of a tree that is being searched.

Also, Komodo uses floating point for evaluation and so (for instance) an interrupt can shave off the high order bits used for intermediate calculations so that even the same version can produce slightly different results at times.

Excuse the interrupt Dann, but does that sound very strange. An interrupt can cause the result of a floating point calculation to be rounded off or terminated early Would that not invalidate our computers for any serious mathematical, statistical or otherwise scientific application. Or are these interrupts designed into Komodo only? Could the intermediate results not be stored and resumed correctly after the interrupt? How long has this been going on and can we blame Bill Gates

Nondeterministic behaviour with Stockfish -different behaviour compared to a stock compile I mean- was I recall the result of a wrong optimization behaviour in the compilation, but I would not know if Komodo is less deterministic. Is that not problematic for bughunting for instance, I suppose that is why there is a switch?

Eelco

It is definitely problematic for numerical calculations and can make things less repeatable.

Depending upon optimization settings, intermediate results can be stored in different formats. (So for instance float*float might actually use double or even long double for intermediate results, depending upon the compiler settings). You can also choose to tell the compiler to do things with extreme determination, but this is slower and since we are not calculating your phone bill, nobody will care if we lose the penny. In GCC, for instance, if you tell it to use the optimization setting :
-ffast-math
it can play fast and loose with the rules. (The fastest Komodo version *is* compiled with -ffast-math option.)

No matter how careful you are with floating point math, there are still some surprises for many people. For instance, most people are surprised to hear that if you add up a long column of floating point numbers you will almost always get a different answer if you add up the column forwards than if you add up the column backwards [*].

Everyone who considers writing a floating point program should read this paper very carefully (It is the best that I know of):
"What Every Computer Scientist Should Know About. Floating-Point Arithmetic."
by DAVID GOLDBERG

[*] No, I am not kidding. If you have a degree in Numerical Analysis, this fact is not even surprising to you at all.

[fern]

Don:
Do you have an idea about which is the most efective GUi for Komodo running on 32 bits windows?
I have Chessbase -fritz 12- GUI, Arena and Shredder. what do you advice?

fern

[Don]

Don:
Do you have an idea about which is the most efective GUi for Komodo running on 32 bits windows?
I have Chessbase -fritz 12- GUI, Arena and Shredder. what do you advice?

fern

You are asking the wrong person! I am not a windows fan and don't use windows at all.

In Linux I use xboard because it's snappy. I like Arena but I won't use it as I prefer native applications (you must use wine and I cannot run 64 bit engines in Linux using Arena.)

Probably you will get a religious holy war if you ask the group this question

Don