Komodo 5 running for the IPON

[lkaufman]

I have evidence that our compile, while fine for Intel machines, was unsuitable for AMD machines. If this is confirmed it would explain our disappointing IPON results, and we may have to make another compile for AMD users.

[IWB]

Hi Larry,

I have evidence that our compile, while fine for Intel machines, was unsuitable for AMD machines. If this is confirmed it would explain our disappointing IPON results, and we may have to make another compile for AMD users.

I answer to both posts here:

1. Timing:

Not a single loss on time in 1270 games (for no one). Most games are won, so it is a bit difficult to judge the timing as towards the end the opponents start to use more time ... but in general, compared to the one or two engines I know a bit more in detail it seems to be absolutly ok. For sure it is not using just 50% of its time ...

2. Compile:

I have a litttle "single" benchmark on inwoba.de in the archive section. It wasn't updated for a while but with 20 Engines this will not change much. You can see that, at same clock speed, the average difference between an i7 and the Phenom2 is 16/13% in nodes (depending the viewpoint)

These are the numbers of Komodo 5 with this Benchmark conditions:
Phenom 2@3206MHz in 5:01.234 min: 290931548 or 943 kN/s
Intel i7@2806MHz in 5:01.343 min: 320779195 or 1040 kN/s

Ph2@1GHz = 294 kN/s
i7@1GHz = 370 kN/s

That is a difference of 26%/21% in other words: You are 10%/8% slower than the average.

The old Komodo 3 was better regarding this but if my test can show a 10% difference ... How many Elo are that? I have some doubts!?

Anyhow, I do not like the idea of having different compiles as no one is having different compiles. If you make a compile especially for my setup it feels a bit like "cheating". Just wait for the CEGT 40/20 rating, usually they are VERY similar to mine.

But I agree, that the AMD is handicaped by most compiles and is looking worse than it has to be. If the programmers would take more care of this the difference of 16% would shring for sure!


Bye
Ingo

[IWB]

...
Anyhow, I do not like the idea of having different compiles as no one is having different compiles. If you make a compile especially for my setup it feels a bit like "cheating
...

Forget about this. Wrong thinking! I am absolutly fine with a new compile as long as it is not handicaping the Intel performance (in Nodes/s) ...

Bye
Ingo

[Vinvin]

now :

Code: Select all

Komodo 5 - Critter 1.6a (2973)	19.5 -11.5	 62.90% Perf=3064
Komodo 5 - Stockfish 2.2.2 JA (2966)	14.5 -15.5	 48.33% Perf=2955

Awesome against Critter and deceiving against SF, we will see if the pattern still ...

Now :
Komodo 5 - Critter 1.6a (2973) 51.5 - 39.5 56.59% Perf=3019
Komodo 5 - Stockfish 2.2.2 JA (2966) 48.0 - 42.0 53.33% Perf=2989

The pattern clearly reduced.

Current perf : 3004. +22 Elo pts over Komodo 4.

[IWB]

Current perf : 3004. +22 Elo pts over Komodo 4.

That is just a best guess. The starting elo are calculated with bayes while the calculation during the run is a pure calulation with the elo formula. For now, this is just an "over the thumb" calculation and might change a few elo up or down with the final result.

Bye
Ingo

[gleperlier]

Current perf : 3004. +22 Elo pts over Komodo 4.

That is just a best guess. The starting elo are calculated with bayes while the calculation during the run is a pure calulation with the elo formula. For now, this is just an "over the thumb" calculation and might change a few elo up or down with the final result.

Bye
Ingo

And we will have to way also all CCRL ratings etc.

. I know you are doing great job with IPON but it's only one time control, one condition of game, one GUI...

We need to wait for many other rating list, many other time controls, many other openings etc.

Cheers,

Gab

[lkaufman]

...
Anyhow, I do not like the idea of having different compiles as no one is having different compiles. If you make a compile especially for my setup it feels a bit like "cheating
...

Forget about this. Wrong thinking! I am absolutly fine with a new compile as long as it is not handicaping the Intel performance (in Nodes/s) ...

Bye
Ingo

A 10% speed difference is supposed to be about 10 elo at blitz levels, but we generally see higher elo differences for speedups than this 1 for 1 rule indicates. For example, on our network of testers, we currently show 10.4 elo as the gain for the final compile, although the speedup was only about 5.5%. We're trying to measure the slowdown of the AMD compile now, to decide whether a new compile is worthwhile. We optimized for Intel because most people have Intel, but now it appears that we may have just crippled the AMD results with almost no benefit for Intel.
In any case, your current result of +23 elo is only 13 behind our own +36 result, so the difference and the 10% figure you mention for AMD slowdown agree almost perfectly. I guess we have no problem with Ponder or with the time control.

[IWB]

My guess is lower than 10 Elo for 10% but with the confidence I can reach with my few games all these small numbers are in vain ...

Ponder and time is not an issue here. It runs smooth and easy

Bye
Ingo

[Don]

...
Anyhow, I do not like the idea of having different compiles as no one is having different compiles. If you make a compile especially for my setup it feels a bit like "cheating
...

Forget about this. Wrong thinking! I am absolutly fine with a new compile as long as it is not handicaping the Intel performance (in Nodes/s) ...

Bye
Ingo

We would try to produce a single binary - I don't like having multiple binaries. But if I cannot produce efficient binaries I would release 2 binaries, to be used on the appropriate platform. So it would not be cheating - you just use the appropriate binary.

Note that it's technically possible to wrap this up in such a way that the binary automatically detects which platform it is running on and just runs the appropriate code. That would be a lot of work but it's possible to do. Another way is a wrapper script that detects the platform and runs the appropriate binary. But I would prefer to avoid any of this.

In the worse case, if it turns out that one platform has to be severely crippled to run well on the other, I would make 2 binaries. But we are not there yet ...

[IWB]

It seems that for IPON as well as for CCRL setting mm 1 1 and then scale to 1 gives the most accurate results. Or you could use Ordo. What you see in Adam posts as 340 instead of 400 (all these fitted numbers must be compared with the default 400 of the logistic) is giving the magnitude of the compression, in this case some 15%. Not negligible at all. Hope you use Bayeselo mm 1 1 scale 1 prior 0.1 or Ordo.

Kai

I have some doubts that I will use that.

1. I do not understand what it is doing (but I will read into it a bit more when i have some time left)
2. That put my list far off of any comparibility with the ohers.

This is how it looks with mm 01:

Code: Select all

   1 Houdini 2.0 STD          3026    9    9  5400   78%  2791   26% 
   2 Houdini 1.5a             3018   10   10  4000   79%  2775   26% 
   3 Komodo 5                 3007   11   11  2700   75%  2816   33% 
   4 Critter 1.4a             2983    9    9  4450   77%  2772   32% 
   5 Komodo 4                 2982    9    9  4850   75%  2781   30% 
   6 Critter 1.6a             2973   10   10  3150   70%  2823   40% 

and this with mm 11, scale 1, prior 0.1:

Code: Select all

   1 Houdini 2.0 STD          3084   11   10  5400   78%  2789   26% 
   2 Houdini 1.5a             3074   12   12  4000   79%  2769   26% 
   3 Komodo 5                 3060   14   13  2700   75%  2821   33% 
   4 Critter 1.4a             3031   11   11  4450   77%  2765   32% 
   5 Komodo 4                 3030   11   11  4850   75%  2777   30% 
   6 Critter 1.6a             3018   12   12  3150   70%  2829   40% 

that is 58 Elo difference for H2.0! My list is loosing any possibility to be compared with others ...
I might be convinced if ALL are doing it.

Bye
Ingo