Status Quo 2016-1 , Stockfish - Komodo

[ernest]

Regarding the asymmetric confidence intervals of Fritz GUI (I suppose it is Fritz GUI), I managed to get most of the numbers. At this moment:

Code: Select all

Komodo 9.3 64-bit x24 - Stockfish 030116 64 POPCNT; 39 Games, +5 =30 -4, 51.3%, TP=+9 Elo, 68%->[-11,+65],
                        95%->[-31,+124], 99.7%->[-51,+191], predicted moves=63.6%

I reached the following equations for the score bounds after many trials. I hope no typos:

Code: Select all

µ = (wins + 0.5*draws)/games

Assuming that Fritz GUI reports 1-sigma, 2-sigma and 3-sigma confidence intervals, for z-sigma confidence interval:

(Lower score bound) = µ + z·ln(2·µ) - (2·z/3)·sqrt[µ·(1 - µ)/(games - 1)]
(Upper score bound) = µ + z·ln(2·µ) + (2·z/3)·sqrt[µ·(1 - µ)/(games - 1)]

Hi Jesus,

Your work is simply great!

I have always wondered WTF that asymmetry (or skew) in the ChessBase GUI confidence intervals was.

Now you pinpointed it, it is the term z·ln(2·µ)
(indeed, when wins=losses µ=1/2, the term is zero and there is no asymmetry
but as soon as wins-losses =1 or more, the asymmetry reappears in the ChessBase GUI confidence intervals )

But how can that rather large asymmetry be interpreted ?
Looks crazy !...

[ernest]

In other terms, Jesus, you reached the formula used by the ChessBase GUI.

But is that formula for the confidence intervals (error-bars) correct ?

[ernest]

When µ is near 0.5, say inside [40%, 60%], letting v=µ-0.5,
your formula
(Lower score bound) = µ + z·ln(2·µ) - (2·z/3)·sqrt[µ·(1 - µ)/(games - 1)]
(Upper score bound) = µ + z·ln(2·µ) + (2·z/3)·sqrt[µ·(1 - µ)/(games - 1)]

becomes approximately :
Upper/Lower = 0.5 + v + z·2·v ± z/3/sqrt(games)

You can see the asymmetric term z·2·v
and the error-bar ± z/3/sqrt(games), which really looks strange !...

Indeed I can verify the asymmetric term z·2·v (multiplying by 700 for Elo) in some of the ChessBase GUI displays, but in some others not.
But still have no idea on what statistics theory this asymmetry could lie...

And look at 2 examples I generated with the ChessBase GUI :
(by forcing resignations in a rapid match I can experiment with ad-lib scores)

Stock syz - Stock; 156 Games, +24 =107 -25, 49.7% TP=-2 Elo,
68%->[-30,+10], 95%->[-58,+22], 99.7->[-87,+34],

Stock syz - Stock; 157 Games, +25 =107 -25, 50.0% TP=0 Elo,
68%->[-28,+28], 95%->[-56,+56], 99.7->[-85,+85],

The 1st example is not far from your formula,
in the 2nd example the error-bar size is (stupidly) 50% larger.

I indeed think your formula is a step in the right direction, to look at "understanding" the ChessBase GUI error-bars.
But probably those ChessBase GUI error-bars are just based on wrong (and idiotic) theory.
This can be verified for instance by using the correct tool elostat, applied to a match result.

P.S. Perhaps ChessBase (if gently asked) will answer on what theory their error-bar calculations are based,
but I'm not too optimistic about that...

[JJJ]

Komodo 9.3 still better than Stockfish at these condition. Nice to see.
Does someone knows about the next TCEC season ?

[Ajedrecista]

Hello Ernest:

Hi Jesus,

Your work is simply great!

I have always wondered WTF that asymmetry (or skew) in the ChessBase GUI confidence intervals was.

Now you pinpointed it, it is the term z·ln(2·µ)
(indeed, when wins=losses µ=1/2, the term is zero and there is no asymmetry
but as soon as wins-losses =1 or more, the asymmetry reappears in the ChessBase GUI confidence intervals )

But how can that rather large asymmetry be interpreted ?
Looks crazy !...

Thanks for your feedback and enthusiasm but my suggested formula has been proven wrong... it was too nice to be true!

I think that the true formula has the form µ + a(µ, n)·z ± b(µ, n)·z·sqrt[µ·(1 - µ)/(n - 1)] with a(0.5, n) = 0 and b(0.5, n) = 1. I realized long time ago that upper and lower bounds for score = 1/2 are score ± sqrt[score*(1 - score)/(games - 1)] = 0.5 ± 0.5/sqrt(n - 1), which always works in this special case.

It is curious that when score = 0.5, the width of the confidence intervals are certain values but when there is one win or lose then the same confidence intervals enlarge a lot.

Regards from Spain.

Ajedrecista.

[Werner]

Hi engine lovers, and a happy new year 2016

I have started an engine macht between Komodo and Stockfish.

Engines use 24 cpu, no hyperthreading, ponder OFF
8 GB of Hashtables
Syzyggy 3-4-5-6
100 games, 60 + 0 timer
using a set of 50 startpossitions 2- 8 moves
Komodo contemp = 0

Will post later games and crosstable
for intrested observers you can watch the games with 15 sec update frequency:
http://clemens-keck.de/screen/

regards, Clemens Keck

Thanks Clemens, seems to be ready. No Display of end result?
Best wishes
Werner

[majortom]

Komodo wins: +12 -8 =80

[Nordlandia]

Table memory for Komodo used?

[Hugo]

Hi All

match is done.


congratulations to the winner.
Replay and games download can be found here:
http://clemens-keck.de/ReplayZone/statusquo2016.htm

thrilling match it was to me

best regards, clemens Keck

[arunsoorya1309]

Great tournament ...... but why not against Sf7 and rather against a dev version ...... the dev version could have bugs which we don't know so far

thanks again for running this