Komodo 9.3 well below 9.2 in CCRL/40

[melajara]

See http://www.computerchess.org.uk/ccrl/40 ... t_all.html

9.3 is trailing 45 ELO on 1 core / 64bits and 32 ELO on 4 cores / 64 bits vs 9.2

Granted, there are only 176 and 124 games so far, but the tendency for both platforms is negative. Something is wrong here, default contempt?

[Leto]

I'm testing Komodo 9.3 x64 12CPU for CEGT 40/4 and I have 750 games so far and it's scoring 64.9%, I think it will end up slightly higher than K9.2 x64 C0 12CPU but I'm just guessing by the score.

It scored 97.5 out of 200 (48.7%) against Stockfish TCEC8 12CPU, so there's a chance that Komodo 9.3 x64 12CPU will start at second place on the list. In comparison Komodo 9.2 x64 C0 12CPU scored only 46%, and Komodo 9.2 x64 12CPU (the default) scored only 44.7% against the TCEC 8 Stockfish.

So at least at 40/4 everything seems fine to me.

40/4 ratinglist: http://www.husvankempen.de/nunn/40_4_Ra ... liste.html

[gerold]

There is sure a lot of versions of Komodo in there.

[Leto]

And Wolfgang at CEGT 40/4 seems to be getting normal results as well with the 1CPU version, it seems like it will go on the list with +20 elo over K9.2:

http://cegt.forumieren.com/t474-testing-komodo-9-3

He's now working on the 4CPU version. Unfortunately it seems that K9.2 x64 4CPU hasn't been tested yet for that list so I'll see if I have enough time to test that for a bit.

[Leto]

Looking at the CCRL results I think what's causing it on the 4CPU version is Bouquet 1.8. K93's performance there is -60 elo. On the 1CPU version you see -68 against Equinox and -65 against Sting and -84 against BlackMamba. But all these matches have about 10 games each, so I think with more games K9.3 should pull ahead of K92.

[melajara]

Thanks for your detailed answers with sensible comments,
statistical fluke then

[Dr.Wael Deeb]

Looking at the CCRL results I think what's causing it on the 4CPU version is Bouquet 1.8. K93's performance there is -60 elo. On the 1CPU version you see -68 against Equinox and -65 against Sting and -84 against BlackMamba. But all these matches have about 10 games each, so I think with more games K9.3 should pull ahead of K92.

Every chess engine & it's subversions has it's own nemesis....

I see this on daily basis building my rating list regards,
Dr.D

[Norm Pollock]

I had assumed that the strong victory by Komodo in tcec8 was mainly due to a big improvement from 9.2 to 9.3. It seems now any improvement was minimal.

I now assume the big victory was mainly due to some previously undetected bugs in SF as well as some decline in performance resulting from the new search.

[Leto]

I had assumed that the strong victory by Komodo in tcec8 was mainly due to a big improvement from 9.2 to 9.3. It seems now any improvement was minimal.

I now assume the big victory was mainly due to some previously undetected bugs in SF as well as some decline in performance resulting from the new search.

I have another theory. Perhaps Komodo 9.3 is about 25 elo stronger than Stockfish TCEC8 at TCEC8 time control, but either equal or slightly weaker than Stockfish TCEC8 at faster time controls.

[APassionForCriminalJustic]

I had assumed that the strong victory by Komodo in tcec8 was mainly due to a big improvement from 9.2 to 9.3. It seems now any improvement was minimal.

I now assume the big victory was mainly due to some previously undetected bugs in SF as well as some decline in performance resulting from the new search.

Komodo 9.2 is already super strong. People who know their stuff, like myself, already knew that Komodo 9.3x was only about 10 ELO stronger versus 9.2 (clearly you didn't get the memo). What you're basically saying is Komodo won because Stockfish got worse; Komodo won as a result of it being stronger. Stockfish is actually super strong at least on my 28 core. Bugs and all that are excuses...