Komodo 9.3 well below 9.2 in CCRL/40

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
melajara
Posts: 213
Joined: Thu Dec 16, 2010 3:39 pm

Komodo 9.3 well below 9.2 in CCRL/40

Post by melajara » Sun Dec 06, 2015 2:47 pm

See http://www.computerchess.org.uk/ccrl/40 ... t_all.html

9.3 is trailing 45 ELO on 1 core / 64bits and 32 ELO on 4 cores / 64 bits vs 9.2

Granted, there are only 176 and 124 games so far, but the tendency for both platforms is negative. Something is wrong here, default contempt?
Per ardua ad astra

User avatar
Leto
Posts: 2028
Joined: Thu May 04, 2006 1:40 am
Location: Dune

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by Leto » Sun Dec 06, 2015 3:16 pm

I'm testing Komodo 9.3 x64 12CPU for CEGT 40/4 and I have 750 games so far and it's scoring 64.9%, I think it will end up slightly higher than K9.2 x64 C0 12CPU but I'm just guessing by the score.

It scored 97.5 out of 200 (48.7%) against Stockfish TCEC8 12CPU, so there's a chance that Komodo 9.3 x64 12CPU will start at second place on the list. In comparison Komodo 9.2 x64 C0 12CPU scored only 46%, and Komodo 9.2 x64 12CPU (the default) scored only 44.7% against the TCEC 8 Stockfish.

So at least at 40/4 everything seems fine to me.

40/4 ratinglist: http://www.husvankempen.de/nunn/40_4_Ra ... liste.html

gerold
Posts: 10121
Joined: Wed Mar 08, 2006 11:57 pm
Location: van buren,missouri

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by gerold » Sun Dec 06, 2015 3:19 pm

There is sure a lot of versions of Komodo in there.

User avatar
Leto
Posts: 2028
Joined: Thu May 04, 2006 1:40 am
Location: Dune

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by Leto » Sun Dec 06, 2015 5:26 pm

And Wolfgang at CEGT 40/4 seems to be getting normal results as well with the 1CPU version, it seems like it will go on the list with +20 elo over K9.2:

http://cegt.forumieren.com/t474-testing-komodo-9-3

He's now working on the 4CPU version. Unfortunately it seems that K9.2 x64 4CPU hasn't been tested yet for that list so I'll see if I have enough time to test that for a bit.

User avatar
Leto
Posts: 2028
Joined: Thu May 04, 2006 1:40 am
Location: Dune

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by Leto » Sun Dec 06, 2015 5:52 pm

Looking at the CCRL results I think what's causing it on the 4CPU version is Bouquet 1.8. K93's performance there is -60 elo. On the 1CPU version you see -68 against Equinox and -65 against Sting and -84 against BlackMamba. But all these matches have about 10 games each, so I think with more games K9.3 should pull ahead of K92.

melajara
Posts: 213
Joined: Thu Dec 16, 2010 3:39 pm

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by melajara » Sun Dec 06, 2015 10:03 pm

Thanks for your detailed answers with sensible comments,
statistical fluke then :-)
Per ardua ad astra

User avatar
Dr.Wael Deeb
Posts: 9635
Joined: Wed Mar 08, 2006 7:44 pm
Location: Amman,Jordan

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by Dr.Wael Deeb » Mon Dec 07, 2015 4:52 am

Leto wrote:Looking at the CCRL results I think what's causing it on the 4CPU version is Bouquet 1.8. K93's performance there is -60 elo. On the 1CPU version you see -68 against Equinox and -65 against Sting and -84 against BlackMamba. But all these matches have about 10 games each, so I think with more games K9.3 should pull ahead of K92.
Every chess engine & it's subversions has it's own nemesis....

I see this on daily basis building my rating list regards,
Dr.D
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….

Norm Pollock
Posts: 1017
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by Norm Pollock » Mon Dec 07, 2015 12:58 pm

I had assumed that the strong victory by Komodo in tcec8 was mainly due to a big improvement from 9.2 to 9.3. It seems now any improvement was minimal.

I now assume the big victory was mainly due to some previously undetected bugs in SF as well as some decline in performance resulting from the new search.

User avatar
Leto
Posts: 2028
Joined: Thu May 04, 2006 1:40 am
Location: Dune

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by Leto » Mon Dec 07, 2015 2:48 pm

Norm Pollock wrote:I had assumed that the strong victory by Komodo in tcec8 was mainly due to a big improvement from 9.2 to 9.3. It seems now any improvement was minimal.

I now assume the big victory was mainly due to some previously undetected bugs in SF as well as some decline in performance resulting from the new search.
I have another theory. Perhaps Komodo 9.3 is about 25 elo stronger than Stockfish TCEC8 at TCEC8 time control, but either equal or slightly weaker than Stockfish TCEC8 at faster time controls.

APassionForCriminalJustic
Posts: 415
Joined: Sat May 24, 2014 7:16 am

Re: Komodo 9.3 well below 9.2 in CCRL/40

Post by APassionForCriminalJustic » Mon Dec 07, 2015 5:01 pm

Norm Pollock wrote:I had assumed that the strong victory by Komodo in tcec8 was mainly due to a big improvement from 9.2 to 9.3. It seems now any improvement was minimal.

I now assume the big victory was mainly due to some previously undetected bugs in SF as well as some decline in performance resulting from the new search.
Komodo 9.2 is already super strong. People who know their stuff, like myself, already knew that Komodo 9.3x was only about 10 ELO stronger versus 9.2 (clearly you didn't get the memo). What you're basically saying is Komodo won because Stockfish got worse; Komodo won as a result of it being stronger. Stockfish is actually super strong at least on my 28 core. Bugs and all that are excuses...

Post Reply