Komodo 2.01 is out!

Laskos · Post by **Laskos** » Sat Jun 11, 2011 4:04 pm

Don wrote:
Laskos wrote:Wow, seems a serious improvement. After 400 games 1s + 0.1s
Code: Select all
    Program                            Score     %      Elo    +   -    Draws

  1 Komodo64 2.01 64 bit           &#58; 262.5/400  65.6    3256   29  29   31.2 %
  2 Komodo64 1.3 JA                &#58; 137.5/400  34.4    3144   29  29   31.2 %
112 +/- 29 Elo points (95% confidence) improvement in self-play, probably a little less in a gauntlet, but the new Reptilian seems the level of SF 2.01. Will leave it for more games, then a gauntlet.

Kai
We believe it is over 100 ELO but it does not always come out the way we think

Komodo 1.3 was supposed to be about 50 ELO in our private testing but when it came out it was much less.

I think the Stockfish team also expected a lot more with version 2.1. It could be because we are all forced to test pretty fast to get statistical confidence in any changes.

We now test more against foreign opponents than we used to (although we have always done some of that) and we have gradually increased out time control to give us a better picture of how we will do at "real" time controls.

Because Komodo is now so much stronger we can no longer handicap the programs we play against which also means it takes a lot longer to test. But I think increasing the time and using foreign opponents more has been a benefit and gives us a better picture of the actual ELO gain despite the fact that increases our resource burden.

Some things we observed: Critter is exceptionally strong at time controls less than 1 minute per game and so is Robbolito. Relative to these programs Stockfish does not look so good until the time control increases, then it looks better. Critter appears to be the least scalable of the programs we test against, but it's difficult to say that for sure, we are extrapolating from out own tests and results we see from rating lists. We believe our program scales very well and we also think Stockfish is exceptional in this area.

We would like to test against Houdini too and it would be the last program we could reasonably handicap to advantage. However we develop and test on Linux because it is has much better behavior for massive testing, but there is no Linux version of Houdini. We can run the 32 bit version using wine but that throws away a lot of the benefit (although the 32 bit version is still stronger than Komodo and Stockfish, at least for now.) But we get crashes and time losses for Houdini.

We have observed that as the programs get stronger, fast time control tests are becoming less reliable predictors of strength. The correlation is still quite high most of the time, but not always. There are some things that clearly test well at game in 3 seconds but test poorly at game in 1 minute and it never used to be that way for us.

Don

Do you have a good control of extrachess losses (on time, illegal moves, etc.) ? I am using LittleBlitzer which has an excellent control on the reasons of losses. Besides that, in ultrashort games, do you see how much time engines are using generally? For example, I cannot go much lower than 100ms increment (with 1000ms basis), the time used by engines becomes erratic, even if they are not losing on time.

Anyway, here are my last results at 1s + 0.1s (average game length ~15 sec)

Self-play finished

Code: Select all


    Program                             Score     %    Av.Op.  Elo    +   -    Draws

  1 Komodo64 2.01 64 bit           &#58; 657.5/1000  65.8   3143   3257   18  18   34.1 %
  2 Komodo64 1.3 JA                &#58; 342.5/1000  34.2   3257   3143   18  18   34.1 %

+114 +/- 18 Elo points (95% confidence) improvement.

Gauntlets in progress

Code: Select all

    Program                            Score     %    Av.Op.  Elo    +   -    Draws

  Komodo64 2.01 64 bit           &#58; 1213.5/3100  39.1   3200   3123   10  10   28.7 %

  1 Houdini 1.5a x64               &#58; 463.0/620  74.7   3123   3309   27  26   22.3 %
  2 Deep Rybka 4.1 x64             &#58; 397.5/620  64.1   3123   3222   23  23   32.1 %
  3 Ivanhoe B47cBx64-1             &#58; 382.0/620  61.6   3123   3204   23  23   30.6 %
  4 Stockfish 2.1 JA 64bit         &#58; 342.5/620  55.2   3123   3158   23  23   28.2 %
  5 Critter 1.01 64-bit            &#58; 301.5/620  48.6   3123   3112   23  23   30.2 %

and

Code: Select all


    Program                            Score     %    Av.Op.  Elo    +   -    Draws

  Komodo64 1.3 JA                 &#58; 377.0/1400  26.9   3200   3026   17  17   26.9 %
  
  1 Houdini 1.5a x64               &#58; 228.0/280  81.4   3026   3281   44  43   19.3 %
  2 Deep Rybka 4.1 x64             &#58; 213.0/280  76.1   3026   3225   38  38   27.1 %
  3 Ivanhoe B47cBx64-1             &#58; 211.0/280  75.4   3026   3219   38  37   27.9 %
  4 Stockfish 2.1 JA 64bit         &#58; 191.5/280  68.4   3026   3158   35  34   33.2 %
  5 Critter 1.01 64-bit            &#58; 179.5/280  64.1   3026   3125   36  36   26.8 %

+97 +/- 20 Elo points (95% confidence) improvement. As of now, the new Reptile seems to be somewhere in between Critter and Stockfish. There are some weird individual scores, but I pretty much trust my testing. The second gauntlet I must continue to decrease somehow the error margins.

What I observed: the new Reptile is 15%-20% slower than 1.3 one, but goes a little deeper. It's by far the slowest of all engines tested (including Rybka times 14). I will put it on Sim03 test, if I find my old data file, as it seems to have acquired a lot in both eval and search

Thanks and congratulations for this huge improvement.
Kai

Wolfgang · Post by **Wolfgang** » Sat Jun 11, 2011 6:31 pm

Don wrote: ...

I am also working on getting some faster binaries compiled with the help of Jim Ablett and Dann Corbit, but due to some compiler issues that will be later. However the supplied binaries are pretty good, just not quite as good as what Jim Ablett has been able to produce.

...

Hi,

first of all: Thanks for the update!!

A few questions concerning possibly faster compiles:
- when do you expect them to be released?
- what increase in speed (percentage) do you expect? Will it be signifcantly faster?

And consequently: Does it make sense to start testing now or shall I wait for the JA or DC compile? I am one of the two main testers for CEGT-blitzlist and can hardly wait starting...

Thanks and best wishes
Wolfgang
CEGT-Team

Don · Post by **Don** » Sat Jun 11, 2011 6:45 pm

FlavusSnow wrote:Great release!

Can Komodo play Chess960 (Fischer Random) correctly?

Have you thought about supporting it in the future?

Yes, this is definitely a consideration for a future version.

Sarciness · Post by **Sarciness** » Sat Jun 11, 2011 6:51 pm

Just to let you know that I have had a few crashes too during infinite analysis when going through my own games.

Running Windows 7 (64-bit) and 64 bit version of Komodo 2.01 64.

El Gringo · Post by **El Gringo** » Sat Jun 11, 2011 7:31 pm

Hi,

results test suite sts 1-13 :

Code: Select all

Komodo 2.01 x64	90	85	79	85	92	86	76	66	81	85	79	76	83	1063
Komodo 1.3 x64 	94	80	77	77	86	84	79	71	77	81	76	83	81	1046

Q6660 @ 3Ghz

Best
Johan

JWorcester · Post by **JWorcester** » Sat Jun 11, 2011 7:49 pm

Well after 236 games at 2+2 vs Houdini, Komodo is -90 ELO behind which equates to a pretty impressive improvement.

I'll report more fully once I've finished the games. The games were run on a Q6600 overclocked to 2.9GHz with each pair of engines sharing one core with ponder off.

tomgdrums · Post by **tomgdrums** » Sat Jun 11, 2011 7:58 pm

Komodo seems to be crashing a bit. (I haven't tried the new version)

How does Komodo work with the Shredder Classic GUI?

Don · Post by **Don** » Sat Jun 11, 2011 8:09 pm

tomgdrums wrote:Komodo seems to be crashing a bit. (I haven't tried the new version)

How does Komodo work with the Shredder Classic GUI?

Yes, Komodo appears to be crashing on occasion and I'm trying to fix that. The problem is that I cannot make it crash on Linux and I don't develop or test in windows. I do have a windows dual boot laptop but it's not crashing in arena. Or if it is I don't see where it is being reported.

It's hard to debug something that is not producing errors in my own tests. So I have asked a few people to help me debug and I'm working on the problem.

Don

kranium · Post by **kranium** » Sat Jun 11, 2011 8:49 pm

Don wrote:
tomgdrums wrote:Komodo seems to be crashing a bit. (I haven't tried the new version)

How does Komodo work with the Shredder Classic GUI?
Yes, Komodo appears to be crashing on occasion and I'm trying to fix that. The problem is that I cannot make it crash on Linux and I don't develop or test in windows. I do have a windows dual boot laptop but it's not crashing in arena. Or if it is I don't see where it is being reported.

It's hard to debug something that is not producing errors in my own tests. So I have asked a few people to help me debug and I'm working on the problem.

Don

regardless of any small issues/problems,
a big congrats is due to you (and Larry) for your tremendous effort on this excellent engine!

stevenaaus · Post by **stevenaaus** » Sat Jun 11, 2011 10:35 pm

Linux64 bit sse4 won't run here ... illegal instruction.
I have an Q9400 with sse4_1 but no sse4_2

cat /proc/cpuinfo

Code: Select all

processor       &#58; 3
vendor_id       &#58; GenuineIntel
cpu family      &#58; 6
model           &#58; 23
model name      &#58; Intel&#40;R&#41; Core&#40;TM&#41;2 Quad CPU    Q9400  @ 2.66GHz
stepping        &#58; 10
cpu MHz         &#58; 2003.000
cache size      &#58; 3072 KB
physical id     &#58; 0
siblings        &#58; 4
core id         &#58; 3
cpu cores       &#58; 4
apicid          &#58; 3
initial apicid  &#58; 3
fpu             &#58; yes
fpu_exception   &#58; yes
cpuid level     &#58; 13
wp              &#58; yes
flags           &#58; fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority
bogomips        &#58; 5333.30
clflush size    &#58; 64
cache_alignment &#58; 64
address sizes   &#58; 36 bits physical, 48 bits virtual
power management&#58;

gdb komodo-201-64-sse4

Code: Select all

This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see&#58;
<http&#58;//www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/komodo-201-64-sse4...&#40;no debugging symbols found&#41;...done.
&#40;gdb&#41; r
Starting program&#58; /usr/local/bin/komodo-201-64-sse4 

Program received signal SIGILL, Illegal instruction.
0x0000000000415ae6 in ?? ()
&#40;gdb&#41; bt
#0  0x0000000000415ae6 in ?? ()
#1  0x000000000042704e in ?? ()
#2  0x0000000000400369 in ?? ()
#3  0x00007fffffffe1c8 in ?? ()
#4  0x0000000000000000 in ?? ()
&#40;gdb&#41; si

Program terminated with signal SIGILL, Illegal instruction.

Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!

Re: Komodo 2.01 is out!