what is wrong with Komodo Dragon 2 here?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
frcha
Posts: 211
Joined: Thu Jan 28, 2010 4:47 pm
Contact:

what is wrong with Komodo Dragon 2 here?

Post by frcha » Wed Jun 09, 2021 11:52 pm

I had noticed previously that Dragon 2 seemed a bit weaker than dragon 1 and so I started a gauntlet match with KD1 and KD 2 against SF13_noNN.

Note this position Dragon 2 has a winning advantage but then it plays Qxf3??



I have never seen this before with any engine !
Also when I later analyze with KD2 it shows correct move so its really mystifying.
I really think this needs to be looked into.

Whole pgn is below. This is played on Arena 256h poff timecontrol using 6s+1s

Also for ref. here is the tournament so far.
Engine Score St
1: Stockfish_13_noNN 56.5/128 ииииииииииииииииииииииииииииииии
2: Lc0_0.27 19.5/32 =====1===1====1=0==1=1=1=1=====1
3: Fire_8.NN.1 19.0/32 10====1==1===0===1=11==1=====1==
4: KomodoDragon-1 17.0/32 ======1====1001==0=====1===1=0=1
5: KomodoDragon-2 16.0/32 ====0=0==10========1=======1=1=0

128 of 400 games played
Name of the tournament: FireDragonTest
Level: Blitz 0:06/1
Hardware: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz with 31.7 GB Memory
Operating system: Windows 10 Home Home Edition (Build 9200) 64 bit






Cornfed
Posts: 269
Joined: Sun Apr 26, 2020 9:40 pm
Full name: Brian D. Smith

Re: what is wrong with Komodo Dragon 2 here?

Post by Cornfed » Thu Jun 10, 2021 1:17 am

I don't have Dragon...but I have a feeling no one is going to be able to replicate this...doesn't look possible.

frcha
Posts: 211
Joined: Thu Jan 28, 2010 4:47 pm
Contact:

Re: what is wrong with Komodo Dragon 2 here?

Post by frcha » Thu Jun 10, 2021 12:16 pm

When you purchase dragon 2 you also get dragon 1, so good value if you dont have dragon 1 . However ... issues as stated are accurate - dont know about replication of exact position but anyone can replicate the poor performance vs SF noNN compared to KD 1.

- I just finished the tournament and there is no obvious mis-configuration. Dragon 2 does have table memory increased to 128M but then so does dragon 1. The results of the Gauntlet Tournament which also includes lc0 and Fire is below:

I suspect it is the time control being too short maybe? There is a setting called reduction (default setting ) which is different in Dragon2 compared to dragon 1 - perhaps the Komodo programmers might have more insight.

Also very few people are testing vs SF nonn. I suspect Dragon 2 does a little better against others engines judging from engine list e.g check https://ccrl.chessdom.com/ccrl/404/cgi/ ... 4-bit_8CPU making it comparable in ELO.

8 Cores ponder off. lc0 on RTX 2070 super using J94 net.


Great performance by Fire 8NN - much weaker performance from KD2 than expected:


-----------------Fire_8.NN.1-----------------
Fire_8.NN.1 - Stockfish_13_noNN : 57.0/100 19-5-76 (10====1==1===0===1=11==1=====1==========0========11==1=1===0=========11====1=======1=======1=1==01==) 57% +49
-----------------KomodoDragon-1-----------------
KomodoDragon-1 - Stockfish_13_noNN : 55.5/100 20-9-71 (======1====1001==0=====1===1=0=1==1====1==0110=1=1===10===0==1=1=========1===1=========11=======01==) 56% +42
-----------------KomodoDragon-2-----------------
KomodoDragon-2 - Stockfish_13_noNN : 52.5/100 13-8-79 (====0=0==10========1=======1=1=00==========11====1===11=========0======1===========1========1==1=00=) 53% +21
-----------------Lc0_0.27-----------------
Lc0_0.27 - Stockfish_13_noNN : 59.0/100 23-5-72 (=====1===1====1=0==1=1=1=1=====1=====101=111====1==0=========1=====11=====01=========11========101=1) 59% +63

lkaufman
Posts: 4884
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: what is wrong with Komodo Dragon 2 here?

Post by lkaufman » Thu Jun 10, 2021 11:18 pm

frcha wrote:
Wed Jun 09, 2021 11:52 pm
I had noticed previously that Dragon 2 seemed a bit weaker than dragon 1 and so I started a gauntlet match with KD1 and KD 2 against SF13_noNN.

Note this position Dragon 2 has a winning advantage but then it plays Qxf3??



I have never seen this before with any engine !
Also when I later analyze with KD2 it shows correct move so its really mystifying.
I really think this needs to be looked into.

Whole pgn is below. This is played on Arena 256h poff timecontrol using 6s+1s

Also for ref. here is the tournament so far.
Engine Score St
1: Stockfish_13_noNN 56.5/128 ииииииииииииииииииииииииииииииии
2: Lc0_0.27 19.5/32 =====1===1====1=0==1=1=1=1=====1
3: Fire_8.NN.1 19.0/32 10====1==1===0===1=11==1=====1==
4: KomodoDragon-1 17.0/32 ======1====1001==0=====1===1=0=1
5: KomodoDragon-2 16.0/32 ====0=0==10========1=======1=1=0

128 of 400 games played
Name of the tournament: FireDragonTest
Level: Blitz 0:06/1
Hardware: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz with 31.7 GB Memory
Operating system: Windows 10 Home Home Edition (Build 9200) 64 bit





I get Qxf5 on any depth, even one ply. It's hard to fix a nonreproducible mistake. I never observe blunders like this. Time shouldn't be an issue with a full second increment. As for the results, the difference between 55.5 and 52.5 out of 100 is not statistically significant. Dragon 2 generally outperforms Dragon 1, regardless of opponent, but the margin is below the margin of error for runs of a hundred or so games.
Komodo rules!

frcha
Posts: 211
Joined: Thu Jan 28, 2010 4:47 pm
Contact:

Re: what is wrong with Komodo Dragon 2 here?

Post by frcha » Fri Jun 11, 2021 1:31 am

I get Qxf5 on any depth, even one ply. It's hard to fix a nonreproducible mistake. I never observe blunders like this.
Please check this position


This one can be reproduced if you give 256h no multipv maybe just 1 core though you can see it briefly at 8 cores:

Komodo Dragon 2 plays Bxe6?


Modern Times
Posts: 2895
Joined: Thu Jun 07, 2012 9:02 pm

Re: what is wrong with Komodo Dragon 2 here?

Post by Modern Times » Fri Jun 11, 2021 1:51 am

Never considers that move here: (1 thread on my laptop)

Code: Select all


Engine: Dragon by Komodo Chess 2 64-bit (256 MB)
by Don Dailey, Larry Kaufman, Mark Lefler,
Found 145 WDL and 145 DTZ tablebase files (up to 5-man).
 20.00	 0:01 	-0.16--	1.Rd2 Nh5 (1.522.406) 912  TB:4
 20.00	 0:01 	-0.05++	1.b3 axb3 2.Bxb3 (1.727.331) 920  TB:8
 20.00	 0:02 	+0.07++	1.b3 axb3 2.Bxb3 (1.958.275) 916  TB:158
 20.00	 0:02 	+0.23++	1.b3 axb3 2.Bxb3 (2.149.177) 920  TB:184
 20.00	 0:02 	+0.18--	1.b3 axb3 (2.311.145) 915  TB:236
 20.00	 0:02 	+0.25++	1.b3 axb3 2.Bxb3 (2.559.886) 919  TB:236
 20.00	 0:02 	+0.15--	1.b3 axb3 (2.592.260) 918  TB:251
 20.00	 0:02 	 0.00 	1.b3 axb3 2.Bxb3 Ng4+ 3.Rxg4 Rd5+ 4.Bxd5 Rxd5+ 5.Kf4 fxg4 6.Ke4 e5 7.dxe5 Rxc5 8.Rd4 Ke6 9.Rd6+ Kf7 10.Kd4 Rc2 11.Rd7+ Ke6 12.Rd6+ Kf7 (2.646.281) 919  TB:251
 21.00	 0:03 	+0.07++	1.b3 axb3 2.Bxb3 (2.763.514) 920  TB:277
 21.01	 0:03 	+0.05--	1.b3 axb3 (2.817.692) 917  TB:282
 21.01	 0:03 	+0.08++	1.b3 axb3 2.Bxb3 (2.885.690) 917  TB:291
 21.01	 0:03 	+0.04--	1.b3 axb3 (2.915.722) 918  TB:292
 21.01	 0:03 	+0.02 	1.b3 axb3 2.Bxb3 Ng4+ 3.Rxg4 Rd5+ 4.Bxd5 Rxd5+ 5.Kf4 fxg4 6.Re3 Rxd4+ 7.Re4 Rd5 8.Rxb4 Rxc5 9.Kxg4 Ra5 10.Rb2 Kf7 11.f4 c5 12.Rb7+ Kf8 13.Rc7 Rxa2 14.Rc8+ (2.954.029) 917  TB:293
 22.01	 0:03 	+0.01 	1.b3 axb3 2.Bxb3 Ng4+ 3.Rxg4 Rd5+ 4.Bxd5 Rxd5+ 5.Kf4 fxg4 6.Re3 Rxd4+ 7.Re4 Rd5 8.Rxb4 Rxc5 9.Kxg4 Ra5 10.Rb7+ Kf8 11.Rb2 Ke7 12.f4 c5 13.Rb7+ Kf8 14.Rc7 (3.526.455) 921  TB:342
 23.01	 0:04 	 0.00 	1.b3 axb3 2.Bxb3 Ng4+ 3.Rxg4 Rd5+ 4.Bxd5 Rxd5+ 5.Kf4 fxg4 6.Re3 Rxd4+ 7.Re4 Rd5 8.Rxb4 Rxc5 9.Kxg4 Ra5 10.Rb7+ Kf8 11.Rb2 Ke7 12.f4 Ra7 13.Rc2 Kd6 14.Rd2+ (3.885.210) 922  TB:377
 24.01	 0:05 	+0.01 	1.b3 axb3 2.Bxb3 Ng4+ 3.Rxg4 Rd5+ 4.Bxd5 Rxd5+ 5.Kf4 fxg4 6.Re3 Rxd4+ 7.Re4 Rd5 8.Rxb4 Rxc5 9.Kxg4 Ra5 10.Rb7+ Kf8 11.Rc7 Rxa2 12.Rxc6 Rxf2 13.Rxe6 Kf7 14.Ra6 (4.808.720) 929  TB:872
 25.01	 0:06 	+0.01 	1.b3 axb3 2.Bxb3 Ng4+ 3.Rxg4 Rd5+ 4.Bxd5 Rxd5+ 5.Kf4 fxg4 6.Re3 Rxd4+ 7.Re4 Rd5 8.Rxb4 Rxc5 9.Kxg4 Ra5 10.Rb7+ Kf8 11.Rc7 Rxa2 12.Rxc6 Rxf2 13.Rxe6 Kf7 14.Ra6 (5.846.156) 939  TB:971
 26.01	 0:07 	+0.01 	1.b3 axb3 2.Bxb3 Ng4+ 3.Rxg4 Rd5+ 4.Bxd5 Rxd5+ 5.Kf4 fxg4 6.Re3 Rxd4+ 7.Re4 Rd5 8.Rxb4 Rxc5 9.Kxg4 Ra5 10.Rb7+ Kf8 11.Rc7 Rxa2 12.Rxc6 Rxf2 13.Rxe6 Kf7 14.Ra6 (7.127.658) 938  TB:1.317
 27.01	 0:08 	+0.02 	1.b3 axb3 2.Bxb3 Ng4+ 3.Rxg4 Rd5+ 4.Bxd5 Rxd5+ 5.Kf4 fxg4 6.Ke4 e5 7.dxe5 Rxc5 8.Rd4 Ke6 9.Rxb4 Rxe5+ 10.Kd3 Rxg5 11.Rc4 Kd5 12.a4 Rf5 13.Rxg4 Rxf2 14.Rg5+ (8.359.960) 937  TB:2.020
 28.01	 0:11 	-0.05--	1.b3 a3 (10.580.054) 936  TB:4.330
 28.01	 0:11 	-0.14--	1.b3 a3 (11.175.491) 939  TB:4.334
 28.02	 0:13 	-0.03++	1.Rd2 Ng4+ 2.Rxg4 (13.125.649) 948  TB:4.436
 28.01	 0:15 	-0.02 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.a4 Rc3+ 14.Kd2 (15.019.266) 950  TB:4.854
 29.01	 0:18 	-0.02 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.a4 Rc3+ 14.Kd2 (17.522.625) 944  TB:5.293
 30.01	 0:22 	-0.02 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.a4 Kxg5 14.Rc7 (21.533.525) 947  TB:6.689
 31.01	 0:25 	-0.02 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.a4 Kxg5 14.Rc7 (24.551.662) 951  TB:7.227
 32.01	 0:29 	-0.02 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.a4 Kxg5 14.Rc7 (27.871.154) 944  TB:7.467
 33.01	 0:37 	-0.01 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.Rc7 Kxg5 14.a4 (36.139.160) 955  TB:13.067
 34.01	 0:47 	-0.01 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.Rc7 Kxg5 14.a4 (45.442.561) 954  TB:18.610
 35.01	 0:57 	-0.01 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.Rc7 Kxg5 14.a4 (55.594.289) 962  TB:23.363
 36.01	 1:09 	-0.01 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.Rc7 Kxg5 14.a4 (68.098.990) 977  TB:34.857
 37.01	 1:26 	-0.01 	1.Rd2 Ng4+ 2.Rxg4 Rd5+ 3.Bxd5 Rxd5+ 4.Kf4 e5+ 5.Ke3 fxg4 6.Rd3 Ke6 7.Ke4 a3 8.bxa3 bxa3 9.Rxa3 Rxd4+ 10.Ke3 Rc4 11.Ra7 Rxc5 12.Rxg7 Kf5 13.Rc7 Kxg5 14.a4 (85.553.720) 988  TB:51.964
best move: Rd3-d2 time: 1:39.907 min  n/s: 993.379  nodes: 99.195.884 TB: 51.964 
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

lkaufman
Posts: 4884
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: what is wrong with Komodo Dragon 2 here?

Post by lkaufman » Fri Jun 11, 2021 3:02 am

frcha wrote:
Fri Jun 11, 2021 1:31 am
I get Qxf5 on any depth, even one ply. It's hard to fix a nonreproducible mistake. I never observe blunders like this.
Please check this position


This one can be reproduced if you give 256h no multipv maybe just 1 core though you can see it briefly at 8 cores:

Komodo Dragon 2 plays Bxe6?

This is an interesting position, but not for the reason you imagine. Komodo Dragon 2 on one thread does make the mistake 36.Bxe6? up thru 18 ply, then sees the danger at 19 ply and avoids it. This is quite a brilliant combination by Black, it's not surprising to me that an engine needs 19 plies to see it in view of all the pruning and reduction, especially null move in this case. It involves a mating net with quiet rook and king moves after a sacrifice, very hard to see. But Stockfish 13 on one thread plays the same mistake all the way thru 25 ply, and wakes up at 26 ply! So the mystery is, why does Komodo see it 7 plies earlier than Stockfish? Of course the programs differ, but the reductions and prunings aren't so different as to readily account for a seven ply difference! There's probably some subtle explanation. Anyway, this is not something for Komodo to fix, 19 plies is fine to see this combination.
Komodo rules!

Post Reply