Komodo 9.3 fails to resolve iterations ...

[tpoppins]

... before moving onto the next ones.

I'd observed a lot of truncated (2-3 plies) PVs in K9.3's output for a while already, but it wasn't until I did some extra-deep searches for Dann Corbit's Somebody needs a good pounding. Wait... It's me. thread that I realized that some of K9.3's iterations are never resolved to a full PV.

Here's a short example, edited down to d=20 through d=30 (all PVs truncated to 13 moves to avoid lines wrapping on smaller screens):

Code: Select all

FEN: r1bq1rk1/ppp3pp/2pb1n2/4N3/3P4/8/PPP1QPPP/RNB1K2R b KQ - 0 9

Komodo 9.3 64-bit:
 20+	00:01	 32,647k	16,350k	+0.10	9. ... Re8 10.0-0 c5
 20+	00:02	 34,284k	16,160k	+0.17	9. ... Re8 10.0-0 c5
 20+	00:02	 35,398k	16,093k	+0.27	9. ... Re8 10.0-0 c5
 20+	00:03	 45,777k	15,048k	+0.42	9. ... Re8 10.0-0 c5
 20-	00:03	 46,554k	14,996k	+0.30	9. ... Re8 10.0-0
 20	00:03	 46,950k	14,973k	+0.27	9. ... Re8 10.0-0 c5 11.Be3 cxd4 12.Bxd4 c5 13.Bc3 b5
 21+	00:03	 53,658k	14,699k	+0.34	9. ... Re8 10.0-0 c5
 21+	00:04	 59,393k	14,476k	+0.41	9. ... Re8 10.0-0 c5
 21-	00:04	 59,701k	14,496k	+0.34	9. ... Re8 10.0-0
 21	00:05	 73,283k	14,065k	+0.29	9. ... Re8 10.c3 c5 11.Be3 Bxe5 12.dxe5 Rxe5 13.0-0 Bg4
 22+	00:05	 81,939k	14,007k	+0.36	9. ... Re8 10.Be3 c5
 22	00:07	 99,636k	13,765k	+0.34	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Nd2 Ng4 13.Ndf3 Bxe5
 23+	00:08	 114,227k	13,738k	+0.41	9. ... Re8 10.0-0 c5
 23-	00:08	 119,107k	13,732k	+0.34	9. ... Re8 10.0-0
 23+	00:08	 123,021k	13,691k	+0.41	9. ... Re8 10.0-0 c5
 24-	00:11	 155,436k	13,705k	+0.34	9. ... Re8 10.0-0
 24+	00:12	 166,159k	13,655k	+0.41	9. ... Re8 10.0-0 c5
 25-	00:13	 183,375k	13,700k	+0.34	9. ... Re8 10.0-0
 25+	00:16	 221,362k	13,644k	+0.41	9. ... Re8 10.0-0 c5
 26	00:19	 260,556k	13,612k	+0.41	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Nd2 Ng4 13.Ndf3 Bxe5
 27	00:24	 339,548k	13,587k	+0.41	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Nd2 Ng4 13.Ndf3 Bxe5
 28-	00:34	 471,545k	13,531k	+0.34	9. ... Re8 10.0-0
 28	00:46	 619,606k	13,346k	+0.28	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Nd2 Be6 13.Rfe1 c5
 29+	01:25	 1,139,976k	13,298k	+0.35	9. ... Re8 10.0-0 c5
 29-	01:35	 1,278,295k	13,367k	+0.28	9. ... Re8 10.0-0
 29	01:37	 1,310,487k	13,387k	+0.29	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Nd2 Be6 13.Bg3 c5
 30+	02:52	 2,410,544k	13,945k	+0.36	9. ... Re8 10.0-0 c5
 30-	03:06	 2,615,646k	14,052k	+0.29	9. ... Re8 10.0-0
 30	03:26	 2,917,448k	14,157k	+0.29	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Nd2 c5 13.Rfe1 Be6

Note that iterations 23-25 are left unresolved. Please correct me if I'm wrong, it's my understanding that an engine is supposed to resolve an iteration before moving onto the next one. That's the way previous Komodo versions and Stockfish operate, at any rate. For comparison, same range of iterations on the same position, with K9.2:

Code: Select all

FEN: r1bq1rk1/ppp3pp/2pb1n2/4N3/3P4/8/PPP1QPPP/RNB1K2R b KQ - 0 9

Komodo 9.2 64-bit:
 20	00:01	 20,133k	13,169k	+0.08	9. ... Re8 10.0-0 c5 11.c3 cxd4 12.cxd4 c5 13.Be3 cxd4
 21+	00:01	 23,432k	13,176k	+0.15	9. ... Re8
 21+	00:01	 25,682k	13,066k	+0.22	9. ... Re8
 21-	00:02	 28,972k	13,079k	+0.15	9. ... Re8 10.0-0
 21	00:02	 32,621k	12,988k	+0.11	9. ... Re8 10.0-0 c5 11.Be3 cxd4 12.Bxd4 Ng4 13.f4 Bxe5
 22+	00:03	 49,455k	12,630k	+0.18	9. ... Re8
 22	00:04	 59,249k	12,452k	+0.17	9. ... Re8 10.Be3 c5 11.0-0 cxd4 12.Bxd4 Ng4 13.f4 Bxe5
 23+	00:05	 66,522k	12,432k	+0.24	9. ... Re8
 23+	00:07	 87,524k	12,223k	+0.31	9. ... Re8
 23-	00:07	 95,132k	12,172k	+0.24	9. ... Re8 10.0-0
 23	00:09	 109,861k	12,018k	+0.31	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Qc4+ Be6 13.Qxd4 c5
 24	00:10	 132,458k	12,095k	+0.32	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Qc4+ Be6 13.Qxd4 Nh5
 25+	00:13	 160,289k	12,261k	+0.39	9. ... Re8
 25+	00:14	 177,604k	12,308k	+0.46	9. ... Re8
 25-	00:19	 244,775k	12,413k	+0.39	9. ... Re8 10.Be3
 25+	00:20	 257,365k	12,432k	+0.47	9. ... Re8
 25-	00:21	 263,688k	12,447k	+0.35	9. ... Re8 10.Be3
 25	00:22	 281,387k	12,491k	+0.51	9. ... Re8 10.Be3 c5 11.0-0 cxd4 12.Bxd4 c5 13.Bc3 b5
 26-	00:24	 309,602k	12,561k	+0.44	9. ... Re8 10.Be3
 26+	00:27	 341,231k	12,629k	+0.51	9. ... Re8
 26-	00:27	 353,953k	12,647k	+0.44	9. ... Re8 10.0-0
 26+	00:36	 484,860k	13,339k	+0.52	9. ... Re8
 26-	00:36	 486,468k	13,366k	+0.40	9. ... Re8 10.0-0
 26	00:36	 486,543k	13,363k	+0.39	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Qc4+ Be6 13.Qxd4 Nh5
 27+	00:43	 578,714k	13,397k	+0.46	9. ... Re8
 27	00:45	 608,754k	13,401k	+0.47	9. ... Re8 10.0-0 c5 11.c3 cxd4 12.cxd4 c5 13.Be3 cxd4
 28	00:50	 679,808k	13,400k	+0.47	9. ... Re8 10.0-0 c5 11.c3 cxd4 12.cxd4 c5 13.Be3 cxd4
 29+	01:06	 900,748k	13,456k	+0.54	9. ... Re8
 29-	01:37	 1,324,736k	13,546k	+0.47	9. ... Re8 10.0-0
 29	01:41	 1,385,000k	13,586k	+0.39	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Qc4+ Be6 13.Qxd4 Nh5
 30	01:46	 1,458,573k	13,637k	+0.39	9. ... Re8 10.0-0 c5 11.Bf4 cxd4 12.Qc4+ Be6 13.Qxd4 Nh5

This gets worse the deeper you go, as per this example from the above thread:

Code: Select all

5rk1/5bpp/8/r2p1P2/3Bp1P1/1BR1P1K1/1n5P/2R5 w - - 0 1

Analysis by Komodo 9.3 64-bit:

<...>
1.Rc7 Rb5 2.Bxd5 Rxd5 3.Bxb2 Ra5 4.Bd4 g6 5.fxg6 hxg6 6.Rf1 Rg5 7.Rb1 Ra5 8.Rcb7 Rd5 9.Rf1 Rg5 10.Bf6 Ra5 11.Bc3 Ra2 12.g5 Rc2 13.Bd4 Rd2 14.Rf4 Rd3 15.Bc5 Rc8 16.Rbxf7 Rd5 17.Bd4 Rxg5+ 18.Kh4 Rh5+ 19.Kg4 Rh7 20.Rxh7 Kxh7 21.Kg5 Kg8 22.Rf6 Rc1 23.Rxg6+ Kf7 24.Rh6 Rc4 25.Kf4 Rc2 26.Kf5 Rf2+ 27.Kxe4 Re2 28.h4 Kg8 29.h5 
  +-  (5.71)   Depth: 41   00:26:19  29225MN, tb=18601775
1.Rc7 Rb5 
  +-  (5.64 --)   Depth: 42   00:27:46  31040MN, tb=20217602
1.Rc7 Rb5 2.Bxd5 
  +-  (5.71 ++)   Depth: 42   00:30:48  34890MN, tb=24131670
1.Rc7 Rb5 2.Bxd5 
  +-  (5.78 ++)   Depth: 43   00:36:19  41579MN, tb=29441905
1.Rc7 Nd3 2.R1c3 
  +-  (5.85 ++)   Depth: 43   00:42:46  49067MN, tb=35748023
1.Rc7 Rb5 2.Bxd5 
  +-  (5.95 ++)   Depth: 43   00:48:28  55414MN, tb=41328402
1.Rc7 g5 2.Bxb2 
  +-  (6.10 ++)   Depth: 43   00:58:16  66127MN, tb=50027158
1.Rc7 Rb5 2.Bxd5 
  +-  (6.32 ++)   Depth: 43   01:09:32  78802MN, tb=60936724
1.Rc7 Rb5 
  +-  (6.15 --)   Depth: 43   01:19:21  91133MN, tb=75647402
1.Rc7 Rb5 2.Bxd5 
  +-  (6.40 ++)   Depth: 43   01:21:39  94102MN, tb=79500375
1.Rc7 Rb5 
  +-  (6.33 --)   Depth: 44   01:24:51  98594MN, tb=88657967
1.Rc7 Rb5 
  +-  (6.26 --)   Depth: 44   01:25:51  100425MN, tb=93003582
1.Rc7 Rb5 
  +-  (6.16 --)   Depth: 44   01:27:06  102920MN, tb=99031958
1.Rc7 Rb5 2.Bxd5 
  +-  (6.24 ++)   Depth: 44   01:31:48  109271MN, tb=105771096
1.Rc7 Rb5 2.Bxd5 
  +-  (6.31 ++)   Depth: 45   01:36:04  114762MN, tb=112396701
1.Rc7 Nc4 2.Bxc4 
  +-  (6.38 ++)   Depth: 45   01:37:02  115913MN, tb=113184750
1.Rc7 Na4 2.Ra1 
  +-  (6.48 ++)   Depth: 45   01:46:39  128045MN, tb=124814103
1.Rc7 Rb5 
  +-  (6.39 --)   Depth: 45   01:54:33  138294MN, tb=134793152
1.Rc7 Rb5 
  +-  (6.17 --)   Depth: 45   01:58:58  145538MN, tb=145541509
1.Rc7 Rb5 
  +-  (5.84 --)   Depth: 45   02:01:08  149576MN, tb=151798140
1.Rc7 Rb5 2.Bxd5 
  +-  (6.09 ++)   Depth: 45   02:10:45  163776MN, tb=172701143
1.Rc7 Rb5 2.Bxd5 
  +-  (6.16 ++)   Depth: 46   02:19:53  175289MN, tb=186525508
1.Rc7 Rb5 2.Bxd5 
  +-  (6.23 ++)   Depth: 46   02:21:56  177937MN, tb=189556499
1.Rc7 Rb5 2.Bxd5 
  +-  (6.33 ++)   Depth: 46   02:26:49  184100MN, tb=195789875
1.Rc7 Rb5 
  +-  (6.24 --)   Depth: 46   02:30:02  188390MN, tb=199802063
1.Rc7 Rb5 2.Bxd5 
  +-  (6.36 ++)   Depth: 46   02:33:21  192534MN, tb=203609164
1.Rc7 Rb5 2.Bxd5 
  +-  (6.43 ++)   Depth: 47   02:39:30  200158MN, tb=212537988
1.Rc7 Nc4 2.Bxc4 
  +-  (6.50 ++)   Depth: 47   03:08:04  236080MN, tb=248990513
1.Rc7 Rb5 
  +-  (6.43 --)   Depth: 47   03:10:35  239481MN, tb=253695274
1.Rc7 Rb5 
  +-  (6.28 --)   Depth: 47   03:17:24  251845MN, tb=292211470
1.Rc7 Rb5 2.Bxd5 
  +-  (6.39 ++)   Depth: 47   03:34:18  278177MN, tb=330230790
1.Rc7 Rb5 2.Bxd5 
  +-  (6.46 ++)   Depth: 48   04:04:39  316748MN, tb=378451463
1.Rc7 Nc4 2.Bxc4 
  +-  (6.53 ++)   Depth: 48   04:10:27  323831MN, tb=383613227
1.Rc7 Nc4 2.Bxc4 
  +-  (6.63 ++)   Depth: 48   04:41:05  362412MN, tb=423018893
1.Rc7 Rb5 
  +-  (6.54 --)   Depth: 48   05:06:38  395738MN, tb=461580314

Five hours of analysis, of which only the first half-hour produced usable output. That translates into 10% efficiency, which does not a good analysis tool make.

The last example was produced under the Deep Fritz 14 interface and the following settings (only those different from default listed):
Table Memory=512 MB
Threads=11
Syzygy Probe Depth=5
Selectivity=65
Hash=16 GB

To eliminate the GUI and my non-default settings as suspects I did the first two examples under Arena 3.5, with 4 GB hash and both engines at default settings. Several runs with each were done to establish that the problem is reproducible (although the affected iterations may differ from run to run). As you can see, the problem is not with the GUI or the settings.

Perhaps it's not too late to have this fixed in the upcoming v9.4? Larry and Mark (and other engine authors) may also find this thread of interest: Always full principal variation.

[mjlef]

This was by design. In some cases, when playing a game, if there is enough evidence a move will be found "best" or "good enough" we found it better to start the next iteration instead of taking the time to try and resolve the full PV. This saves a surprising amount of time, but is annoying for user wanting to see the full line. So in our development version, one of the changes is if Komodo is in UCI Analysis mode, it will spend the extra time to resolve the PV. Expect to see it in a new release soon. Note that regular Komodo 9.3 will eventually reach a depth and show the PV, but as you have noted, this could take more time than practical.

[gordonr]

So in our development version, one of the changes is if Komodo is in UCI Analysis mode, it will spend the extra time to resolve the PV. Expect to see it in a new release soon.

For the new release, what if I'm e.g. solving a hard tactical test where I'm only interested in finding the correct first ply. If I do this in analysis mode, is it going to be significantly slower than if it had been a real game situation? If so, how much slower due to resolving the PV? Is this "resolve PV" a UCI option?

Thanks.

[mjlef]

So in our development version, one of the changes is if Komodo is in UCI Analysis mode, it will spend the extra time to resolve the PV. Expect to see it in a new release soon.

For the new release, what if I'm e.g. solving a hard tactical test where I'm only interested in finding the correct first ply. If I do this in analysis mode, is it going to be significantly slower than if it had been a real game situation? If so, how much slower due to resolving the PV? Is this "resolve PV" a UCI option?

Thanks.

It is automatic whenever using Analysis mode. We did not make it a separate UCI option since GUIs have a limit to the number of options they support. I cannot give you an answer on how much slower it will be since this is very position dependent. Also the specific search has conditions to trigger this so they have to be met before it jumps to the next iteration. This only happens during "regular search" (UCI_AnalyseMode set to false), so assuming your GUI gives you access to that parameter you can control when it is used. Note this command is already part of the UCI specification.

[gordonr]

It is automatic whenever using Analysis mode.

Sorry, I didn't realise this was an explicit option in the UCI specification. I just thought that "analysis mode" meant "go infinite". Thanks for clarifying.

[Leto]

It is automatic whenever using Analysis mode.

Sorry, I didn't realise this was an explicit option in the UCI specification. I just thought that "analysis mode" meant "go infinite". Thanks for clarifying.

Analysis mode is "infinite analysis" in Fritz/Chessbase gui's.

[tpoppins]

Thank you for the prompt reply, Mark.

This was by design. In some cases, when playing a game, if there is enough evidence a move will be found "best" or "good enough" we found it better to start the next iteration instead of taking the time to try and resolve the full PV. This saves a surprising amount of time, but is annoying for user wanting to see the full line.

Yes, I can see the arguments for both sides here. On the one hand, a wait for the full PV can seem interminable, esp. if the box that does the analysis is your only box; although, true, this is not something unique to Komodo (Houdini Tactical is even worse in that respect). On the other hand, it's not just about the full PV (half of it is most likely junk anyway, unless it's a long forced sequence): the evals shown for the fail-highs and fail-lows are for positions that would never be reached via optimal play. That severely limits the usefulness of v9.3 as an analysis tool for those who like to go deep (and hard ;), so it's back to v9.2 for me for the time being.

So in our development version, one of the changes is if Komodo is in UCI Analysis mode, it will spend the extra time to resolve the PV. Expect to see it in a new release soon. Note that regular Komodo 9.3 will eventually reach a depth and show the PV, but as you have noted, this could take more time than practical.

"more time than practical" - I like the sound of it, almost silky-smooth, heh. Well, the important thing is that you guys recognize an ... errr... sub-optimal design decision for what it is and already have a fix ready.

Extra credit for listening to your users and actually implementing some of the things they want to see in, which makes most Stockfish developers' posts on here look ... err, where's my English-PC dictionary again? ... ah yes, what I meant to say is, which makes a rather favorable impression next to some of Stockfish developers' posts on here.

For the new release, what if I'm e.g. solving a hard tactical test where I'm only interested in finding the correct first ply.

I think Sting, Mate Finder and Houdini Tactical are a lot better suited for a task like that; though an idea of a Swiss Knife-like engine is enticing (I wouldn't mind if Komodo could do all the things those three can) I don't know how practical an implementation might be.

This only happens during "regular search" (UCI_AnalyseMode set to false), so assuming your GUI gives you access to that parameter you can control when it is used. Note this command is already part of the UCI specification.

Yes, but AFAIK, most popular GUIs don't give the user that access - CB/Fritz and Arena immediately come to mind. If GUI limits/screen real estate are a concern, you could consider ditching the two Book options - the book hasn't been updated in years, so I wonder if anyone at all still uses it. That would make room for Gordon's req and one of my own - a "Suppress Fail-Highs and Fail-Lows in Analysis Output" option à la Dann's build of Mate Finder.

[mjlef]

It is automatic whenever using Analysis mode.

Sorry, I didn't realise this was an explicit option in the UCI specification. I just thought that "analysis mode" meant "go infinite". Thanks for clarifying.

Analysis mode is "infinite analysis" in Fritz/Chessbase gui's.

I spend way too much time programming around GUI issues. But anyway, we will have Komodo accept a "go infinite" command as meaning to set the UCI parameter for analysis mode (UCI_AnalyseMode). Anyone wishing to try something radical, like following the specs in their GUI can find them near the bottom of this page:

http://www.shredderchess.com/download.html