The 600 most interesting positions in the world, part 1

[Dann Corbit]

Here are half of them, which I describe as "the 300 Spartans, by day":
http://cap.connx.com/EPD/300d.epd.bz2

Now, if you download those positions, you are going to object.
"Those positions are not interesting at all! In fact, they are all well known opening positions with well established theory. Those are probably among the most boring positions in the world!"

But now, for my claim:
I have spent a good deal of time with computer analysis of the most commonly played positions. For those positions reached at least 1000 times in real games in my "database of high quality games" I have 17,435 positions where the analysis by computer agrees completely with the move chosen by opening books, human experts, etc. But I also have about 600 positions where the computer analysis and the human analysis simply do not agree, after a search of at least 26 plies.

For most of these positions, the popular choice is probably the correct choice, but the computer analysis simply has not found it yet. So my search goes like this:
Analyse the positions in a batch and reprocess them. For all of those still not solved, simply double the time and do it again. Currently, I am at one hour per position (though some of the positions have been analyzed at much longer time scale).

So, why are these positions interesting?
1. I guess that for 1% of the puzzle positions, there are real novelties found by the computer analysis. For these positions, we may have a trap to spring on our opponent.
2. For the other positions, it seems that computers do not understand the positions properly. So it seems likely that when we fall out of book, the computer may not know what to do if the book lines are not deep enough.
3. Though the positions seem simple, there must be more than meets the eye, since a very deep computer analysis does not see the simple solution.
4. These positions come up all the time in games. So you as a human are going to play them if you play chess. If you are a chess engine author, your chess engine is going to play them frequenly also.
5. Why is this 6% different than the other 93% that the computer programs can solve easily?

OK, OK, time to qualify my subject string:
The 600 most interesting positions in the world, part 1 {to me anyway}.

Here is a sample position:
[d]r1bqkb1r/pppp1ppp/2n2n2/4p3/2P5/2N2N2/PP1PPPPP/R1BQKB1R w KQkq - acd 34; ce 0; pv e4 Bb4 d3 d6 Be2 O-O O-O Bg4 Bg5 h6 Be3 Bc5 Nd5 Nd7 Nd2 Bxe2 Qxe2 Nd4 Bxd4 Bxd4 Nf3 c6 Ne3 Bxe3 fxe3 a5 g3 Nc5 Nh4 g6 Rf2 Qe7 Raf1 a4 Qd2 Kh7 Qe2 Kg8; bm e4; pm g3 {23077} e3 {4077} a3 {1324} d4 {829} d3 {745} e4 {591} Qc2 {224} Qa4 {102} b3 {95}; id "300SpartansDaytime.300";

Here is a discussing of the non-position components:

acd 34;
Depth of analysis was 34 plies (analysis engine will be a top level engine such as Rybka 3 or Stockfish 1.71)

ce 0;
Computer score was 0 centipawns

pv e4 Bb4 d3 d6 Be2 O-O O-O Bg4 Bg5 h6 Be3 Bc5 Nd5 Nd7 Nd2 Bxe2 Qxe2 Nd4 Bxd4 Bxd4 Nf3 c6 Ne3 Bxe3 fxe3 a5 g3 Nc5 Nh4 g6 Rf2 Qe7 Raf1 a4 Qd2 Kh7 Qe2 Kg8;
The computer's best guess at a plan is shown in the above trajectory.

bm e4;
The computer's key move choice was e4

pm g3 {23077} e3 {4077} a3 {1324} d4 {829} d3 {745} e4 {591} Qc2 {224} Qa4 {102} b3 {95};
For this position, from my database of high quailty games, the list of possible predicted moves is found above. The number of times the move occured in actual games is listed beside the move choice in curly-brace comments. Move choices that occurred less than 10 times are not included in the list. The dominatingly most popular move for this position is g3 (NOT the computer choice of e4) which occurred 23,077 times in actual games in the database. The move e4 happend only 591 times in the database. The database has been carefully filtered so that only top level GMs, correspondence players, and engines are contained in it.

id "300SpartansDaytime.300";
This is the tag to uniquely identify the position.

[Marek Soszynski]

The set of positions could form a kind of test for the most human engine (or least computer-like). I'd very much like to see how reputedly human engines fare - Rybka Human, HIARCS, and Komodo.

[Dann Corbit]

The set of positions could form a kind of test for the most human engine (or least computer-like). I'd very much like to see how reputedly human engines fare - Rybka Human, HIARCS, and Komodo.

The other half will be available in a few days (currently, the other set has 304 positions and I want to solve 4 more before I release it so that it will be a symmetrical set of 300 + 300 positions.)

[Dann Corbit]

The set of positions could form a kind of test for the most human engine (or least computer-like). I'd very much like to see how reputedly human engines fare - Rybka Human, HIARCS, and Komodo.

Another thing worthy of note is that some of the disagreements between human and computer do not arise until a very deep search has been performed.

A classic example of this is the opening position of chess. The pedagogic answer is "1.e4" but the computer analysis answer is "1.d4" currently. Now, all the strong engines will pick 1.e4 after just a few seconds of analysis. But at ply 30 or 31, they all seem to switch to 1.d4. I guess that they will switch back eventually, but I simply have not found it yet.

So the test may not produce the desired result since many of the positions are initially solved with the common move, but reject it with a deeper search (and you can often see the engine bouncing back and forth between various answers as the plies increase).

[Allard Siemelink]

Interesting tessuite!
I ran a quick test at 2 seconds per position. This probably means nothing yet Spark found human 268 moves out of 300. On average agreeing
with 3200 human players.

Perhaps it is an idea to add scores for the human moves based on frequency of play?
For example, the most frequently played human move could always get 10 points, and all others something less, based on some formula.
E.g. points=10*f/fmax, where fmax is the frequency of the move played most often and f is the frequency of the move in question.
E.g. for the opening position you have e4=540408, d4=516324 c4=222732 Nf3=185503 g3=19536 b3=4189 and so on.
Then, we would have the following points: e4=10, d4=9 Nf3=3, g3=0, b3=0

[Dann Corbit]

Interesting tessuite!
I ran a quick test at 2 seconds per position. This probably means nothing yet Spark found human 268 moves out of 300. On average agreeing
with 3200 human players.

Perhaps it is an idea to add scores for the human moves based on frequency of play?
For example, the most frequently played human move could always get 10 points, and all others something less, based on some formula.
E.g. points=10*f/fmax, where fmax is the frequency of the move played most often and f is the frequency of the move in question.
E.g. for the opening position you have e4=540408, d4=516324 c4=222732 Nf3=185503 g3=19536 b3=4189 and so on.
Then, we would have the following points: e4=10, d4=9 Nf3=3, g3=0, b3=0

A logical score might be:
10 * (Count for position chosen) / (Count for most frequent move) as you have suggested.
Or possibly as an alternative:
100 * (Count for position chosen) / (Count for all move choices)

There are other notions that might also work well.

[Allard Siemelink]

A logical score might be:
10 * (Count for position chosen) / (Count for most frequent move) as you have suggested.
Or possibly as an alternative:
100 * (Count for position chosen) / (Count for all move choices)

There are other notions that might also work well.

It would be great if you could add either one of these, or an alternate notion (which?).
I for one would be interested how the various engines
(including Spark of course) compare when we tally these scores.

How much time per position would be reasonable, you'd say?

[Dann Corbit]

A logical score might be:
10 * (Count for position chosen) / (Count for most frequent move) as you have suggested.
Or possibly as an alternative:
100 * (Count for position chosen) / (Count for all move choices)

There are other notions that might also work well.

It would be great if you could add either one of these, or an alternate notion (which?).
I for one would be interested how the various engines
(including Spark of course) compare when we tally these scores.

How much time per position would be reasonable, you'd say?

Hard to say. I think that the answers will definitely move over time.

Right now, I am running them at one hour, but I guess that time control will be a little unpopular.

[perejaslav]

In the second position pv-line is totally incorrect

rnbqkb1r/pp2pp1p/3p1np1/8/3NP3/2N5/PPP2PPP/R1BQKB1R w KQkq - acd 26; ce 16; pv a3 Bg7 Bb5+ Bd7 O-O O-O Bg5 Bxb5 Ndxb5 Nc6 f3?? Qb6+ Kh1 Rfc8; bm a3; pm Be3 {4980} Be2 {614} Bc4 {401} g3 {193} f4 {57} Bb5+ {55} Bg5 {46} f3 {43} h3 {37} Nd5 {21}; id "300SpartansDaytime.002";

I think it's better to remove such pv-lines from thе Suite at all.

[Marc MP]

This one is hard to believe (number 2):

[D]rnbqkb1r/pp2pp1p/3p1np1/8/3NP3/2N5/PPP2PPP/R1BQKB1R w KQk - 0 1

The bm is a3. oops! Did both SF and R3 agreed on this one??

Interesting experiment anyways!