Utterly neutral {and so theoretically useful for testing}

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

Dann Corbit
Posts: 12537
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Utterly neutral {and so theoretically useful for testing

Post by Dann Corbit »

michiguel wrote:
Dann Corbit wrote:
bob wrote:
Dann Corbit wrote:
bob wrote:
Dann Corbit wrote:There is some overlap with other test suites, as these positions (for instance) are also found in the Hyatt Quiet test suite:

Code: Select all

r1b2rk1/1pq2ppp/p1p1pn2/8/1b2P3/2NQB3/PPP1BPPP/R4RK1 w - - bm Rad1; id "Hyatt-Quiet.0548"; c0 "Rad1=10 ";
r1bqk2r/5ppp/p1n1p3/1pnpP3/5P2/2N2N2/PPPQ2PP/R3KB1R w KQkq - bm Qf2; id "Hyatt-Quiet.1396"; c0 "Qf2=10 ";
r2q1rk1/pb1nbppp/1pp1pn2/3p4/2PP1B2/5NP1/PPQ1PPBP/RN1R2K1 w - - bm Nc3; id "Hyatt-Quiet.1883"; c0 "Nc3=10 ";
r2q1rk1/pp2bpp1/2npbn1p/4p3/4P3/2N2N1P/PPP1BPP1/R1BQR1K1 w - - bm Bf1; id "Hyatt-Quiet.2060"; c0 "Bf1=10 ";
r2q1rk1/pp2ppbp/3pbnp1/8/2P4Q/2N3P1/PP2PPBP/R1B2RK1 w - - bm Bxb7; id "Hyatt-Quiet.2103"; c0 "Bxb7=10 ";
r2qk2r/3bbppp/p1nppn2/1p4B1/4PP2/2N2N2/PPPQ2PP/2KR1B1R w kq - bm Bxf6; id "Hyatt-Quiet.2274"; c0 "Bxf6=10 ";
r3kb1r/1bqn1ppp/pp1ppn2/8/P2NPP2/2N1BB2/1PP3PP/R2QK2R w KQkq - bm Qe2; id "Hyatt-Quiet.2729"; c0 "Qe2=10 ";
rn1q1rk1/1b2bppp/pp1p1n2/4p3/4P3/1NN1B3/PPP1BPPP/R2Q1R1K w - - bm f3; id "Hyatt-Quiet.2941"; c0 "f3=10 ";
rnbqk2r/1p2ppb1/p2p4/6pp/3NP1n1/2N3B1/PPP1BPPP/R2QK2R w KQkq - bm Bxg4; id "Hyatt-Quiet.3821"; c0 "Bxg4=10 ";
I don't want "quiet" or "==0" type positions for testing. I want a cross-section of what I might expect to encounter in real games. Of particular interest is positions with some sort of imbalance giving both sides chances to play for a win... That is, I want positions that are not dead lost or won, but also not dead even, which you rarely see in a chess game.
These are all from real games and are all frequently played. Both sides have chances to win. They are selected because neither white nor black has a clear advantage, so the normal imbalance for white is reduced.
Bert Gower and I discovered years ago that we didn't want to steer the game toward "equal" positions out of the book. We preferred positions where the eval jumped around as the search went deeper, where we had some plusses and minuses (really equal positions are way too drawish). For example, a weak queenside pawn structure in return for some king-side attacking chances. Each side then has something to "play with" and the better evaluation and search will usually find a way to exploit its own advantage without letting the opponent take advantage of his plusses...
Most of the positions in this collection are played literally thousands of times in real games. I guess that they have chances for both sides almost all the time. However, like every other collection of starting positions, it will have some drawbacks.
Are they available as PGNs?

One of the good thing about this set, is that if you find that your engine is consistently losing in one of this positions compared to others, there something to take a look at.

I observed this behavior by accident a while ago. The first position I had, I think from Nunn, Gaviota almost lost every game. Since it was the first, I almost always looked at the game after I started gauntlets. It allowed to spot a terrible tuning I had for passed pawns. I always pondered the idea to make this "hole-hunting" in openings more systematic.

Miguel
These positions are derived from a database of 4 million games of very high quality. It contains highly ranked correspondence games, super GM games, and high level computer competitors at long time control. Every position in the list has many, many games related to it. Even the least popular in the list will have at least one hundred games. All of them are fairly early in the opening phase, and if you look at them you will recognize quite a few of them. If you have questions about any particular one, I can give you a collection of related games.

All of these positions have been analyzed for several minutes by one or more top ranked computer programs. They are also selected because none of them has a single best move from the position in my database (IOW, all of them have at least three distinct moves that were played by those participants in the games).

Be that as it may, I guess that some people will not find any use for it. My intention is to have a neutral opening set especially useful for small swiss tournaments where we do not get to face all of the opponents as both colors. The hope is to remove the white bias to some extent without (at the same time) enhancing black's chances above those of white's.

I have no idea if I have succeeded in my goals for this test set, as I have not yet performed any experiments with it.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Utterly neutral {and so theoretically useful for testing

Post by bob »

michiguel wrote:
Dann Corbit wrote:
bob wrote:
Dann Corbit wrote:
bob wrote:
Dann Corbit wrote:There is some overlap with other test suites, as these positions (for instance) are also found in the Hyatt Quiet test suite:

Code: Select all

r1b2rk1/1pq2ppp/p1p1pn2/8/1b2P3/2NQB3/PPP1BPPP/R4RK1 w - - bm Rad1; id "Hyatt-Quiet.0548"; c0 "Rad1=10 ";
r1bqk2r/5ppp/p1n1p3/1pnpP3/5P2/2N2N2/PPPQ2PP/R3KB1R w KQkq - bm Qf2; id "Hyatt-Quiet.1396"; c0 "Qf2=10 ";
r2q1rk1/pb1nbppp/1pp1pn2/3p4/2PP1B2/5NP1/PPQ1PPBP/RN1R2K1 w - - bm Nc3; id "Hyatt-Quiet.1883"; c0 "Nc3=10 ";
r2q1rk1/pp2bpp1/2npbn1p/4p3/4P3/2N2N1P/PPP1BPP1/R1BQR1K1 w - - bm Bf1; id "Hyatt-Quiet.2060"; c0 "Bf1=10 ";
r2q1rk1/pp2ppbp/3pbnp1/8/2P4Q/2N3P1/PP2PPBP/R1B2RK1 w - - bm Bxb7; id "Hyatt-Quiet.2103"; c0 "Bxb7=10 ";
r2qk2r/3bbppp/p1nppn2/1p4B1/4PP2/2N2N2/PPPQ2PP/2KR1B1R w kq - bm Bxf6; id "Hyatt-Quiet.2274"; c0 "Bxf6=10 ";
r3kb1r/1bqn1ppp/pp1ppn2/8/P2NPP2/2N1BB2/1PP3PP/R2QK2R w KQkq - bm Qe2; id "Hyatt-Quiet.2729"; c0 "Qe2=10 ";
rn1q1rk1/1b2bppp/pp1p1n2/4p3/4P3/1NN1B3/PPP1BPPP/R2Q1R1K w - - bm f3; id "Hyatt-Quiet.2941"; c0 "f3=10 ";
rnbqk2r/1p2ppb1/p2p4/6pp/3NP1n1/2N3B1/PPP1BPPP/R2QK2R w KQkq - bm Bxg4; id "Hyatt-Quiet.3821"; c0 "Bxg4=10 ";
I don't want "quiet" or "==0" type positions for testing. I want a cross-section of what I might expect to encounter in real games. Of particular interest is positions with some sort of imbalance giving both sides chances to play for a win... That is, I want positions that are not dead lost or won, but also not dead even, which you rarely see in a chess game.
These are all from real games and are all frequently played. Both sides have chances to win. They are selected because neither white nor black has a clear advantage, so the normal imbalance for white is reduced.
Bert Gower and I discovered years ago that we didn't want to steer the game toward "equal" positions out of the book. We preferred positions where the eval jumped around as the search went deeper, where we had some plusses and minuses (really equal positions are way too drawish). For example, a weak queenside pawn structure in return for some king-side attacking chances. Each side then has something to "play with" and the better evaluation and search will usually find a way to exploit its own advantage without letting the opponent take advantage of his plusses...
Most of the positions in this collection are played literally thousands of times in real games. I guess that they have chances for both sides almost all the time. However, like every other collection of starting positions, it will have some drawbacks.
Are they available as PGNs?

One of the good thing about this set, is that if you find that your engine is consistently losing in one of this positions compared to others, there something to take a look at.

I observed this behavior by accident a while ago. The first position I had, I think from Nunn, Gaviota almost lost every game. Since it was the first, I almost always looked at the game after I started gauntlets. It allowed to spot a terrible tuning I had for passed pawns. I always pondered the idea to make this "hole-hunting" in openings more systematic.

Miguel
I have a pretty simple approach to this. If Crafty loses both games from the same position, that suggests attention is needed. I usually take such positions and play 4 or 8 games (rather than just 2) as you can still get that random case where you lose as black and white based mainly on luck. If you lose all 8, you can't play that particular position very well...
h1a8
Posts: 508
Joined: Fri Jun 04, 2010 7:23 am

Re: Utterly neutral {and so theoretically useful for testing

Post by h1a8 »

Thanks Dan. I would like to try these in Fritz GUI. Do you have pgn of it or can tell me how I can use these in Fritz?
Dann Corbit
Posts: 12537
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Utterly neutral {and so theoretically useful for testing

Post by Dann Corbit »

h1a8 wrote:Thanks Dan. I would like to try these in Fritz GUI. Do you have pgn of it or can tell me how I can use these in Fritz?
This possibly might work:
http://cap.connx.com/chess-engines/new- ... al.pgn.bz2

It was compressed with bzip2. You can decompress it with 7-zip, among others.
User avatar
Peter Skinner
Posts: 1763
Joined: Sun Feb 26, 2006 1:49 pm
Location: Edmonton, Alberta, Canada
Full name: Peter Skinner

Re: Utterly neutral {and so theoretically useful for testing

Post by Peter Skinner »

Dann Corbit wrote: These positions are derived from a database of 4 million games of very high quality. It contains highly ranked correspondence games, super GM games, and high level computer competitors at long time control.
Is this database available for download? It might come in handy for an opening book I am working on.

Peter
I was kicked out of Chapters because I moved all the Bibles to the fiction section.
User avatar
Werner
Posts: 2871
Joined: Wed Mar 08, 2006 10:09 pm
Location: Germany
Full name: Werner Schüle

Re: Utterly neutral {and so theoretically useful for testing

Post by Werner »

Probably the best way is to mix significant imbalanced positions (static vs. dynamic, static vs. different static, dynamic vs. different dynamic) with some not so imbalanced positions (quiet?) since BOTH do occur in real games. This not only gives a more accurate feeling of which engine is stronger but it can tell where lies the more strength.

I would also test early endgame positions. Early endgame strength is also a must for analysis.
This was exactly the intention of an engine test in a former German Computerchess Journal! It´s a pity that I have only 10 such positions - not more :)
Werner
h1a8
Posts: 508
Joined: Fri Jun 04, 2010 7:23 am

Re: Utterly neutral {and so theoretically useful for testing

Post by h1a8 »

Dann Corbit wrote:
h1a8 wrote:Thanks Dan. I would like to try these in Fritz GUI. Do you have pgn of it or can tell me how I can use these in Fritz?
This possibly might work:
http://cap.connx.com/chess-engines/new- ... al.pgn.bz2

It was compressed with bzip2. You can decompress it with 7-zip, among others.
Thank you it works. I'm testing now.
Dann Corbit
Posts: 12537
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Utterly neutral {and so theoretically useful for testing

Post by Dann Corbit »

h1a8 wrote:
Dann Corbit wrote:
h1a8 wrote:Thanks Dan. I would like to try these in Fritz GUI. Do you have pgn of it or can tell me how I can use these in Fritz?
This possibly might work:
http://cap.connx.com/chess-engines/new- ... al.pgn.bz2

It was compressed with bzip2. You can decompress it with 7-zip, among others.
Thank you it works. I'm testing now.
I will be interested to see the results.