OKE - Opening Knowledge Engines

[Dann Corbit]

I think the massive Lichess stuff perhaps is good for NN engines. Or to extract games on elo. Else I see no usefulness. But perhaps you might want to elaborate.

What I mean is there are 700 million games here:
https://database.lichess.org/

I have a parser that will collect win loss draw (and any other available information) from the games.
You can (of course) filter for Elo, but I do not think that is as important as most people think because the bad moves will still lose and the good moves will still win.
So you get data with this format (current Lichess data has time control data for the moves):
[Epd], [Engine/Player], [acs], [pm], [result]
When you have a million instances of an Epd record (there are many billions of Epd records in the data from Lichess) then you can get a good idea of how good a move is from the outcome percentages.
https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

[Dann Corbit]

1. you need to start later in the opening

I think it is completely irrelevant where you start. A fully closed position is difficult, like locked positions, but some computers can find good answers.

2. you have to carefully check the scoring system, which cannot be done automatically

Math you can do in your head you can do with a computer.
I do agree, however, that this is a very difficult problem

3. you need to decide whom to trust (SF LC0 Komodo Houdini, all can fail here too, moreover
it would introduce bias trusting one of them, it seems inevitable to consult opening encyclopedies
based on decades of human knowledge, crosschecked by all of them and then add scoring intelligently,
not black/white or 99:1, if the average result will be the same)

I have no special trust in any engine. I do not necessarily even trust it when all the engines agree. I trust when all factors combine in agreement (win/loss/draw, stats, GM opinion, and computer eval) and otherwise I still have doubts.

[Guenther]

1. you need to start later in the opening

I think it is completely irrelevant where you start. A fully closed position is difficult, like locked positions, but some computers can find good answers.

It is totally relevant because with only 2! up to 4 moves you are left with much too much equal moves, which are just a matter of taste/style.
2-4 moves is the current discussion for OKE, it is incomprehensible for me how you can dispute about this.
The whole discussion lacks of logic as most threads here, thus I leave it here.

[Rebel]

I think you could create a very good statistics based book by using the lichees data.
It's the crowd wisdom metaphor.
True, you can't get a perfect book jut via statistics.
But you can't get a perfect book any other way either.
If you can build a perfect book then you can solve chess.
We're not there yet.

I don't know, whom you want to answer Dann (I guess Ed), but we are not anymore in a thread about building
an opening book, but about constructing an opening test for engines. (OKE)

With the current methodology it fails miserably. I repeat it again, the idea is good, but

1. you need to start later in the opening
2. you have to carefully check the scoring system, which cannot be done automatically
3. you need to decide whom to trust (SF LC0 Komodo Houdini, all can fail here too, moreover
it would introduce bias trusting one of them, it seems inevitable to consult opening encyclopedies
based on decades of human knowledge, crosschecked by all of them and then add scoring intelligently,
not black/white or 99:1, if the average result will be the same)

Your point is well understood yet it is not helpful to repeat it without answering my comments above. On the red - it only fails if you are expecting a perfect solution, there isn't and yours is mission impossible also as exaplained in the post you ignored so far.

What the evidence shows is that there is a lot to improve in an area which did not get the attention it deserved, or worse, is neglected since rating lists give programmers the first 4-6-10 moves in advance, so why bother.

I can tell in advance that current run (2-moves.epd - 6728 positions) compared to the old one (404 positions) does not look very different, the pattern is clear in the meantime. 2-moves.epd attached. Please start the monk work making the manual corrections

[Laskos]

Well Ed, why bother? We have the excellent "Strategic Test Suite" which "consists of series of themed test suites designed to evaluate chess engine's long term understanding of strategical and positional concepts" (self-description). Dann and his Rybka were the main contributors to it. Here are some current standings on "strategical and positional concepts" (have no patience testing more engines):

Code: Select all

Stuck to the solution from 1s to 2s of thinking
4 i7 cores at 3.80GHz
RTX 2070 GPU

Houdini 1.5a        score=1339/1500 [averages on correct positions: depth=8.4 time=0.15 nodes=1320734]
Komodo 13.02        score=1319/1500 [averages on correct positions: depth=10.6 time=0.16 nodes=1064120]
Stockfish dev       score=1284/1500 [averages on correct positions: depth=10.8 time=0.18 nodes=1111414]  
Texel 1.07          score=1241/1500 [averages on correct positions: depth=8.9 time=0.22 nodes=1379498]
Arasan 21.0         score=1195/1500 [averages on correct positions: depth=9.5 time=0.24 nodes=1017178]
Lc0 v21.2 ID42524   score=1177/1500 [averages on correct positions: depth=3.8 time=0.14 nodes=1580]
Fruit 2.1           score= 993/1500 [averages on correct positions: depth=5.2 time=0.23 nodes=505132]

I guess your suite might converge to STS as standings go. Good!

[Rebel]

Well Ed, why bother? We have the excellent "Strategic Test Suite" which "consists of series of themed test suites designed to evaluate chess engine's long term understanding of strategical and positional concepts" (self-description). Dann and his Rybka were the main contributors to it. Here are some current standings on "strategical and positional concepts" (have no patience testing more engines):

Code: Select all

Stuck to the solution from 1s to 2s of thinking
4 i7 cores at 3.80GHz
RTX 2070 GPU

Houdini 1.5a        score=1339/1500 [averages on correct positions: depth=8.4 time=0.15 nodes=1320734]
Komodo 13.02        score=1319/1500 [averages on correct positions: depth=10.6 time=0.16 nodes=1064120]
Stockfish dev       score=1284/1500 [averages on correct positions: depth=10.8 time=0.18 nodes=1111414]  
Texel 1.07          score=1241/1500 [averages on correct positions: depth=8.9 time=0.22 nodes=1379498]
Arasan 21.0         score=1195/1500 [averages on correct positions: depth=9.5 time=0.24 nodes=1017178]
Lc0 v21.2 ID42524   score=1177/1500 [averages on correct positions: depth=3.8 time=0.14 nodes=1580]
Fruit 2.1           score= 993/1500 [averages on correct positions: depth=5.2 time=0.23 nodes=505132]

I guess your suite might converge to STS as standings go. Good!

You can't compare STS with OKE. STS was created with the help of the top engines of that time, all the Rybka derivatives will top the STS list not only Houdini 1.5

[Dann Corbit]

1. you need to start later in the opening

I think it is completely irrelevant where you start. A fully closed position is difficult, like locked positions, but some computers can find good answers.

It is totally relevant because with only 2! up to 4 moves you are left with much too much equal moves, which are just a matter of taste/style.
2-4 moves is the current discussion for OKE, it is incomprehensible for me how you can dispute about this.
The whole discussion lacks of logic as most threads here, thus I leave it here.

Most chess positions have 2 to 4 good choices. The reason a lot of test sets only have one good move is that they were designed for this.
When you analyze a position with several engines, you will often get more than one choice.
Then you analyze with multi-pv and you get a plethora of choices.
When you combine this with the won/lost/drawn statistics for each of the moves you have a very good understanding of the choices.

I don't understand why you think it is so absurd to try.

Let's look at the opening position:
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq -
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 70; bm c4; pm e4 {2379495} d4 {1725526} Nf3 {483679} c4 {479713} g3 {39820} f4 {35589} b3 {27764} Nc3 {13010} e3 {2686} b4 {2458} g4 {2450} c3 {1654} d3 {1649} a3 {1150} h3 {717} a4 {552} h4 {425} f3 {363} Nh3 {82} Na3 {28}; pv c4 c5 Nf3 Nf6 g3 d5 d4 dxc4 Qa4 Nc6 dxc5 e6 Bg2 Bxc5 Nc3 Qa5 Qxc4 O-O O-O Bd7 Ne4 Be7 Bd2 Qf5 Nxf6 Bxf6 Bc3 e5 Nd2 Rac8 Qb3 Rc7 Bd5 b5 a3 a6 Rac1 Be7 e3 Rfc8 f4 Be6 Bxe6 Qxe6 Qxe6 fxe6 Ne4 a5 Bxe5 Nxe5 Rxc7 Rxc7 fxe5 Rc4 Rf4 Bc5 Nxc5 Rxc5 Kf1 Rc1 Kg2 Rc5 b4 axb4 axb4 Rc2 Rf2 Rc3 Kf3 Rc4 Rd2 Rxb4 Rd7 g5 g4 Rb2 Rb7 Rxh2 Rxb5; white_wins 2132; black_wins 5580; draws 1068; Opening Fundamental opening position;
computer analysis said c4 at a depth of 70.
The other choices analyzed:
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm a3; pv a3 c5 c3 d5 d4 e6 Bf4 Nf6 e3 Bd6 Nf3 Nc6 dxc5 Bxc5 c4 0-0 Nc3 d4 exd4 Nxd4 Be2 Nxf3+ Bxf3 Bd4 Nb5 Bxb2 Rb1 e5 Rxb2 Qxd1+ Bxd1 exf4 Nc7 Rb8 Na6 Ra8 Nc7;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm a4; pv a4 d5 d4 Bf5 c3 e6 Bf4 Bd6 e3 Bxf4 exf4 Ne7 Nf3 0-0 Be2 Nd7 0-0 c5 dxc5 Nxc5 Nd4 Qb6 Ra2 Bg6 g3 Nc6 a5 Qc7 Re1 Rac8 Ra3 Ne4 a6 Nxd4 Qxd4 b6 Nd2 Nd6 Qe3 Rfd8 Raa1 Bf5 g4 Be4 Nxe4 Nxe4 Rad1 h6;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm b3; pv b3 d5 Nf3 Nf6 Bb2 Bg4 e3 e6 Be2 h6 0-0 Nbd7 c4 c6 d4 Bd6 c5 Bc7 Nbd2 Bf5 b4 0-0 a4 Ne4 b5 Nxd2 Nxd2 Qf6 Nf3 Rab8 Rc1 Rfd8 Bc3 Qe7 h3 Be4 Nd2 Bf5 Nf3;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm b4; pv b4 e5 Bb2 Bxb4 Bxe5 Nf6 e3 0-0 c4 d5 Nf3 Bf5 a3 Bd6 d4 Bxe5 dxe5 Ne4 cxd5 c6 Be2 cxd5 0-0 Qe7 Qxd5 Rd8 Qb3 Nc6 Nc3 Nc5 Qa2 Nxe5 Nd4 Bg6 Nd5 Qd7 Nf4 Be4 Rfd1 Qe7 f3 Bc6 Nf5 Qf6 Nd4 Rac8 Rac1 b6 Qc2;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm c3; pv c3 d5 Nf3 c5 d4 Nf6 Bf4 Nc6 e3 Nh5 dxc5 Nxf4 exf4 e6 b4 Qf6 g3 Bd7 Bg2 Nxb4 Ne5 Nc6 Nxd7 Kxd7 0-0 Kc7 Na3 Bxc5 Nb5+ Kb8 c4 a6 cxd5 exd5 Rc1 Bb4 Qxd5 axb5 Qxb5 Ra5 Qb6 Ra6 Qb5 Ra5;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 55; bm d4; pv d4 d5 c4 e6 Nc3 Nf6 cxd5 exd5 Bg5 c6 e3 Bd6 Bd3 O-O Qc2 h6 Bh4 Re8 Nge2 Qe7 O-O Nbd7 h3 Nf8 Bg3 b6 Rfd1 Bd7 Rab1 a5 Bxd6 Qxd6 Nf4 Ne6 Nce2 Rac8 Nxe6 Qxe6 Nf4 Qe7 Ba6 Rb8 Be2 Ne4 Bd3 Rbc8 Rbc1 Qg5 a3 c5 Bxe4 dxe4 dxc5 Rxc5 Qd2 Rxc1 Qxc1;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm e3; pv e3 Nf6 Nf3 d5 c4 e6 b3 Be7 d4 0-0 Bd3 dxc4 bxc4 c5 Bb2 b6 0-0 Bb7 Nc3 cxd4 exd4 Nc6 Rc1 Re8 Qd2 Bd6 Nb5 Bb4 Qe3 Bf8 h3 a6 Nc3 Ne7 Ne4 Nxe4 Bxe4 Nf5 Bxf5 exf5 Qb3 Be4 Rfe1 Qc7 d5 Bd6 Bd4 Rab8 a4 Bf4;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm e4; pv e4 e6 d4 d5 Nd2 c5 exd5 Qxd5 Ngf3 cxd4 Bc4 Qd6 0-0 Nf6 Re1 Be7 Nb3 Nc6 Nbxd4 Nxd4 Nxd4 0-0 c3 Qc7 Bd3 Rd8 Qf3 Bd6 h3 Be5 Be3 Bd7 Rad1 h6 Nb5 Bxb5 Bxb5 Rd5 Rxd5 Nxd5 Bd3 Rd8 Qe4 f5 Qc4 a6 Bd2 Qd6 Qb3 Bh2+ Kh1;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm f3; pv f3 e5 Nc3 Nc6 e4 Nf6 Bb5 Nd4 Nge2 c6 Bd3 Bc5 Na4 Bd6 0-0 Ne6 Bc4 0-0 Bxe6 fxe6 d3 b5 Nac3 Bc5+ Kh1 d5 a3 Qb6 Qe1 Bb7 Bg5 Bd6 Nd1 c5 Nf2 Qc7 Ng3 a5 Qd2 Bc6 exd5 exd5 Nf5 h6 Nxd6 Qxd6 Bxf6 Rxf6 Ng4;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm f4; pv f4 Nf6 Nf3 c5 Nc3 d5 e3 g6 Bb5+ Bd7 Qe2 Nc6 Ne5 Qc7 Nxd7 Qxd7 d3 Bg7 e4 0-0 Bd2 a6 Bxc6 Qxc6 e5 Nd7 0-0-0 d4 Ne4 f5 exf6 exf6 Rhe1 Rfe8 Qf3 f5 Ng5 Qxf3 Nxf3 Nb6 g3 Nd5 Ne5 b6 Nc4 Bf6 b3 b5 Ne5 Bxe5 fxe5 Kf7 Kb2;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm g3; pv g3 c5 Bg2 Nc6 Nf3 g6 c4 Bg7 Nc3 d6 0-0 Bf5 d3 Qd7 Rb1 Bh3 a3 Nf6 b4 Bxg2 Kxg2 0-0 Bg5 Nh5 Qd2 Bxc3 Qxc3 f6 Bh6 Rfe8 b5 Ne5 Nxe5 dxe5 f3 Ng7 Bxg7 Kxg7 a4 Red8 a5 Rac8 Ra1 Ra8 Rfb1 Rac8 Qd2 Ra8 b6 Qd4 bxa7;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm g4; pv g4 d5 c4 Bxg4 Qb3 dxc4 Qxb7 Nd7 Na3 Rb8 Qg2 Be6 Nh3 Nb6 Ng5 Bd5 e4 h6 Nh7 Rxh7 exd5 Qxd5 Qxd5 Nxd5 Nxc4 e6 b3 Nb4 Ke2 Ne7 Bb2 g5 h4 g4 Bg2 Ng6 Be5 Nxe5 Nxe5 f5 Nc6 Nxc6 Bxc6+ Ke7 Rac1 Kf6 d4 Bd6 Rhe1 Rb6 Rc2 h5 Rec1 a6 Rc4 Ba3 R1c2;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm h3; pv h3 d5 d4 c5 e3 Nf6 c4 cxd4 exd4 Nc6 Nf3 g6 c5 Bg7 Nc3 0-0 Bb5 Ne4 0-0 Qa5 Bxc6 bxc6 Bf4 Nxc3 bxc3 Qxc3 Re1 Bf5 Rxe7 Rfe8 Qe1 Qxe1+ Raxe1 Be4 Rxe8+ Rxe8 Bd6 Rc8 Ng5 Bf5 Nf3 Be4;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm h4; pv h4 d5 d4 c5 c3 Nc6 Nf3 e6 Bf4 Bd6 Bxd6 Qxd6 e3 Nf6 Bb5 0-0 0-0 Bd7 a4 a6 Bxc6 Bxc6 Ne5 Ne4 Nxc6 bxc6 Nd2 Nxd2 dxc5 Qxc5 Qxd2 e5 h5 h6 Qe2 Rab8 e4 a5 exd5 cxd5 Rad1 Rfe8 Rfe1 f6 Rd2 d4 Qd3 Qd5 cxd4;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm Na3; pv Na3 e5 c4 c6 d4 exd4 Nf3 Nf6 Qxd4 d5 e3 a6 cxd5 cxd5 Nc2 Nc6 Qd1 Bd6 Be2 0-0 0-0 Ne4 Ncd4 Re8 Bd2 Bd7 Rc1 Qb6 Bc3 Rac8 h3 h6 Nxc6 bxc6 Bd3 Bf5 Nd4 Bg6 Bxe4 Bxe4 b4 c5 bxc5 Bxc5 Qg4 Qg6 Qxg6 Bxg6 Rfd1 Be4;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm Nc3; pv Nc3 d5 d4 Nf6 Bf4 e6 Nb5 Bd6 Nxd6+ cxd6 c3 Nc6 Nf3 0-0 Bg3 Qe7 e3 e5 Be2 a6 0-0 h6 Re1 Bf5 h3 Rfe8 Nd2 Qd7 Qb3 Qc7 Bh4 g5 Bg3 Bg6 Qd1 Kg7 Rc1 Rad8 Qb3 Na5 Qb4 Nc6 Qb3;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm Nf3; pv Nf3 d5 d4 Nf6 c4 c6 cxd5 cxd5 Nc3 Nc6 Bf4 Bf5 e3 e6 Ne5 Nxe5 Bxe5 Nd7 Bg3 h5 h3 Rc8 Rc1 Qb6 Qd2 h4 Bf4 Bd6 Bxd6 Qxd6 Bd3 Bxd3 Qxd3 0-0 0-0 Nf6 Rc2 Qb8 Rfc1 a6 e4 Qf4 e5 Ne4 Nxe4 Rxc2 Rxc2 dxe4 Qe3;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; bm Nh3; pv Nh3 d5 d4 Nf6 Nf4 c5 e3 Nc6 Bb5 Qb6 Nc3 e6 0-0 Be7 dxc5 Qxc5 Nd3 Qd6 b3 0-0 Ne2 Nb8 Ba4 a6 b4 Bd7 Bb3 a5 bxa5 Nc6 a4 Rfc8 Bb2 e5 c4 Be6 c5 Qc7 Nc3 Qd8 Nb5 Ne4 Nxe5 Nxa5 c6 bxc6 Nd4 Qb6 f3 Nxb3;

Which of these evaluations seems in error to you?
It should be apparent that we can use depth and evaluation and statistical results to come up with a pretty good score for each move.

[Rebel]

I think the massive Lichess stuff perhaps is good for NN engines. Or to extract games on elo. Else I see no usefulness. But perhaps you might want to elaborate.

What I mean is there are 700 million games here:
https://database.lichess.org/

I have a parser that will collect win loss draw (and any other available information) from the games.
You can (of course) filter for Elo, but I do not think that is as important as most people think because the bad moves will still lose and the good moves will still win.
So you get data with this format (current Lichess data has time control data for the moves):
[Epd], [Engine/Player], [acs], [pm], [result]
When you have a million instances of an Epd record (there are many billions of Epd records in the data from Lichess) then you can get a good idea of how good a move is from the outcome percentages.
https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

Got it.

Volume rules.

[Dann Corbit]

Left the scores off the alternatives. Here they are with scores:

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 70; bm c4; pm e4 {2379495} d4 {1725526} Nf3 {483679} c4 {479713} g3 {39820} f4 {35589} b3 {27764} Nc3 {13010} e3 {2686} b4 {2458} g4 {2450} c3 {1654} d3 {1649} a3 {1150} h3 {717} a4 {552} h4 {425} f3 {363} Nh3 {82} Na3 {28}; pv c4 c5 Nf3 Nf6 g3 d5 d4 dxc4 Qa4 Nc6 dxc5 e6 Bg2 Bxc5 Nc3 Qa5 Qxc4 O-O O-O Bd7 Ne4 Be7 Bd2 Qf5 Nxf6 Bxf6 Bc3 e5 Nd2 Rac8 Qb3 Rc7 Bd5 b5 a3 a6 Rac1 Be7 e3 Rfc8 f4 Be6 Bxe6 Qxe6 Qxe6 fxe6 Ne4 a5 Bxe5 Nxe5 Rxc7 Rxc7 fxe5 Rc4 Rf4 Bc5 Nxc5 Rxc5 Kf1 Rc1 Kg2 Rc5 b4 axb4 axb4 Rc2 Rf2 Rc3 Kf3 Rc4 Rd2 Rxb4 Rd7 g5 g4 Rb2 Rb7 Rxh2 Rxb5; Opening Fundamental opening position; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 0; bm a3; pv a3 c5 c3 d5 d4 e6 Bf4 Nf6 e3 Bd6 Nf3 Nc6 dxc5 Bxc5 c4 0-0 Nc3 d4 exd4 Nxd4 Be2 Nxf3+ Bxf3 Bd4 Nb5 Bxb2 Rb1 e5 Rxb2 Qxd1+ Bxd1 exf4 Nc7 Rb8 Na6 Ra8 Nc7; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce -15; bm a4; pv a4 d5 d4 Bf5 c3 e6 Bf4 Bd6 e3 Bxf4 exf4 Ne7 Nf3 0-0 Be2 Nd7 0-0 c5 dxc5 Nxc5 Nd4 Qb6 Ra2 Bg6 g3 Nc6 a5 Qc7 Re1 Rac8 Ra3 Ne4 a6 Nxd4 Qxd4 b6 Nd2 Nd6 Qe3 Rfd8 Raa1 Bf5 g4 Be4 Nxe4 Nxe4 Rad1 h6; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 0; bm b3; pv b3 d5 Nf3 Nf6 Bb2 Bg4 e3 e6 Be2 h6 0-0 Nbd7 c4 c6 d4 Bd6 c5 Bc7 Nbd2 Bf5 b4 0-0 a4 Ne4 b5 Nxd2 Nxd2 Qf6 Nf3 Rab8 Rc1 Rfd8 Bc3 Qe7 h3 Be4 Nd2 Bf5 Nf3; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce -12; bm b4; pv b4 e5 Bb2 Bxb4 Bxe5 Nf6 e3 0-0 c4 d5 Nf3 Bf5 a3 Bd6 d4 Bxe5 dxe5 Ne4 cxd5 c6 Be2 cxd5 0-0 Qe7 Qxd5 Rd8 Qb3 Nc6 Nc3 Nc5 Qa2 Nxe5 Nd4 Bg6 Nd5 Qd7 Nf4 Be4 Rfd1 Qe7 f3 Bc6 Nf5 Qf6 Nd4 Rac8 Rac1 b6 Qc2; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 0; bm c3; pv c3 d5 Nf3 c5 d4 Nf6 Bf4 Nc6 e3 Nh5 dxc5 Nxf4 exf4 e6 b4 Qf6 g3 Bd7 Bg2 Nxb4 Ne5 Nc6 Nxd7 Kxd7 0-0 Kc7 Na3 Bxc5 Nb5+ Kb8 c4 a6 cxd5 exd5 Rc1 Bb4 Qxd5 axb5 Qxb5 Ra5 Qb6 Ra6 Qb5 Ra5; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 55; ce 11; bm d4; pv d4 d5 c4 e6 Nc3 Nf6 cxd5 exd5 Bg5 c6 e3 Bd6 Bd3 O-O Qc2 h6 Bh4 Re8 Nge2 Qe7 O-O Nbd7 h3 Nf8 Bg3 b6 Rfd1 Bd7 Rab1 a5 Bxd6 Qxd6 Nf4 Ne6 Nce2 Rac8 Nxe6 Qxe6 Nf4 Qe7 Ba6 Rb8 Be2 Ne4 Bd3 Rbc8 Rbc1 Qg5 a3 c5 Bxe4 dxe4 dxc5 Rxc5 Qd2 Rxc1 Qxc1; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 10; bm e3; pv e3 Nf6 Nf3 d5 c4 e6 b3 Be7 d4 0-0 Bd3 dxc4 bxc4 c5 Bb2 b6 0-0 Bb7 Nc3 cxd4 exd4 Nc6 Rc1 Re8 Qd2 Bd6 Nb5 Bb4 Qe3 Bf8 h3 a6 Nc3 Ne7 Ne4 Nxe4 Bxe4 Nf5 Bxf5 exf5 Qb3 Be4 Rfe1 Qc7 d5 Bd6 Bd4 Rab8 a4 Bf4; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 13; bm e4; pv e4 e6 d4 d5 Nd2 c5 exd5 Qxd5 Ngf3 cxd4 Bc4 Qd6 0-0 Nf6 Re1 Be7 Nb3 Nc6 Nbxd4 Nxd4 Nxd4 0-0 c3 Qc7 Bd3 Rd8 Qf3 Bd6 h3 Be5 Be3 Bd7 Rad1 h6 Nb5 Bxb5 Bxb5 Rd5 Rxd5 Nxd5 Bd3 Rd8 Qe4 f5 Qc4 a6 Bd2 Qd6 Qb3 Bh2+ Kh1; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce -47; bm f3; pv f3 e5 Nc3 Nc6 e4 Nf6 Bb5 Nd4 Nge2 c6 Bd3 Bc5 Na4 Bd6 0-0 Ne6 Bc4 0-0 Bxe6 fxe6 d3 b5 Nac3 Bc5+ Kh1 d5 a3 Qb6 Qe1 Bb7 Bg5 Bd6 Nd1 c5 Nf2 Qc7 Ng3 a5 Qd2 Bc6 exd5 exd5 Nf5 h6 Nxd6 Qxd6 Bxf6 Rxf6 Ng4; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce -17; bm f4; pv f4 Nf6 Nf3 c5 Nc3 d5 e3 g6 Bb5+ Bd7 Qe2 Nc6 Ne5 Qc7 Nxd7 Qxd7 d3 Bg7 e4 0-0 Bd2 a6 Bxc6 Qxc6 e5 Nd7 0-0-0 d4 Ne4 f5 exf6 exf6 Rhe1 Rfe8 Qf3 f5 Ng5 Qxf3 Nxf3 Nb6 g3 Nd5 Ne5 b6 Nc4 Bf6 b3 b5 Ne5 Bxe5 fxe5 Kf7 Kb2; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 10; bm g3; pv g3 c5 Bg2 Nc6 Nf3 g6 c4 Bg7 Nc3 d6 0-0 Bf5 d3 Qd7 Rb1 Bh3 a3 Nf6 b4 Bxg2 Kxg2 0-0 Bg5 Nh5 Qd2 Bxc3 Qxc3 f6 Bh6 Rfe8 b5 Ne5 Nxe5 dxe5 f3 Ng7 Bxg7 Kxg7 a4 Red8 a5 Rac8 Ra1 Ra8 Rfb1 Rac8 Qd2 Ra8 b6 Qd4 bxa7; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce -85; bm g4; pv g4 d5 c4 Bxg4 Qb3 dxc4 Qxb7 Nd7 Na3 Rb8 Qg2 Be6 Nh3 Nb6 Ng5 Bd5 e4 h6 Nh7 Rxh7 exd5 Qxd5 Qxd5 Nxd5 Nxc4 e6 b3 Nb4 Ke2 Ne7 Bb2 g5 h4 g4 Bg2 Ng6 Be5 Nxe5 Nxe5 f5 Nc6 Nxc6 Bxc6+ Ke7 Rac1 Kf6 d4 Bd6 Rhe1 Rb6 Rc2 h5 Rec1 a6 Rc4 Ba3 R1c2; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 0; bm h3; pv h3 d5 d4 c5 e3 Nf6 c4 cxd4 exd4 Nc6 Nf3 g6 c5 Bg7 Nc3 0-0 Bb5 Ne4 0-0 Qa5 Bxc6 bxc6 Bf4 Nxc3 bxc3 Qxc3 Re1 Bf5 Rxe7 Rfe8 Qe1 Qxe1+ Raxe1 Be4 Rxe8+ Rxe8 Bd6 Rc8 Ng5 Bf5 Nf3 Be4; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce -28; bm h4; pv h4 d5 d4 c5 c3 Nc6 Nf3 e6 Bf4 Bd6 Bxd6 Qxd6 e3 Nf6 Bb5 0-0 0-0 Bd7 a4 a6 Bxc6 Bxc6 Ne5 Ne4 Nxc6 bxc6 Nd2 Nxd2 dxc5 Qxc5 Qxd2 e5 h5 h6 Qe2 Rab8 e4 a5 exd5 cxd5 Rad1 Rfe8 Rfe1 f6 Rd2 d4 Qd3 Qd5 cxd4; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce -41; bm Na3; pv Na3 e5 c4 c6 d4 exd4 Nf3 Nf6 Qxd4 d5 e3 a6 cxd5 cxd5 Nc2 Nc6 Qd1 Bd6 Be2 0-0 0-0 Ne4 Ncd4 Re8 Bd2 Bd7 Rc1 Qb6 Bc3 Rac8 h3 h6 Nxc6 bxc6 Bd3 Bf5 Nd4 Bg6 Bxe4 Bxe4 b4 c5 bxc5 Bxc5 Qg4 Qg6 Qxg6 Bxg6 Rfd1 Be4; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 0; bm Nc3; pv Nc3 d5 d4 Nf6 Bf4 e6 Nb5 Bd6 Nxd6+ cxd6 c3 Nc6 Nf3 0-0 Bg3 Qe7 e3 e5 Be2 a6 0-0 h6 Re1 Bf5 h3 Rfe8 Nd2 Qd7 Qb3 Qc7 Bh4 g5 Bg3 Bg6 Qd1 Kg7 Rc1 Rad8 Qb3 Na5 Qb4 Nc6 Qb3; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce 18; bm Nf3; pv Nf3 d5 d4 Nf6 c4 c6 cxd5 cxd5 Nc3 Nc6 Bf4 Bf5 e3 e6 Ne5 Nxe5 Bxe5 Nd7 Bg3 h5 h3 Rc8 Rc1 Qb6 Qd2 h4 Bf4 Bd6 Bxd6 Qxd6 Bd3 Bxd3 Qxd3 0-0 0-0 Nf6 Rc2 Qb8 Rfc1 a6 e4 Qf4 e5 Ne4 Nxe4 Rxc2 Rxc2 dxe4 Qe3; 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - acd 46; ce -40; bm Nh3; pv Nh3 d5 d4 Nf6 Nf4 c5 e3 Nc6 Bb5 Qb6 Nc3 e6 0-0 Be7 dxc5 Qxc5 Nd3 Qd6 b3 0-0 Ne2 Nb8 Ba4 a6 b4 Bd7 Bb3 a5 bxa5 Nc6 a4 Rfc8 Bb2 e5 c4 Be6 c5 Qc7 Nc3 Qd8 Nb5 Ne4 Nxe5 Nxa5 c6 bxc6 Nd4 Qb6 f3 Nxb3; 

[Laskos]

Well Ed, why bother? We have the excellent "Strategic Test Suite" which "consists of series of themed test suites designed to evaluate chess engine's long term understanding of strategical and positional concepts" (self-description). Dann and his Rybka were the main contributors to it. Here are some current standings on "strategical and positional concepts" (have no patience testing more engines):

Code: Select all

Stuck to the solution from 1s to 2s of thinking
4 i7 cores at 3.80GHz
RTX 2070 GPU

Houdini 1.5a        score=1339/1500 [averages on correct positions: depth=8.4 time=0.15 nodes=1320734]
Komodo 13.02        score=1319/1500 [averages on correct positions: depth=10.6 time=0.16 nodes=1064120]
Stockfish dev       score=1284/1500 [averages on correct positions: depth=10.8 time=0.18 nodes=1111414]  
Texel 1.07          score=1241/1500 [averages on correct positions: depth=8.9 time=0.22 nodes=1379498]
Arasan 21.0         score=1195/1500 [averages on correct positions: depth=9.5 time=0.24 nodes=1017178]
Lc0 v21.2 ID42524   score=1177/1500 [averages on correct positions: depth=3.8 time=0.14 nodes=1580]
Fruit 2.1           score= 993/1500 [averages on correct positions: depth=5.2 time=0.23 nodes=505132]

I guess your suite might converge to STS as standings go. Good!

You can't compare STS with OKE. STS was created with the help of the top engines of that time, all the Rybka derivatives will top the STS list not only Houdini 1.5

Well, after seeing your methodology of building the suite, I think I will stick to my opening suite, which I didn't appreciated too much, maybe I lack a sense of being an "expert" (I always feel a patzer in Chess).

The opening suite was built manually and pretty slowly almost three years ago, before any Lc0, and the boost of confidence came with Lc0 too:

Code: Select all

4 i7 cores at 3.80GHz
RTX 2070 GPU

Openings1000.epd test-suite (stuck to the solution from 1s to 2s of thinking):

Lc0 v21.2 ID42524  757/1000
Lc0 v21.2 ID32930  727/1000
Lc0 v21.2 ID11261  723/1000
Stockfish_dev      574/1000
Houdini 6.03       558/1000
Komodo 13.02       556/1000
Xiphos 0.5         513/1000
Booot 6.3.1        494/1000
Andscacs 0.95      484/1000
Laser 1.7          480/1000
Ethereal 11.25     467/1000
Texel 1.07         419/1000
Rodent III         376/1000
Fruit 2.1          348/1000
BikJump 2.01       276/1000
Predateur 2.2.1    265/1000

Here is the suite:
http://s000.tinyupload.com/?file_id=854 ... 6503996473

The number of positions is 1000, 5 times the same 200 positions, for less noise. My methodology hardly allows for more than 200 positions in some reasonable amount of time (several weekends spent).