For testing engines: 3.698 balanced opening lines, can produce short wins ...

[Frank Quisinsky]

Hi there,

shortly:
In the FEOBOS project 10 engines work together to evaluate over 80,000 positions. Klaus Wlotzka and I calculated the most balanced opening lines with an Excel spreadsheet (over 16 million formulas). The project with a documentation of 60 pages is available on my website (since a while I am not working on it). The project ended around 2017.

Since that time I use one of the 5 FEOBOS databases (book database with contempt 3 and especially with more E00-E99 positions with contempt 3-5) for testing engines (my FCP tours or rating systems with longer time controls).

I have the opinion, maybe I am wrong, that the most interesting are the lines that can produce short wins. So I collected the short wins from my official tournaments for some years.

Now I do the following:

Readme:

Code: Select all

No magic!?
feobos-6m-v2.1.bkt (opening book for Shredder GUI by Stefan Meyer-Kahlen)

Step 1 - collected games, 1:0 games (below move number 60 to mate) and 0:1 games (below move number 68 to mate)
         games played with feobos-20.1-contempt_3-5_tuned-v2.bkt
         games are from FCP-Tourney-2021, FCP-Tourney-2022, KI-Ratings, FCP-Tourney-2024 (round 01-15)

Step 2 - games with an eval higher than 0.60 (white) / -0.75 (black), 3 moves after the end of the opening line, deleted.
         The tool looking in white eval for the third own evaluation of engines.
         Tool: universal-pgn-epd-tool v2 by Ferdinand Mosca
         https://www.amateurschach.de (looking under downloads)

Step 3 - truncate to 6 moves
         truncate test.pgn 12
         Tool: truncate by Norm Pollock
         http://www.nk-qy.info/40h/

Step 4 - game results set to 1/2:1/2, different other optimizations (have a look in the *.pgn header)
         I used Notepad++ v8.6
         https://notepad-plus-plus.org/

Step 5 - move transposition with pgn-extract deleated
         pgn-extract --fuzzydepth 0 --duplicates dupes.pgn --output unique-beta-1.pgn test.pgn
         final result = 4.131 lines ... book version 2.0
         Tool: pgn-extract v22.11 by David J. Barnes
         https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/

Step 6 - EPD analysis by Stockfish 16 NN running at 4.4Ghz on Intel i9-10900k, 10-core, 15 seconds per position
         lines are deleted with an eval more as:
         white: + 0.65 max.
         black: - 0,30 max.
         final result = 3.698 lines ... book version 2.1 (433 lines deleted)
         Stockfish engine by the Stockfish developers:
         https://stockfishchess.org/
         https://github.com/official-stockfish/Stockfish/releases
         https://abrok.eu/stockfish/

         Please be inform that all the important pgn-extract / *.epd parameters can be found in this directory!
         One of the older readme's from the FEOBOS project by Klaus Wlotzka and me.

The final feobos-6m-v2.1.pgn
The final feobos-6m-v2.1.epd
The final feobos-6m-v2.1.bkt ... Shredder's book format / opening book
can be found in this directory!

My hope is that this small and easy balanced opening book reduces the number of draws by 3-4%!



Frank Quisinsky, Germany / Gutweiler

Note: This is not my main database, I am working for a test set... the lines from the main database go 9 moves deep. I am not finished with my test set and need more material. One of the reasons for my FCP tourney's.

I will reduce the draw rate with balanced lines!

Here is a database with 3.698 lines.
Very sure you can create with this material a strong own test set with balanced positions.
---

What is really bad ...

Have a look here:
(all book lines are draw in PGN, have a look what is missed):

Code: Select all

E80         3     50.0%   :      0+   :      3=   :      0-
E81         8     50.0%   :      0+   :      8=   :      0-
E82         1     50.0%   :      0+   :      1=   :      0-
E83 missed
E84         3     50.0%   :      0+   :      3=   :      0-
E85 missed
E86 missed
E87 missed
E88 missed
E89 missed

E90         6     50.0%   :      0+   :      6=   :      0-
E91         1     50.0%   :      0+   :      1=   :      0-
E92         2     50.0%   :      0+   :      2=   :      0-
E93         1     50.0%   :      0+   :      1=   :      0-
E94         3     50.0%   :      0+   :      3=   :      0-
E95         1     50.0%   :      0+   :      1=   :      0-
E96 missed
E97         2     50.0%   :      0+   :      2=   :      0-
E98 missed
E99 missed

I prefer such opening systems myself. But all the interesting Kings-Indian lines can't produce short games with balanced lines. I fight against it, I try again and again lines with Eng matches I found in chess books and so on. But the advantage of black or white is too big if lines produce short games.

66 ECO codes are missing from the database. Possibly the engines found the way to a missing ECO code.

Much more extreme is the database I created for my test set ... I work with a depth of 9 moves. About 175 missed ECO codes, can't be produced with the high level of engines today short games and if the more random because for different eng-eng constellation all is again draw.

Do you know this:
49x E11, 52x B12, 49x B06, 57x B07 or 46x A25 lines produced short games.
No wonder GMs like these systems!

Or did you know that such lines mostly ended in draws?

Code: Select all

A60         5     50.0%   :      0+   :      5=   :      0-
A61 missed
A62         3     50.0%   :      0+   :      3=   :      0-
A63         4     50.0%   :      0+   :      4=   :      0-
A64 missed
A65         7     50.0%   :      0+   :      7=   :      0-
A66 missed
A67 missed
A68 missed
A69         2     50.0%   :      0+   :      2=   :      0-

A70 missed
A71 missed
A72 missed
A73 missed
A74 missed
A75 missed
A76 missed
A77         1     50.0%   :      0+   :      1=   :      0-
A78 missed
A79 missed

---

Again, a long time work I have a lot of fun here with my main work around openings.
Maybe you like it for testing engines.

I will use the new book for my next FCP-Tourney-2024-MA (MA for Move-Average) and a time control 66 minutes per game + 6 seconds per move with 6-pieces.

Have a nice second christmas day!

Best
Frank

[Frank Quisinsky]

Ah ...

The *.epd / *.pgn to the 3.698 lines can be found in my current version of my *.zip file to the running FCP-Tourney-2024 (ended today in the evening).

v0.15.8
https://www.amateurschach.de/download/_ ... y-2024.zip

Best
Frank

End of 2024 I will public the final result of my test-set.
This one is only 6-moves deep for experiments with balanced and 100% bugfree lines.

Note:
Different engines like to give an high eval, but you can be sure the lines are balanced!

Example:

Code: Select all

Tournament : pgn\fcp-tourney-2024.pgn
Output     : pgn\fcp-tourney-2024.txt
Games      : 12.300
Type       : white
White POV  : y

Engine                      Games   [1]   [2]   [3]   [4]   [5]   [6]   [7]   [8]   Result
Stockfish 16 NN               300  0.33  0.36  0.37  0.39  0.41  0.43  0.45  0.48    79.0%
Dragon 3.3 NN (Komodo)        300  0.38  0.38  0.40  0.41  0.42  0.44  0.46  0.48    78.3%
Berserk 12 NN                 300  0.38  0.40  0.41  0.42  0.44  0.47  0.49  0.51    76.0%
RubiChess 20230918 NN         300  0.33  0.34  0.36  0.37  0.38  0.39  0.41  0.44    70.3%
Caissa 1.14.1 NN              300  0.53  0.55  0.57  0.58  0.61  0.62  0.65  0.68    69.7%
CSTal 2.00 NN                 300  0.56  0.58  0.60  0.62  0.63  0.66  0.70  0.74    67.8%
rofChade 3.1 NN               300  0.55  0.57  0.57  0.59  0.61  0.60  0.61  0.62    67.2%
Clover 6.1 NN                 300  0.44  0.47  0.48  0.51  0.53  0.55  0.59  0.61    66.7%
Igel 3.5.0 NN                 300  0.65  0.66  0.68  0.69  0.71  0.74  0.74  0.78    66.5%
Uralochka 3.40a NN            300  0.29  0.30  0.34  0.36  0.38  0.39  0.41  0.45    66.2%
Rebel EAS NN                  300  0.75  0.77  0.77  0.78  0.77  0.82  0.80  0.84    65.8%
Obsidian 8.0 NN               300  0.39  0.41  0.42  0.44  0.45  0.47  0.49  0.51    65.0%
Revenge 3.0 NN                300  0.34  0.36  0.38  0.39  0.41  0.43  0.45  0.47    64.3%
Koivisto 9.2 NN dev (hotfix)  299  0.54  0.56  0.59  0.60  0.64  0.66  0.66  0.68    64.5%
Viridithas 11.0.0 NN          300  0.35  0.36  0.37  0.38  0.40  0.40  0.41  0.43    63.7%
Alexandria 5.1.0 NN           300  0.70  0.72  0.72  0.75  0.76  0.77  0.82  0.83    63.5%
Seer 2.7.0 NN                 300  0.46  0.48  0.48  0.49  0.51  0.52  0.53  0.53    62.3%
SlowChess Blitz 2.9 NN        299  0.34  0.34  0.33  0.35  0.35  0.35  0.36  0.38    61.7%
Arasan 24.0 NN                300  0.80  0.84  0.87  0.91  0.90  0.93  0.97  0.98    59.8%
Minic 3.39 NN                 300  0.89  0.93  0.92  0.92  0.95  0.95  0.97  0.95    58.5%
Fire 9.2 NN                   300  0.20  0.23  0.24  0.25  0.25  0.26  0.26  0.27    58.0%
Fritz 19 NN (Gingko)          300  0.34  0.36  0.36  0.36  0.36  0.38  0.38  0.39    57.3%
Booot 7.2 NN                  300  0.41  0.44  0.46  0.48  0.51  0.52  0.53  0.55    56.5%
Carp 3.0.1 NN                 300  0.31  0.32  0.33  0.33  0.33  0.34  0.34  0.34    54.7%
Superultra 1.0 NN             300  0.45  0.46  0.47  0.47  0.48  0.47  0.47  0.49    54.2%
Wasp 6.61 NN dev              300  0.42  0.42  0.42  0.42  0.43  0.42  0.43  0.42    53.7%
Velvet 5.3.0 NN               300  0.32  0.34  0.36  0.37  0.40  0.41  0.41  0.44    53.2%
Black Marlin 8.0 NN           300  0.46  0.47  0.48  0.49  0.50  0.50  0.51  0.52    51.5%
Altair 6.0.0 NN               300  0.33  0.34  0.36  0.37  0.37  0.37  0.37  0.37    51.5%
Stormphrax 3.0.0 NN           300  0.31  0.32  0.32  0.32  0.32  0.32  0.32  0.31    51.2%
Akimbo 0.7.0 NN               300  0.70  0.74  0.77  0.82  0.80  0.82  0.82  0.83    49.7%
Texel 1.10 NN                 300  0.32  0.33  0.34  0.35  0.34  0.34  0.35  0.34    48.5%
BlackCore 6.0 NN              300  0.37  0.38  0.40  0.39  0.40  0.41  0.43  0.41    47.5%
Marvin 6.2.0 NN               300  0.38  0.41  0.42  0.43  0.43  0.42  0.41  0.39    46.7%
Pawn 2.0 NN                   300  0.27  0.29  0.31  0.31  0.31  0.31  0.31  0.29    45.5%
Smallbrain 7.0 NN             300  0.32  0.34  0.35  0.34  0.35  0.34  0.33  0.33    45.3%
Devre 4.0 NN                  300  0.46  0.47  0.47  0.47  0.48  0.49  0.47  0.49    44.8%
Halogen 11.4 NN               300  0.45  0.45  0.45  0.46  0.46  0.45  0.44  0.42    44.5%
Nemorino 6.11 NN dev          300  0.45  0.46  0.49  0.48  0.49  0.49  0.49  0.50    43.8%
Chess.cpp 4.0 NN              300  0.32  0.34  0.33  0.34  0.30  0.31  0.29  0.30    42.0%
Willow 3.1 NN                 300  0.43  0.44  0.45  0.46  0.46  0.45  0.46  0.46    41.5%
                                1  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00     0.0%

Code: Select all

Tournament : pgn\fcp-tourney-2024.pgn
Output     : pgn\fcp-tourney-2024.txt
Games      : 12.300
Type       : black
White POV  : y

Engine                      Games    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]   Result
Stockfish 16 NN               300  -0.30  -0.26  -0.23  -0.21  -0.19  -0.16  -0.14  -0.12    57.7%
Dragon 3.3 NN (Komodo)        300  -0.36  -0.32  -0.30  -0.27  -0.24  -0.22  -0.19  -0.17    57.2%
CSTal 2.00 NN                 300  -0.51  -0.49  -0.47  -0.44  -0.40  -0.36  -0.34  -0.32    55.7%
Berserk 12 NN                 300  -0.32  -0.30  -0.27  -0.26  -0.23  -0.21  -0.19  -0.17    55.0%
RubiChess 20230918 NN         300  -0.28  -0.27  -0.27  -0.25  -0.24  -0.23  -0.21  -0.20    54.2%
Caissa 1.14.1 NN              300  -0.51  -0.48  -0.45  -0.43  -0.41  -0.39  -0.35  -0.33    53.5%
Revenge 3.0 NN                300  -0.34  -0.32  -0.31  -0.31  -0.30  -0.28  -0.29  -0.28    53.3%
Igel 3.5.0 NN                 300  -0.61  -0.59  -0.58  -0.57  -0.52  -0.48  -0.46  -0.45    52.5%
Uralochka 3.40a NN            300  -0.26  -0.27  -0.25  -0.25  -0.26  -0.24  -0.23  -0.22    51.7%
Clover 6.1 NN                 300  -0.39  -0.38  -0.38  -0.37  -0.36  -0.33  -0.32  -0.31    51.2%
rofChade 3.1 NN               300  -0.49  -0.46  -0.43  -0.43  -0.40  -0.40  -0.39  -0.36    51.0%
Obsidian 8.0 NN               300  -0.36  -0.36  -0.33  -0.31  -0.29  -0.27  -0.25  -0.23    51.0%
Seer 2.7.0 NN                 300  -0.44  -0.42  -0.41  -0.39  -0.37  -0.35  -0.33  -0.32    50.5%
Rebel EAS NN                  300  -0.68  -0.64  -0.61  -0.58  -0.57  -0.51  -0.51  -0.47    50.3%
Alexandria 5.1.0 NN           300  -0.64  -0.61  -0.55  -0.53  -0.49  -0.45  -0.44  -0.42    49.5%
Koivisto 9.2 NN dev (hotfix)  299  -0.47  -0.46  -0.43  -0.41  -0.38  -0.38  -0.36  -0.33    49.5%
Viridithas 11.0.0 NN          300  -0.31  -0.29  -0.29  -0.28  -0.27  -0.26  -0.25  -0.24    49.2%
Arasan 24.0 NN                300  -0.74  -0.71  -0.69  -0.68  -0.62  -0.59  -0.56  -0.54    48.5%
SlowChess Blitz 2.9 NN        299  -0.32  -0.32  -0.31  -0.30  -0.29  -0.28  -0.27  -0.26    47.2%
Fire 9.2 NN                   300  -0.18  -0.17  -0.16  -0.15  -0.15  -0.15  -0.15  -0.15    43.7%
Minic 3.39 NN                 300  -0.82  -0.82  -0.81  -0.81  -0.79  -0.77  -0.81  -0.79    43.3%
Carp 3.0.1 NN                 300  -0.26  -0.25  -0.24  -0.24  -0.23  -0.23  -0.23  -0.22    42.7%
Fritz 19 NN (Gingko)          300  -0.33  -0.32  -0.31  -0.30  -0.29  -0.29  -0.30  -0.29    42.5%
Booot 7.2 NN                  300  -0.41  -0.40  -0.40  -0.40  -0.40  -0.40  -0.40  -0.41    40.0%
Altair 6.0.0 NN               300  -0.31  -0.31  -0.30  -0.29  -0.28  -0.27  -0.27  -0.26    38.2%
Black Marlin 8.0 NN           300  -0.43  -0.40  -0.39  -0.38  -0.37  -0.36  -0.35  -0.35    37.2%
Superultra 1.0 NN             300  -0.42  -0.40  -0.40  -0.41  -0.40  -0.40  -0.40  -0.40    36.7%
Stormphrax 3.0.0 NN           300  -0.26  -0.26  -0.25  -0.23  -0.24  -0.24  -0.24  -0.24    35.7%
Wasp 6.61 NN dev              300  -0.26  -0.26  -0.27  -0.28  -0.29  -0.31  -0.33  -0.35    33.5%
Velvet 5.3.0 NN               300  -0.30  -0.30  -0.31  -0.31  -0.33  -0.32  -0.34  -0.35    33.3%
Akimbo 0.7.0 NN               300  -0.67  -0.67  -0.65  -0.63  -0.63  -0.64  -0.67  -0.67    31.2%
Devre 4.0 NN                  300  -0.44  -0.45  -0.43  -0.44  -0.46  -0.44  -0.46  -0.47    30.2%
Texel 1.10 NN                 300  -0.32  -0.32  -0.32  -0.31  -0.32  -0.33  -0.33  -0.34    30.2%
BlackCore 6.0 NN              300  -0.33  -0.33  -0.30  -0.30  -0.30  -0.31  -0.32  -0.33    29.3%
Halogen 11.4 NN               300  -0.44  -0.44  -0.42  -0.43  -0.42  -0.43  -0.43  -0.45    29.2%
Nemorino 6.11 NN dev          300  -0.44  -0.43  -0.43  -0.43  -0.43  -0.43  -0.44  -0.44    29.2%
Pawn 2.0 NN                   300  -0.26  -0.26  -0.26  -0.26  -0.27  -0.27  -0.28  -0.30    28.3%
Smallbrain 7.0 NN             300  -0.30  -0.30  -0.30  -0.31  -0.31  -0.33  -0.35  -0.37    27.3%
Chess.cpp 4.0 NN              300  -0.28  -0.27  -0.29  -0.31  -0.30  -0.31  -0.32  -0.33    27.2%
Marvin 6.2.0 NN               300  -0.39  -0.40  -0.41  -0.40  -0.42  -0.43  -0.46  -0.47    26.5%
Willow 3.1 NN                 300  -0.39  -0.38  -0.38  -0.39  -0.40  -0.42  -0.42  -0.46    25.5%
                                1   0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00     0.0%

What Engines evaluate with my FEOBOS Contempt 3-5 Standard book.
Different engines like to give a very high eval for the first 8 moves!

Best
Frank

[Frank Quisinsky]

My bad English ...
Sorry about that!

What I want to say is ...
With short-win databases I can produce a lower draw rate for eng-eng matches on TOP level.
That's my opinion I wrote before ... maybe I'm wrong here.
But my results are always the same on this topic.

And I forgot to write ...
I have been testing version 2.0 here on different PCs for the last few weeks.
I can reduce the draws for TOP engines with the 6-move database by about 4.4% when I compare it with my own FEOBOS standard book. That's a great result. With my 9 move test set (still working on it) I have at the moment only 371 lines where TOP-engines did not finish with a draw in many of the eng-eng constellations. But again, this test set isn't finished yet and I need some more test games. I will do that in 2024. I hope the engines (participating in my FCP-Tourney-2024-MP) can produce more lines with the 6-moves book, which I can use for the 9-moves test-set.

One of the reasons that all tactically strong engines must be in the field and can help me for my work. I need the full power of possible aggressiveness for this, because everything is much more complicated for a balanced test-set.

Best regards
Frank

[Volker Pittlik]

My bad English ...
Sorry about that!
...

Little Hint:

Me:
Falls man sich unsicher im Englischen ist, kann man Texte in Deutsch verfassen und von ChatGPT (oder einer anderen Maschine) übersetzen lassen. Das funktioniert meistens mindestens zufriedenszellend, wie man an diesem Beispiel sieht.
ChatGPT

Chat GPT:
If one is uncertain in English, they can compose their text in German and have it translated by ChatGPT (or another machine). This usually works satisfactorily, as demonstrated in this example.

Gruss und Frohe (Rest-)Festtage

Volker

[Frank Quisinsky]

Hi Volker,

I know, but I'm too lazy for that.
I think my English is better than 25 years ago and most people understand it.
If not, the others will have to learn Frankish, so easy!



Merry christmas for you!
Du alter Haudegen Du ... for english speaker the translation should be a problem!

Best
Frank