Modular opening book SF analysed 87417 pos., beta-1

[Frank Quisinsky]

Hi Ferdinand,

must drive to my professional work now ...

Shortly ...
I made an update on my detail page to the book project.
Version 1.03 is now available (small and big).

- alpha files (without develops files) now in small directory too. Important here the alpha.pgn and alpha.epd file.

Different to the big file only:
In the big file are the log files of engine anlysis and development files for the alpha.pgn database. All other files are available in the small download file. So 99% of users have enough with the small download file.

- new readme file (explain how I can create the important *.epd files with the tools by Norm). I think important. Norm have good readmes but in this case I have all what is important for my project in a short readme.

- some changes in my other readme files and much spelling error corrections.

- better data structure inside the *.zip file for better understandings.

- some other small changes!

###

Later if you are ready with your tool ...

I would like to have your new tool in download area and in the small and big project *.zip files.

Later, the project is ready, most need your tool only for creating own files with the database basic files.

I am sure you will give your aggreement.

I like that project ... that is the final result of three years FCP Rating List. So the games I produced made sense for much more interesting things.

###

Best
Frank

[Frank Quisinsky]

I wrote:
I like that project ... that is the final result of three years FCP Rating List. So the games I produced made sense for much more interesting things.

But most I like ...
All engines programmers I am used engines for FCP Rating List are an important part of the project.

And all of this programmers have now directly a great basics for engine improvements and engine testing.

Because ... 80% of baisc games comes from FCP-Rating-List.
In 30-40% of that games engines try to find better moves after the ECO start started ... games based on FCP Live Book.



Thats what I like most!

Best
Frank

PS: On the others hand ... basics are 3 moves after ECO code formed only. And most after ECO code formed is good known theory. But often we can find games, no Grandmaster played ... moves comes by engines! Really a great combination between Grandmaster chess, correspondence and computer chess.

[Ferdy]

Just some basic functionality, from readme,

Code: Select all

Game Extractor v1.0

Extract games based on criteria.txt file.

Files required:
* pgn-extract.exe
* criteria.txt
* your ref pgn file
* your ref epd file

Open criteria.txt to define your ref epd and pgn files as well
as your choice of engine score and number of queens.

If your ref pgn filename is big.pgn the output filenames would be
filtered_big.pgn, and unfiltered_big.pgn. The filtered_big.pgn is
the file that you are interested based from the criteria.txt.
Duplicates in filtered_big.pgn file are also removed.

How to run?
Just double-click gameExtractor.exe or run the batch file run_game_extractor.bat.
This program is tested under windows 7.

Contents of criteria.txt

Code: Select all

start
refepd, opening.epd
refpgn, big.pgn
minscorecp, -30
maxscorecp, +30
mincntqueen, 0
maxcntqueen, 2
end

pgn-extract.exe is not included in the package, you need to download it at its official site.

Sample run.

Code: Select all

Game Extractor 1.0


Filtering FEN based on criteria.txt ...
elapsed: 8.0s

Extracting games based from filtered FEN using pgn-extract ...
elapsed: 7.7s


Done!!

Download game extractor.
https://drive.google.com/file/d/0BwAOsu ... sp=sharing

[Frank Quisinsky]

Ups ...
I have to try out it ... directly!

Will do it ...
Later today!

Best
Frank

[Frank Quisinsky]

Hi Ferdinand,

I edit the criteria.txt
"alpha.pgn and alpha.epd"
I copy pgn-extract.exe (v17.37) in the same directory.

I edit to 0.5 and -0.3

Code: Select all

start
refepd, alpha.epd
refpgn, alpha.pgn
minscorecp, -30
maxscorecp, +50
mincntqueen, 0
maxcntqueen, 2
end

OS: Windows 10 Prof. x64

After all I got the message:

"Unable to open fenTag.txt for reading."

What is "fenTag.txt"?
Not in the directory from yourself.

Best
Frank

PS: Stefan Pohl will be happy. The good idea by Stefan Pohl is included!!
THANKS!!

[Frank Quisinsky]

Hi Ferdinand,

I added the *.epd without eval ...

Typical Frank ... forget my last message, the other readers too.

It works ...
FANTASTIC ...

More as fantastic ...
Later ...

Best
Frank

[Frank Quisinsky]

Hi Ferdinand,

but the final result can't be right.

Example: Alpha.pgn = 87.714 games, Alpha.epd = 87.417 the same with ce after SF analysis.

If I set ... 0.50 / -0.30 I should have the same result I found out.

77.976 should be the final result ... if I made no mistakes in my work I do by hand.

But the result with your tool is: 63.541

with the setting:

Code: Select all

start
refepd, alpha.epd
refpgn, alpha.pgn
minscorecp, -30
maxscorecp, +50
mincntqueen, 0
maxcntqueen, 2
end

The program give me now a new PGN with reject games and a second new PGN with the good games.

For a check what is reject ... I need the same in *.epd.
*.epd is very important too. So I can added the reject games later in a new database for a check with other engines (is Stockfish right or not).

two new *.epd databases ...
Reject in *.epd too and the new *.epd with good positions.

With other words I need 4 new files.

reject in *.pgn file Format
reject in *.epd file Format
good games in *.pgn file Format
good games in *.epd file Format

Best
Frank

No idea why the final result in an other.
Maybe it have to do with the parameters ...

Code: Select all

mincntqueen, 0
maxcntqueen, 2

[Ferdy]

Hi Ferdinand,

I edit the criteria.txt
"alpha.pgn and alpha.epd"
I copy pgn-extract.exe (v17.37) in the same directory.

I edit to 0.5 and -0.3

Code: Select all

start
refepd, alpha.epd
refpgn, alpha.pgn
minscorecp, -30
maxscorecp, +50
mincntqueen, 0
maxcntqueen, 2
end

OS: Windows 10 Prof. x64

After all I got the message:

"Unable to open fenTag.txt for reading."

What is "fenTag.txt"?
Not in the directory from yourself.

Best
Frank

PS: Stefan Pohl will be happy. The good idea by Stefan Pohl is included!!
THANKS!!

"fenTag.txt" This is the file created by the tool. It contains lines with fen based from criteria.txt.

Code: Select all

FEN "r2qkb1r/ppp1pppp/5n2/8/3P2b1/5N2/PPP2PPP/R1BQKB1R w KQkq - 0 1"

[...]

This file is given to pgn-extract and pgn-extract will search this position in the ref pgn, once found it will save the game in a file as output.

That explains it, if your ref epd has no ce then this file will not be created by the tool, and pgn-extract would complain that this file is missing. I will improve error checking of the tool later.

[Ferdy]

Hi Ferdinand,

but the final result can't be right.

Example: Alpha.pgn = 87.714 games, Alpha.epd = 87.417 the same with ce after SF analysis.

If I set ... 0.50 / -0.30 I should have the same result I found out.

77.976 should be the final result ... if I made no mistakes in my work I do by hand.

But the result with your tool is: 63.541

with the setting:

Code: Select all

start
refepd, alpha.epd
refpgn, alpha.pgn
minscorecp, -30
maxscorecp, +50
mincntqueen, 0
maxcntqueen, 2
end

I have downloaded new-opening-book_beta-v1.03-big (got the alpha.pgn and alpha.epd), is this 77.976 games that you mentioned available for download? Does that file contain no duplicates?

The criteria.txt is fine.

The program give me now a new PGN with reject games and a second new PGN with the good games.

For a check what is reject ... I need the same in *.epd.
*.epd is very important too. So I can added the reject games later in a new database for a check with other engines (is Stockfish right or not).

two new *.epd databases ...
Reject in *.epd too and the new *.epd with good positions.

With other words I need 4 new files.

reject in *.pgn file Format
reject in *.epd file Format
good games in *.pgn file Format
good games in *.epd file Format

I can create those as output in next version.

[Frank Quisinsky]

Alpha = 87.417 games (before Stockfish analysed)
can be found in alpha directory (my download file 1.03).

During the Stockfish analysis I made an update.
Can be found in alpha subdirectory = 4.728 games.
Important ... I am searching to codes I haven't enough.

This " 4.728 update-games" are checked by SF on an other PC (0.50 and -0.30) too for two times. Same ECO-Code length as I am using for the main database alpha.pgn with 87.417 games. In the 4.728 games are doubles but the move transpositions are included only.

---

After Stockfish analysis I have 77.976 games, reject with 0.50 and -0.30 9.391 games, and reject 50 games by hand ... unusual first moves) = 9.441!

77.976 + 9.441 = 87.417 games

77.976 + 4.728 checked new games I added as update I have ...
82.704 games (called beta-1.pgn).

At the moment Komodo analysed this 82.704 games as next engine!

---

I saw in your readme that your tool recject the doubles.

You wrote:
"Duplicates in filtered_big.pgn file are also removed."

Better is to do that with a parameter in your "criteria.txt".
reject doubles yes / no

---

If you tool not reject the dublicates your tool must have the final results with 0.50 / -0.30 = 9.339 found from the 87.417 *.pgn file called Alpha.pgn in my download version 1.03.

Best
Frank

PS: Yes it would be much better if Output = 4 files with EPD files. But I am sure in your tool must be an error. I am searching at the moment ... searching all without doubles! Maybe I am wrong! Later ...