Modular opening book SF analysed 87417 pos., beta-1

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Game Extractor v1.2

Post by Ferdy »

Version 1.2

Sample run.

Code: Select all

Game Extractor v1.2

File: criteria.txt

start
refepd, st-ko-analysis.epd
refpgn, alpha.pgn
minscorecp, -30
maxscorecp, +30
mincntqueen, 0
maxcntqueen, 2
end

Available analyzing engines in ref epd:
[0]
[1] Komodo 10.3 4C
[2] Stockfish 16012017 BMI2 x64

Enter number of analyzing engine, 0 to ignore? 0

(1) Filtering FEN based on criteria.txt ...
elapsed: 29.5s

(2) Extracting unique games based from filtered FEN and extracting non-unique ga
mes from unfiltered FEN using pgn-extract ...
elapsed: 8.4s

(3) Extracting unique games from unfiltered pgn file using pgn-extract ...
elapsed: 2.8s

(4) Extracting non-unique games based from filtered FEN using pgn-extract ...
elapsed: 7.6s

(5) Deleting temp file fenTag.txt ...

Output files:
in_criteria_unique_alpha.pgn
in_criteria_not_unique_alpha.pgn
not_in_criteria_unique_alpha.pgn
not_in_criteria_not_unique_alpha.pgn
in_criteria_st-ko-analysis.epd
not_in_criteria_st-ko-analysis.epd


Done!!

Press enter key to exit
Download exe, readme and criteria.txt files
https://drive.google.com/file/d/0BwAOsu ... sp=sharing
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Game Extractor v1.2

Post by Frank Quisinsky »

It works!
I will make an update in the next hour.

Ferdinand,

different small things if you like ...

1. I think better to give the 6 output files a name, started with "_" ... so the Output are on first position in the directory with all the files.

2. I have an *.epd with Stockfish and Komodo files and can see ...

0 = ...
1 = Komodo
2 = Stockfish

Better here without "0"

3. Your program give me now 6 Output files.
Here I think a log file with the result can be better ...

_in_criteria_unique_x.pgn = found = x
_in_criteria_not_unique_x.pgn = found x
_in_criteria_x.epd = found x

_not_in_criteria_unique_x.pgn = found = x
_not_in_criteria_not_unique_x.pgn = found x
_not_in_criteria_x.epd = found x

Not "found" ... better "inside"

4. The pop-up after I started your tool should not closed if ready, better is to hold the pop-up open with the result, same I can later found in a text file, maybe with the name result.txt.

Again, if you like only ... not very important what I wrote here!

Today I have a lot of time and can test more in detail.

Best
Frank
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Modular opening book SF analysed 87417 pos., beta-1

Post by Frank Quisinsky »

Hi Dann,

the first analysis by Komodo and Stockfish are not very important, cleaning only.

Here I reject with 0.50 and -0.30 with 30 seconds per positions.

Much more important is the file beta-v03 ... the main file for now. After I have beta-v03 ... 10 engines are searching now with the main file beta-v03_endpos-n.epd.

Please note:
I reject at first with the 312.550 "FCP Rating List" database. All important ECO codes - 3 moves after ECO code formed - and main lines are inside (not all but most important ... think so) and all evals are inside with the univeralpgnepdtool v2 by Ferdinand. Can be found on my download are too. Later I added new grandmasters theory, lines to ECO codes I haven't enough. During FCP Rating List are still running I am looking game by game for bad moves and reject by Hand in my FCP Live Book. So the database from 312.550 is very strong but for sure ... absolutely possible that book-looses are inside ... should be not to many.

With other words:
Most of the bad lines in the first database for this project comes from newer games I added (newer grandmasters theory games).

After cleaning with Stockfish and Komodo with 30 seconds per move ... the main database is beta-v03 with move transpositions ... later important for opening books.

Best
Frank

And with the new tool by Ferdinand you can reject the main database with your own settings and can do interesting things ...
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Game Extractor v1.2

Post by Ferdy »

Frank Quisinsky wrote: Not "found" ... better "inside"
I cannot understand that.
Frank Quisinsky wrote: 4. The pop-up after I started your tool should not closed if ready, better is to hold the pop-up open with the result, same I can later found in a text file, maybe with the name result.txt.
I also cannot understand that.
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Modular opening book SF analysed 87417 pos., beta-1

Post by Frank Quisinsky »

Hi Dann,

later if 10 engines are done with the beta-v03_endpos-n.epd I am sure we can from reject lines create a test-set for openings. Can looking in Excel and compare the results by engines.

:-)

All will be need 6 months with 1 minute by position (if all analysis are done).

But my main interest is to reject bad lines for new balanced opening books, 3-moves after ECO code formed.

Best
Frank
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Game Extractor v1.2

Post by Frank Quisinsky »

Hi Ferdinand,

3:
If your program is done I can see ...

Output files:
in_criteria_unique_beta-v03.pgn
in_criteria_not_unique_beta-v03.pgn
not_in_criteria_unique_beta-v03.pgn
not_in_criteria_not_unique_beta-v03.pgn
in_criteria_houdini-analysis.epd
not in criteria houdini-analysis.epd ("_" missed)

better is ...

Output files:
_ (started with "_")

_in_criteria_unique_beta-v03.pgn (inside = x games)
_in_criteria_not_unique_beta-v03.pgn (inside = x games)
_not_in_criteria_unique_beta-v03.pgn (inside = x games)
_not_in_criteria_not_unique_beta-v03.pgn (inside = x games)
_in_criteria_houdini-analysis.epd (inside = x positions)
_not_in_criteria_houdini-analysis.epd (inside = x positions).

And it would nice to have the complete output of your program in a new "log" *.txt file. Maybe with the name the epd have ...

Example:
houdini-analysis.log

And to 4 ...
Again a mistake by myself.
If your program is ready the Windows with the result isn't automatically closed.

Forget point 4!!
I am to old for all this.

Best
Frank
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Game Extractor v1.2

Post by Frank Quisinsky »

Hi Ferdinand,

I added your tool v1.2 on my download area with beta-v03.pgn and the first Houdini analysis and wrote an additional readme.

Files can be found in the download *.zip to the project ...
in your directory
too.

Best
Frank

Code: Select all

Here can be found the files for game-extractor 1.2 by Ferdinand Mosca (Philippines)!
Ferdinand is still working on it! ... Please have patience!
Engines are still working on the analysis! ... Please have patience!

At the moment an example for engine analysis is included only:
v01: After the first 7.177 of 26.146 beta-v03_endpos-n.epd Houdini 5.01 Pro x64 analysis!
  
- Intel® Core™ i7-5930K, 4.3 GHz
- 6 cores, used 10 cores (4x Hyperthreading)
- Windows 10 Professional x64
- 4Gb for hash tables
- 60 seconds per end position

  
Note 1:
All final engine analyses will be available in this download file later too!
v01 - v10 = 10 engine will be analyse the beta-v03_endpos-n.epd file.

For more information ...
Please have a look under:
http://www.amateurschach.de/main/_new-opening-book.htm

Note 2:
Please copy "pgn-extract.exe" by David J. Barnes in the same directory!!
Otherwise the program by Ferdinand Mosca doesn't work!
https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/

Settings:
Can be give with "criteria.txt".
Please have a look in the readme by Ferdinand Mosca (Philippines).



March 12th, 2017 (09:27)
Frank Quisinsky, Trier (Germany)
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Game Extractor v1.2

Post by Ferdy »

Frank Quisinsky wrote: 2. I have an *.epd with Stockfish and Komodo files and can see ...

0 = ...
1 = Komodo
2 = Stockfish

Better here without "0"
The purpose of 0 is when a user does not care about the analyzing engine.
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Game Extractor v1.2

Post by Ferdy »

Frank Quisinsky wrote: 3:
If your program is done I can see ...

Output files:
in_criteria_unique_beta-v03.pgn
in_criteria_not_unique_beta-v03.pgn
not_in_criteria_unique_beta-v03.pgn
not_in_criteria_not_unique_beta-v03.pgn
in_criteria_houdini-analysis.epd
not in criteria houdini-analysis.epd ("_" missed)
Are you sure about the following?
not in criteria houdini-analysis.epd ("_" missed)
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Game Extractor v1.2

Post by Ferdy »

Frank Quisinsky wrote: And it would nice to have the complete output of your program in a new "log" *.txt file. Maybe with the name the epd have ...

Example:
houdini-analysis.log
I will not implement logging, the user can copy the text in console and paste it in a text processor.
This is why I add a pause,
"Press enter key to exit"
so that a user can save what is in console.

New version with output game and epd counts.

Code: Select all

Game Extractor v1.3

File: criteria.txt

start
refepd, st-ko-analysis.epd
refpgn, alpha.pgn
minscorecp, -30
maxscorecp, +30
mincntqueen, 0
maxcntqueen, 2
end

Available analyzing engines in ref epd:
[0]
[1] Komodo 10.3 4C
[2] Stockfish 16012017 BMI2 x64

Enter number of analyzing engine, 0 to ignore? 0

(1) Filtering FEN based on criteria.txt ...
elapsed: 24.6s

(2) Extracting unique games based from filtered FEN and extracting non-unique ga
mes from unfiltered FEN using pgn-extract ...
elapsed: 8.0s

(3) Extracting unique games from unfiltered pgn file using pgn-extract ...
elapsed: 2.6s

(4) Extracting non-unique games based from filtered FEN using pgn-extract ...
elapsed: 8.5s

(5) Deleting temp file fenTag.txt ...

Output files:
_in_criteria_unique_alpha.pgn (games = 21749)
_in_criteria_not_unique_alpha.pgn (games = 75445)
_not_in_criteria_unique_alpha.pgn (games = 16388)
_not_in_criteria_not_unique_alpha.pgn (games = 65668)
_in_criteria_st-ko-analysis.epd (positions = 39633)
_not_in_criteria_st-ko-analysis.epd (positions = 14039)


Done!!

Press enter key to exit