Modular opening book SF analysed 87417 pos., beta-1
Moderators: hgm, Rebel, chrisw
-
- Posts: 6811
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Update v1.03 ... short question!
Hi Ferdinand,
must drive to my professional work now ...
Shortly ...
I made an update on my detail page to the book project.
Version 1.03 is now available (small and big).
- alpha files (without develops files) now in small directory too. Important here the alpha.pgn and alpha.epd file.
Different to the big file only:
In the big file are the log files of engine anlysis and development files for the alpha.pgn database. All other files are available in the small download file. So 99% of users have enough with the small download file.
- new readme file (explain how I can create the important *.epd files with the tools by Norm). I think important. Norm have good readmes but in this case I have all what is important for my project in a short readme.
- some changes in my other readme files and much spelling error corrections.
- better data structure inside the *.zip file for better understandings.
- some other small changes!
###
Later if you are ready with your tool ...
I would like to have your new tool in download area and in the small and big project *.zip files.
Later, the project is ready, most need your tool only for creating own files with the database basic files.
I am sure you will give your aggreement.
I like that project ... that is the final result of three years FCP Rating List. So the games I produced made sense for much more interesting things.
###
Best
Frank
must drive to my professional work now ...
Shortly ...
I made an update on my detail page to the book project.
Version 1.03 is now available (small and big).
- alpha files (without develops files) now in small directory too. Important here the alpha.pgn and alpha.epd file.
Different to the big file only:
In the big file are the log files of engine anlysis and development files for the alpha.pgn database. All other files are available in the small download file. So 99% of users have enough with the small download file.
- new readme file (explain how I can create the important *.epd files with the tools by Norm). I think important. Norm have good readmes but in this case I have all what is important for my project in a short readme.
- some changes in my other readme files and much spelling error corrections.
- better data structure inside the *.zip file for better understandings.
- some other small changes!
###
Later if you are ready with your tool ...
I would like to have your new tool in download area and in the small and big project *.zip files.
Later, the project is ready, most need your tool only for creating own files with the database basic files.
I am sure you will give your aggreement.
I like that project ... that is the final result of three years FCP Rating List. So the games I produced made sense for much more interesting things.
###
Best
Frank
-
- Posts: 6811
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: The engine programmers ... like most!
I wrote:
I like that project ... that is the final result of three years FCP Rating List. So the games I produced made sense for much more interesting things.
But most I like ...
All engines programmers I am used engines for FCP Rating List are an important part of the project.
And all of this programmers have now directly a great basics for engine improvements and engine testing.
Because ... 80% of baisc games comes from FCP-Rating-List.
In 30-40% of that games engines try to find better moves after the ECO start started ... games based on FCP Live Book.
![Smile :-)](./images/smilies/icon_smile.gif)
Thats what I like most!
Best
Frank
PS: On the others hand ... basics are 3 moves after ECO code formed only. And most after ECO code formed is good known theory. But often we can find games, no Grandmaster played ... moves comes by engines! Really a great combination between Grandmaster chess, correspondence and computer chess.
I like that project ... that is the final result of three years FCP Rating List. So the games I produced made sense for much more interesting things.
But most I like ...
All engines programmers I am used engines for FCP Rating List are an important part of the project.
And all of this programmers have now directly a great basics for engine improvements and engine testing.
Because ... 80% of baisc games comes from FCP-Rating-List.
In 30-40% of that games engines try to find better moves after the ECO start started ... games based on FCP Live Book.
![Smile :-)](./images/smilies/icon_smile.gif)
Thats what I like most!
Best
Frank
PS: On the others hand ... basics are 3 moves after ECO code formed only. And most after ECO code formed is good known theory. But often we can find games, no Grandmaster played ... moves comes by engines! Really a great combination between Grandmaster chess, correspondence and computer chess.
-
- Posts: 4840
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: The engine programmers ... like most!
Just some basic functionality, from readme,
Contents of criteria.txt
pgn-extract.exe is not included in the package, you need to download it at its official site.
Sample run.
Download game extractor.
https://drive.google.com/file/d/0BwAOsu ... sp=sharing
Code: Select all
Game Extractor v1.0
Extract games based on criteria.txt file.
Files required:
* pgn-extract.exe
* criteria.txt
* your ref pgn file
* your ref epd file
Open criteria.txt to define your ref epd and pgn files as well
as your choice of engine score and number of queens.
If your ref pgn filename is big.pgn the output filenames would be
filtered_big.pgn, and unfiltered_big.pgn. The filtered_big.pgn is
the file that you are interested based from the criteria.txt.
Duplicates in filtered_big.pgn file are also removed.
How to run?
Just double-click gameExtractor.exe or run the batch file run_game_extractor.bat.
This program is tested under windows 7.
Code: Select all
start
refepd, opening.epd
refpgn, big.pgn
minscorecp, -30
maxscorecp, +30
mincntqueen, 0
maxcntqueen, 2
end
Sample run.
Code: Select all
Game Extractor 1.0
Filtering FEN based on criteria.txt ...
elapsed: 8.0s
Extracting games based from filtered FEN using pgn-extract ...
elapsed: 7.7s
Done!!
https://drive.google.com/file/d/0BwAOsu ... sp=sharing
-
- Posts: 6811
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: The engine programmers ... like most!
Ups ...
I have to try out it ... directly!
Will do it ...
Later today!
Best
Frank
I have to try out it ... directly!
Will do it ...
Later today!
Best
Frank
-
- Posts: 6811
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Stefan will be happy ... first beta!
Hi Ferdinand,
I edit the criteria.txt
"alpha.pgn and alpha.epd"
I copy pgn-extract.exe (v17.37) in the same directory.
I edit to 0.5 and -0.3
OS: Windows 10 Prof. x64
After all I got the message:
"Unable to open fenTag.txt for reading."
What is "fenTag.txt"?
Not in the directory from yourself.
Best
Frank
PS: Stefan Pohl will be happy. The good idea by Stefan Pohl is included!!
THANKS!!
I edit the criteria.txt
"alpha.pgn and alpha.epd"
I copy pgn-extract.exe (v17.37) in the same directory.
I edit to 0.5 and -0.3
Code: Select all
start
refepd, alpha.epd
refpgn, alpha.pgn
minscorecp, -30
maxscorecp, +50
mincntqueen, 0
maxcntqueen, 2
end
After all I got the message:
"Unable to open fenTag.txt for reading."
What is "fenTag.txt"?
Not in the directory from yourself.
Best
Frank
PS: Stefan Pohl will be happy. The good idea by Stefan Pohl is included!!
THANKS!!
-
- Posts: 6811
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: It works ... computer chess can be wonderful!
Hi Ferdinand,
I added the *.epd without eval ...
Typical Frank ... forget my last message, the other readers too.
It works ...
FANTASTIC ...
More as fantastic ...
Later ...
Best
Frank
I added the *.epd without eval ...
Typical Frank ... forget my last message, the other readers too.
It works ...
FANTASTIC ...
More as fantastic ...
Later ...
Best
Frank
-
- Posts: 6811
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: But ...
Hi Ferdinand,
but the final result can't be right.
Example: Alpha.pgn = 87.714 games, Alpha.epd = 87.417 the same with ce after SF analysis.
If I set ... 0.50 / -0.30 I should have the same result I found out.
77.976 should be the final result ... if I made no mistakes in my work I do by hand.
But the result with your tool is: 63.541
with the setting:
The program give me now a new PGN with reject games and a second new PGN with the good games.
For a check what is reject ... I need the same in *.epd.
*.epd is very important too. So I can added the reject games later in a new database for a check with other engines (is Stockfish right or not).
two new *.epd databases ...
Reject in *.epd too and the new *.epd with good positions.
With other words I need 4 new files.
reject in *.pgn file Format
reject in *.epd file Format
good games in *.pgn file Format
good games in *.epd file Format
Best
Frank
No idea why the final result in an other.
Maybe it have to do with the parameters ...
but the final result can't be right.
Example: Alpha.pgn = 87.714 games, Alpha.epd = 87.417 the same with ce after SF analysis.
If I set ... 0.50 / -0.30 I should have the same result I found out.
77.976 should be the final result ... if I made no mistakes in my work I do by hand.
But the result with your tool is: 63.541
with the setting:
Code: Select all
start
refepd, alpha.epd
refpgn, alpha.pgn
minscorecp, -30
maxscorecp, +50
mincntqueen, 0
maxcntqueen, 2
end
For a check what is reject ... I need the same in *.epd.
*.epd is very important too. So I can added the reject games later in a new database for a check with other engines (is Stockfish right or not).
two new *.epd databases ...
Reject in *.epd too and the new *.epd with good positions.
With other words I need 4 new files.
reject in *.pgn file Format
reject in *.epd file Format
good games in *.pgn file Format
good games in *.epd file Format
Best
Frank
No idea why the final result in an other.
Maybe it have to do with the parameters ...
Code: Select all
mincntqueen, 0
maxcntqueen, 2
-
- Posts: 4840
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Stefan will be happy ... first beta!
"fenTag.txt" This is the file created by the tool. It contains lines with fen based from criteria.txt.Frank Quisinsky wrote:Hi Ferdinand,
I edit the criteria.txt
"alpha.pgn and alpha.epd"
I copy pgn-extract.exe (v17.37) in the same directory.
I edit to 0.5 and -0.3
OS: Windows 10 Prof. x64Code: Select all
start refepd, alpha.epd refpgn, alpha.pgn minscorecp, -30 maxscorecp, +50 mincntqueen, 0 maxcntqueen, 2 end
After all I got the message:
"Unable to open fenTag.txt for reading."
What is "fenTag.txt"?
Not in the directory from yourself.
Best
Frank
PS: Stefan Pohl will be happy. The good idea by Stefan Pohl is included!!
THANKS!!
Code: Select all
FEN "r2qkb1r/ppp1pppp/5n2/8/3P2b1/5N2/PPP2PPP/R1BQKB1R w KQkq - 0 1"
[...]
That explains it, if your ref epd has no ce then this file will not be created by the tool, and pgn-extract would complain that this file is missing. I will improve error checking of the tool later.
-
- Posts: 4840
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: But ...
I have downloaded new-opening-book_beta-v1.03-big (got the alpha.pgn and alpha.epd), is this 77.976 games that you mentioned available for download? Does that file contain no duplicates?Frank Quisinsky wrote:Hi Ferdinand,
but the final result can't be right.
Example: Alpha.pgn = 87.714 games, Alpha.epd = 87.417 the same with ce after SF analysis.
If I set ... 0.50 / -0.30 I should have the same result I found out.
77.976 should be the final result ... if I made no mistakes in my work I do by hand.
But the result with your tool is: 63.541
with the setting:
Code: Select all
start refepd, alpha.epd refpgn, alpha.pgn minscorecp, -30 maxscorecp, +50 mincntqueen, 0 maxcntqueen, 2 end
The criteria.txt is fine.
I can create those as output in next version.Frank Quisinsky wrote: The program give me now a new PGN with reject games and a second new PGN with the good games.
For a check what is reject ... I need the same in *.epd.
*.epd is very important too. So I can added the reject games later in a new database for a check with other engines (is Stockfish right or not).
two new *.epd databases ...
Reject in *.epd too and the new *.epd with good positions.
With other words I need 4 new files.
reject in *.pgn file Format
reject in *.epd file Format
good games in *.pgn file Format
good games in *.epd file Format
-
- Posts: 6811
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: But ...
Alpha = 87.417 games (before Stockfish analysed)
can be found in alpha directory (my download file 1.03).
During the Stockfish analysis I made an update.
Can be found in alpha subdirectory = 4.728 games.
Important ... I am searching to codes I haven't enough.
This " 4.728 update-games" are checked by SF on an other PC (0.50 and -0.30) too for two times. Same ECO-Code length as I am using for the main database alpha.pgn with 87.417 games. In the 4.728 games are doubles but the move transpositions are included only.
---
After Stockfish analysis I have 77.976 games, reject with 0.50 and -0.30 9.391 games, and reject 50 games by hand ... unusual first moves) = 9.441!
77.976 + 9.441 = 87.417 games
77.976 + 4.728 checked new games I added as update I have ...
82.704 games (called beta-1.pgn).
At the moment Komodo analysed this 82.704 games as next engine!
---
I saw in your readme that your tool recject the doubles.
You wrote:
"Duplicates in filtered_big.pgn file are also removed."
Better is to do that with a parameter in your "criteria.txt".
reject doubles yes / no
---
If you tool not reject the dublicates your tool must have the final results with 0.50 / -0.30 = 9.339 found from the 87.417 *.pgn file called Alpha.pgn in my download version 1.03.
Best
Frank
PS: Yes it would be much better if Output = 4 files with EPD files. But I am sure in your tool must be an error. I am searching at the moment ... searching all without doubles! Maybe I am wrong! Later ...
can be found in alpha directory (my download file 1.03).
During the Stockfish analysis I made an update.
Can be found in alpha subdirectory = 4.728 games.
Important ... I am searching to codes I haven't enough.
This " 4.728 update-games" are checked by SF on an other PC (0.50 and -0.30) too for two times. Same ECO-Code length as I am using for the main database alpha.pgn with 87.417 games. In the 4.728 games are doubles but the move transpositions are included only.
---
After Stockfish analysis I have 77.976 games, reject with 0.50 and -0.30 9.391 games, and reject 50 games by hand ... unusual first moves) = 9.441!
77.976 + 9.441 = 87.417 games
77.976 + 4.728 checked new games I added as update I have ...
82.704 games (called beta-1.pgn).
At the moment Komodo analysed this 82.704 games as next engine!
---
I saw in your readme that your tool recject the doubles.
You wrote:
"Duplicates in filtered_big.pgn file are also removed."
Better is to do that with a parameter in your "criteria.txt".
reject doubles yes / no
---
If you tool not reject the dublicates your tool must have the final results with 0.50 / -0.30 = 9.339 found from the 87.417 *.pgn file called Alpha.pgn in my download version 1.03.
Best
Frank
PS: Yes it would be much better if Output = 4 files with EPD files. But I am sure in your tool must be an error. I am searching at the moment ... searching all without doubles! Maybe I am wrong! Later ...
Last edited by Frank Quisinsky on Fri Mar 03, 2017 10:21 am, edited 2 times in total.