Hi Ferdinand,
OK, wonderful ...
I will update if I have version 1.3 on my site.
Best
Frank
Modular opening book SF analysed 87417 pos., beta-1
Moderators: hgm, Rebel, chrisw
-
Frank Quisinsky
- Posts: 6808
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Game Extractor v1.3
Last edited by Frank Quisinsky on Sun Mar 12, 2017 1:33 pm, edited 1 time in total.
-
Frank Quisinsky
- Posts: 6808
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Game Extractor v1.2
Hm ...
No, if I try it again ... all is right.
Maybe it have to do with overwritten ...
Means, after the first try I got the output files and after the second try output files are overwritten in the same directory.
Forget it ... not important!
Best
Frank
On sunday ... all that work!
Thanks Ferdinand again!!
No, if I try it again ... all is right.
Maybe it have to do with overwritten ...
Means, after the first try I got the output files and after the second try output files are overwritten in the same directory.
Forget it ... not important!
Best
Frank
On sunday ... all that work!
Thanks Ferdinand again!!
-
Ferdy
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
-
Frank Quisinsky
- Posts: 6808
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Game Extractor v1.3
Hi Ferdinand,
Version 1.3 added on download area and in project files.
Now we need v-01 (after Houdini analyzes, need a while, still running). With the first version users can do much more.
Best
Frank
Version 1.3 added on download area and in project files.
Now we need v-01 (after Houdini analyzes, need a while, still running). With the first version users can do much more.
Best
Frank
-
Frank Quisinsky
- Posts: 6808
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Game Extractor v1.2
Hi Ferdinand,
OK, understand it ...
All is fine now!
From my view ... no more ideas!
Program working perfect and can be used for the future.
Best
Frank
OK, understand it ...
All is fine now!
From my view ... no more ideas!
Program working perfect and can be used for the future.
Best
Frank
-
Ferdy
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Modular opening book SF analysed 87417 pos., beta-1
Chess Opening Wizard 2016 Build 68.
Filtered unique pgn file with ce -/+1000 on Ae stockfish, import it to COW 2016, also import the epd with ce, backsolved it and I get this. Best is d4 with +24 cp eval.
Best defense for black after d4 is
Filtered unique pgn file with ce -/+1000 on Ae stockfish, import it to COW 2016, also import the epd with ce, backsolved it and I get this. Best is d4 with +24 cp eval.
Best defense for black after d4 is
-
Frank Quisinsky
- Posts: 6808
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Modular opening book SF analysed 87417 pos., beta-1
Hi Ferdinand,
the software isn't free available!
Very nice, yes (eval average) ... but again a new GUI for things we have in x others.
Important is to create books with all the balanced positions to different opening book file format ... working on it, at first for Shredder Classic GUI.
Furthermore, users can import later the database basics to each favorite GUIs / engine book or GUI opening book.
I am thinking it would be nice to have the eval from 10 engines as comment in *.pgn ... later if all is ready!
Again, very nice what BookUp do ... and with the database we create BookUp will be have great basics too. So BookUp fans can find additional material.
Best
Frank
the software isn't free available!
Very nice, yes (eval average) ... but again a new GUI for things we have in x others.
Important is to create books with all the balanced positions to different opening book file format ... working on it, at first for Shredder Classic GUI.
Furthermore, users can import later the database basics to each favorite GUIs / engine book or GUI opening book.
I am thinking it would be nice to have the eval from 10 engines as comment in *.pgn ... later if all is ready!
Again, very nice what BookUp do ... and with the database we create BookUp will be have great basics too. So BookUp fans can find additional material.
Best
Frank
-
Dann Corbit
- Posts: 12541
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Modular opening book SF analysed 87417 pos., beta-1
Analyzed positions from Frank's book
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=572461
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=572461
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
Dann Corbit
- Posts: 12541
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Modular opening book SF analysed 87417 pos., beta-1
From the TCEC tournament results, I have changed my mind about book building.
I used to think that book positions should be between 0 and 35 centipawns.
I think that is a mistake. All that leads to is a billion boring draws.
I think a better book has all positions between 55 and 25 centipawns in absolute value (e.g. 53, -50, 26 and -29 are all fine but -13 and 5 are too low and 60 and -67 are too high). Any position in the tournament book should be sanity checked to 36 plies depth before use by a top 3 engine.
This format assumes that the positions will be played equally by all engines in the contest both as white and black.
So, for instance, if position 17 is chosen, then every engine should play position 17 both as white and as black.
I used to think that book positions should be between 0 and 35 centipawns.
I think that is a mistake. All that leads to is a billion boring draws.
I think a better book has all positions between 55 and 25 centipawns in absolute value (e.g. 53, -50, 26 and -29 are all fine but -13 and 5 are too low and 60 and -67 are too high). Any position in the tournament book should be sanity checked to 36 plies depth before use by a top 3 engine.
This format assumes that the positions will be played equally by all engines in the contest both as white and black.
So, for instance, if position 17 is chosen, then every engine should play position 17 both as white and as black.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
Frank Quisinsky
- Posts: 6808
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Modular opening book SF analysed 87417 pos., beta-1
Hi Dann,
I try to select out 3-fold-repetion before the project started with my "FCP Live Book".
1. During the time "FCP Rating List" was still running I delete lines with 3-fold repetition directly with "priority e" in my "FCP Live Book". And in my download file to my Rating List you can see all that lines, collect in a database.
2. Around 80% of lines I am using for my opening book project comes from my 312.550 strong FCP Rating List database with the setting:
- average from 67 participant engines with 0.45 / -0.25
I am using "Universal PGN EPD tool v2" by Ferdinand here.
After I create that database I reject all the fast draw games undo move 30 from the result too.
With the result that 80% of my project database have a very high contempt itself.
Note:
My long years optimized "FCP Live Book" is after three years development time on a very high level. All good know lines in theory can be play.
---
After three years looking in evaluation from FCP Rating List games, believe me ... I know the evaluations from participant engines. Because I am looking in the games, three moves after the "FCP Live Book" ended for bad lines. Bad lines I select out from test-run to test-run ... over 7.600 lines I select out during the FCP Rating List was still running. Working the most of the time in the last three years on it. Because my main interest was from minute 1 ... with eng-eng games to create a strong database for a new opening book. ELO of engines isn’t my main topic. Playing styles and opening book was all the time in my main interest. Most games I select out comes from correspondence games … grandmaster games have a much higher theory quality.
Very interesting is that I added in the beginning of FCP Rating List lines undo 6 moves in my book. So engines can find out better theory after move 6. After each test-run new moves ... between 6-12 I added in my book, the reason that the last version from FCP Live Book was version 4.49 … from test-run to test-run I made FCP Live Book update.
---
After all, the database I created to all 500 ECO codes (always, three moves after ECO Code formed) is on a very high level. You can find many balanced lines, never played in GM chess. The main database is the result of more as three years hard work because I do the same with SWCR Rating List, but in times SWCR Rating List are still running, never I have such a strong book I have in times from my FCP Rating List with my much better FCP Live Book. Means that most of basics comes from the very old SWCR database, included later in FCP Rating List. Of Course again with new GM and correspondence chess theory. So I am working many years on that.
---
I have the same opinion that 0.35 is to hard (TCEC isn't important for me, I am sorry for that). I am looking in the TCEC book lines but many of the lines are not good enough for the complete group of participant engines.
Example:
Stockfish and Komodo reject the main files I have in the last months with the setting 0.50 / -0.30 and now Houdini is still running. After the first 10.000 Analysis Houdini found with 0.65 / -0.45 again over 3% of lines Komodo and Stockfish not haven’t seen, better for Komodo and Stockfish fully OK. Houdini is ofter much more aggressive after the opening book moves comparing to Stockfish and Komodo. And Houdini have around the same level Stockfish / Komodo have. To optimize an opening book with one engine only is totally wrong because Stockfish can't see all, Komodo can't see all ... not important how strong are these engines. OK, Houdini is more speculative but in most of cases founded.
OK, Houdini have clearly an higher evaluation in comparing to Stockfish and Komodo. For that reason I can't use for each engine the same setting. I can’t say … I analyzed opening positions with one engine only. I can’t say I am using the best engine for it. If Stockfish do it alone … 10% of book moves are for others not playable.
An additional good example is Fizbo ... I am thinking with Fizbo I have to give:
0.80 / -0.50 … maybe higher!!
---
After all my knowledge in looking the games from FCP Rating List I know the positions engines like Fizbo or Houdini will give me very high evaluation. Because I do that three years for 67 engines and made here a lot of stats.
Never we can create an optimal opening book, an optimal rating list system but with more and more knowledge we can do it better and better. And with all my knowledge and ideas I try now to create now "my best result". I am sure on the end of the day, better in six months the result I can present will be very strong.
---
The basic database I have is now beta-03.pgn (with and without move transpositions, doubles to end of lines).
And this database, believe me ... three moves after ECO codes was formed (for an example: C54 formed after move 6 ... my lines goes undo move 9, E96 formed after 9 moves ... my lines goes undo move 12) is very strong and very balanced with a very high contempt itself.
Most of perhaps good known missed lines are reject before ... most of reasons are 3-fold-repetition for it, will not have such lines in an opening book for engine-engine testing.
CONTEMPT of opening book should be as high as possible … VERY IMPORTANT !!!!!!!!!! … for really a good opening book!
---
36 plies depth isn't possible.
I am using for analyzing 6 cores i7 with 4.3 Ghz.
Average depth for Houdini is 24, for Komodo around 30 for Stockfish around 32 after 1-minute time for each position. I think for find out bad lines, three moves after ECO codes formed ... it is absolutely enough. For 36 plies Houdini running for one position a lot of days and I have without move Transposition over 26.000 to check ... not possible for me and absolutely not necessary. Three moves only … after ECO codes was formed I don’t need a super computer with the result that engines will give me a very high selective depth.
---
If I reject with 0.35 to around 70 ECO codes no more material will be included in the final results. Not in my interest because engine should give us to all possible openings better lines. So the database must be very strong and all ECO codes should be included.
But for test-set 0.35 can be interesting. With the tool by Ferdinand users can reject from beta-03 with own settings very easy!
---
After all I think most of balanced positions you can produce with all ECO codes I have in my database. Maybe different very rarely older lines can be missed.
Again nothing is perfect but I can try to give my best and the result today is fantastic.
Engines can search with my database ... with eng-eng matches ... better ways in good known theory. So the games we produced are on a much more higher level without opening book errors. Maybe other will used the database, others made here own … not important for me. I will have it, so I have to create it. Because available material isn’t good enough for myself.
I have more interest to produce my own with the knowledge I have and I can be sure ... that's the best for myself I can do. And with such material I can present optimal engine-engine result ... give with my work a little bit back what all the programmers do for me. More or less my thanks!
Best
Frank
I am so happy that Ferdinand / Norm / David helps here. Ferdinand is really a great programmer ... my favorite programmer because Ferdinand is not looking on here won things only and give the community his helps. The tool he created will be very important for the future for opening book devcelopers. And only with good opening books we can create good material with the available engines.
I try to select out 3-fold-repetion before the project started with my "FCP Live Book".
1. During the time "FCP Rating List" was still running I delete lines with 3-fold repetition directly with "priority e" in my "FCP Live Book". And in my download file to my Rating List you can see all that lines, collect in a database.
2. Around 80% of lines I am using for my opening book project comes from my 312.550 strong FCP Rating List database with the setting:
- average from 67 participant engines with 0.45 / -0.25
I am using "Universal PGN EPD tool v2" by Ferdinand here.
After I create that database I reject all the fast draw games undo move 30 from the result too.
With the result that 80% of my project database have a very high contempt itself.
Note:
My long years optimized "FCP Live Book" is after three years development time on a very high level. All good know lines in theory can be play.
---
After three years looking in evaluation from FCP Rating List games, believe me ... I know the evaluations from participant engines. Because I am looking in the games, three moves after the "FCP Live Book" ended for bad lines. Bad lines I select out from test-run to test-run ... over 7.600 lines I select out during the FCP Rating List was still running. Working the most of the time in the last three years on it. Because my main interest was from minute 1 ... with eng-eng games to create a strong database for a new opening book. ELO of engines isn’t my main topic. Playing styles and opening book was all the time in my main interest. Most games I select out comes from correspondence games … grandmaster games have a much higher theory quality.
Very interesting is that I added in the beginning of FCP Rating List lines undo 6 moves in my book. So engines can find out better theory after move 6. After each test-run new moves ... between 6-12 I added in my book, the reason that the last version from FCP Live Book was version 4.49 … from test-run to test-run I made FCP Live Book update.
---
After all, the database I created to all 500 ECO codes (always, three moves after ECO Code formed) is on a very high level. You can find many balanced lines, never played in GM chess. The main database is the result of more as three years hard work because I do the same with SWCR Rating List, but in times SWCR Rating List are still running, never I have such a strong book I have in times from my FCP Rating List with my much better FCP Live Book. Means that most of basics comes from the very old SWCR database, included later in FCP Rating List. Of Course again with new GM and correspondence chess theory. So I am working many years on that.
---
I have the same opinion that 0.35 is to hard (TCEC isn't important for me, I am sorry for that). I am looking in the TCEC book lines but many of the lines are not good enough for the complete group of participant engines.
Example:
Stockfish and Komodo reject the main files I have in the last months with the setting 0.50 / -0.30 and now Houdini is still running. After the first 10.000 Analysis Houdini found with 0.65 / -0.45 again over 3% of lines Komodo and Stockfish not haven’t seen, better for Komodo and Stockfish fully OK. Houdini is ofter much more aggressive after the opening book moves comparing to Stockfish and Komodo. And Houdini have around the same level Stockfish / Komodo have. To optimize an opening book with one engine only is totally wrong because Stockfish can't see all, Komodo can't see all ... not important how strong are these engines. OK, Houdini is more speculative but in most of cases founded.
OK, Houdini have clearly an higher evaluation in comparing to Stockfish and Komodo. For that reason I can't use for each engine the same setting. I can’t say … I analyzed opening positions with one engine only. I can’t say I am using the best engine for it. If Stockfish do it alone … 10% of book moves are for others not playable.
An additional good example is Fizbo ... I am thinking with Fizbo I have to give:
0.80 / -0.50 … maybe higher!!
---
After all my knowledge in looking the games from FCP Rating List I know the positions engines like Fizbo or Houdini will give me very high evaluation. Because I do that three years for 67 engines and made here a lot of stats.
Never we can create an optimal opening book, an optimal rating list system but with more and more knowledge we can do it better and better. And with all my knowledge and ideas I try now to create now "my best result". I am sure on the end of the day, better in six months the result I can present will be very strong.
---
The basic database I have is now beta-03.pgn (with and without move transpositions, doubles to end of lines).
And this database, believe me ... three moves after ECO codes was formed (for an example: C54 formed after move 6 ... my lines goes undo move 9, E96 formed after 9 moves ... my lines goes undo move 12) is very strong and very balanced with a very high contempt itself.
Most of perhaps good known missed lines are reject before ... most of reasons are 3-fold-repetition for it, will not have such lines in an opening book for engine-engine testing.
CONTEMPT of opening book should be as high as possible … VERY IMPORTANT !!!!!!!!!! … for really a good opening book!
---
36 plies depth isn't possible.
I am using for analyzing 6 cores i7 with 4.3 Ghz.
Average depth for Houdini is 24, for Komodo around 30 for Stockfish around 32 after 1-minute time for each position. I think for find out bad lines, three moves after ECO codes formed ... it is absolutely enough. For 36 plies Houdini running for one position a lot of days and I have without move Transposition over 26.000 to check ... not possible for me and absolutely not necessary. Three moves only … after ECO codes was formed I don’t need a super computer with the result that engines will give me a very high selective depth.
---
If I reject with 0.35 to around 70 ECO codes no more material will be included in the final results. Not in my interest because engine should give us to all possible openings better lines. So the database must be very strong and all ECO codes should be included.
But for test-set 0.35 can be interesting. With the tool by Ferdinand users can reject from beta-03 with own settings very easy!
---
After all I think most of balanced positions you can produce with all ECO codes I have in my database. Maybe different very rarely older lines can be missed.
Again nothing is perfect but I can try to give my best and the result today is fantastic.
Engines can search with my database ... with eng-eng matches ... better ways in good known theory. So the games we produced are on a much more higher level without opening book errors. Maybe other will used the database, others made here own … not important for me. I will have it, so I have to create it. Because available material isn’t good enough for myself.
I have more interest to produce my own with the knowledge I have and I can be sure ... that's the best for myself I can do. And with such material I can present optimal engine-engine result ... give with my work a little bit back what all the programmers do for me. More or less my thanks!
Best
Frank
I am so happy that Ferdinand / Norm / David helps here. Ferdinand is really a great programmer ... my favorite programmer because Ferdinand is not looking on here won things only and give the community his helps. The tool he created will be very important for the future for opening book devcelopers. And only with good opening books we can create good material with the available engines.