Polyglot extension tool

[Dann Corbit]

The problem here is depth.
For an opening position, 36 plies is bare minimum and any new calculations I perform go at least 40 (38 or 39 plies is really good enough, but I figure it will age too fast and be useless in 4 years).
If it is a rather closed position or a gambit, figure 42 plies before you can trust it.

I found that the orangutan opening needs 50 plies before the strongest engines even start to understand it.
Engines do understand 1.d4 pretty well

Engines also have trouble with all the fianchetto Indian style openings.

[Norm Pollock]

Hi thanks ED, that helps a lot.

I don't know how many positions there are in the bin book, how can I know this? If I export the bin book to pgn with Lucas Chess I get 5000 different opening lines. (5000 games).

How can I split the epd file in 5 parts? Is there some software to do that? Or do I have to do it in a text editor?

There are several utils that will do that for you:

40H-EPD - https://40h.000webhostapp.com/
SOMU - http://rebel13.nl/download/utilities.html#somu

But if you only have 5000 EPD lines I would use the advice from Guenther.

Dealing with a large epd file should also include some cleanup. It is likely that the file you are dealing with does not need to be cleaned up, but it wouldn't hurt to check. There are 3 cleanup issues that I check:
1. Are there any records with false "en passant" target squares? If so, this could enable duplicate records to go undetected.
2. Are there any duplicate records? Using duplicates will give extra weight to certain opening positions if the file is used for openings.
3. Are there are records where there are an excessive number of promoted pieces (beyond 1 per side). Removes "composed" positions that are unrealistic.

In my epd tool collection, I have detection and remedies for these issues.
1. epdEnPass to detect false "en passant" target squares.
2. epdOccur to flag duplicates
3. epdExtra to flag records with excessive promotions.

40H-EPD - https://40h.000webhostapp.com/

[Rebel]

The problem here is depth.

And another problem is volume.

But you have to start somewhere and keep updating. 10 million positions from 3000+ engines is not a bad start. Maybe in 2-3 years the minimum elo can be increased to 3100.

For an opening position, 36 plies is bare minimum and any new calculations I perform go at least 40 (38 or 39 plies is really good enough, but I figure it will age too fast and be useless in 4 years).
If it is a rather closed position or a gambit, figure 42 plies before you can trust it.

I found that the orangutan opening needs 50 plies before the strongest engines even start to understand it.
Engines do understand 1.d4 pretty well

Engines also have trouble with all the fianchetto Indian style openings.

And with the stone-wall, probably with more strategic openings.

Besides the comp advice there will be a human advice as well. I have a test database with the games of Magnus Carlsen only and then after 1.e4 c5 the overview looks as follows:

Code: Select all

Reference : consult\3000.dat
Positions : 10.013.033

Moves found for rnbqkbnr/pp1ppppp/8/2p5/4P3/8/PPPP1PPP/RNBQKBNR w KQkq c6

Move Depth Score Engine
g1f3   20  0.28  Stockfish 7
g1f3   21  0.28  Stockfish 6
g1f3   18  0.63  Stockfish 4
g1f3   20  0.34  Stockfish DD
g1f3   20  0.34  Komodo 10
g1f3   18  0.37  Komodo 9
b1c3   19  0.23  Komodo 8
g1f3   19  0.25  Komodo 7
b1c3   18  0.31  Komodo 6
g1f3   18  0.23  Komodo 6
g1f3   18  0.23  Komodo 5
b1c3   19  0.27  Komodo 5
g1f3   18  0.33  Houdini 3
b1c3   17  0.18  Fire 5
g1f3   16  0.28  Fizbo 1.7
g1f3   18  0.21  Fritz 15
b1c3   20  0.30  Fritz 15
g1f3   22  0.18  Chiron 2
g1f3   22  0.25  Chiron 3
g1f3   18  0.21  Gull 2
g1f3   16  0.17  Gull 3
b1c3   19  0.27  Equinox
g1f3   20  0.18  Equinox
g1f3   16  0.20  Critter 1.6
b1c3   16  0.15  Critter 1.6
g1f3   18  0.21  Texel 1.05
b1c3   16  0.46  Naum 4.6
h2h3   17  0.50  Naum 4.6

Reference : consult\gm.dat
Positions : 86.372

Moves found for rnbqkbnr/pp1ppppp/8/2p5/4P3/8/PPPP1PPP/RNBQKBNR w KQkq c6

Move            Player          Game
g1f3 d7d6 d2d4  Carlsen         Carlsen - Gelfand ½-½
g1f3 b8c6 d2d4  Shirov          Shirov - Carlsen ½-½
g1f3 b8c6 d2d4  Topalov         Topalov - Carlsen ½-½
g1f3 b8c6 f1b5  Nunn            Nunn - Carlsen 0-1
g1f3 b8c6 b1c3  Svidler         Svidler - Carlsen 1-0
b1c3 g7g6 h2h4  Carlsen         Carlsen - N.N. 0-1
g1f3 b8c6 d2d4  Morozevich      Morozevich - Carlsen ½-½
g1f3 e7e6 d2d4  Giri            Giri - Carlsen 1-0
b2b3 b8c6 c1b2  Carlsen         Carlsen - Ehlvest 1-0
g1f3 b8c6 d2d4  Anand,V         Anand,V - Carlsen ½-½
b1c3 b8c6 g1f3  Nakamura        Nakamura - Carlsen ½-½
g1f3 d7d6 d2d4  Leko            Leko - Carlsen ½-½
g1f3 b8c6 d2d4  Kamsky          Kamsky - Carlsen 1-0
g1e2 d7d6 b1c3  Carlsen         Carlsen - Nakamura 0-1
g1f3 d7d6 d2d4  Ivanchuk        Ivanchuk - Carlsen ½-½
g1f3 d7d6 f1b5  Bacrot          Bacrot - Carlsen 0-1
g1f3 d7d6 d2d4  Polgar,J        Polgar,J - Carlsen 0-1
g1f3 d7d6 d2d4  Short           Short - Carlsen ½-½
g1f3 b8c6 c2c3  Nakamura        Nakamura - Carlsen ½-½
g1f3 d7d6 d2d4  Adams           Adams - Carlsen ½-½
b1c3 b8c6 g2g3  Short           Short - Carlsen ½-½

I am sure these guys know how to play fianchetto Indian style openings.

[Dann Corbit]

I have a chest chuckle every time I see this:
"90% of coding is debugging, the other 10% is writing bugs."

[Jonathan003]

Hi

Does someone know how it's possible to change the notation in a pgn database.

So the notation is like

1. d4 d5 2. Nf3 Nf6 3. Bf4

instead of

1.d4 d5 2.Nf3 Nf6 3.Bf4

To be able to copie and past the moves here

http://www.ficgs.com/wikichess.html

[Jonathan003]

I find it myself.

It can be done by copying to a new pgn database in ChessX

I wanted to use the website for annotating my repertoire. But it's not very useful I think.

[Jonathan003]

Hi Ed

Poly 1.1 is very useful for me for creating my chess repertoire for over the board games.
I use it in conjunction with the functions of Lucas Chess to import only best line or best lines from a bin book to a openings guide.

I use the big database that comes with powerbook 2019 from chessbase to change the weight of the moves of a bin book with poly 1.1

I just change the names of the player with chess assistant 2019, to be able to use the function 'weight on learn' from poly 1.1.

Can you explain me a bit how this function works? Do it use back solving or mini-max like they cal it?

In addition to be able to set the value 100% for the moves with the highest learn value, or with the highest Stockfish score. I would like to be able to to ad a high value to the best 2 or best 3 moves. (If it are good moves). So the score of the best moves would 50% and 50% or 33%, 33% and 33%.

I want to be able to choose for max one best move, two best moves or three best moves.

If two moves have a score of 50% booth moves of the bin book wil be imported in the opening lines, if I choose to import al best lines, for on color, from a bin book in Lucas Chess. If three moves with a score of 33%, 33% and 33% al three moves wil be imported.

I hope you can help me with that.

[Rebel]

I am not following, first you wanted the best move based on the SF score always on top (100%) and now you want to vary. Surely it can be done but to avoid disappointment just tell me how you want to have it.

Code: Select all

Book  Weight Score Depth Learn
d2d4  51.60%     6   20     0
e2e4  41.96%    19   20     0
c2c4   4.13%     5   20     0
g1f3   2.31%    -1   20     0

How should the weight percentages look like?

And after 1.e4 same question.

Code: Select all

Book  Weight Score Depth Learn
c7c5  38.85%   -32   20     0
e7e5  22.38%   -33   20     0
e7e6  15.20%   -25   20     0
c7c6   8.63%   -34   20     0
d7d6   5.71%   -37   20     0
d7d5   5.46%   -47   20     0
g8f6   3.77%   -52   20     0

[Jonathan003]

Hi Ed

I understand your confusion.

I try to explain why I want multiple best lines. (I want to be able to choose for one best move, two best move or tree best moves)

I want multiple best moves with the same score, because if I import al best moves (for one color) from a bin book, in Lucas Chess, and a moves have for exemple 51% weight and the other move has a weight of 49% only the move with the 51% weight wil be imported. The weight has to be 50% and 50% for booth moves to be imported,(Limitations of Lucas Chess at the moment). If tree moves have a score of 33% al tree moves will be imported if I choose to import al best moves in Lucas Chess.

I want to make two repertoires. A narrow repertoire with only one best candidate move, for the side where the repertoire is build for.
If I know that repertoire by hart. I want to widen my repertoire to allow more variety of play.

And I don't want to ad bad moves to the repertoire. That's why I want to use the 'weight on learn' function from poly.

I don't know anything about programming. So I don't know if and how this can be programmed.

Basically I want to make pgn files from bin books. For my white repertoire I want to ad only the best (or multiple best moves),for white and al possible answers for black. For my black repertoire I want to ad only the best (or multiple best moves), for black and al possible answers for white.

I hope you understand a bit better what I want now.

[Rebel]

Hi Ed

I understand your confusion.

I try to explain why I want multiple best lines. (I want to be able to choose for one best move, two best move or tree best moves)

I want multiple best moves with the same score, because if I import al best moves (for one color) from a bin book, in Lucas Chess, and a moves have for exemple 51% weight and the other move has a weight of 49% only the move with the 51% weight wil be imported. The weight has to be 50% and 50% for booth moves to be imported,(Limitations of Lucas Chess at the moment). If tree moves have a score of 33% al tree moves will be imported if I choose to import al best moves in Lucas Chess.

I want to make two repertoires. A narrow repertoire with only one best candidate move, for the side where the repertoire is build for.
If I know that repertoire by hart. I want to widen my repertoire to allow more variety of play.

And I don't want to ad bad moves to the repertoire. That's why I want to use the 'weight on learn' function from poly.

I don't know anything about programming. So I don't know if and how this can be programmed.

Basically I want to make pgn files from bin books. For my white repertoire I want to ad only the best (or multiple best moves),for white and al possible answers for black. For my black repertoire I want to ad only the best (or multiple best moves), for black and al possible answers for white.

I hope you understand a bit better what I want now.

I understood the first time But look at the part I bolded. You can make a book from PGN but not the other way around, technically impossible. However your wish makes sense and I will figure out some formula that recalculates the weights that represents the SF10 scores.

So:

Code: Select all

Book  Weight Score Depth Learn
d2d4  51.60%   0.06  20    0
e2e4  41.96%   0.19  20    0
c2c4   4.13%   0.05  20    0
g1f3   2.31%  -0.01  20    0

becomes something like:

Code: Select all

Book  Weight Score Depth Learn
e2e4  80.00%   0.19  20    0      
d2d4  10.00%   0.06  20    0      
c2c4  10.00%   0.05  20    0      
g1f3   0.00%  -0.01  20    0  

since the difference of 0.13 (0.19-0.06) with the second best move.

Stay tuned.