Need a tool ... question for Norm!

Frank Quisinsky · Post by **Frank Quisinsky** » Thu Nov 10, 2016 5:49 am

Hi Norm,

we have A00-E99 and a lot of games!
we have truncate!

Now I need a tool I can say ...
truncate 3 moves after building ECO code
or truncate 4 moves after building ECO code

Example:

Code: Select all

B80     06 = 09       a
B81     06 = 09       b
B82     06 = 12!      b
B83     06 = 10!      c
B84     07 = 11!      b
B85     07 = 10       a
B86     07 = 10       c
B87     07 = 10       c
B88     07 = 10       c
B89     07 = 10       c

That means:
B80 for an example ... ECO code started with move 06 ... I need the games three moves later ... opening is populare (priority A ... but not important for that what I need). Priority is later important!

B84 for a second example ... ECO code started with move 07 ... but here I need the games four moves later because very important lines goes undo move 11. So I set "!" ... if I need more as three moves after ECO code ended I set "!"

Now a database with 100.000 games!

I can give in a table A00 - E99 the length of moves and truncate do what I need. So truncate must only looking in a table with 500 entries for A00-E99.

Such a tool I need, maybe you can add in the truncate you have such a possibility.

Best
Frank

Norm Pollock · Post by **Norm Pollock** » Thu Nov 10, 2016 7:39 am

Hi Frank,

If I understand it, you want to truncate, for example, all B80 games, after a certain number of moves.

Some of my tools may help, I think.

ecoExtract (pgn tool) can be used 500 times to extract games into files for each of the 500 ECOs.

Then you can use truncate on each of the 500 files.

Another approach: I do have a tool ecoSplit (pgn tool) that separates into 5 files - A, B, C, D, E. I would have to extend that tool into options for either 5, 50 or 500 files. I will look at that possibility when I have a chance. But right now that tool does not yet exist.

In either case, you would have to truncate each of the 500 files individually because they have differing number of plies.

-Norm

Btw, on a personal note, this month marks exactly 10 years from the start of releasing my "40H" tools. Jim Ablett was the one who encouraged me and showed me how to compile the binaries from Java.

Frank Quisinsky · Post by **Frank Quisinsky** » Thu Nov 10, 2016 8:07 am

Hi Norm,

yes, your work all the years is indeed more as good!
I am a big fan from that what you do for us.

My congrats!

Yes, I can create with your tools what I like to do but today in the morning I am thinking it must be more easy.

One *.pgn file and one order to a tool.

My rating list ended soon. After three years I have now enough games for the next steps in opening book development. To all important 500 ECO codes and lines engines produced game material.

I hope you have next year again a bit time for me?

I need later ... again ... a *.epd file with end positions of games. I will give the end positions 8 different engines for 1-minute analysis.

I try it out very often in self work but I don't know why ... you explain it me more times ... I am stupid here!

Best
Frank

Norm Pollock · Post by **Norm Pollock** » Thu Nov 10, 2016 1:46 pm

Frank Quisinsky wrote:
...

I need later ... again ... a *.epd file with end positions of games. I will give the end positions 8 different engines for 1-minute analysis.

I try it out very often in self work but I don't know why ... you explain it me more times ... I am stupid here!

Best
Frank

I'll try to explain again.

Be sure you have the latest versions of "pgn-extract" and "epdFin" (40H-EPD tool) available in the same folder. Then execute as follows (watch out for the double hyphen, and no space after hyphens):

Code: Select all

         PGN-Extract -Wepd --nofauxep -s -otemp.epd alpha.pgn

         epdFin temp.epd

Output is outF.epd, which contains the ending position of each game in alpha.pgn. "temp.epd" is a large file, are you getting an overflow error?

Frank Quisinsky · Post by **Frank Quisinsky** » Thu Nov 10, 2016 2:06 pm

Hi Norm,

OK, later this day ...
I will try it again and will give you shortly the problems I created.

Later ...

Best
Frank

Norm Pollock · Post by **Norm Pollock** » Sat Nov 12, 2016 7:28 am

Hi Frank,

Here is a new tool that will create 500 files named ecoA00.pgn to ecoE99.pgn from a pgn file. The games are put into the file corresponding to the eco code in each game. Also there is a file ecoN.pgn for games without an eco tag.

usage:

eco500 alpha.pgn

download link:

http://www.mediafire.com/file/hkcinakkb ... eco500.exe

As for the rest of your project, it might be accomplished using a command script, but that is outside my skill set.

-Norm

Frank Quisinsky · Post by **Frank Quisinsky** » Sat Nov 12, 2016 8:45 am

Hi Norm,

ah ... I try it out yesterday in the morning!
I think I have understand now to create the *.epd with end postions.

I have to try out directly your new work.

Now all can be very easy ...
I can create with truncate the book length for each of the 500 databases. After this one I can sort out the doubles and after this one I can start 8 different engines for analyses of end position for find out the last bad moves.

For me important that my work around my FCP Rating List can be used now. I will have more as 300.000 45-minutes games. With this database I can do lot in book optimization because all important lines played ... for all of the 500 ECO codes with he FCP Live Book I create for it. This FCP Live Book I can make now much better.

A lot of work ... after my Rating list ended. I think I Need 6-9 months from my free time.

With such a strong database I have after I can create the polyglod book, the Wasp Book and a new Shredder book.

Your tools are really very valueable !!!

Best
Frank

Code: Select all

Work file 1:

Frank Quisinsky, Trier, Germany
File created: 09.11.2016
Last changes: 10.11.2016

Priorities:
"a" =  58 = 11.6%
"b" = 107 = 21,4%
"c" = 143 = 28,6%
"d" = 192 = 38,4%


        Move depth    Priorities
        min. = 6      "a" - "d"

A00     01 = 06       c
A01     01 = 06       c
A02     01 = 06       d
A03     01 = 06       c
A04     01 = 06       c
A05     01 = 06       c
A06     02 = 06       c
A07     02 = 06       b
A08     03 = 06       d
A09     02 = 06       d

A10     01 = 06       d
A11     04 = 07       c
A12     03 = 06       d
A13     04 = 07       c
A14     04 = 07       d
A15     03 = 06       d
A16     02 = 06       c
A17     03 = 06       c
A18     04 = 07       c
A19     03 = 06       d

A20     02 = 06       b
A21     02 = 06       b
A22     03 = 06       c
A23     03 = 06       d
A24     05 = 08       d
A25     05 = 08       b
A26     05 = 08       b
A27     03 = 06       d
A28     04 = 07       b
A29     04 = 07       a

A30     03 = 06       a
A31     04 = 07       c
A32     05 = 08       c
A33     05 = 08       b
A34     03 = 09!      a
A35     03 = 07!      c
A36     05 = 11!      b
A37     05 = 10!      b
A38     07 = 10       c
A39     07 = 10       c

A40     01 = 06       b
A41     04 = 07       b
A42     04 = 07       c
A43     02 = 10!      b
A44     02 = 09!      d
A45     02 = 06       b
A46     03 = 07!      b
A47     03 = 06       c
A48     03 = 08!      b
A49     03 = 06       c

A50     03 = 08!      d
A51     03 = 06       d
A52     04 = 09!      d
A53     04 = 07       c
A54     05 = 10!      d
A55     05 = 10!      c
A56     03 = 06       c
A57     04 = 08!      b
A58     06 = 11!      c
A59     07 = 12!      c

A60     06 = 09       c
A61     06 = 10!      c
A62     09 = 12       c
A63     11 = 14       d
A64     11 = 14       d
A65     07 = 10       b
A66     07 = 13!      d
A67     08 = 12!      d
A68     09 = 12       d
A69     09 = 12       d

A70     08 = 11       b
A71     08 = 12!      d
A72     09 = 13!      d
A73     09 = 12       d
A74     10 = 13       d
A75     10 = 13       d
A76     10 = 13       d
A77     10 = 13       d
A78     11 = 14       d
A79     11 = 14       d

A80     02 = 06       c
A81     02 = 09!      c
A82     04 = 07       d
A83     04 = 07       d
A84     02 = 06       c
A85     03 = 06       c
A86     05 = 08       d
A87     07 = 10       c
A88     07 = 10       b
A89     07 = 10       c

A90     04 = 07       b
A91     05 = 08       d
A92     07 = 11!      d
A93     08 = 11       d
A94     08 = 11       d
A95     07 = 10       d
A96     07 = 10       d
A97     08 = 11       d
A98     08 = 11       d
A99     08 = 11       d

B00     01 = 06       c
B01     01 = 06       a
B02     03 = 06       d
B03     03 = 10!      c
B04     04 = 11!      b
B05     04 = 11!      d
B06     02 = 06       a
B07     04 = 07       a
B08     04 = 08!      b
B09     04 = 08!      c

B10     02 = 06       b
B11     03 = 06       b
B12     03 = 08!      a
B13     05 = 08       b
B14     05 = 09!      b
B15     05 = 08       c
B16     05 = 08       d
B17     04 = 08!      b
B18     07 = 10       b
B19     07 = 12!      a

B20     02 = 06       c
B21     03 = 07!      c
B22     02 = 10!      a
B23     02 = 06       b
B24     05 = 08       d
B25     06 = 09       c
B26     06 = 10       d
B27     02 = 08!      c
B28     02 = 08!      d
B29     02 = 10!      d

B30     03 = 08!      a
B31     03 = 08!      a
B32     04 = 09!      b
B33     05 = 11!      a
B34     07 = 11!      d
B35     07 = 11!      c
B36     05 = 11!      b
B37     06 = 09       c
B38     06 = 09       b
B39     07 = 10       d

B40     05 = 09!      b
B41     05 = 09!      c
B42     05 = 12!      a
B43     05 = 09!      b
B44     05 = 11!      c
B45     05 = 12!      c
B46     05 = 08       b
B47     06 = 09       b
B48     07 = 10       a
B49     07 = 10       c

B50     03 = 07!      b
B51     03 = 07!      a
B52     03 = 07!      b
B53     04 = 09!      b
B54     06 = 09       c
B55     06 = 09       c
B56     06 = 09       c
B57     06 = 09       c
B58     07 = 10       c
B59     07 = 11!      d

B60     06 = 10!      c
B61     07 = 13!      c
B62     07 = 10       d
B63     09 = 12       c
B64     09 = 12       d
B65     09 = 12       c
B66     08 = 11       b
B67     09 = 12       b
B68     11 = 14       d
B69     11 = 14       d

B70     06 = 09       b
B71     06 = 10!      d
B72     08 = 11       d
B73     09 = 12       d
B74     09 = 12       d
B75     07 = 10       d
B76     09 = 12       c
B77     09 = 12       c
B78     10 = 14!      c
B79     10 = 17!      d

B80     06 = 09       a
B81     06 = 09       b
B82     06 = 12!      b
B83     06 = 10!      c
B84     07 = 11!      b
B85     07 = 10       a
B86     07 = 10       c
B87     07 = 10       c
B88     07 = 10       c
B89     07 = 10       c

B90     06 = 09       a
B91     06 = 09       c
B92     06 = 12!      b
B93     06 = 09       c
B94     06 = 10!      c
B95     07 = 10       d
B96     07 = 11!      c
B97     07 = 10       c
B98     09 = 12       d
B99     09 = 14!      c

C00     03 = 06       b
C01     03 = 06       b
C02     03 = 06       b
C03     04 = 07       c
C04     04 = 07       d
C05     07 = 10       d
C06     07 = 10       b
C07     04 = 07       b
C08     05 = 08       c
C09     05 = 08       c

C10     03 = 06       a
C11     04 = 07       a
C12     04 = 07       c
C13     06 = 09       b
C14     06 = 09       c
C15     04 = 07       c
C16     04 = 07       c
C17     05 = 08       d
C18     07 = 10       a
C19     07 = 10       c

C20     07 = 10       d
C21     01 = 06       d
C22     03 = 06       d
C23     02 = 06       d
C24     02 = 06       c
C25     02 = 06       d
C26     03 = 06       c
C27     03 = 06       d
C28     03 = 06       d
C29     03 = 06       d

C30     02 = 06       d
C31     04 = 07       d
C32     04 = 07       d
C33     03 = 06       d
C34     03 = 06       d
C35     03 = 06       d
C36     03 = 06       d
C37     04 = 07       d
C38     04 = 07       d
C39     04 = 07       d

C40     02 = 06       d
C41     02 = 06       a
C42     03 = 06       a
C43     03 = 06       b
C44     04 = 07       d
C45     04 = 07       a
C46     04 = 07       d
C47     04 = 07       c
C48     04 = 07       c
C49     04 = 07       c

C50     04 = 07       c
C51     05 = 08       d
C52     05 = 08       d
C53     05 = 08       d
C54     06 = 09       a
C55     05 = 08       b
C56     05 = 08       d
C57     05 = 08       d
C58     05 = 08       d
C59     08 = 11       d

C60     08 = 11       c
C61     03 = 06       d
C62     03 = 06       d
C63     03 = 06       c
C64     03 = 06       d
C65     04 = 07       b
C66     04 = 07       c
C67     04 = 08!      a
C68     05 = 08       b
C69     05 = 08       c

C70     04 = 07       d
C71     04 = 07       d
C72     05 = 08       d
C73     05 = 08       d
C74     05 = 08       d
C75     06 = 09       d
C76     06 = 09       c
C77     05 = 08       b
C78     05 = 08       a
C79     05 = 08       d

C80     09 = 12       c
C81     09 = 12       d
C82     09 = 12       c
C83     09 = 12       b
C84     06 = 09       b
C85     06 = 09       c
C86     06 = 09       c
C87     06 = 09       c
C88     08 = 11       a
C89     08 = 11       b

C90     09 = 12       b
C91     09 = 12       b
C92     09 = 12       a
C93     09 = 12       c
C94     10 = 13       d
C95     10 = 13       a
C96     11 = 14       b
C97     12 = 15       c
C98     12 = 15       d
C99     12 = 15       b

D00     03 = 06       b
D01     03 = 06       d
D02     03 = 06       a
D03     03 = 06       b
D04     03 = 06       d
D05     03 = 06       c
D06     02 = 06       d
D07     02 = 06       c
D08     05 = 08       d
D09     05 = 08       d

D10     03 = 06       a
D11     05 = 08       a
D12     03 = 06       a
D13     04 = 07       b
D14     06 = 09       c
D15     05 = 08       a
D16     05 = 08       b
D17     06 = 09       a
D18     09 = 12       b
D19     09 = 12       b

D20     03 = 06       a
D21     04 = 07       c
D22     04 = 07       d
D23     04 = 07       b
D24     04 = 07       c
D25     04 = 07       c
D26     06 = 09       c
D27     07 = 10       a
D28     08 = 11       d
D29     08 = 11       d

D30     03 = 06       a
D31     04 = 07       a
D32     06 = 09       c
D33     07 = 10       d
D34     07 = 10       b
D35     06 = 09       b
D36     06 = 09       b
D37     05 = 08       a
D38     05 = 08       a
D39     05 = 08       b

D40     05 = 08       c
D41     07 = 10       b
D42     07 = 10       c
D43     05 = 08       a
D44     05 = 08       b
D45     06 = 09       a
D46     06 = 09       a
D47     08 = 11       b
D48     11 = 14       b
D49     11 = 14       d

D50     04 = 07       d
D51     06 = 09       d
D52     06 = 09       c
D53     06 = 09       c
D54     06 = 09       d
D55     07 = 10       c
D56     09 = 12       c
D57     09 = 12       d
D58     08 = 11       b
D59     08 = 11       d

D60     07 = 10       d
D61     07 = 10       d
D62     07 = 10       d
D63     08 = 11       d
D64     09 = 12       d
D65     09 = 12       d
D66     09 = 12       d
D67     12 = 15       d
D68     13 = 16       d
D69     13 = 16       d

D70     04 = 07       c
D71     06 = 09       d
D72     06 = 09       c
D73     06 = 09       c
D74     08 = 11       d
D75     08 = 11       d
D76     07 = 10       b
D77     06 = 09       c
D78     07 = 10       b
D79     07 = 10       b

D80     04 = 07       b
D81     04 = 07       d
D82     05 = 08       d
D83     06 = 09       d
D84     06 = 12!      d
D85     07 = 10       a
D86     08 = 11       d
D87     10 = 13       b
D88     13 = 16       d
D89     13 = 16       d

D90     05 = 08       b
D91     05 = 08       c
D92     06 = 09       c
D93     06 = 09       c
D94     06 = 09       a
D95     06 = 09       d
D96     07 = 10       d
D97     07 = 10       b
D98     09 = 12       d
D99     09 = 12       d

E00     04 = 07       c
E01     04 = 07       b
E02     05 = 08       d
E03     06 = 09       d
E04     05 = 08       a
E05     05 = 08       a
E06     06 = 09       c
E07     07 = 10       c
E08     08 = 11       b
E09     08 = 11       c

E10     04 = 07       c
E11     03 = 07       a
E12     05 = 08       a
E13     05 = 08       b
E14     04 = 10!      b
E15     05 = 08       a
E16     05 = 09!      b
E17     07 = 10       b
E18     09 = 12       b
E19     09 = 12       d

E20     04 = 07       a
E21     04 = 07       c
E22     05 = 08       d
E23     05 = 08       d
E24     07 = 10       d
E25     07 = 10       c
E26     06 = 09       d
E27     06 = 09       d
E28     07 = 10       d
E29     07 = 10       d

E30     06 = 09       d
E31     04 = 10!      d
E32     04 = 08!      a
E33     04 = 07       c
E34     05 = 09!      c
E35     05 = 09!      c
E36     06 = 09       c
E37     06 = 11!      c
E38     05 = 08       c
E39     05 = 08       b

E40     04 = 09!      d
E41     05 = 08       c
E42     05 = 08       b
E43     05 = 08       c
E44     05 = 08       d
E45     05 = 08       d
E46     05 = 08       b
E47     05 = 08       d
E48     06 = 09       c
E49     06 = 11!      c

E50     05 = 08       d
E51     06 = 09       d
E52     06 = 09       c
E53     07 = 10       d
E54     08 = 12!      b
E55     08 = 10       c
E56     09 = 12       d
E57     09 = 12       d
E58     09 = 12       d
E59     09 = 12       c

E60     06 = 09       b
E61     05 = 08       b
E62     07 = 10       b
E63     07 = 10       b
E64     07 = 10       d
E65     08 = 11       d
E66     08 = 11       c
E67     08 = 11       d
E68     09 = 12       b
E69     09 = 12       b

E70     05 = 08       b
E71     05 = 09!      c
E72     05 = 08       d
E73     06 = 09       c
E74     07 = 10       d
E75     07 = 10       d
E76     10 = 13       d
E77     07 = 10       d
E78     09 = 12       d
E79     09 = 12       d

E80     05 = 08       d
E81     06 = 09       b
E82     06 = 09       d
E83     08 = 11       d
E84     08 = 11       c
E85     07 = 10       d
E86     07 = 10       d
E87     07 = 10       d
E88     08 = 11       d
E89     08 = 12!      d

E90     06 = 09       a
E91     06 = 09       b
E92     07 = 10       a
E93     08 = 11       d
E94     08 = 11       a
E95     09 = 12       d
E96     09 = 12       d
E97     09 = 12       a
E98     10 = 13       d
E99     10 = 13       c

Jesse Gersenson · Post by **Jesse Gersenson** » Sun Nov 13, 2016 11:43 am

Frank Quisinsky wrote:
Code: Select all
A00     01 = 06       c
A11     04 = 07       c
A34     03 = 09!      a

(My script below assumes you save this table as datafile.txt.)

I'm not sure I understand the next step. Is it to do this

truncate ecoA00.pgn 6
truncate ecoA11.pgn 7
truncate ecoA34.pgn 9

and, perhaps, save this output as truncated-ecoA00.pgn, truncated-ecoA11.pgn, etc?

I'd do this in Linux and bash -- off the top of my head and without tweaking for performance:

Code: Select all

#!/bin/bash

for ecoclass in A B C D E; do
 for ecocode in $(seq -w 1 99); do

	depth=`grep "$ecoclass$ecocode" datafile.txt | cut -b14-15`
	if [  ! -z "$depth" ]
    then 
	    wine truncate.exe "eco$ecoclass$ecocode".pgn $depth
		mv outU.pgn truncated-eco$ecoclass$ecocode".pgn

	fi
	
 done
done

may need to adjust the numbers 14-15. Those are cutting each line of datafile.txt at a certain byte range. Perhaps it will need to be change to 12-13 or 13-14 or 15-16, etc.

if someone knows, please tell me how how to run Norm's java files from the command line... i.e. how to run truncate.class on file helloworld.pgn.

Jesse Gersenson · Post by **Jesse Gersenson** » Sun Nov 13, 2016 12:03 pm

java truncate helloworld.pgn 4

Exception in thread "main" java.lang.UnsupportedClassVersionError: truncate : Unsupported major.minor version 52.0

default-jre version 2:1.7-52 installed.

Norm Pollock · Post by **Norm Pollock** » Sun Nov 13, 2016 5:49 pm

Jesse Gersenson wrote:java truncate helloworld.pgn 4

Exception in thread "main" java.lang.UnsupportedClassVersionError: truncate : Unsupported major.minor version 52.0

default-jre version 2:1.7-52 installed.

Have to first figure out which version of java is installed. Run:

java -version

in the folder where you have "truncate.class". The version should be 6, 7 or 8. The higher the better.

The error message could indicate a problem with the installation. These links may be helpful if you want to reinstall JRE.

https://launchpad.net/ubuntu/wily/amd64 ... s/2:1.7-52

http://tutorialforlinux.com/2015/09/30/ ... -gnulinux/
-Norm

Need a tool ... question for Norm!

Need a tool ... question for Norm!

Re: Need a tool ... question for Norm!

Re: Congrats ... 10 years Norm's Toolbox!!

Re: Congrats ... 10 years Norm's Toolbox!!

Re: Congrats ... 10 years Norm's Toolbox!!

Re: Congrats ... 10 years Norm's Toolbox!!

Re: Congrats ... 10 years Norm's Toolbox!!

Re: Congrats ... 10 years Norm's Toolbox!!

Re: Congrats ... 10 years Norm's Toolbox!!

Re: Congrats ... 10 years Norm's Toolbox!!