Utility to remove duplicate positions from a PGN?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

Utility to remove duplicate positions from a PGN?

Post by sincerelybrent » Fri Mar 29, 2019 4:22 pm

Is there a utility to remove duplicate positions from a PGN? Thanks

User avatar
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

Re: Utility to remove duplicate positions from a PGN?

Post by sincerelybrent » Fri Mar 29, 2019 5:40 pm

Looks like pgnscanner is the answer. Is it common for it to keep finding absolute or positional double positions, even after they've been removed? The first round, it found 9 doubles. I removed them. Then it found 4. I removed them. Then it found 5. I removed them. Now it found 4 more. The total number of positions in the file keeps going down by the number of positions I remove, so I know it's reading current and correct information.

Dann Corbit
Posts: 10096
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Utility to remove duplicate positions from a PGN?

Post by Dann Corbit » Fri Mar 29, 2019 5:49 pm

Duplicate positions, or duplicate games?

pgn-extract removes duplicate games with the -d flag.

I do not understand why you would want to remove duplicate positions unless you are analyzing EPD.
And in that case it is trivial to pipe the sorted EPD to uniq first and get a distinct list
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

User avatar
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

Re: Utility to remove duplicate positions from a PGN?

Post by sincerelybrent » Fri Mar 29, 2019 6:44 pm

I was creating my own opening pgn for engine testing (both Silver suites, some Noomen, some Balsa, some Pohl, some Lc0, some TCEC, some of my own...). I found the duplicates.

Frank Quisinsky
Posts: 4852
Joined: Wed Nov 18, 2009 6:16 pm
Location: Trier, Germany
Contact:

Re: Utility to remove duplicate positions from a PGN?

Post by Frank Quisinsky » Sat Mar 30, 2019 6:44 am

Hi,

or looking in the FEOBOS documentation for such things (included in download file _basics.zip).
Here I added a file: readme-to-the-important-parameters.txt

http://www.amateurschach.de (menue FEOBOS).

Here you can find the information about tools, parmeters and also the files from FEOBOS without move transpositions and doubles. And of course sorted files with a ranking system for opening books in *.epd, *.pgn and all the other material.

Best
Frank
I like computer chess!

Post Reply