Need pgn utility to generate 'position startpos moves' data

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Jesse Gersenson
Posts: 593
Joined: Sat Aug 20, 2011 9:43 am

Need pgn utility to generate 'position startpos moves' data

Post by Jesse Gersenson »

I wrote a command line bash script to automatically evaluate each move in a pgn file with a uci engine and then do some simple math on the evaluation.

The script parsed the pgn file using this php pgn parser https://github.com/DHTMLGoodies/chessParser and passed fen positions to the uci engine like this:

Code: Select all

position fen rnbqkbnr/ppp1pppp/8/8/4p3/2N5/PPPP1PPP/R1BQKBNR w KQkq - 0 3
Works ok, but it overlooks the 50 move rule and move repetition and maybe some other stuff.

So, is there a pgn utility which will take a game in pgn format and output the moves prior to and including the current move? I.e. something I could use to populate uci's "position startpos moves"?

I'll need something like the following output to feed my script, what tool can generate this from a pgn file?

Code: Select all

position startpos moves e2e4 
position startpos moves e2e4 e7e5
position startpos moves e2e4 e7e5 f2f4
edit: afterthought
I'd like my script to eventually be able to run on windows as a command line tool. I've never written anything for windows. Is there any chance a bash script, using a php parser, can become an exe file?
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: Need pgn utility to generate 'position startpos moves' d

Post by zullil »

Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Need pgn utility to generate 'position startpos moves' d

Post by Ferdy »

A standard pgn format can be converted to uci format using pgn-extract. The output is like this, single line per game.

Code: Select all

e2e4 d7d5 e4d5 g8f6 d2d4 f6d5 g1f3 e7e6 f1e2 f8e7 e1g1 [...] 1-0

d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8e7 [...] 0-1

[...]
Command is like this, use the latest version as it supports -Wuci option.

Code: Select all

pgn-extract-17-14 --notags -Wuci -onew.pgn input.pgn
Based from above output I think outputting it to your intended format is not that difficult.
Jesse Gersenson
Posts: 593
Joined: Sat Aug 20, 2011 9:43 am

Re: Need pgn utility to generate 'position startpos moves' d

Post by Jesse Gersenson »

Ferdy wrote:[snip]

Code: Select all

e2e4 d7d5 e4d5 g8f6 d2d4 f6d5 g1f3 e7e6 f1e2 f8e7 e1g1 [...] 1-0

d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8e7 [...] 0-1

[...]
[/snip]
Ah, and just use the space character as a delimiter!

Code: Select all

cut -d' ' -f1-1 new.pgn
cut -d' ' -f1-2 new.pgn
cut -d' ' -f1-3 new.pgn
...
That's easy, thanks.

Now just have to figure out how to compile a .exe from xubuntu...

Code: Select all

#!/bin/bash
echo 'how in the world do i get this bash script to be an exe?'
ZirconiumX
Posts: 1334
Joined: Sun Jul 17, 2011 11:14 am

Re: Need pgn utility to generate 'position startpos moves' d

Post by ZirconiumX »

Jesse Gersenson wrote:
Ferdy wrote:[snip]

Code: Select all

e2e4 d7d5 e4d5 g8f6 d2d4 f6d5 g1f3 e7e6 f1e2 f8e7 e1g1 [...] 1-0

d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8e7 [...] 0-1

[...]
[/snip]
Ah, and just use the space character as a delimiter!

Code: Select all

cut -d' ' -f1-1 new.pgn
cut -d' ' -f1-2 new.pgn
cut -d' ' -f1-3 new.pgn
...
That's easy, thanks.

Now just have to figure out how to compile a .exe from xubuntu...

Code: Select all

#!/bin/bash
echo 'how in the world do i get this bash script to be an exe?'
Bash scripts cannot be made into exes. You could get MinGW and use bash.exe and interpret it from that, but you cannot compile a bash script.

Just copy the file over to xubuntu and run the shell script and copy it back.

Matthew:out
Some believe in the almighty dollar.

I believe in the almighty printf statement.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Need pgn utility to generate 'position startpos moves' d

Post by bob »

Jesse Gersenson wrote:I wrote a command line bash script to automatically evaluate each move in a pgn file with a uci engine and then do some simple math on the evaluation.

The script parsed the pgn file using this php pgn parser https://github.com/DHTMLGoodies/chessParser and passed fen positions to the uci engine like this:

Code: Select all

position fen rnbqkbnr/ppp1pppp/8/8/4p3/2N5/PPPP1PPP/R1BQKBNR w KQkq - 0 3
Works ok, but it overlooks the 50 move rule and move repetition and maybe some other stuff.

So, is there a pgn utility which will take a game in pgn format and output the moves prior to and including the current move? I.e. something I could use to populate uci's "position startpos moves"?

I'll need something like the following output to feed my script, what tool can generate this from a pgn file?

Code: Select all

position startpos moves e2e4 
position startpos moves e2e4 e7e5
position startpos moves e2e4 e7e5 f2f4
edit: afterthought
I'd like my script to eventually be able to run on windows as a command line tool. I've never written anything for windows. Is there any chance a bash script, using a php parser, can become an exe file?
If you can compile C, Crafty has the ability to read a PGN file and spit out a random FEN string, one per game. I can tell you how to adjust the "move # range" so that you can get positions between move x and move y, and it does randomly choose black or white to move as well. This is what I use to create my test positions for cluster testing. It is definitely a hack, because I use the book create command, but when compiled correctly, it will spit out positions into a file as it builds a book.
User avatar
hgm
Posts: 27792
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Need pgn utility to generate 'position startpos moves' d

Post by hgm »

Jesse Gersenson wrote:I wrote a command line bash script to automatically evaluate each move in a pgn file with a uci engine and then do some simple math on the evaluation.
It seems to me you are doing this in an unnecessarily hard way, and that what you still need to reach the goal is more complex than using a completely different method. There are many GUIs that can annotate a set of games in a PGN file on each move with scores and PVs from any engine. Why not use those, and then write a script to to do the calculation on the scores read from the annotated file?
Jesse Gersenson
Posts: 593
Joined: Sat Aug 20, 2011 9:43 am

Re: Need pgn utility to generate 'position startpos moves' d

Post by Jesse Gersenson »

hgm wrote:There are many GUIs that can annotate a set of games in a PGN file on each move with scores and PVs from any engine. Why not use those, and then write a script to to do the calculation on the scores read from the annotated file?
My plan is to compile my simple script, working version is 800 bytes, into an executable for linux and another for windows. When it's working, I'll try to revamp the analysis process to, for example, search deeper when blunders are spotted, or perhaps run a second search using another engine. Perhaps the gui's offer analysis but I'd guess their 'blunder checking' is as primitive as my 800 character bash script.

Indirect perks - I'll learn to compile and code in C++, and have fun for a few weeks.

Am figuring out how to compile. my OS is linux. am totally new to compiling and c/c++.

try #1: testA.cpp

Code: Select all

#include <iostream>
#include <string>
using namespace std;

int main&#40;) &#123;
        string who;
        string name;
        who = "myself";
        cout << "i'll kick " << who << " in the pants if this works" << endl;
        //int age = 0;
        //cout << "how many legs do you have?\n";
        //cin >> age;
        //name="jesse";
        //cout << name << " has "<< age << " legs\n";
        return 0;
&#125;
linux compile: success

Code: Select all

$ g++ testA.cpp -o main

$ file main
main&#58; ELF 64-bit LSB executable, x86-64, version 1 &#40;SYSV&#41;, dynamically linked &#40;uses shared libs&#41;, for GNU/Linux 2.6.24, BuildID&#91;sha1&#93;=0x0b6421fd11a8f9edca8eb982357cc2dc707ace86, not stripped

$ ./main
i'll kick myself in the pants if this works
windows 32-bit & 64-bit compiles: failed

Code: Select all

$ i586-mingw32msvc-cc testA.cpp -o main.exe
/tmp/ccBzFHZg.o&#58;main.cpp&#58;(.text+0xe&#41;&#58; undefined reference to `std&#58;&#58;basic_string<char, std&#58;&#58;char_traits<char>, std&#58;&#58;allocator<char> >&#58;&#58;size&#40;) const'
/tmp/ccBzFHZg.o&#58;main.cpp&#58;(.text+0x52&#41;&#58; undefined reference to `std&#58;&#58;basic_string<char, std&#58;&#58;char_traits<char>, std&#58;&#58;allocator<char> >&#58;&#58;operator&#91;&#93;&#40;unsigned int&#41; const'
/tmp/ccBzFHZg.o&#58;main.cpp&#58;(.text+0x8b&#41;&#58; undefined reference to `std&#58;&#58;basic_string<char, std&#58;&#58;char_traits<char>, std&#58;&#58;allocator<char> >&#58;&#58;operator&#91;&#93;&#40;unsigned int&#41; const'
Why did testA.cpp fail to compile as an exe and what should I try?

try #2: hello.c

Code: Select all

$ cat hello.c
#include "stdio.h"  // Include the standard C library
 int main&#40;)
&#123;
  printf&#40;"If I were a compiler, what would I ouput? &#58;wq \n");
  return 0;
&#125;
32-bit exe compile - runs
$ i586-mingw32msvc-cc hello.c -o test.exe
$ file test.exe
test.exe: PE32 executable (console) Intel 80386, for MS Windows
$ wine test.exe
If I were a compiler, what would I ouput? :wq

64-bit exe compile - runs
$ x86_64-w64-mingw32-gcc hello.c -o test.exe
$ file test.exe
test.exe: PE32+ executable (console) x86-64, for MS Windows
$ wine test.exe
If I were a compiler, what would I ouput? :wq

64-bit linux compile - runs
$ gcc hello.c -o test
$ file test
test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x3973bbf76244d6ea2f8ace0a23a638a6f94625c9, not stripped
Jesse Gersenson
Posts: 593
Joined: Sat Aug 20, 2011 9:43 am

Re: Need pgn utility to generate 'position startpos moves' d

Post by Jesse Gersenson »

$ i586-mingw32msvc-cc testA.cpp -o main.exe

ah, is the problem C++ code can't be compiled with a 'cc' compiler?

tried this and it gave less errors:

$ x86_64-w64-mingw32-g++ testA.cpp -o test.exe
$ file test.exe
test.exe: PE32+ executable (console) x86-64, for MS Windows

$ wine test.exe
err:module:import_dll Library libgcc_s_sjlj-1.dll (which is needed by L"Z:\\home\\jesse\\eclipse\\testA\\src\\test.exe") not found
err:module:import_dll Library libstdc++-6.dll (which is needed by L"Z:\\home\\jesse\\eclipse\\testA\\src\\test.exe") not found
err:module:LdrInitializeThunk Main exe initialization for L"Z:\\home\\jesse\\eclipse\\testA\\src\\test.exe" failed, status c0000135

missing some dll files. geez, it never ends. anyone know what I'm doing wrong?
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: Need pgn utility to generate 'position startpos moves' d

Post by zullil »

Jesse Gersenson wrote:
windows 32-bit & 64-bit compiles: failed

Code: Select all

$ i586-mingw32msvc-cc testA.cpp -o main.exe
/tmp/ccBzFHZg.o&#58;main.cpp&#58;(.text+0xe&#41;&#58; undefined reference to `std&#58;&#58;basic_string<char, std&#58;&#58;char_traits<char>, std&#58;&#58;allocator<char> >&#58;&#58;size&#40;) const'
/tmp/ccBzFHZg.o&#58;main.cpp&#58;(.text+0x52&#41;&#58; undefined reference to `std&#58;&#58;basic_string<char, std&#58;&#58;char_traits<char>, std&#58;&#58;allocator<char> >&#58;&#58;operator&#91;&#93;&#40;unsigned int&#41; const'
/tmp/ccBzFHZg.o&#58;main.cpp&#58;(.text+0x8b&#41;&#58; undefined reference to `std&#58;&#58;basic_string<char, std&#58;&#58;char_traits<char>, std&#58;&#58;allocator<char> >&#58;&#58;operator&#91;&#93;&#40;unsigned int&#41; const'
Why did testA.cpp fail to compile as an exe and what should I try?
Any different if you try

Code: Select all

$ i586-mingw32msvc-cc testA.cpp -o main.exe -lstdc++
[EDIT] Nevermind. Was responding to the first of your posts before the followup appeared.