Automated tool to append "bm" PV to EPD file.

[MikeGL]

I wonder if there is already a tool that can parse an EPD file and
append SF9 PV analysis as 'bm' on each position in EPD file.

I wanted to know if there's an automated tool that can append the PV lines on an
EPD with around 13,000+ positions on this EPD file (specifically this epd file):
http://www.talkchess.com/forum/viewtopi ... _view=flat


Maybe F. Mosca or E. Schroder had already built previously a similar tool which I
missed? Would be appreciated if you can point me to the right direction.

[Jesse Gersenson]

Here's a quick hack based around Norm Pollock's epdConvert.
Assumes you are running in the same folder as epdConvert.

Code: Select all

#!/bin/bash
inputfile="rand_studies.epd"
outputfile="outEPD.pgn"
motor=stockfish

# thinking time in ms
movetime=100

# should be (movetime divided by 1000) + 0.1
movetimesleep=0.2

java epdConvert "$inputfile"

cat "$outputfile" | grep FEN | cut -f2 -d\" | while read i;

do 
echo "$i bmdddddddd";
(echo "setoption name Hash value 32";sleep 0.03;echo "position fen $i";sleep 0.03;echo "go movetime $movetime";sleep "$movetimesleep"; echo "quit";)|./$motor | grep pv | tail -1 | sed 's/.*pv //g';

done | sed ':a;N;$!ba;s/bmdddddddd\n/bm /g' > output.txt

[Jesse Gersenson]

sample output

Code: Select all

1B1Q1R1n/5p2/B3k3/2r1P1pP/2p5/6K1/8/7b w - - 0 1 bm d8d1 c5d5 f8e8 e6d7 a6b5
1B1k4/8/1b6/4p2P/8/8/8/1K6 w - - 0 1 bm h5h6 b6d4 b8a7 d4c3 b1c2 c3a1 a7d4 a1d4 c2d3 d4a1 d3e4 d8e7 h6h7 e7d6 h7h8q a1d4 e4d3 d6e6 h8f8 e6d5 f8g8 d5d6 g8f7 d6c6 f7e8 c6c5 e8b8 c5c6 b8a8 c6d6 a8g8 d6c5 g8c4 c5b6 d3c2 b6a7 c4g8

if you want to test, change this
cat "$outputfile" | grep FEN | cut -f2 -d\" | while read i;
to
cat "$outputfile" | grep FEN | cut -f2 -d\" | head -5 | while read i;

[Norm Pollock]

If you can create a file of pv values, then the tool epdInsert (from 40H-EPD tools) can append a pv opcode to the end of each line of an end file.

The process is explained further with the tool bmOpcode. It extracts the pv from Arena analysis log after doing an engine analysis using Arena and an engine.

[Jesse Gersenson]

My first draft was not what you asked for.

Here is my second try
http://termbin.com/9mud

output:
1B1Q1R1n/5p2/B3k3/2r1P1pP/2p5/6K1/8/7b w - - 0 1 bm d8d1 c5d5 f8e8 e6d7 a6b5
1B1k4/8/1b6/4p2P/8/8/8/1K6 w - - 0 1 bm h5h6 b6d4

[MikeGL]

Here's a quick hack based around Norm Pollock's epdConvert.
Assumes you are running in the same folder as epdConvert.

Code: Select all

#!/bin/bash
inputfile="rand_studies.epd"
outputfile="outEPD.pgn"
motor=stockfish

# thinking time in ms
movetime=100

# should be (movetime divided by 1000) + 0.1
movetimesleep=0.2

java epdConvert "$inputfile"

cat "$outputfile" | grep FEN | cut -f2 -d" | while read i;

do 
echo "$i bmdddddddd";
(echo "setoption name Hash value 32";sleep 0.03;echo "position fen $i";sleep 0.03;echo "go movetime $movetime";sleep "$movetimesleep"; echo "quit";)|./$motor | grep pv | tail -1 | sed 's/.*pv //g';

done | sed ':a;N;$!ba;s/bmdddddddd\n/bm /g' > output.txt

Thanks for the script, that is truly guru level scripting.
Was first confused because you mixed a Win32 executable in your script,
and I thought it was Win64 Subsystem for Linux (WSL) and I don't have Win10 64bit.
Then I realized I only needed to download 40H-Java.7z of Norm and your
bash script would work in a pure Linux Mint environment.

I also just noticed someone asked and solved a similar problem at stackexchange using only Arena (could work on both PGN and EPD), https://chess.stackexchange.com/questio ... -full-game

But your solution is better and more elegant.
Thanks again, and thanks to Norm Pollock also for those useful PGN and EPD tools.

[MikeGL]

If you can create a file of pv values, then the tool epdInsert (from 40H-EPD tools) can append a pv opcode to the end of each line of an end file.

The process is explained further with the tool bmOpcode. It extracts the pv from Arena analysis log after doing an engine analysis using Arena and an engine.

Thanks a lot for this alternative solution, and also thanks for those PGN and EPD utils/tools.

[Jesse Gersenson]

It would be much faster if the binary were called only once. But, it would require error checking, and thinking!?

Final try, with comments

Code: Select all

#!/bin/bash
INPUTFILE="rand_studies.epd"
OUTPUTFILE="outEPD.pgn"
DATAOUTPUT='output.txt'

#motor settings
MOTOR=cfish-32
HASH=8
THREADS=2
COUNTER=0

# thinking time in ms
MOVETIME=130

# should be (movetime divided by 1000) + 0.1
MOVETIMESLEEP=0.18
SHORTSLEEP=0.001

# generates pgn file, $OUTPUTFILE, with a FEN string
# uses '40H Java Class Files' from https://komodochess.com/pub/40H-pgn-utilities/
java epdConvert "$INPUTFILE" > /dev/null

# loop through the pgn file, grep out the FEN line, cut out the fen
# and store the line as variable $fen
cat "$OUTPUTFILE" | grep FEN | cut -f2 -d\" | while read fen;
do
  # counter is used to grab the correct line from $INPUTFILE
  let COUNTER=COUNTER+1

  # print the EPD, located by line number. tr strips the line break
  sed -n "$COUNTER"p "$INPUTFILE" | tr -d '\012\015'

  # send the following lines to the engine and, grep the engine's output for 
  # the last 'pv' line
  (
	echo "setoption name Hash value $HASH" && sleep "$SHORTSLEEP";
	echo "setoption name Threads value $THREADS" && sleep "$SHORTSLEEP";
	echo "position fen $fen" && sleep "$SHORTSLEEP";
	echo "go movetime $MOVETIME";
	sleep "$MOVETIMESLEEP"; 
	echo "quit";
  )|./$MOTOR | grep pv | tail -1 | sed 's/.*pv / bm /g' ;

# output everything to $DATAOUTPUT
done > "$DATAOUTPUT"

Output is here:
https://komodochess.com/pub/rand_studies-solved.epd

[MikeGL]

It would be much faster if the binary were called only once. But, it would require error checking, and thinking!?

Final try, with comments

Code: Select all

#!/bin/bash
INPUTFILE="rand_studies.epd"
OUTPUTFILE="outEPD.pgn"
DATAOUTPUT='output.txt'

#motor settings
MOTOR=cfish-32
HASH=8
THREADS=2
COUNTER=0

# thinking time in ms
MOVETIME=130

# should be (movetime divided by 1000) + 0.1
MOVETIMESLEEP=0.18
SHORTSLEEP=0.001

# generates pgn file, $OUTPUTFILE, with a FEN string
# uses '40H Java Class Files' from https://komodochess.com/pub/40H-pgn-utilities/
java epdConvert "$INPUTFILE" > /dev/null

# loop through the pgn file, grep out the FEN line, cut out the fen
# and store the line as variable $fen
cat "$OUTPUTFILE" | grep FEN | cut -f2 -d" | while read fen;
do
  # counter is used to grab the correct line from $INPUTFILE
  let COUNTER=COUNTER+1

  # print the EPD, located by line number. tr strips the line break
  sed -n "$COUNTER"p "$INPUTFILE" | tr -d '\012\015'

  # send the following lines to the engine and, grep the engine's output for 
  # the last 'pv' line
  (
	echo "setoption name Hash value $HASH" && sleep "$SHORTSLEEP";
	echo "setoption name Threads value $THREADS" && sleep "$SHORTSLEEP";
	echo "position fen $fen" && sleep "$SHORTSLEEP";
	echo "go movetime $MOVETIME";
	sleep "$MOVETIMESLEEP"; 
	echo "quit";
  )|./$MOTOR | grep pv | tail -1 | sed 's/.*pv / bm /g' ;

# output everything to $DATAOUTPUT
done > "$DATAOUTPUT"

Output is here:
https://komodochess.com/pub/rand_studies-solved.epd

Thanks, still studying and learning some parts of this script.
Looks more difficult than those studies.

p.s. I guess the supplied solutions of SF9 (or was it cfish-32) were not
correct on positions 4,8,19,23,25,26,29,32 and others up to end of file,
after reviewing your supplied attached epd file. Maybe engine needs
longer time to consider the correct move. But never mind, I am also
making a run with SF9 using your above script.


Thanks, don't know what to call the updated script because the
original one you've given in previous post was already excellent.

[Jesse Gersenson]

...the original one you've given in previous post was already excellent.

The first version output fen + pv. Subsequent versions output epd + pv. This is just speculation, as I am can not able to tell the difference between EPD and FEN...

If it turns out to be broken, let me know. It does not do any error checking, so badly formed epd would cause problems - as would a range of other unexpected conditions...