I wonder if there is already a tool that can parse an EPD file and
append SF9 PV analysis as 'bm' on each position in EPD file.
I wanted to know if there's an automated tool that can append the PV lines on an
EPD with around 13,000+ positions on this EPD file (specifically this epd file):
http://www.talkchess.com/forum/viewtopi ... _view=flat
Maybe F. Mosca or E. Schroder had already built previously a similar tool which I
missed? Would be appreciated if you can point me to the right direction.
Automated tool to append "bm" PV to EPD file.
Moderators: hgm, Rebel, chrisw
-
- Posts: 1010
- Joined: Thu Sep 01, 2011 2:49 pm
Automated tool to append "bm" PV to EPD file.
I told my wife that a husband is like a fine wine; he gets better with age. The next day, she locked me in the cellar.
-
- Posts: 593
- Joined: Sat Aug 20, 2011 9:43 am
Re: Automated tool to append "bm" PV to EPD file.
Here's a quick hack based around Norm Pollock's epdConvert.
Assumes you are running in the same folder as epdConvert.
Assumes you are running in the same folder as epdConvert.
Code: Select all
#!/bin/bash
inputfile="rand_studies.epd"
outputfile="outEPD.pgn"
motor=stockfish
# thinking time in ms
movetime=100
# should be (movetime divided by 1000) + 0.1
movetimesleep=0.2
java epdConvert "$inputfile"
cat "$outputfile" | grep FEN | cut -f2 -d\" | while read i;
do
echo "$i bmdddddddd";
(echo "setoption name Hash value 32";sleep 0.03;echo "position fen $i";sleep 0.03;echo "go movetime $movetime";sleep "$movetimesleep"; echo "quit";)|./$motor | grep pv | tail -1 | sed 's/.*pv //g';
done | sed ':a;N;$!ba;s/bmdddddddd\n/bm /g' > output.txt
-
- Posts: 593
- Joined: Sat Aug 20, 2011 9:43 am
Re: Automated tool to append "bm" PV to EPD file.
sample output
if you want to test, change this
cat "$outputfile" | grep FEN | cut -f2 -d\" | while read i;
to
cat "$outputfile" | grep FEN | cut -f2 -d\" | head -5 | while read i;
Code: Select all
1B1Q1R1n/5p2/B3k3/2r1P1pP/2p5/6K1/8/7b w - - 0 1 bm d8d1 c5d5 f8e8 e6d7 a6b5
1B1k4/8/1b6/4p2P/8/8/8/1K6 w - - 0 1 bm h5h6 b6d4 b8a7 d4c3 b1c2 c3a1 a7d4 a1d4 c2d3 d4a1 d3e4 d8e7 h6h7 e7d6 h7h8q a1d4 e4d3 d6e6 h8f8 e6d5 f8g8 d5d6 g8f7 d6c6 f7e8 c6c5 e8b8 c5c6 b8a8 c6d6 a8g8 d6c5 g8c4 c5b6 d3c2 b6a7 c4g8
cat "$outputfile" | grep FEN | cut -f2 -d\" | while read i;
to
cat "$outputfile" | grep FEN | cut -f2 -d\" | head -5 | while read i;
-
- Posts: 1056
- Joined: Thu Mar 09, 2006 4:15 pm
- Location: Long Island, NY, USA
Re: Automated tool to append "bm" PV to EPD file.
If you can create a file of pv values, then the tool epdInsert (from 40H-EPD tools) can append a pv opcode to the end of each line of an end file.
The process is explained further with the tool bmOpcode. It extracts the pv from Arena analysis log after doing an engine analysis using Arena and an engine.
The process is explained further with the tool bmOpcode. It extracts the pv from Arena analysis log after doing an engine analysis using Arena and an engine.
-
- Posts: 593
- Joined: Sat Aug 20, 2011 9:43 am
Re: Automated tool to append "bm" PV to EPD file.
My first draft was not what you asked for.
Here is my second try
http://termbin.com/9mud
output:
1B1Q1R1n/5p2/B3k3/2r1P1pP/2p5/6K1/8/7b w - - 0 1 bm d8d1 c5d5 f8e8 e6d7 a6b5
1B1k4/8/1b6/4p2P/8/8/8/1K6 w - - 0 1 bm h5h6 b6d4
Here is my second try
http://termbin.com/9mud
output:
1B1Q1R1n/5p2/B3k3/2r1P1pP/2p5/6K1/8/7b w - - 0 1 bm d8d1 c5d5 f8e8 e6d7 a6b5
1B1k4/8/1b6/4p2P/8/8/8/1K6 w - - 0 1 bm h5h6 b6d4
-
- Posts: 1010
- Joined: Thu Sep 01, 2011 2:49 pm
Re: Automated tool to append "bm" PV to EPD file.
Thanks for the script, that is truly guru level scripting.Jesse Gersenson wrote:Here's a quick hack based around Norm Pollock's epdConvert.
Assumes you are running in the same folder as epdConvert.
Code: Select all
#!/bin/bash inputfile="rand_studies.epd" outputfile="outEPD.pgn" motor=stockfish # thinking time in ms movetime=100 # should be (movetime divided by 1000) + 0.1 movetimesleep=0.2 java epdConvert "$inputfile" cat "$outputfile" | grep FEN | cut -f2 -d" | while read i; do echo "$i bmdddddddd"; (echo "setoption name Hash value 32";sleep 0.03;echo "position fen $i";sleep 0.03;echo "go movetime $movetime";sleep "$movetimesleep"; echo "quit";)|./$motor | grep pv | tail -1 | sed 's/.*pv //g'; done | sed ':a;N;$!ba;s/bmdddddddd\n/bm /g' > output.txt
Was first confused because you mixed a Win32 executable in your script,
and I thought it was Win64 Subsystem for Linux (WSL) and I don't have Win10 64bit.
Then I realized I only needed to download 40H-Java.7z of Norm and your
bash script would work in a pure Linux Mint environment.
I also just noticed someone asked and solved a similar problem at stackexchange using only Arena (could work on both PGN and EPD), https://chess.stackexchange.com/questio ... -full-game
But your solution is better and more elegant.
Thanks again, and thanks to Norm Pollock also for those useful PGN and EPD tools.
Last edited by MikeGL on Mon Feb 19, 2018 8:23 pm, edited 1 time in total.
I told my wife that a husband is like a fine wine; he gets better with age. The next day, she locked me in the cellar.
-
- Posts: 1010
- Joined: Thu Sep 01, 2011 2:49 pm
Re: Automated tool to append "bm" PV to EPD file.
Thanks a lot for this alternative solution, and also thanks for those PGN and EPD utils/tools.Norm Pollock wrote:If you can create a file of pv values, then the tool epdInsert (from 40H-EPD tools) can append a pv opcode to the end of each line of an end file.
The process is explained further with the tool bmOpcode. It extracts the pv from Arena analysis log after doing an engine analysis using Arena and an engine.
I told my wife that a husband is like a fine wine; he gets better with age. The next day, she locked me in the cellar.
-
- Posts: 593
- Joined: Sat Aug 20, 2011 9:43 am
Re: Automated tool to append "bm" PV to EPD file.
It would be much faster if the binary were called only once. But, it would require error checking, and thinking!?
Final try, with comments
Output is here:
https://komodochess.com/pub/rand_studies-solved.epd
Final try, with comments
Code: Select all
#!/bin/bash
INPUTFILE="rand_studies.epd"
OUTPUTFILE="outEPD.pgn"
DATAOUTPUT='output.txt'
#motor settings
MOTOR=cfish-32
HASH=8
THREADS=2
COUNTER=0
# thinking time in ms
MOVETIME=130
# should be (movetime divided by 1000) + 0.1
MOVETIMESLEEP=0.18
SHORTSLEEP=0.001
# generates pgn file, $OUTPUTFILE, with a FEN string
# uses '40H Java Class Files' from https://komodochess.com/pub/40H-pgn-utilities/
java epdConvert "$INPUTFILE" > /dev/null
# loop through the pgn file, grep out the FEN line, cut out the fen
# and store the line as variable $fen
cat "$OUTPUTFILE" | grep FEN | cut -f2 -d\" | while read fen;
do
# counter is used to grab the correct line from $INPUTFILE
let COUNTER=COUNTER+1
# print the EPD, located by line number. tr strips the line break
sed -n "$COUNTER"p "$INPUTFILE" | tr -d '\012\015'
# send the following lines to the engine and, grep the engine's output for
# the last 'pv' line
(
echo "setoption name Hash value $HASH" && sleep "$SHORTSLEEP";
echo "setoption name Threads value $THREADS" && sleep "$SHORTSLEEP";
echo "position fen $fen" && sleep "$SHORTSLEEP";
echo "go movetime $MOVETIME";
sleep "$MOVETIMESLEEP";
echo "quit";
)|./$MOTOR | grep pv | tail -1 | sed 's/.*pv / bm /g' ;
# output everything to $DATAOUTPUT
done > "$DATAOUTPUT"
https://komodochess.com/pub/rand_studies-solved.epd
-
- Posts: 1010
- Joined: Thu Sep 01, 2011 2:49 pm
Re: Automated tool to append "bm" PV to EPD file.
Thanks, still studying and learning some parts of this script.Jesse Gersenson wrote:It would be much faster if the binary were called only once. But, it would require error checking, and thinking!?
Final try, with commentsOutput is here:Code: Select all
#!/bin/bash INPUTFILE="rand_studies.epd" OUTPUTFILE="outEPD.pgn" DATAOUTPUT='output.txt' #motor settings MOTOR=cfish-32 HASH=8 THREADS=2 COUNTER=0 # thinking time in ms MOVETIME=130 # should be (movetime divided by 1000) + 0.1 MOVETIMESLEEP=0.18 SHORTSLEEP=0.001 # generates pgn file, $OUTPUTFILE, with a FEN string # uses '40H Java Class Files' from https://komodochess.com/pub/40H-pgn-utilities/ java epdConvert "$INPUTFILE" > /dev/null # loop through the pgn file, grep out the FEN line, cut out the fen # and store the line as variable $fen cat "$OUTPUTFILE" | grep FEN | cut -f2 -d" | while read fen; do # counter is used to grab the correct line from $INPUTFILE let COUNTER=COUNTER+1 # print the EPD, located by line number. tr strips the line break sed -n "$COUNTER"p "$INPUTFILE" | tr -d '\012\015' # send the following lines to the engine and, grep the engine's output for # the last 'pv' line ( echo "setoption name Hash value $HASH" && sleep "$SHORTSLEEP"; echo "setoption name Threads value $THREADS" && sleep "$SHORTSLEEP"; echo "position fen $fen" && sleep "$SHORTSLEEP"; echo "go movetime $MOVETIME"; sleep "$MOVETIMESLEEP"; echo "quit"; )|./$MOTOR | grep pv | tail -1 | sed 's/.*pv / bm /g' ; # output everything to $DATAOUTPUT done > "$DATAOUTPUT"
https://komodochess.com/pub/rand_studies-solved.epd
Looks more difficult than those studies.
p.s. I guess the supplied solutions of SF9 (or was it cfish-32) were not
correct on positions 4,8,19,23,25,26,29,32 and others up to end of file,
after reviewing your supplied attached epd file. Maybe engine needs
longer time to consider the correct move. But never mind, I am also
making a run with SF9 using your above script.
Thanks, don't know what to call the updated script because the
original one you've given in previous post was already excellent.
I told my wife that a husband is like a fine wine; he gets better with age. The next day, she locked me in the cellar.
-
- Posts: 593
- Joined: Sat Aug 20, 2011 9:43 am
Re: Automated tool to append "bm" PV to EPD file.
The first version output fen + pv. Subsequent versions output epd + pv. This is just speculation, as I am can not able to tell the difference between EPD and FEN...MikeGL wrote:
...the original one you've given in previous post was already excellent.
If it turns out to be broken, let me know. It does not do any error checking, so badly formed epd would cause problems - as would a range of other unexpected conditions...