I do not fully understand the conditions but if I understand correctly best move need to lead to advantage.
I think that this condition is wrong and a position when there is only one move that is not losing also can be a tactical exercise.
I think that what you basically need is not clearly winning second best move and not clearly losing first best move.
program to generare epd file of tactical exercises from pgn
Moderators: hgm, Rebel, chrisw
-
- Posts: 10297
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: program to generare epd file of tactical exercises from pgn
I am implementing complexity number or bestmove1 changes per iteration. And remove the easy move at 20ms method. As the analyzing engine changes its mind on bestmove1, it will be recorded. But this depends on the engine too as there are positions that even at iter 1 it alredy got the bestmove.Laskos wrote: ↑Mon Mar 11, 2019 10:30 am Ferdy, when you will release the tool? It already seems quite good. It might be used to build new, pretty unseen sort of tactical test-suites, which are in touch with actual games, and represent an actual and important aspect of the game-play. Lc0 showed pretty conclusively to me that almost all tactical test-suites as of now have been built with wrong preconceptions, for example "to be hard for AB null-move LMR searchers" and represent game-play only marginally as a side-effect.
When ready, please release it, otherwise I will pester you with PM .
I also included the complexity number and bestscore2 in the epd to possibly improve the algo while knowing these metrics.
Ending with complexity 0
[d]8/4k3/4p2p/2K4P/2P5/8/8/8 w - - bm Kc6; ce 31980; sm Kc6; acd 51; acs 15; fmvn 75; hmvc 1; pv Kc6 e5 Kd5 Kf6 c5; c0 "Carlsen, Magnus - Popov, Ivan RUS, World Blitz 2018, St Petersburg RUS, 2018.12.29, R1.1"; c1 "Complexity: 0"; c2 "bestscore2: 0"; c3 "Analyzing engine: Stockfish 10 64 POPCNT";
Middle with complexity 6, in the actual game white played Bd5 see sm opcode.
[d]6rk/Q6p/1p1pq3/1Pp1b1r1/P1P2p2/5B1P/4RPP1/3R1K2 w - - bm Rxe5; ce 315; sm Bd5; acd 26; acs 15; fmvn 38; hmvc 1; pv Rxe5 Rxe5; c0 "Kazhgaleyev, Murtas - Nepomniachtchi, Ian, World Blitz 2018, St Petersburg RUS, 2018.12.29, R1.4"; c1 "Complexity: 6"; c2 "bestscore2: 31"; c3 "Analyzing engine: Stockfish 10 64 POPCNT";
So the engine feels the complexity of this position.
I will release a beta 1 hr from now, prepare your python 3 and latest python-chess v0.26.0.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: program to generare epd file of tactical exercises from pgn
That one is normal, if score difference between the move found by computer and the game move is big that can made a test position.kinderchocolate wrote: ↑Mon Mar 11, 2019 11:21 amhttps://github.com/vitogit/lichess-tactics-generator
Lichess simply compare two consecutive moves. If both players miss an important move according to a computer, it'd be flagged as a tactical exercice.
I remember the old lichess puzzles where the best move is mate in 11 and if your move is mate in 12, you failed to solve the puzzle
I studied lots of tactics and let the engine analyze it at multipv 2, I discovered that most of those positions, bestscore1 is winning or has the advantage of 2 or more pawns but the bestscore2 from multipv 2 is close to zero or worst. But if the bestscore1 already has a mating score, bestscore2 can be 500cp or less.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: program to generare epd file of tactical exercises from pgn
Most interesting puzzles are like that. I tried to make the tool setting more flexible so that user can adjust 6 thresholds.
I don't think so, people enjoy puzzle that has a reward, finding the move that will win or give him an advantage is more satisfying.
True.
That is possible. The tool can handle that and more when user adjust the 6 thresholds.
Code: Select all
minbest1score1 = 1000 # cp, stm is winning
minbest1score2 = 500 # cp, stm has decisive advantage
minbest1score3 = 200 # cp, stm has moderate
maxbest2score1 = 200 # cp, stm 2nd top move max score threshold 1
maxbest2score2 = 100 # cp, stm 2nd top move max score threshold 2
maxbest2score3 = 50 # cp, stm 2nd top move max score threshold 3
Code: Select all
def interesting_pos(board, bs1, bs2, mib1s1, mib1s2, mib1s3, mab2s1, mab2s2, mab2s3):
"""
board: board position
bs1: bestscore1 from multipv 1
bs2: bestscore2 from multipv 2
mib1s1: minimum best score1, threshold 1
mib1s2: minimum score2, threshold 2
mib1s3: minimum score3, threshold 3
mib1s1 > mib1s2 > mib1s3
mab2s1: maximum best score2, threshold 1
mab2s2: maximum best score2, threshold 2
mab2s3: maximum best score2, threshold 3
mab2s1 > mab2s2 > mab2s3
"""
if bs1 >= mib1s1:
# mate score
if bs1 >= 30000 and bs2 <= 2*mab2s1:
return True
if bs2 <= mab2s1:
return True
elif bs1 >= mib1s2:
if bs2 <= mab2s2:
return True
elif bs1 >= mib1s3:
if bs2 <= mab2s3:
return True
print('Not an interesting pos: {}'.format(board.fen()))
return False
Position with high complexity has the tendency to be a good test.
This one has 3.
[d]4r3/pbp1r3/3p2pk/8/P3PP2/1P2R1N1/5K2/2R5 w - - bm Nf5+; ce 531; sm f5; acd 27; acs 15; fmvn 36; hmvc 2; pv Nf5+ Kh7 Nxe7 Rxe7 Rec3; c0 "Potapov, Pavel - Wang, Hao, World Blitz 2018, St Petersburg RUS, 2018.12.29, R1.10"; c1 "Complexity: 3"; c2 "bestscore2: 9"; c3 "Analyzing engine: Stockfish 10 64 POPCNT";
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: program to generare epd file of tactical exercises from pgn
Python source, save it as icpg.py or anyname.
Code: Select all
"""
icpg.py
Interesting Chess Position Generator
Read games and analyze positions with engine and save interesting positions
based on some criteria
Requirements:
python 3
python-chess v0.26.0
Dev log:
* Parse moves in the game in reverse.
* Exit search early when stm score is way below minimum score threshold
* Save interesting position when stm is winning and the 2nd bestscore from
multipv 2 is not winning.
* Save interesting position when stm position has a big advantage or having
mate scores and the 2nd bestscore from multipv 2 is below user defined param.
"""
import argparse
import chess.pgn
import chess.engine
def interesting_pos(board, bs1, bs2, mib1s1, mib1s2, mib1s3, mab2s1, mab2s2, mab2s3):
"""
board: board position
bs1: bestscore1 from multipv 1
bs2: bestscore2 from multipv 2
mib1s1: minimum best score1, threshold 1
mib1s2: minimum score2, threshold 2
mib1s3: minimum score3, threshold 3
mib1s1 > mib1s2 > mib1s3
mab2s1: maximum best score2, threshold 1
mab2s2: maximum best score2, threshold 2
mab2s3: maximum best score2, threshold 3
mab2s1 > mab2s2 > mab2s3
"""
if bs1 >= mib1s1:
# mate score
if bs1 >= 30000 and bs2 <= 2*mab2s1:
return True
if bs2 <= mab2s1:
return True
elif bs1 >= mib1s2:
if bs2 <= mab2s2:
return True
elif bs1 >= mib1s3:
if bs2 <= mab2s3:
return True
print('Not an interesting pos: {}'.format(board.fen()))
return False
def analyze_game(game, enginefn, hash_val, thread_val, analysis_start_move_num,
outepdfn, gcnt, mintime=1.0, maxtime=2.0,
minscorediffcheck=25, minbest1score1=2000,
minbest1score2=1000, minbest1score3=500,
maxbest2score1=300, maxbest2score2=200,
maxbest2score3=100, weightsfile=None):
""" """
multipv_value = 2
# Define analyzing engine
engine = chess.engine.SimpleEngine.popen_uci(enginefn)
engname = engine.id['name']
try:
engine.configure({"Hash": hash_val})
except:
pass
try:
engine.configure({"Threads": thread_val})
except:
pass
# Lc0 weight setting
if weightsfile is not None:
try:
engine.configure({"WeightsFile": weightsfile})
except:
pass
limit = chess.engine.Limit(time=maxtime)
# Copy orig game header to our header
ev = game.headers['Event']
si = game.headers['Site']
da = game.headers['Date']
ro = game.headers['Round']
wp = game.headers['White']
bp = game.headers['Black']
res = game.headers['Result']
# Skip draw result if minbestscore1 is 500 or more
if res == '1/2-1/2':
return
c0_val = wp + ' - ' + bp + ', ' + ev + ', ' + si + ', ' + da + ', R' + ro
poscnt = 0
# Parse move in reverse
game_end = game.end()
curboard = game_end.board()
while curboard:
board = curboard
fmvn = board.fullmove_number
stm = board.turn
if fmvn == 1 and stm == chess.WHITE:
print('startpos')
break
if fmvn < analysis_start_move_num:
print('move start limit is reached, exit from this game')
break
g_move = board.pop()
curboard = board
# If side to move is in check, skip this position
if board.is_check():
print()
print(board.fen())
print('Skip this position, stm is in check\n')
continue
# Print the fen before g_move is made on the board
poscnt += 1
print()
print('game {} / position {}'.format(gcnt, poscnt))
print(board.fen())
print(board)
print('game move: {}'.format(g_move))
# Run engine in multipv 2
print('{} is searching at multipv {} ...'.format(engname, multipv_value))
bm1, bm2, depth = None, None, None
is_exitearly, is_lowscore = False, False
raw_pv = None
bestmovechanges = 0 # Start comparing bestmove1 at depth 4
tmpmove, oldtmpmove = None, None
with engine.analysis(board, limit, multipv=multipv_value) as analysis:
for info in analysis:
try:
multipv = info['multipv']
depth = info['depth']
if info['score'].is_mate():
s = info['score'].relative.score(mate_score=32000)
else:
s = info['score'].relative.score()
pv = info['pv'][0:5]
t = info['time']
if multipv == 1:
bm1 = pv[0]
bs1 = s
raw_pv = pv
# Exit early if score is below half of minbest1score3
if not is_lowscore and t >= mintime \
and bs1 < minbest1score3/2:
is_lowscore = True
print('Exit search early, current best score is only {} and it is still below half of minbest1score3 or {}/2={}'.format(
bs1, minbest1score3, minbest1score3//2))
break
# Record bestmove move changes to determine position complexity
if 'depth' in info and 'pv' in info \
and 'score' in info \
and not 'lowerbound' in info \
and not 'upperbound' in info \
and depth >= 4:
tmpmove = info['pv'][0]
if tmpmove is not None and tmpmove != oldtmpmove:
assert oldtmpmove is not None, 'oldtmp move is None at depth {}'.format(depth)
bestmovechanges += 1
elif multipv == 2:
bm2 = pv[0]
bs2 = s
if not is_exitearly and t >= mintime \
and bs1 - bs2 < minscorediffcheck:
is_exitearly = True
print('Exit search early, scorediff={} is below minscorediff of {}'.format(
bs1-bs2, minscorediffcheck))
break
oldtmpmove = tmpmove
except:
pass
print('Search is done!!'.format(engname))
print('game move : {}'.format(g_move))
print('complexity : {}'.format(bestmovechanges))
print('best move 1 : {}, best score 1: {}'.format(bm1, bs1))
print('best move 2 : {}, best score 2: {}'.format(bm2, bs2))
print('scorediff : {}'.format(bs1 - bs2))
# Don't save positions if score is already bad
if bs1 < minbest1score3:
print('Skip this position, score {} is below minbest1score3 of {}'.format(bs1, minbest1score3))
continue
# If complexity is 1 or less and if bestmove1 is a capture, skip this position
if board.is_capture(bm1) and bestmovechanges <= 1:
print('Skip this position, bm1 is a capture and position complexity is below 2')
continue
if bs1 - bs2 < minbest1score3 - maxbest2score3:
print('Skip this position, actual min score diff of {} is below user defined min score diff of {}'.format(
bs1 - bs2, minbest1score3 - maxbest2score3))
continue
# Save epd if criteria is satisfied
if interesting_pos(board, bs1, bs2, minbest1score1, minbest1score2,
minbest1score3, maxbest2score1, maxbest2score2,
maxbest2score3):
print('Save this position!!')
ae_oper = 'Analyzing engine: ' + engname
complexity_oper = 'Complexity: ' + str(bestmovechanges)
bs2_oper = 'bestscore2: ' + str(bs2)
new_epd = board.epd(
bm = bm1,
ce = bs1,
sm = g_move,
acd = depth,
acs = int(t),
fmvn = board.fullmove_number,
hmvc = board.halfmove_clock,
pv = raw_pv,
c0 = c0_val,
c1 = complexity_oper,
c2 = bs2_oper,
c3 = ae_oper)
print(new_epd)
with open(outepdfn, 'a') as f:
f.write('{}\n'.format(new_epd))
# break # 1 pos per game only
engine.quit()
def main():
parser = argparse.ArgumentParser(prog='Interesting Chess Position Generator v0.1 beta',
description='Generates interesting position using engine and ' +
'some criteria', epilog='%(prog)s')
parser.add_argument("-i", "--inpgn", help="input pgn file",
required=True)
parser.add_argument("-o", "--outepd", help="output epd file, default=interesting.epd",
default='interesting.epd', required=False)
parser.add_argument("-e", "--engine", help="engine file or path",
required=True)
parser.add_argument("-t", "--threads", help="engine threads (default=1)",
default=1, type=int, required=False)
parser.add_argument("-a", "--hash", help="engine hash in MB (default=128)",
default=128, type=int, required=False)
parser.add_argument("-w", "--weight", help="weight file for NN engine",
required=False)
parser.add_argument("-n", "--mintime", help="analysis minimum time in sec (default=2.0)",
default=2.0, type=float, required=False)
parser.add_argument("-x", "--maxtime", help="analysis maximum time in sec (default=10.0)",
default=10.0, type=float, required=False)
args = parser.parse_args()
pgnfn = args.inpgn
outepdfn = args.outepd
thread_val = args.threads
hash_val = args.hash
enginefn = args.engine
weightsfile = args.weight
mintime = args.mintime
maxtime = args.maxtime
maxgame = 1000 # Number of games to be processed in the given pgn file
start_move = 16 # Stop the analysis when this move no. is reached
minscorediffcheck = 50 # cp
minbest1score1 = 1000 # cp, stm is winning
minbest1score2 = 500 # cp, stm has decisive advantage
minbest1score3 = 200 # cp, stm has moderate
maxbest2score1 = 200 # cp, stm 2nd top move max score threshold 1
maxbest2score2 = 100 # cp, stm 2nd top move max score threshold 2
maxbest2score3 = 50 # cp, stm 2nd top move max score threshold 3
print('pgn file: {}\n'.format(pgnfn))
print('Conditions:')
print('mininum time : {}s'.format(mintime))
print('maximum time : {}s'.format(maxtime))
print('mininum score diff check : {}'.format(minscorediffcheck))
print('mininum best 1 score 1 : {}'.format(minbest1score1))
print('mininum best 1 score 2 : {}'.format(minbest1score2))
print('mininum best 1 score 3 : {}'.format(minbest1score3))
print('maximum best 2 score 1 : {}'.format(maxbest2score1))
print('maximum best 2 score 2 : {}'.format(maxbest2score2))
print('maximum best 2 score 3 : {}'.format(maxbest2score3))
print('stm is not in check : {}'.format('Yes'))
print('stm position is not bad : {}'.format('Yes'))
print('start analysis move number : {}'.format(start_move))
# Save games from pgn file to game list
game_list = []
with open(pgnfn, 'r') as pgn:
game = chess.pgn.read_game(pgn)
while game:
game_list.append(game)
if len(game_list) >= maxgame:
break
game = chess.pgn.read_game(pgn)
gcnt = 0
for g in game_list:
gcnt += 1
analyze_game(g,
enginefn,
hash_val,
thread_val,
start_move,
outepdfn,
gcnt,
mintime=mintime,
maxtime=maxtime,
minscorediffcheck=minscorediffcheck,
minbest1score1=minbest1score1,
minbest1score2=minbest1score2,
minbest1score3=minbest1score3,
maxbest2score1=maxbest2score1,
maxbest2score2=maxbest2score2,
maxbest2score3=maxbest2score3,
weightsfile=weightsfile)
if __name__ == '__main__':
main()
Not all params are settable. So adjust your settings thru the code.
Typically this.
Code: Select all
minbest1score1 = 1000 # cp, stm is winning
minbest1score2 = 500 # cp, stm has decisive advantage
minbest1score3 = 200 # cp, stm has moderate
maxbest2score1 = 200 # cp, stm 2nd top move max score threshold 1
maxbest2score2 = 100 # cp, stm 2nd top move max score threshold 2
maxbest2score3 = 50 # cp, stm 2nd top move max score threshold 3
Typical command line is:
Code: Select all
python icpg.py --inpgn bataviagmb19.pgn --engine sf10.exe --threads 1 --hash 128 --mintime 5.0 --maxtime 15.0
To get help
python icpg.py -h
Code: Select all
Generates interesting position using engine and some criteria
optional arguments:
-h, --help show this help message and exit
-i INPGN, --inpgn INPGN
input pgn file
-o OUTEPD, --outepd OUTEPD
output epd file, default=interesting.epd
-e ENGINE, --engine ENGINE
engine file or path
-t THREADS, --threads THREADS
engine threads (default=1)
-a HASH, --hash HASH engine hash in MB (default=128)
-w WEIGHT, --weight WEIGHT
weight file for NN engine
-n MINTIME, --mintime MINTIME
analysis minimum time in sec (default=2.0)
-x MAXTIME, --maxtime MAXTIME
analysis maximum time in sec (default=10.0)
Interesting Chess Position Generator v0.1 beta
-
- Posts: 215
- Joined: Sun Feb 24, 2008 2:08 am
Re: program to generare epd file of tactical exercises from pgn
Ferdy
I tried this but get an error.
Traceback (most recent call last):
File "icpg.py", line 353, in <module>
main()
File "icpg.py", line 349, in main
weightsfile=weightsfile)
File "icpg.py", line 76, in analyze_game
engine = chess.engine.SimpleEngine.popen_uci(enginefn)
AttributeError: module 'chess.engine' has no attribute 'SimpleEngine'
Command line:
python icpg.py --inpgn games.pgn --engine stockfish.exe --threads 2 --hash 256 --mintime 1.0 --maxtime 2.0
Stockfish is in the same directory as icpg.py.
Do you know what the problem is thanks.
I tried this but get an error.
Traceback (most recent call last):
File "icpg.py", line 353, in <module>
main()
File "icpg.py", line 349, in main
weightsfile=weightsfile)
File "icpg.py", line 76, in analyze_game
engine = chess.engine.SimpleEngine.popen_uci(enginefn)
AttributeError: module 'chess.engine' has no attribute 'SimpleEngine'
Command line:
python icpg.py --inpgn games.pgn --engine stockfish.exe --threads 2 --hash 256 --mintime 1.0 --maxtime 2.0
Stockfish is in the same directory as icpg.py.
Do you know what the problem is thanks.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: program to generare epd file of tactical exercises from pgn
Did you install python-chess v0.26.0?JohnS wrote: ↑Tue Mar 12, 2019 1:12 am Ferdy
I tried this but get an error.
Traceback (most recent call last):
File "icpg.py", line 353, in <module>
main()
File "icpg.py", line 349, in main
weightsfile=weightsfile)
File "icpg.py", line 76, in analyze_game
engine = chess.engine.SimpleEngine.popen_uci(enginefn)
AttributeError: module 'chess.engine' has no attribute 'SimpleEngine'
Command line:
python icpg.py --inpgn games.pgn --engine stockfish.exe --threads 2 --hash 256 --mintime 1.0 --maxtime 2.0
Stockfish is in the same directory as icpg.py.
Do you know what the problem is thanks.
-
- Posts: 215
- Joined: Sun Feb 24, 2008 2:08 am
Re: program to generare epd file of tactical exercises from pgn
Oops. I had version 0.21 installed. It works now thanks!Ferdy wrote: ↑Tue Mar 12, 2019 3:20 amDid you install python-chess v0.26.0?JohnS wrote: ↑Tue Mar 12, 2019 1:12 am Ferdy
I tried this but get an error.
Traceback (most recent call last):
File "icpg.py", line 353, in <module>
main()
File "icpg.py", line 349, in main
weightsfile=weightsfile)
File "icpg.py", line 76, in analyze_game
engine = chess.engine.SimpleEngine.popen_uci(enginefn)
AttributeError: module 'chess.engine' has no attribute 'SimpleEngine'
Command line:
python icpg.py --inpgn games.pgn --engine stockfish.exe --threads 2 --hash 256 --mintime 1.0 --maxtime 2.0
Stockfish is in the same directory as icpg.py.
Do you know what the problem is thanks.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: program to generare epd file of tactical exercises from pgn
v0.2 beta released.
Changes:
Score thresholds are also modified. You can change it by modifying the source.
icpg.py source
Changes:
Code: Select all
Dev log:
v0.2 beta
* Added --log flag to enable logging
* Relocate engine initialization to the main() was in analyze_game(), this
would avoid Lc0 engine from duplicating in memory. Lc0 would not quit
from uci quit command after analyzing a game.
* When analyzing engine is Lc0, set SmartPruningFactor to 0, this would
avoid pruning the analysis time.
* Added --skipdraw flag to skip games with draw results
icpg.py source
Code: Select all
"""
icpg.py
Interesting Chess Position Generator
Read games and analyze positions with engine and save interesting positions
based on user defined criteria via score thresholds against engine bestscore1
and bestscore2 from mulitpv 2 analysis results.
Requirements:
python 3
python-chess v0.26.0
Analysis engine that supports multipv and movetime
Dev log:
v0.2 beta
* Added --log flag to enable logging
* Relocate engine initialization to the main() was in analyze_game(), this
would avoid Lc0 engine from duplicating in memory. Lc0 would not quit
from uci quit command after analyzing a game.
* When analyzing engine is Lc0, set SmartPruningFactor to 0, this would
avoid pruning the analysis time.
* Added --skipdraw flag to skip games with draw results
v0.1 beta
* Parse moves in the game in reverse.
* Exit search early when stm score is way below minimum score threshold
* Save interesting positions depending on the engine multipv scores and
user defined score thresholds
"""
import argparse
import logging
import chess.pgn
import chess.engine
VERSION = 'v0.2 beta'
def interesting_pos(board, bs1, bs2, mib1s1, mib1s2, mib1s3, mab2s1, mab2s2, mab2s3):
"""
board: board position
bs1: bestscore1 from multipv 1
bs2: bestscore2 from multipv 2
mib1s1: minimum best score1, threshold 1
mib1s2: minimum score2, threshold 2
mib1s3: minimum score3, threshold 3
mib1s1 > mib1s2 > mib1s3
mab2s1: maximum best score2, threshold 1
mab2s2: maximum best score2, threshold 2
mab2s3: maximum best score2, threshold 3
mab2s1 > mab2s2 > mab2s3
"""
if bs1 >= mib1s1:
# mate score
if bs1 >= 30000 and bs2 <= 2*mab2s1:
return True
if bs2 <= mab2s1:
return True
elif bs1 >= mib1s2:
if bs2 <= mab2s2:
return True
elif bs1 >= mib1s3:
if bs2 <= mab2s3:
return True
print('Not an interesting pos: {}'.format(board.fen()))
return False
def analyze_game(game, engine, enginefn, hash_val, thread_val,
analysis_start_move_num,
outepdfn, gcnt, engname, mintime=1.0, maxtime=2.0,
minscorediffcheck=25, minbest1score1=2000,
minbest1score2=1000, minbest1score3=500,
maxbest2score1=300, maxbest2score2=200,
maxbest2score3=100, weightsfile=None, skipdraw=True):
""" """
limit = chess.engine.Limit(time=maxtime)
# Copy orig game header to our epd output
ev = game.headers['Event']
si = game.headers['Site']
da = game.headers['Date']
ro = game.headers['Round']
wp = game.headers['White']
bp = game.headers['Black']
res = game.headers['Result']
# If result of this game is a draw and skipdraw is true, we skip it
if skipdraw and res == '1/2-1/2':
return
c0_val = wp + ' - ' + bp + ', ' + ev + ', ' + si + ', ' + da + ', R' + ro
poscnt = 0
# Parse move in reverse
game_end = game.end()
curboard = game_end.board()
while curboard:
board = curboard
fmvn = board.fullmove_number
stm = board.turn
if fmvn == 1 and stm == chess.WHITE:
print('startpos')
break
if fmvn < analysis_start_move_num:
print('move start limit is reached, exit from this game')
break
g_move = board.pop()
curboard = board
# If side to move is in check, skip this position
if board.is_check():
print()
print(board.fen())
print('Skip this position, stm is in check\n')
continue
# Print the fen before g_move is made on the board
poscnt += 1
print()
print('game {} / position {}'.format(gcnt, poscnt))
print(board.fen())
print(board)
print('game move: {}'.format(g_move))
# Run engine in multipv 2
print('{} is searching at multipv {} ...'.format(engname, 2))
bm1, bm2, depth = None, None, None
raw_pv = None
bestmovechanges = 0 # Start comparing bestmove1 at depth 4
tmpmove, oldtmpmove = None, None
with engine.analysis(board, limit, multipv=2) as analysis:
for info in analysis:
try:
multipv = info['multipv']
depth = info['depth']
if info['score'].is_mate():
s = info['score'].relative.score(mate_score=32000)
else:
s = info['score'].relative.score()
pv = info['pv'][0:5]
t = info['time']
if multipv == 1:
bm1 = pv[0]
bs1 = s
raw_pv = pv
# Exit early if score is below half of minbest1score3
if t >= mintime and bs1 < minbest1score3/2:
print('Exit search early, current best score is only {} and it is still below half of minbest1score3 or {}/2={}'.format(
bs1, minbest1score3, minbest1score3//2))
break
# Record bestmove move changes to determine position complexity
if 'depth' in info and 'pv' in info \
and 'score' in info \
and not 'lowerbound' in info \
and not 'upperbound' in info \
and depth >= 4:
tmpmove = info['pv'][0]
if tmpmove is not None and tmpmove != oldtmpmove:
assert oldtmpmove is not None, 'oldtmp move is None at depth {}'.format(depth)
bestmovechanges += 1
elif multipv == 2:
bm2 = pv[0]
bs2 = s
if t >= mintime and bs1 - bs2 < minscorediffcheck:
print('Exit search early, scorediff={} is below minscorediff of {}'.format(
bs1-bs2, minscorediffcheck))
break
oldtmpmove = tmpmove
except:
pass
print('Search is done!!'.format(engname))
print('game move : {}'.format(g_move))
print('complexity : {}'.format(bestmovechanges))
print('best move 1 : {}, best score 1: {}'.format(bm1, bs1))
print('best move 2 : {}, best score 2: {}'.format(bm2, bs2))
print('scorediff : {}'.format(bs1 - bs2))
# Don't save positions if score is already bad
if bs1 < minbest1score3:
print('Skip this position, score {} is below minbest1score3 of {}'.format(bs1, minbest1score3))
continue
# If complexity is 1 or less and if bestmove1 is a capture, skip this position
if board.is_capture(bm1) and bestmovechanges <= 1:
print('Skip this position, bm1 is a capture and position complexity is below 2')
continue
if bs1 - bs2 < minbest1score3 - maxbest2score3:
print('Skip this position, actual min score diff of {} is below user defined min score diff of {}'.format(
bs1 - bs2, minbest1score3 - maxbest2score3))
continue
# Save epd if criteria is satisfied
if interesting_pos(board, bs1, bs2, minbest1score1, minbest1score2,
minbest1score3, maxbest2score1, maxbest2score2,
maxbest2score3):
print('Save this position!!')
ae_oper = 'Analyzing engine: ' + engname
complexity_oper = 'Complexity: ' + str(bestmovechanges)
bs2_oper = 'bestscore2: ' + str(bs2)
new_epd = board.epd(
bm = bm1,
ce = bs1,
sm = g_move,
acd = depth,
acs = int(t),
fmvn = board.fullmove_number,
hmvc = board.halfmove_clock,
pv = raw_pv,
c0 = c0_val,
c1 = complexity_oper,
c2 = bs2_oper,
c3 = ae_oper)
print(new_epd)
with open(outepdfn, 'a') as f:
f.write('{}\n'.format(new_epd))
def main():
parser = argparse.ArgumentParser(prog='Interesting Chess Position Generator {}'.format(VERSION),
description='Generates interesting position using engine and ' +
'some criteria', epilog='%(prog)s')
parser.add_argument("-i", "--inpgn", help="input pgn file",
required=True)
parser.add_argument("-o", "--outepd", help="output epd file, default=interesting.epd",
default='interesting.epd', required=False)
parser.add_argument("-e", "--engine", help="engine file or path",
required=True)
parser.add_argument("-t", "--threads", help="engine threads (default=1)",
default=1, type=int, required=False)
parser.add_argument("-a", "--hash", help="engine hash in MB (default=128)",
default=128, type=int, required=False)
parser.add_argument("-w", "--weight", help="weight file for NN engine",
required=False)
parser.add_argument("-n", "--mintime", help="analysis minimum time in sec (default=2.0)",
default=2.0, type=float, required=False)
parser.add_argument("-x", "--maxtime", help="analysis maximum time in sec (default=10.0)",
default=10.0, type=float, required=False)
parser.add_argument("--skipdraw", help="a flag to skip games with draw results",
action="store_true")
parser.add_argument("--log", help="a flag to save logs in a file",
action="store_true")
args = parser.parse_args()
pgnfn = args.inpgn
outepdfn = args.outepd
thread_val = args.threads
hash_val = args.hash
enginefn = args.engine
weightsfile = args.weight
mintime = args.mintime
maxtime = args.maxtime
skipdraw = args.skipdraw
maxgame = 1000 # Number of games to be processed in the given pgn file
start_move = 16 # Stop the analysis when this move no. is reached
# Adjust score thresholds to save interesting positions
minscorediffcheck = 50 # cp
minbest1score1 = 200 # cp, stm is winning
minbest1score2 = 50 # cp, stm has decisive advantage
minbest1score3 = -50 # cp, stm has moderate
maxbest2score1 = 0 # cp, stm 2nd top move max score threshold 1
maxbest2score2 = -100 # cp, stm 2nd top move max score threshold 2
maxbest2score3 = -200 # cp, stm 2nd top move max score threshold 3
print('pgn file: {}\n'.format(pgnfn))
print('Conditions:')
print('mininum time : {}s'.format(mintime))
print('maximum time : {}s'.format(maxtime))
print('mininum score diff check : {}'.format(minscorediffcheck))
print('mininum best 1 score 1 : {}'.format(minbest1score1))
print('mininum best 1 score 2 : {}'.format(minbest1score2))
print('mininum best 1 score 3 : {}'.format(minbest1score3))
print('maximum best 2 score 1 : {}'.format(maxbest2score1))
print('maximum best 2 score 2 : {}'.format(maxbest2score2))
print('maximum best 2 score 3 : {}'.format(maxbest2score3))
print('stm is not in check : {}'.format('Yes'))
print('stop analysis move number : {}'.format(start_move))
# Save games from pgn file to game list
game_list = []
with open(pgnfn, 'r') as pgn:
game = chess.pgn.read_game(pgn)
while game:
game_list.append(game)
if len(game_list) >= maxgame:
break
game = chess.pgn.read_game(pgn)
# Define analyzing engine
engine = chess.engine.SimpleEngine.popen_uci(enginefn)
engname = engine.id['name']
if args.log:
logfn = '_'.join(engname.split()) + '_icpg_log.txt'
logging.basicConfig(level=logging.DEBUG, filename=logfn,
filemode='w', format='%(asctime)s [%(levelname)s] %(message)s')
# Set Lc0 SmartPruningFactor to 0 to avoid analysis time pruning
if 'lc0' in engname.lower():
try:
engine.configure({"SmartPruningFactor": 0})
except:
pass
else:
try:
engine.configure({"Hash": hash_val})
except:
pass
try:
engine.configure({"Threads": thread_val})
except:
pass
# For NN engine that uses uci option WeightsFile similar to Lc0
if weightsfile is not None:
try:
engine.configure({"WeightsFile": weightsfile})
except:
pass
gcnt = 0
for g in game_list:
gcnt += 1
analyze_game(g,
engine,
enginefn,
hash_val,
thread_val,
start_move,
outepdfn,
gcnt,
engname,
mintime=mintime,
maxtime=maxtime,
minscorediffcheck=minscorediffcheck,
minbest1score1=minbest1score1,
minbest1score2=minbest1score2,
minbest1score3=minbest1score3,
maxbest2score1=maxbest2score1,
maxbest2score2=maxbest2score2,
maxbest2score3=maxbest2score3,
weightsfile=weightsfile,
skipdraw=skipdraw)
engine.quit()
if __name__ == '__main__':
main()
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: program to generare epd file of tactical exercises from pgn
Released v0.3 beta
icpg.py
Sample generated positions.
[d]1qr4k/p5p1/7p/P2Q4/2R2P2/2p3K1/6PP/8 b - - bm c2; ce 845; sm c2; acd 27; acs 15; fmvn 40; hmvc 0; pv c2 Qd3 c1=Q Rxc1 Rxc1; c0 "Navara, David - So, Wesley, Champions Showdown Blitz, Saint Louis USA, 2019.02.23, R1.3"; c1 "Complexity: 0"; c2 "bestscore2: 0"; c3 "Analyzing engine: Stockfish 10 64 POPCNT";
[d]3r4/1k3p1p/5p2/1p1r1b2/1b1p1B2/1N3B2/1PP2PPP/5RK1 w - - bm Nxd4; ce 165; sm Nxd4; acd 27; acs 15; fmvn 23; hmvc 2; pv Nxd4 Bg4 Be4 f5 Bxd5+; c0 "Jumabayev, Rinat - Gagare, Shalmali, 2nd Sharjah Masters 2018, Sharjah UAE, 2018.04.12, R1.21"; c1 "Complexity: 2"; c2 "bestscore2: 10"; c3 "Analyzing engine: Stockfish 10 64 POPCNT";
Sample command line to enable the pin.
Code: Select all
Dev log:
v0.3 beta
* Added option --pin to save only interesting positions when a piece of
not stm is pinned.
* minscorediffcheck is no longer an option but is calculated as:
minscorediffcheck = minbest1score3 - maxbest2score3
icpg.py
Code: Select all
"""
icpg.py
Interesting Chess Position Generator
Read games and analyze positions with engine and save interesting positions
based on user defined criteria via score thresholds against engine bestscore1
and bestscore2 from mulitpv 2 analysis results.
Requirements:
python 3
python-chess v0.26.0
Analysis engine that supports multipv and movetime
Dev log:
v0.3 beta
* Added option --pin to save only interesting positions when a piece of
not stm is pinned.
* minscorediffcheck is no longer an option but is calculated as:
minscorediffcheck = minbest1score3 - maxbest2score3
v0.2 beta
* Added --log flag to enable logging
* Relocate engine initialization to the main() was in analyze_game(), this
would avoid Lc0 engine from duplicating in memory. Lc0 would not quit
from uci quit command after analyzing a game.
* When analyzing engine is Lc0, set SmartPruningFactor to 0, this would
avoid pruning the analysis time.
* Added --skipdraw flag to skip games with draw results
v0.1 beta
* Parse moves in the game in reverse.
* Exit search early when stm score is way below minimum score threshold
* Save interesting positions depending on the engine multipv scores and
user defined score thresholds
"""
import argparse
import logging
import chess.pgn
import chess.engine
VERSION = 'v0.3 beta'
def interesting_pos(board, bs1, bs2, mib1s1, mib1s2, mib1s3, mab2s1, mab2s2, mab2s3):
"""
board: board position
bs1: bestscore1 from multipv 1
bs2: bestscore2 from multipv 2
mib1s1: minimum best score1, threshold 1
mib1s2: minimum score2, threshold 2
mib1s3: minimum score3, threshold 3
mib1s1 > mib1s2 > mib1s3
mab2s1: maximum best score2, threshold 1
mab2s2: maximum best score2, threshold 2
mab2s3: maximum best score2, threshold 3
mab2s1 > mab2s2 > mab2s3
"""
if bs1 >= mib1s1:
# mate score
if bs1 >= 30000 and bs2 <= 2*mab2s1:
return True
if bs2 <= mab2s1:
return True
elif bs1 >= mib1s2:
if bs2 <= mab2s2:
return True
elif bs1 >= mib1s3:
if bs2 <= mab2s3:
return True
print('Not an interesting pos: {}'.format(board.fen()))
return False
def abs_pinned(board, color):
""" Returns true if one or more pieces of color color is pinned """
for sq in chess.SQUARES:
if board.is_pinned(color, sq):
return True
return False
def analyze_game(game, engine, enginefn, hash_val, thread_val,
analysis_start_move_num,
outepdfn, gcnt, engname, mintime=1.0, maxtime=2.0,
minscorediffcheck=25, minbest1score1=2000,
minbest1score2=1000, minbest1score3=500,
maxbest2score1=300, maxbest2score2=200,
maxbest2score3=100, weightsfile=None, skipdraw=True,
pin=None):
""" """
limit = chess.engine.Limit(time=maxtime)
# Copy orig game header to our epd output
ev = game.headers['Event']
si = game.headers['Site']
da = game.headers['Date']
ro = game.headers['Round']
wp = game.headers['White']
bp = game.headers['Black']
res = game.headers['Result']
# If result of this game is a draw and skipdraw is true, we skip it
if skipdraw and res == '1/2-1/2':
return
c0_val = wp + ' - ' + bp + ', ' + ev + ', ' + si + ', ' + da + ', R' + ro
poscnt = 0
# Parse move in reverse
game_end = game.end()
curboard = game_end.board()
while curboard:
board = curboard
fmvn = board.fullmove_number
stm = board.turn
if fmvn == 1 and stm == chess.WHITE:
print('startpos')
break
if fmvn < analysis_start_move_num:
print('move start limit is reached, exit from this game')
break
g_move = board.pop()
curboard = board
# Print the fen before g_move is made on the board
poscnt += 1
print()
print('game {} / position {}'.format(gcnt, poscnt))
print(board.fen())
print(board)
print('game move: {}\n'.format(g_move))
# Skip this position if --pin is set and no one of the not stm piece is pinned
if pin and not abs_pinned(board, board.turn ^ 1):
print('Skip this position no one of the pieces of the not stm is pinned')
print(board.fen())
print()
continue
# If side to move is in check, skip this position
if board.is_check():
print()
print(board.fen())
print('Skip this position, stm is in check\n')
continue
# Run engine in multipv 2
print('{} is searching at multipv {} ...'.format(engname, 2))
bm1, bm2, depth = None, None, None
raw_pv = None
bestmovechanges = 0 # Start comparing bestmove1 at depth 4
tmpmove, oldtmpmove = None, None
with engine.analysis(board, limit, multipv=2) as analysis:
for info in analysis:
try:
multipv = info['multipv']
depth = info['depth']
if info['score'].is_mate():
s = info['score'].relative.score(mate_score=32000)
else:
s = info['score'].relative.score()
pv = info['pv'][0:5]
t = info['time']
if multipv == 1:
bm1 = pv[0]
bs1 = s
raw_pv = pv
# Exit early if score is below half of minbest1score3
if t >= mintime and bs1 < minbest1score3/2:
print('Exit search early, current best score is only {} and it is still below half of minbest1score3 or {}/2={}'.format(
bs1, minbest1score3, minbest1score3//2))
break
# Record bestmove move changes to determine position complexity
if 'depth' in info and 'pv' in info \
and 'score' in info \
and not 'lowerbound' in info \
and not 'upperbound' in info \
and depth >= 4:
tmpmove = info['pv'][0]
if tmpmove is not None and tmpmove != oldtmpmove:
assert oldtmpmove is not None, 'oldtmp move is None at depth {}'.format(depth)
bestmovechanges += 1
elif multipv == 2:
bm2 = pv[0]
bs2 = s
if t >= mintime and bs1 - bs2 < minscorediffcheck:
print('Exit search early, scorediff={} is below minscorediff of {}'.format(
bs1-bs2, minscorediffcheck))
break
oldtmpmove = tmpmove
except:
pass
print('Search is done!!'.format(engname))
print('game move : {}'.format(g_move))
print('complexity : {}'.format(bestmovechanges))
print('best move 1 : {}, best score 1: {}'.format(bm1, bs1))
print('best move 2 : {}, best score 2: {}'.format(bm2, bs2))
print('scorediff : {}'.format(bs1 - bs2))
# Don't save positions if score is already bad
if bs1 < minbest1score3:
print('Skip this position, score {} is below minbest1score3 of {}'.format(bs1, minbest1score3))
continue
# If complexity is 1 or less and if bestmove1 is a capture, skip this position
if board.is_capture(bm1) and bestmovechanges <= 1:
print('Skip this position, bm1 is a capture and position complexity is below 2')
continue
if bs1 - bs2 < minbest1score3 - maxbest2score3:
print('Skip this position, actual min score diff of {} is below user defined min score diff of {}'.format(
bs1 - bs2, minbest1score3 - maxbest2score3))
continue
# Save epd if criteria is satisfied
if interesting_pos(board, bs1, bs2, minbest1score1, minbest1score2,
minbest1score3, maxbest2score1, maxbest2score2,
maxbest2score3):
print('Save this position!!')
ae_oper = 'Analyzing engine: ' + engname
complexity_oper = 'Complexity: ' + str(bestmovechanges)
bs2_oper = 'bestscore2: ' + str(bs2)
new_epd = board.epd(
bm = bm1,
ce = bs1,
sm = g_move,
acd = depth,
acs = int(t),
fmvn = board.fullmove_number,
hmvc = board.halfmove_clock,
pv = raw_pv,
c0 = c0_val,
c1 = complexity_oper,
c2 = bs2_oper,
c3 = ae_oper)
print(new_epd)
with open(outepdfn, 'a') as f:
f.write('{}\n'.format(new_epd))
def main():
parser = argparse.ArgumentParser(prog='Interesting Chess Position Generator {}'.format(VERSION),
description='Generates interesting position using engine and ' +
'some criteria', epilog='%(prog)s')
parser.add_argument('-i', '--inpgn', help='input pgn file',
required=True)
parser.add_argument('-o', '--outepd', help='output epd file, default=interesting.epd',
default='interesting.epd', required=False)
parser.add_argument('-e', '--engine', help='engine file or path',
required=True)
parser.add_argument('-t', '--threads', help='engine threads (default=1)',
default=1, type=int, required=False)
parser.add_argument('-a', '--hash', help='engine hash in MB (default=128)',
default=128, type=int, required=False)
parser.add_argument('-w', '--weight', help='weight file for NN engine',
required=False)
parser.add_argument('-n', '--mintime', help='analysis minimum time in sec (default=2.0)',
default=2.0, type=float, required=False)
parser.add_argument('-x', '--maxtime', help='analysis maximum time in sec (default=10.0)',
default=10.0, type=float, required=False)
parser.add_argument('--skipdraw', help='a flag to skip games with draw results',
action='store_true')
parser.add_argument('--log', help='a flag to save logs in a file',
action='store_true')
parser.add_argument('--pin', help='a flag when enabled will only save interesting' +
'position if not stm piece is pinned', action='store_true')
args = parser.parse_args()
pgnfn = args.inpgn
outepdfn = args.outepd
thread_val = args.threads
hash_val = args.hash
enginefn = args.engine
weightsfile = args.weight
mintime = args.mintime
maxtime = args.maxtime
skipdraw = args.skipdraw
pin = args.pin
maxgame = 1000 # Number of games to be processed in the given pgn file
start_move = 16 # Stop the analysis when this move no. is reached
# Adjust score thresholds to save interesting positions
minbest1score1 = 500 # cp, stm is winning
minbest1score2 = 300 # cp, stm has decisive advantage
minbest1score3 = 100 # cp, stm has moderate
maxbest2score1 = 200 # cp, stm 2nd top move max score threshold 1
maxbest2score2 = 100 # cp, stm 2nd top move max score threshold 2
maxbest2score3 = 25 # cp, stm 2nd top move max score threshold 3
minscorediffcheck = minbest1score3 - maxbest2score3
print('pgn file: {}\n'.format(pgnfn))
print('Conditions:')
print('mininum time : {}s'.format(mintime))
print('maximum time : {}s'.format(maxtime))
print('mininum score diff check : {}'.format(minscorediffcheck))
print('mininum best 1 score 1 : {}'.format(minbest1score1))
print('mininum best 1 score 2 : {}'.format(minbest1score2))
print('mininum best 1 score 3 : {}'.format(minbest1score3))
print('maximum best 2 score 1 : {}'.format(maxbest2score1))
print('maximum best 2 score 2 : {}'.format(maxbest2score2))
print('maximum best 2 score 3 : {}'.format(maxbest2score3))
print('stm is not in check : {}'.format('Yes'))
print('stop analysis move number : {}'.format(start_move))
# Save games from pgn file to game list
game_list = []
with open(pgnfn, 'r') as pgn:
game = chess.pgn.read_game(pgn)
while game:
game_list.append(game)
if len(game_list) >= maxgame:
break
game = chess.pgn.read_game(pgn)
# Define analyzing engine
engine = chess.engine.SimpleEngine.popen_uci(enginefn)
engname = engine.id['name']
if args.log:
logfn = '_'.join(engname.split()) + '_icpg_log.txt'
logging.basicConfig(level=logging.DEBUG, filename=logfn,
filemode='w', format='%(asctime)s [%(levelname)s] %(message)s')
# Set Lc0 SmartPruningFactor to 0 to avoid analysis time pruning
if 'lc0' in engname.lower():
try:
engine.configure({"SmartPruningFactor": 0})
except:
pass
else:
try:
engine.configure({"Hash": hash_val})
except:
pass
try:
engine.configure({"Threads": thread_val})
except:
pass
# For NN engine that uses uci option WeightsFile similar to Lc0
if weightsfile is not None:
try:
engine.configure({"WeightsFile": weightsfile})
except:
pass
gcnt = 0
for g in game_list:
gcnt += 1
analyze_game(g,
engine,
enginefn,
hash_val,
thread_val,
start_move,
outepdfn,
gcnt,
engname,
mintime=mintime,
maxtime=maxtime,
minscorediffcheck=minscorediffcheck,
minbest1score1=minbest1score1,
minbest1score2=minbest1score2,
minbest1score3=minbest1score3,
maxbest2score1=maxbest2score1,
maxbest2score2=maxbest2score2,
maxbest2score3=maxbest2score3,
weightsfile=weightsfile,
skipdraw=skipdraw,
pin=pin)
engine.quit()
if __name__ == '__main__':
main()
Sample generated positions.
[d]1qr4k/p5p1/7p/P2Q4/2R2P2/2p3K1/6PP/8 b - - bm c2; ce 845; sm c2; acd 27; acs 15; fmvn 40; hmvc 0; pv c2 Qd3 c1=Q Rxc1 Rxc1; c0 "Navara, David - So, Wesley, Champions Showdown Blitz, Saint Louis USA, 2019.02.23, R1.3"; c1 "Complexity: 0"; c2 "bestscore2: 0"; c3 "Analyzing engine: Stockfish 10 64 POPCNT";
[d]3r4/1k3p1p/5p2/1p1r1b2/1b1p1B2/1N3B2/1PP2PPP/5RK1 w - - bm Nxd4; ce 165; sm Nxd4; acd 27; acs 15; fmvn 23; hmvc 2; pv Nxd4 Bg4 Be4 f5 Bxd5+; c0 "Jumabayev, Rinat - Gagare, Shalmali, 2nd Sharjah Masters 2018, Sharjah UAE, 2018.04.12, R1.21"; c1 "Complexity: 2"; c2 "bestscore2: 10"; c3 "Analyzing engine: Stockfish 10 64 POPCNT";
Sample command line to enable the pin.
Code: Select all
python icpg.py --inpgn champshowb19.pgn --engine sf10.exe --threads 1 --hash 128 --mintime 5.0 --maxtime 15.0 --pin