MIP - Material Imbalance Pruning

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

User avatar
Rebel
Posts: 7040
Joined: Thu Aug 18, 2011 12:04 pm
Full name: Ed Schröder

MIP - Material Imbalance Pruning

Post by Rebel »

As far as I know Material Imbalance Pruning is a new technique that offers quite some new features to improve the search, or even the evaluation. Possibilities -

1. Reductions
2. Pruning
3. Lazy eval
4. Score corrections.

more...
90% of coding is debugging, the other 10% is writing bugs.
User avatar
hgm
Posts: 27931
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: MIP - Material Imbalance Pruning

Post by hgm »

Is this really different from evaluating the material score in centi-Pawns, and taking any decisions based on that? Rybka already used a 'material table' that was used to correct the simplistic sum-of-piece-values evaluation, and with proper tuning of the piece values (with the usual interpolation from opening to end-game) the corrections for particular combinations (except for the very pawn-poor end-game positions, which tend to be extra drawish) should be very minor.

I could also imagine that King Safety would be a much more important factor in deciding whether you can afford hefty reduction than, say, the number of Pawns you are ahead.
User avatar
Rebel
Posts: 7040
Joined: Thu Aug 18, 2011 12:04 pm
Full name: Ed Schröder

Re: MIP - Material Imbalance Pruning

Post by Rebel »

The Rybka MIT was more complex and had different goals, MIP is a different and more simple method based on massive statistical data (13.7 billion positions) and figure out by trial on error its possibilities, I gave 4 ideas that could be tried.
90% of coding is debugging, the other 10% is writing bugs.
User avatar
pohl4711
Posts: 2515
Joined: Sat Sep 03, 2011 7:25 am
Location: Berlin, Germany
Full name: Stefan Pohl

Re: MIP - Material Imbalance Pruning

Post by pohl4711 »

Rebel wrote: Sun Jun 23, 2024 8:12 pm As far as I know Material Imbalance Pruning is a new technique that offers quite some new features to improve the search, or even the evaluation. Possibilities -

1. Reductions
2. Pruning
3. Lazy eval
4. Score corrections.

more...
If you want to filter big pgn bases for games with material imbalance super quickly (perhaps for net-learning or so), you can use pgn-extract (I copy some parts of my InterestingWinsSearch Tool (you can use the imbalance file from the bin-folder of the tool) in here)
https://www.sp-cc.de/files/interesting_ ... ch_tool.7z

.bat-file must contain:
pgn-extract --quiet -zimbalance source.pgn --output imbalance.pgn

or (if you want to delete all comments in the output (imbalance.pgn) file)
pgn-extract --quiet -C -N -V -zimbalance source.pgn --output imbalance.pgn

or (only won games):
pgn-extract --quiet -Tr1-0 -Tr0-1 -zimbalance source.pgn --output imbalance.pgn

or (only drawn games):
pgn-extract --quiet -Tr1/2-1/2 -zimbalance source.pgn --output imbalance.pgn

or ((specific engine) combine this with -Tr, if you want, see above):
pgn-extract --quiet -Tp%engine% -zimbalance source.pgn --output imbalance.pgn
or -Tw%engine% or -Tb%engine%, when specific color is needed


imbalance-file (same folder as pgn-extract!) must contain (14 is the number of consecutive plies, one pattern has to be true, you can lower this to 10 or so with an text-editor, if you want to get more search-hits):
14 q1r*l*p* q1<=r1>=l1>=p1>=
14 q*r*l*p* q=r1<=l1>=p2>=
14 q1r*l*p* q1<=r=l3>p=
14 q1r*l*p* q1<=r2>=l=p1<=
14 q*r*l*p* q=r1<=l2>=p1<=
14 q*r*l*p* q=r2>=l2<=p4<=
14 q1r*l*p* q1<=r1>=l=p4>=
14 q1r*l*p* q1<=r=l2>=p3>=
14 q1r*l*p* q1<=r1>=l1>=p2>=
14 q*r*l*p* q=r1<=l1>=p3>=
14 q1r*l*p* q1<=r=l3>p1>=
14 q1r*l*p* q1<=r2>=l=p=
14 q*r*l*p* q=r1<=l2>=p=
14 q*r*l*p* q=r2>=l2<=p3<=
14 q1r*l*p* q1<=r1>=l=p5>=
14 q1r*l*p* q1<=r=l2>=p4>=
14 q1r*l*p* q1<=r1>=l1>=p=
14 q*r*l*p* q=r1<=l1>=p1>=
14 q1r*l*p* q1<=r=l3>p1<=
14 q1r*l*p* q1<=r2>=l=p2<=
14 q*r*l*p* q=r1<=l2>=p2<=
14 q*r*l*p* q=r2>=l2<=p5<=
14 q1r*l*p* q1<=r1>=l=p3>=
14 q1r*l*p* q1<=r=l2>=p2>=
chrisw
Posts: 4360
Joined: Tue Apr 03, 2012 4:28 pm

Re: MIP - Material Imbalance Pruning

Post by chrisw »

pohl4711 wrote: Mon Jun 24, 2024 11:56 am
Rebel wrote: Sun Jun 23, 2024 8:12 pm As far as I know Material Imbalance Pruning is a new technique that offers quite some new features to improve the search, or even the evaluation. Possibilities -

1. Reductions
2. Pruning
3. Lazy eval
4. Score corrections.

more...
If you want to filter big pgn bases for games with material imbalance super quickly (perhaps for net-learning or so), you can use pgn-extract (I copy some parts of my InterestingWinsSearch Tool (you can use the imbalance file from the bin-folder of the tool) in here)
https://www.sp-cc.de/files/interesting_ ... ch_tool.7z

.bat-file must contain:
pgn-extract --quiet -zimbalance source.pgn --output imbalance.pgn

or (if you want to delete all comments in the output (imbalance.pgn) file)
pgn-extract --quiet -C -N -V -zimbalance source.pgn --output imbalance.pgn

or (only won games):
pgn-extract --quiet -Tr1-0 -Tr0-1 -zimbalance source.pgn --output imbalance.pgn

or (only drawn games):
pgn-extract --quiet -Tr1/2-1/2 -zimbalance source.pgn --output imbalance.pgn

or ((specific engine) combine this with -Tr, if you want, see above):
pgn-extract --quiet -Tp%engine% -zimbalance source.pgn --output imbalance.pgn
or -Tw%engine% or -Tb%engine%, when specific color is needed


imbalance-file (same folder as pgn-extract!) must contain (14 is the number of consecutive plies, one pattern has to be true, you can lower this to 10 or so with an text-editor, if you want to get more search-hits):
14 q1r*l*p* q1<=r1>=l1>=p1>=
14 q*r*l*p* q=r1<=l1>=p2>=
14 q1r*l*p* q1<=r=l3>p=
14 q1r*l*p* q1<=r2>=l=p1<=
14 q*r*l*p* q=r1<=l2>=p1<=
14 q*r*l*p* q=r2>=l2<=p4<=
14 q1r*l*p* q1<=r1>=l=p4>=
14 q1r*l*p* q1<=r=l2>=p3>=
14 q1r*l*p* q1<=r1>=l1>=p2>=
14 q*r*l*p* q=r1<=l1>=p3>=
14 q1r*l*p* q1<=r=l3>p1>=
14 q1r*l*p* q1<=r2>=l=p=
14 q*r*l*p* q=r1<=l2>=p=
14 q*r*l*p* q=r2>=l2<=p3<=
14 q1r*l*p* q1<=r1>=l=p5>=
14 q1r*l*p* q1<=r=l2>=p4>=
14 q1r*l*p* q1<=r1>=l1>=p=
14 q*r*l*p* q=r1<=l1>=p1>=
14 q1r*l*p* q1<=r=l3>p1<=
14 q1r*l*p* q1<=r2>=l=p2<=
14 q*r*l*p* q=r1<=l2>=p2<=
14 q*r*l*p* q=r2>=l2<=p5<=
14 q1r*l*p* q1<=r1>=l=p3>=
14 q1r*l*p* q1<=r=l2>=p2>=
Directly parsing packed binary (SF format pgn learning files) in C is going to be orders of magnitude faster.