Stockfish Handicap Matches

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
Rebel
Posts: 6991
Joined: Thu Aug 18, 2011 12:04 pm

Re: Stockfish Handicap Matches

Post by Rebel »

lkaufman wrote: ↑Sun Jun 21, 2020 11:08 pm I don't see much point in creating artificial problems for this purpose, your current method is much better. But the more interesting way to do it in my opinion is to take chrisw's knight odds opening book, and then run gauntlets of top engines like SF and Komodo vs. several relatively weak engines, whatever engines are about the right strength to score 30-70% at knight odds. Knight odds (either White knight removed) is a clearly defined handicap of a nearly constant magnitude, which makes it ideal for the purpose of seeing how much improvement there has been and which top engine is better at giving the handicap. By the way, knight odds is always used rather than bishop in chess because with one bishop removed the game changes much more, you try to put pawns on particular colors, it doesn't feel like chess anymore. Knights are interchangeable, bishops are not. Also knight odds means odds giver has White, otherwise it is called "knight and move" odds.
You are making sense, changing something I consider fun into something useful, just one problem I ran into, namely cute-chess. It changes colors despite "-tournament gauntlet" and without "-repeat" and so I am forced to split the current epd of 10 positions into 10 epd's with 1 position and start 10 separate batch files. Unless there is a solution it's not desirable to use more positions than 20.
90% of coding is debugging, the other 10% is writing bugs.
Top
User avatar
Rebel
Posts: 6991
Joined: Thu Aug 18, 2011 12:04 pm

Re: Stockfish Handicap Matches

Post by Rebel »

chrisw wrote: ↑Sun Jun 21, 2020 11:20 pm Done 5600 EPDs off the start position minus b1 knight, played out all four ply combinations, culled all duplicates, culled all positions where SF11 evaluated more than +/-10 centipawns away from 300 centipawns (SF11 average score for all epds), and am now left with 5600 EPDs.

Link: https://github.com/ChrisWhittington/Che ... t-odds.epd

Will upload for no knight at g1 tomorrow am.

Small randomised sample below

Code: Select all

rnbqkbnr/pppp2pp/5p2/4p3/4P2P/8/PPPP1PP1/RNBQKB1R w KQkq - 0 3
rnbqk1nr/pppp1ppp/3bp3/8/3P4/8/PPP1PPPP/RNBQKBR1 w Qkq - 2 3
rnbqkb1r/pppppp1p/6pn/8/3P2P1/8/PPP1PP1P/RNBQKB1R w KQkq - 0 3
rnbqkbnr/p1p1pppp/1p1p4/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 0 3
rnbqkb1r/ppppnppp/8/4p3/8/2P3P1/PP1PPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbnr/p1ppppp1/8/1p5p/8/1QP5/PP1PPPPP/RNB1KB1R w KQkq - 0 3
rnbqkbnr/pp1ppp1p/2p5/6p1/8/N4P2/PPPPP1PP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/p1ppppp1/7p/1p6/1P6/P7/2PPPPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1p1ppppp/p7/2p5/2P5/4P3/PP1P1PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppp1p1p/4p3/6p1/P7/3P4/1PP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/1ppppppp/p6n/8/1P6/2N5/P1PPPPPP/R1BQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n5/5p2/4P3/3P4/PPP2PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/6p1/7p/3P4/2N5/PPP1PPPP/R1BQKB1R w KQkq - 0 3
r1bqkbnr/pppppp1p/n5p1/8/8/P6P/1PPPPPP1/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 1 3
rnbqkb1r/pp1ppppp/7n/2p5/3P4/8/PPPBPPPP/RN1QKB1R w KQkq - 0 3
r1bqkbnr/pppppppp/8/n3P3/8/8/PPPP1PPP/RNBQKB1R w KQkq - 1 3
rnbqkb1r/pppppppp/8/6P1/4n3/8/PPPPPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbr1/pppppppp/5n2/8/1P2P3/8/P1PP1PPP/RNBQKB1R w KQq - 1 3
rnbq1bnr/pppkpppp/8/3p4/8/NP6/P1PPPPPP/R1BQKB1R w KQ - 1 3
r1bqkbnr/ppppp1pp/n4p2/8/8/1P1P4/P1P1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1pppppp1/p6p/8/3P1B2/8/PPP1PPPP/RN1QKB1R w KQkq - 0 3
rnbqkb1r/pppppp1p/7n/6p1/5P2/8/PPPPP1PP/RNBQKBR1 w Qkq - 1 3
r1bqkbnr/pp1ppppp/n1p5/8/3P4/P7/1PP1PPPP/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/4P3/6P1/PPPP1P1P/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppppp1/n6p/2P5/8/8/PP1PPPPP/RNBQKB1R w KQkq - 1 3
rnbqkbn1/pppppppr/7p/8/7P/1P6/P1PPPPP1/RNBQKB1R w KQq - 1 3
rnbqkbnr/ppp1p1pp/3p4/5p2/6P1/2P5/PP1PPP1P/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pppp1ppp/4p2n/8/8/4P3/PPPPBPPP/RNBQK2R w KQkq - 2 3
rnbqkbnr/pppppp1p/8/8/6p1/P5P1/1PPPPP1P/RNBQKB1R w KQkq - 0 3
1nbqkbnr/1ppppppp/r7/p7/8/NP6/P1PPPPPP/R1BQKB1R w KQk - 2 3
r1bqkb1r/pppppppp/2n4n/8/8/2NP4/PPP1PPPP/R1BQKB1R w KQkq - 3 3
rnbqkbr1/pppppppp/5n2/8/8/P6P/1PPPPPP1/RNBQKB1R w KQq - 1 3
Cool... hopefully there is a solution for the cute-chess obstacle.
90% of coding is debugging, the other 10% is writing bugs.
Top
chrisw
Posts: 4313
Joined: Tue Apr 03, 2012 4:28 pm

Re: Stockfish Handicap Matches

Post by chrisw »

lkaufman wrote: ↑Mon Jun 22, 2020 12:00 am
chrisw wrote: ↑Sun Jun 21, 2020 11:20 pm
lkaufman wrote: ↑Sun Jun 21, 2020 11:08 pm
Rebel wrote: ↑Sun Jun 21, 2020 10:29 pm
chrisw wrote: ↑Sun Jun 21, 2020 9:43 pm I’m tempted to generate a few tens of thousands of test positions. How many would be enough?
I prefer 10 special positions, positions like:

[d]8/4kpbn/p1p3p1/Pp2p2p/1P2Pn2/N1P1BP2/5P1P/5BK1 w - - bm Nxb5; id Karpov - Hansen;
1.Nxb5 instantly wins. Since not every engine finds it quickly enough they should at least play 1.c4 with good winning chances.

A job for Larry?

:D
I don't see much point in creating artificial problems for this purpose, your current method is much better. But the more interesting way to do it in my opinion is to take chrisw's knight odds opening book, and then run gauntlets of top engines like SF and Komodo vs. several relatively weak engines, whatever engines are about the right strength to score 30-70% at knight odds. Knight odds (either White knight removed) is a clearly defined handicap of a nearly constant magnitude, which makes it ideal for the purpose of seeing how much improvement there has been and which top engine is better at giving the handicap. By the way, knight odds is always used rather than bishop in chess because with one bishop removed the game changes much more, you try to put pawns on particular colors, it doesn't feel like chess anymore. Knights are interchangeable, bishops are not. Also knight odds means odds giver has White, otherwise it is called "knight and move" odds.
Done 5600 EPDs off the start position minus b1 knight, played out all four ply combinations, culled all duplicates, culled all positions where SF11 evaluated more than +/-10 centipawns away from 300 centipawns (SF11 average score for all epds), and am now left with 5600 EPDs.

Link: https://github.com/ChrisWhittington/Che ... t-odds.epd

Will upload for no knight at g1 tomorrow am.

Small randomised sample below

Code: Select all

rnbqkbnr/pppp2pp/5p2/4p3/4P2P/8/PPPP1PP1/RNBQKB1R w KQkq - 0 3
rnbqk1nr/pppp1ppp/3bp3/8/3P4/8/PPP1PPPP/RNBQKBR1 w Qkq - 2 3
rnbqkb1r/pppppp1p/6pn/8/3P2P1/8/PPP1PP1P/RNBQKB1R w KQkq - 0 3
rnbqkbnr/p1p1pppp/1p1p4/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 0 3
rnbqkb1r/ppppnppp/8/4p3/8/2P3P1/PP1PPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbnr/p1ppppp1/8/1p5p/8/1QP5/PP1PPPPP/RNB1KB1R w KQkq - 0 3
rnbqkbnr/pp1ppp1p/2p5/6p1/8/N4P2/PPPPP1PP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/p1ppppp1/7p/1p6/1P6/P7/2PPPPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1p1ppppp/p7/2p5/2P5/4P3/PP1P1PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppp1p1p/4p3/6p1/P7/3P4/1PP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/1ppppppp/p6n/8/1P6/2N5/P1PPPPPP/R1BQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n5/5p2/4P3/3P4/PPP2PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/6p1/7p/3P4/2N5/PPP1PPPP/R1BQKB1R w KQkq - 0 3
r1bqkbnr/pppppp1p/n5p1/8/8/P6P/1PPPPPP1/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 1 3
rnbqkb1r/pp1ppppp/7n/2p5/3P4/8/PPPBPPPP/RN1QKB1R w KQkq - 0 3
r1bqkbnr/pppppppp/8/n3P3/8/8/PPPP1PPP/RNBQKB1R w KQkq - 1 3
rnbqkb1r/pppppppp/8/6P1/4n3/8/PPPPPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbr1/pppppppp/5n2/8/1P2P3/8/P1PP1PPP/RNBQKB1R w KQq - 1 3
rnbq1bnr/pppkpppp/8/3p4/8/NP6/P1PPPPPP/R1BQKB1R w KQ - 1 3
r1bqkbnr/ppppp1pp/n4p2/8/8/1P1P4/P1P1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1pppppp1/p6p/8/3P1B2/8/PPP1PPPP/RN1QKB1R w KQkq - 0 3
rnbqkb1r/pppppp1p/7n/6p1/5P2/8/PPPPP1PP/RNBQKBR1 w Qkq - 1 3
r1bqkbnr/pp1ppppp/n1p5/8/3P4/P7/1PP1PPPP/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/4P3/6P1/PPPP1P1P/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppppp1/n6p/2P5/8/8/PP1PPPPP/RNBQKB1R w KQkq - 1 3
rnbqkbn1/pppppppr/7p/8/7P/1P6/P1PPPPP1/RNBQKB1R w KQq - 1 3
rnbqkbnr/ppp1p1pp/3p4/5p2/6P1/2P5/PP1PPP1P/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pppp1ppp/4p2n/8/8/4P3/PPPPBPPP/RNBQK2R w KQkq - 2 3
rnbqkbnr/pppppp1p/8/8/6p1/P5P1/1PPPPP1P/RNBQKB1R w KQkq - 0 3
1nbqkbnr/1ppppppp/r7/p7/8/NP6/P1PPPPPP/R1BQKB1R w KQk - 2 3
r1bqkb1r/pppppppp/2n4n/8/8/2NP4/PPP1PPPP/R1BQKB1R w KQkq - 3 3
rnbqkbr1/pppppppp/5n2/8/8/P6P/1PPPPPP1/RNBQKB1R w KQq - 1 3
Thanks, but something seems very wrong here, because you say it's based on Stockfish average score of -300 centipawns for the positions. But Stockfish 11 evaluation of knight odds position is way worse than -300, it doesn't sound possible to me that the positions could average only down 300 centipawns. Maybe if I check some of the positions I'll get a clue as to what the problem might be.
-290 centipawns on the run with mixed epds, no knight on b1 or g1. The prior was g1 missing (I think).

Well, I can only report what the data says, remember this is all four-ply positions, white to move, duplicates culled.
SF is slightly optimistic anyway according to my tests, for example 80,000 positions taken randomly from CCRL, then normalised for game result, exactly 50/50 black-white wins, PGN actual shows 0.50 average, low nodes search latest LC0 0.51 and low nodes search SF11 0.527. Used to be the other way round, LC0 reporting optimistic and SF not.

The first run was with SF11 given 10ms to analyse, to get some rapid results. Current run, doing now, is at 25 ms, and the average SF eval -295 centipawns
Top
lkaufman
Posts: 5960
Joined: Sun Jan 10, 2010 6:15 am
Location: Maryland USA

Re: Stockfish Handicap Matches

Post by lkaufman »

Rebel wrote: ↑Mon Jun 22, 2020 12:09 am
chrisw wrote: ↑Sun Jun 21, 2020 11:20 pm Done 5600 EPDs off the start position minus b1 knight, played out all four ply combinations, culled all duplicates, culled all positions where SF11 evaluated more than +/-10 centipawns away from 300 centipawns (SF11 average score for all epds), and am now left with 5600 EPDs.

Link: https://github.com/ChrisWhittington/Che ... t-odds.epd

Will upload for no knight at g1 tomorrow am.

Small randomised sample below

Code: Select all

rnbqkbnr/pppp2pp/5p2/4p3/4P2P/8/PPPP1PP1/RNBQKB1R w KQkq - 0 3
rnbqk1nr/pppp1ppp/3bp3/8/3P4/8/PPP1PPPP/RNBQKBR1 w Qkq - 2 3
rnbqkb1r/pppppp1p/6pn/8/3P2P1/8/PPP1PP1P/RNBQKB1R w KQkq - 0 3
rnbqkbnr/p1p1pppp/1p1p4/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 0 3
rnbqkb1r/ppppnppp/8/4p3/8/2P3P1/PP1PPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbnr/p1ppppp1/8/1p5p/8/1QP5/PP1PPPPP/RNB1KB1R w KQkq - 0 3
rnbqkbnr/pp1ppp1p/2p5/6p1/8/N4P2/PPPPP1PP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/p1ppppp1/7p/1p6/1P6/P7/2PPPPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1p1ppppp/p7/2p5/2P5/4P3/PP1P1PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppp1p1p/4p3/6p1/P7/3P4/1PP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/1ppppppp/p6n/8/1P6/2N5/P1PPPPPP/R1BQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n5/5p2/4P3/3P4/PPP2PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/6p1/7p/3P4/2N5/PPP1PPPP/R1BQKB1R w KQkq - 0 3
r1bqkbnr/pppppp1p/n5p1/8/8/P6P/1PPPPPP1/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 1 3
rnbqkb1r/pp1ppppp/7n/2p5/3P4/8/PPPBPPPP/RN1QKB1R w KQkq - 0 3
r1bqkbnr/pppppppp/8/n3P3/8/8/PPPP1PPP/RNBQKB1R w KQkq - 1 3
rnbqkb1r/pppppppp/8/6P1/4n3/8/PPPPPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbr1/pppppppp/5n2/8/1P2P3/8/P1PP1PPP/RNBQKB1R w KQq - 1 3
rnbq1bnr/pppkpppp/8/3p4/8/NP6/P1PPPPPP/R1BQKB1R w KQ - 1 3
r1bqkbnr/ppppp1pp/n4p2/8/8/1P1P4/P1P1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1pppppp1/p6p/8/3P1B2/8/PPP1PPPP/RN1QKB1R w KQkq - 0 3
rnbqkb1r/pppppp1p/7n/6p1/5P2/8/PPPPP1PP/RNBQKBR1 w Qkq - 1 3
r1bqkbnr/pp1ppppp/n1p5/8/3P4/P7/1PP1PPPP/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/4P3/6P1/PPPP1P1P/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppppp1/n6p/2P5/8/8/PP1PPPPP/RNBQKB1R w KQkq - 1 3
rnbqkbn1/pppppppr/7p/8/7P/1P6/P1PPPPP1/RNBQKB1R w KQq - 1 3
rnbqkbnr/ppp1p1pp/3p4/5p2/6P1/2P5/PP1PPP1P/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pppp1ppp/4p2n/8/8/4P3/PPPPBPPP/RNBQK2R w KQkq - 2 3
rnbqkbnr/pppppp1p/8/8/6p1/P5P1/1PPPPP1P/RNBQKB1R w KQkq - 0 3
1nbqkbnr/1ppppppp/r7/p7/8/NP6/P1PPPPPP/R1BQKB1R w KQk - 2 3
r1bqkb1r/pppppppp/2n4n/8/8/2NP4/PPP1PPPP/R1BQKB1R w KQkq - 3 3
rnbqkbr1/pppppppp/5n2/8/8/P6P/1PPPPPP1/RNBQKB1R w KQq - 1 3
Cool... hopefully there is a solution for the cute-chess obstacle.
Something was really wrong, I checked several positions and they showed scores of 4 pawns or more down, as does the initial knight odds position, not -300. If you just want under 20 positions, take off the b1 knight, choose the 3 best White moves by SF11 multiPV, chose the 3 best replies to each of those, and repeat with g1 knight off. 18 positions, totally fair, no silly moves, real knight odds chess! If you want more just choose best 4 or best five for each side.
Komodo rules!
Top
chrisw
Posts: 4313
Joined: Tue Apr 03, 2012 4:28 pm

Re: Stockfish Handicap Matches

Post by chrisw »

lkaufman wrote: ↑Mon Jun 22, 2020 12:30 am
Rebel wrote: ↑Mon Jun 22, 2020 12:09 am
chrisw wrote: ↑Sun Jun 21, 2020 11:20 pm Done 5600 EPDs off the start position minus b1 knight, played out all four ply combinations, culled all duplicates, culled all positions where SF11 evaluated more than +/-10 centipawns away from 300 centipawns (SF11 average score for all epds), and am now left with 5600 EPDs.

Link: https://github.com/ChrisWhittington/Che ... t-odds.epd

Will upload for no knight at g1 tomorrow am.

Small randomised sample below

Code: Select all

rnbqkbnr/pppp2pp/5p2/4p3/4P2P/8/PPPP1PP1/RNBQKB1R w KQkq - 0 3
rnbqk1nr/pppp1ppp/3bp3/8/3P4/8/PPP1PPPP/RNBQKBR1 w Qkq - 2 3
rnbqkb1r/pppppp1p/6pn/8/3P2P1/8/PPP1PP1P/RNBQKB1R w KQkq - 0 3
rnbqkbnr/p1p1pppp/1p1p4/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 0 3
rnbqkb1r/ppppnppp/8/4p3/8/2P3P1/PP1PPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbnr/p1ppppp1/8/1p5p/8/1QP5/PP1PPPPP/RNB1KB1R w KQkq - 0 3
rnbqkbnr/pp1ppp1p/2p5/6p1/8/N4P2/PPPPP1PP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/p1ppppp1/7p/1p6/1P6/P7/2PPPPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1p1ppppp/p7/2p5/2P5/4P3/PP1P1PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppp1p1p/4p3/6p1/P7/3P4/1PP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/1ppppppp/p6n/8/1P6/2N5/P1PPPPPP/R1BQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n5/5p2/4P3/3P4/PPP2PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/6p1/7p/3P4/2N5/PPP1PPPP/R1BQKB1R w KQkq - 0 3
r1bqkbnr/pppppp1p/n5p1/8/8/P6P/1PPPPPP1/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 1 3
rnbqkb1r/pp1ppppp/7n/2p5/3P4/8/PPPBPPPP/RN1QKB1R w KQkq - 0 3
r1bqkbnr/pppppppp/8/n3P3/8/8/PPPP1PPP/RNBQKB1R w KQkq - 1 3
rnbqkb1r/pppppppp/8/6P1/4n3/8/PPPPPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbr1/pppppppp/5n2/8/1P2P3/8/P1PP1PPP/RNBQKB1R w KQq - 1 3
rnbq1bnr/pppkpppp/8/3p4/8/NP6/P1PPPPPP/R1BQKB1R w KQ - 1 3
r1bqkbnr/ppppp1pp/n4p2/8/8/1P1P4/P1P1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1pppppp1/p6p/8/3P1B2/8/PPP1PPPP/RN1QKB1R w KQkq - 0 3
rnbqkb1r/pppppp1p/7n/6p1/5P2/8/PPPPP1PP/RNBQKBR1 w Qkq - 1 3
r1bqkbnr/pp1ppppp/n1p5/8/3P4/P7/1PP1PPPP/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/4P3/6P1/PPPP1P1P/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppppp1/n6p/2P5/8/8/PP1PPPPP/RNBQKB1R w KQkq - 1 3
rnbqkbn1/pppppppr/7p/8/7P/1P6/P1PPPPP1/RNBQKB1R w KQq - 1 3
rnbqkbnr/ppp1p1pp/3p4/5p2/6P1/2P5/PP1PPP1P/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pppp1ppp/4p2n/8/8/4P3/PPPPBPPP/RNBQK2R w KQkq - 2 3
rnbqkbnr/pppppp1p/8/8/6p1/P5P1/1PPPPP1P/RNBQKB1R w KQkq - 0 3
1nbqkbnr/1ppppppp/r7/p7/8/NP6/P1PPPPPP/R1BQKB1R w KQk - 2 3
r1bqkb1r/pppppppp/2n4n/8/8/2NP4/PPP1PPPP/R1BQKB1R w KQkq - 3 3
rnbqkbr1/pppppppp/5n2/8/8/P6P/1PPPPPP1/RNBQKB1R w KQq - 1 3
Cool... hopefully there is a solution for the cute-chess obstacle.
Something was really wrong,
Well. Results are results. The average SF eval for all the 4-ply positions, duplicates culled, is within a whisker of 300 centipawns at 25ms search, default SF11.

Check the epds listed, they ought, under similar search parameters, to result in SF scores of -300 +/- 10

I checked several positions and they showed scores of 4 pawns or more down, as does the initial knight odds position, not -300.
You checked the listed epds for variance from -300? Which ones? What was the variance? What SF11 conditions are you using?

If you just want under 20 positions, take off the b1 knight, choose the 3 best White moves by SF11 multiPV, chose the 3 best replies to each of those, and repeat with g1 knight off. 18 positions, totally fair, no silly moves, real knight odds chess! If you want more just choose best 4 or best five for each side.
Edit: Whoops, rechecked, the mean is indeed -293, but I was selecting EPDs based on -340, so you should find the listed EPDs evaluate at around -340 centipawns. Will correct the EPD dump tomorrow am.
Last edited by chrisw on Mon Jun 22, 2020 12:53 am, edited 1 time in total.
Top
chrisw
Posts: 4313
Joined: Tue Apr 03, 2012 4:28 pm

Re: Stockfish Handicap Matches

Post by chrisw »

Rebel wrote: ↑Mon Jun 22, 2020 12:09 am
chrisw wrote: ↑Sun Jun 21, 2020 11:20 pm Done 5600 EPDs off the start position minus b1 knight, played out all four ply combinations, culled all duplicates, culled all positions where SF11 evaluated more than +/-10 centipawns away from 300 centipawns (SF11 average score for all epds), and am now left with 5600 EPDs.

Link: https://github.com/ChrisWhittington/Che ... t-odds.epd

Will upload for no knight at g1 tomorrow am.

Small randomised sample below

Code: Select all

rnbqkbnr/pppp2pp/5p2/4p3/4P2P/8/PPPP1PP1/RNBQKB1R w KQkq - 0 3
rnbqk1nr/pppp1ppp/3bp3/8/3P4/8/PPP1PPPP/RNBQKBR1 w Qkq - 2 3
rnbqkb1r/pppppp1p/6pn/8/3P2P1/8/PPP1PP1P/RNBQKB1R w KQkq - 0 3
rnbqkbnr/p1p1pppp/1p1p4/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 0 3
rnbqkb1r/ppppnppp/8/4p3/8/2P3P1/PP1PPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbnr/p1ppppp1/8/1p5p/8/1QP5/PP1PPPPP/RNB1KB1R w KQkq - 0 3
rnbqkbnr/pp1ppp1p/2p5/6p1/8/N4P2/PPPPP1PP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/p1ppppp1/7p/1p6/1P6/P7/2PPPPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1p1ppppp/p7/2p5/2P5/4P3/PP1P1PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppp1p1p/4p3/6p1/P7/3P4/1PP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/1ppppppp/p6n/8/1P6/2N5/P1PPPPPP/R1BQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n5/5p2/4P3/3P4/PPP2PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/6p1/7p/3P4/2N5/PPP1PPPP/R1BQKB1R w KQkq - 0 3
r1bqkbnr/pppppp1p/n5p1/8/8/P6P/1PPPPPP1/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 1 3
rnbqkb1r/pp1ppppp/7n/2p5/3P4/8/PPPBPPPP/RN1QKB1R w KQkq - 0 3
r1bqkbnr/pppppppp/8/n3P3/8/8/PPPP1PPP/RNBQKB1R w KQkq - 1 3
rnbqkb1r/pppppppp/8/6P1/4n3/8/PPPPPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbr1/pppppppp/5n2/8/1P2P3/8/P1PP1PPP/RNBQKB1R w KQq - 1 3
rnbq1bnr/pppkpppp/8/3p4/8/NP6/P1PPPPPP/R1BQKB1R w KQ - 1 3
r1bqkbnr/ppppp1pp/n4p2/8/8/1P1P4/P1P1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1pppppp1/p6p/8/3P1B2/8/PPP1PPPP/RN1QKB1R w KQkq - 0 3
rnbqkb1r/pppppp1p/7n/6p1/5P2/8/PPPPP1PP/RNBQKBR1 w Qkq - 1 3
r1bqkbnr/pp1ppppp/n1p5/8/3P4/P7/1PP1PPPP/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/4P3/6P1/PPPP1P1P/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppppp1/n6p/2P5/8/8/PP1PPPPP/RNBQKB1R w KQkq - 1 3
rnbqkbn1/pppppppr/7p/8/7P/1P6/P1PPPPP1/RNBQKB1R w KQq - 1 3
rnbqkbnr/ppp1p1pp/3p4/5p2/6P1/2P5/PP1PPP1P/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pppp1ppp/4p2n/8/8/4P3/PPPPBPPP/RNBQK2R w KQkq - 2 3
rnbqkbnr/pppppp1p/8/8/6p1/P5P1/1PPPPP1P/RNBQKB1R w KQkq - 0 3
1nbqkbnr/1ppppppp/r7/p7/8/NP6/P1PPPPPP/R1BQKB1R w KQk - 2 3
r1bqkb1r/pppppppp/2n4n/8/8/2NP4/PPP1PPPP/R1BQKB1R w KQkq - 3 3
rnbqkbr1/pppppppp/5n2/8/8/P6P/1PPPPPP1/RNBQKB1R w KQq - 1 3
Cool... hopefully there is a solution for the cute-chess obstacle.
Updated. The test set is now mix of knight g1/b1 removed.

https://github.com/ChrisWhittington/Che ... t-odds.epd

Random sample:

Code: Select all

rnb1kbnr/ppqppppp/8/2P5/8/8/P1PPPPPP/R1BQKBNR w KQkq - 1 3
rnbqkbnr/p1pppppp/8/1p6/8/1P4P1/P1PPPP1P/RNBQKB1R w KQkq - 0 3
rnbqkbnr/2pppppp/1p6/p7/8/2N1P3/PPPP1PPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/2pppppp/8/pp6/8/4P3/PPPP1PPP/1RBQKBNR w Kkq - 0 3
rnbqkb1r/pppppppp/8/8/3P2n1/8/PPPBPPPP/R2QKBNR w KQkq - 3 3
rnbqkbnr/ppp1p1pp/3p4/5p2/8/2P4N/PP1PPPPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pp1ppppp/8/7Q/2p1P3/8/PPPP1PPP/RNB1KB1R w KQkq - 0 3
rnbqkbnr/p1p1pppp/8/1p1p4/3P4/2N5/PPP1PPPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/6p1/7p/8/6P1/PPPPPPBP/RNBQK2R w KQkq - 0 3
r1bqkbnr/p1pppppp/np6/8/7P/3P4/PPP1PPP1/RNBQKB1R w KQkq - 1 3
rnbqkbnr/p1p1pppp/3p4/1p6/7P/4P3/PPPP1PP1/R1BQKBNR w KQkq - 0 3
rnbqkbnr/1ppppp1p/p7/6p1/8/4P3/PPPPBPPP/R1BQK1NR w KQkq - 0 3
rnbqkbnr/p1pppp1p/1p6/6p1/8/PP6/2PPPPPP/RNBQKB1R w KQkq - 0 3
rnbqkbr1/pppppppp/7n/8/8/5PP1/PPPPP2P/R1BQKBNR w KQq - 1 3
r1bqkbnr/pppppppp/8/8/1n2PP2/8/PPPP2PP/R1BQKBNR w KQkq - 1 3
r1bqkbnr/ppppppp1/2n5/7p/7P/2N5/PPPPPPP1/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pppppppp/8/8/5P2/3P4/PPP1P1PP/R1BQKBNR w KQkq - 1 3
rnbqkbnr/p1p1pppp/3p4/1p6/4P3/8/PPPPQPPP/RNB1KB1R w KQkq - 0 3
rnbqkb1r/p1pppppp/5n2/1p6/2P5/P7/1P1PPPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pp1ppppp/7n/2p5/1P6/8/PBPPPPPP/R2QKBNR w KQkq - 2 3
rnbqkb1r/1ppppppp/7n/p7/P7/N7/1PPPPPPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/7p/6p1/8/P5P1/1PPPPP1P/RNBQKB1R w KQkq - 0 3
rn1qkbnr/ppp1pppp/3pb3/8/2P5/5N2/PP1PPPPP/R1BQKB1R w KQkq - 1 3
rnbqkbnr/1ppppp1p/8/p5p1/P7/8/1PPPPPPP/RNBQKBR1 w Qkq - 0 3
rnbqkb1r/pppppp1p/5n2/6p1/2P5/7P/PP1PPPP1/R1BQKBNR w KQkq - 0 3
r1bqkbnr/ppppp1pp/n4p2/8/8/1P5P/P1PPPPP1/R1BQKBNR w KQkq - 0 3
rnb1kbnr/ppppqppp/4p3/8/8/2N4P/PPPPPPP1/R1BQKB1R w KQkq - 1 3
rnb1kbnr/pppp1ppp/4p3/8/3P3q/8/PPPBPPPP/RN1QKB1R w KQkq - 2 3
rnbqkbnr/ppp1pp1p/3p2p1/8/2PP4/8/PP2PPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/ppp1p1pp/3p1p2/8/8/P2P4/1PP1PPPP/R1BQKBNR w KQkq - 0 3
rnbqkb1r/ppppp1pp/7n/5p2/P6P/8/1PPPPPP1/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1pppppp1/8/p6p/4P3/5Q2/PPPP1PPP/R1B1KBNR w KQkq - 0 3
rnbqkbnr/p2ppppp/2p5/1p6/2P5/4P3/PP1P1PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pp1pp1pp/5p2/2p5/4P3/6P1/PPPP1P1P/R1BQKBNR w KQkq - 0 3
rnbqkbnr/1pppppp1/B6p/8/8/4P3/PPPP1PPP/R1BQK1NR w KQkq - 0 3
rnbqkb1r/1ppppppp/p6n/8/6P1/2P5/PP1PPP1P/RNBQKB1R w KQkq - 0 3
rnb1kbnr/pppp1ppp/8/4p3/3P3q/7N/PPP1PPPP/R1BQKB1R w KQkq - 1 3
rnbqkbnr/pp1ppp1p/2p5/6p1/3P2P1/8/PPP1PP1P/RNBQKB1R w KQkq - 0 3
r1bqkbnr/ppppppp1/n7/1B5p/4P3/8/PPPP1PPP/RNBQK2R w KQkq - 2 3
rnbqkbnr/pp1ppp1p/2p5/3P2p1/8/8/PPP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/ppp1pppp/7n/3p4/2P5/1Q6/PP1PPPPP/R1B1KBNR w KQkq - 0 3
r1bqkbnr/1ppppppp/2n5/p7/3P3P/8/PPP1PPP1/R1BQKBNR w KQkq - 1 3
rnbqk1nr/ppppppbp/8/6p1/P7/2P5/1P1PPPPP/RNBQKB1R w KQkq - 1 3
r1bqkbnr/pp1ppppp/n1p5/8/3P4/5P2/PPP1P1PP/R1BQKBNR w KQkq - 0 3
rnbqkbnr/pp1ppp1p/2p5/6p1/8/3P1P2/PPP1P1PP/R1BQKBNR w KQkq - 0 3
rnbqkbnr/pppp1p1p/4p3/6p1/2P2P2/8/PP1PP1PP/RNBQKB1R w KQkq - 0 3
r1bqkb1r/pppppppp/n4n2/8/4P3/6P1/PPPP1P1P/R1BQKBNR w KQkq - 1 3
rn1qkbnr/ppp1pppp/3pb3/8/3P4/4P3/PPP2PPP/R1BQKBNR w KQkq - 1 3
rnbqkbnr/p2ppppp/2p5/1p6/4P3/8/PPPP1PPP/R1BQKBNR w KQkq - 0 3
rnbqkbnr/1ppppp1p/p7/6pP/8/8/PPPPPPP1/R1BQKBNR w KQkq - 0 3
rnbqkbnr/ppppp2p/5pp1/8/3P4/5P2/PPP1P1PP/R1BQKBNR w KQkq - 0 3
rnbqkb1r/ppppp1pp/7n/5p2/4P3/6P1/PPPP1P1P/R1BQKBNR w KQkq - 0 3
rnbqkbnr/1ppp1ppp/4p3/p7/8/2P2N2/PP1PPPPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pp1pp1pp/2p5/5p2/8/3P4/PPPBPPPP/RN1QKB1R w KQkq - 0 3
Top
User avatar
Rebel
Posts: 6991
Joined: Thu Aug 18, 2011 12:04 pm

Re: Stockfish Handicap Matches

Post by Rebel »

Results at 40/240

Code: Select all

No. Engine               1     2     3     4     5     6     7     8  Score  Games   Perc   Moves
-------------------------------------------------------------------------------------------------
 1 Stockfish_11      xxxxx   0.0   1.5   1.5   4.0   4.0   5.5   7.0   23.5 /   70 (33.57%)  68.7  
 2 Komodo_14          10.0 xxxxx   0.0   0.0   0.0   0.0   0.0   0.0   10.0 /   10 (100.00%)  58.5  
 3 Ethereal_12.25      8.5   0.0 xxxxx   0.0   0.0   0.0   0.0   0.0    8.5 /   10 (85.00%)  72.9  
 4 rofChade_2.3        8.5   0.0   0.0 xxxxx   0.0   0.0   0.0   0.0    8.5 /   10 (85.00%)  71.8  
 5 Laser_1.7           6.0   0.0   0.0   0.0 xxxxx   0.0   0.0   0.0    6.0 /   10 (60.00%)  72.8  
 6 Benjamin            6.0   0.0   0.0   0.0   0.0 xxxxx   0.0   0.0    6.0 /   10 (60.00%)  76.6  
 7 Rybka_1.0           4.5   0.0   0.0   0.0   0.0   0.0 xxxxx   0.0    4.5 /   10 (45.00%)  67.7  
 8 Fruit_2.1           3.0   0.0   0.0   0.0   0.0   0.0   0.0 xxxxx    3.0 /   10 (30.00%)  60.8
Komodo the perfect 100% score.
90% of coding is debugging, the other 10% is writing bugs.
Top
lkaufman
Posts: 5960
Joined: Sun Jan 10, 2010 6:15 am
Location: Maryland USA

Re: Stockfish Handicap Matches

Post by lkaufman »

Rebel wrote: ↑Mon Jun 22, 2020 2:31 am Results at 40/240

Code: Select all

No. Engine               1     2     3     4     5     6     7     8  Score  Games   Perc   Moves
-------------------------------------------------------------------------------------------------
 1 Stockfish_11      xxxxx   0.0   1.5   1.5   4.0   4.0   5.5   7.0   23.5 /   70 (33.57%)  68.7  
 2 Komodo_14          10.0 xxxxx   0.0   0.0   0.0   0.0   0.0   0.0   10.0 /   10 (100.00%)  58.5  
 3 Ethereal_12.25      8.5   0.0 xxxxx   0.0   0.0   0.0   0.0   0.0    8.5 /   10 (85.00%)  72.9  
 4 rofChade_2.3        8.5   0.0   0.0 xxxxx   0.0   0.0   0.0   0.0    8.5 /   10 (85.00%)  71.8  
 5 Laser_1.7           6.0   0.0   0.0   0.0 xxxxx   0.0   0.0   0.0    6.0 /   10 (60.00%)  72.8  
 6 Benjamin            6.0   0.0   0.0   0.0   0.0 xxxxx   0.0   0.0    6.0 /   10 (60.00%)  76.6  
 7 Rybka_1.0           4.5   0.0   0.0   0.0   0.0   0.0 xxxxx   0.0    4.5 /   10 (45.00%)  67.7  
 8 Fruit_2.1           3.0   0.0   0.0   0.0   0.0   0.0   0.0 xxxxx    3.0 /   10 (30.00%)  60.8
Komodo the perfect 100% score.
That's actually rather surprising, assuming you were still using your original handicap set. Only about five of those ten positions were bad enough to be resigned in games between human GMs. All are of course won for White, but I would have expected Stockfish to hold at least one draw out of the five non-resignable positions even vs. Komodo. Based on the results, I suspect that even tailender Fruit 2.1 would come out ahead of Stockfish 11 at this time limit at knight odds, but perhaps not way ahead. So engines slightly below Fruit 2.1 on the rating lists might be suitable opponents for Stockfish and Komodo in a true knight odds match.
Komodo rules!
Top
lkaufman
Posts: 5960
Joined: Sun Jan 10, 2010 6:15 am
Location: Maryland USA

Re: Stockfish Handicap Matches

Post by lkaufman »

chrisw wrote: ↑Mon Jun 22, 2020 12:42 am
lkaufman wrote: ↑Mon Jun 22, 2020 12:30 am
Rebel wrote: ↑Mon Jun 22, 2020 12:09 am
chrisw wrote: ↑Sun Jun 21, 2020 11:20 pm Done 5600 EPDs off the start position minus b1 knight, played out all four ply combinations, culled all duplicates, culled all positions where SF11 evaluated more than +/-10 centipawns away from 300 centipawns (SF11 average score for all epds), and am now left with 5600 EPDs.

Link: https://github.com/ChrisWhittington/Che ... t-odds.epd

Will upload for no knight at g1 tomorrow am.

Small randomised sample below

Code: Select all

rnbqkbnr/pppp2pp/5p2/4p3/4P2P/8/PPPP1PP1/RNBQKB1R w KQkq - 0 3
rnbqk1nr/pppp1ppp/3bp3/8/3P4/8/PPP1PPPP/RNBQKBR1 w Qkq - 2 3
rnbqkb1r/pppppp1p/6pn/8/3P2P1/8/PPP1PP1P/RNBQKB1R w KQkq - 0 3
rnbqkbnr/p1p1pppp/1p1p4/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 0 3
rnbqkb1r/ppppnppp/8/4p3/8/2P3P1/PP1PPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbnr/p1ppppp1/8/1p5p/8/1QP5/PP1PPPPP/RNB1KB1R w KQkq - 0 3
rnbqkbnr/pp1ppp1p/2p5/6p1/8/N4P2/PPPPP1PP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/p1ppppp1/7p/1p6/1P6/P7/2PPPPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1p1ppppp/p7/2p5/2P5/4P3/PP1P1PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppp1p1p/4p3/6p1/P7/3P4/1PP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/1ppppppp/p6n/8/1P6/2N5/P1PPPPPP/R1BQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n5/5p2/4P3/3P4/PPP2PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/6p1/7p/3P4/2N5/PPP1PPPP/R1BQKB1R w KQkq - 0 3
r1bqkbnr/pppppp1p/n5p1/8/8/P6P/1PPPPPP1/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/8/P1N5/1PPPPPPP/R1BQKB1R w KQkq - 1 3
rnbqkb1r/pp1ppppp/7n/2p5/3P4/8/PPPBPPPP/RN1QKB1R w KQkq - 0 3
r1bqkbnr/pppppppp/8/n3P3/8/8/PPPP1PPP/RNBQKB1R w KQkq - 1 3
rnbqkb1r/pppppppp/8/6P1/4n3/8/PPPPPP1P/RNBQKB1R w KQkq - 1 3
rnbqkbr1/pppppppp/5n2/8/1P2P3/8/P1PP1PPP/RNBQKB1R w KQq - 1 3
rnbq1bnr/pppkpppp/8/3p4/8/NP6/P1PPPPPP/R1BQKB1R w KQ - 1 3
r1bqkbnr/ppppp1pp/n4p2/8/8/1P1P4/P1P1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1pppppp1/p6p/8/3P1B2/8/PPP1PPPP/RN1QKB1R w KQkq - 0 3
rnbqkb1r/pppppp1p/7n/6p1/5P2/8/PPPPP1PP/RNBQKBR1 w Qkq - 1 3
r1bqkbnr/pp1ppppp/n1p5/8/3P4/P7/1PP1PPPP/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppp1pp/2n2p2/8/4P3/6P1/PPPP1P1P/RNBQKB1R w KQkq - 1 3
r1bqkbnr/ppppppp1/n6p/2P5/8/8/PP1PPPPP/RNBQKB1R w KQkq - 1 3
rnbqkbn1/pppppppr/7p/8/7P/1P6/P1PPPPP1/RNBQKB1R w KQq - 1 3
rnbqkbnr/ppp1p1pp/3p4/5p2/6P1/2P5/PP1PPP1P/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pppp1ppp/4p2n/8/8/4P3/PPPPBPPP/RNBQK2R w KQkq - 2 3
rnbqkbnr/pppppp1p/8/8/6p1/P5P1/1PPPPP1P/RNBQKB1R w KQkq - 0 3
1nbqkbnr/1ppppppp/r7/p7/8/NP6/P1PPPPPP/R1BQKB1R w KQk - 2 3
r1bqkb1r/pppppppp/2n4n/8/8/2NP4/PPP1PPPP/R1BQKB1R w KQkq - 3 3
rnbqkbr1/pppppppp/5n2/8/8/P6P/1PPPPPP1/RNBQKB1R w KQq - 1 3
Cool... hopefully there is a solution for the cute-chess obstacle.
Something was really wrong,
Well. Results are results. The average SF eval for all the 4-ply positions, duplicates culled, is within a whisker of 300 centipawns at 25ms search, default SF11.

Check the epds listed, they ought, under similar search parameters, to result in SF scores of -300 +/- 10

I checked several positions and they showed scores of 4 pawns or more down, as does the initial knight odds position, not -300.
You checked the listed epds for variance from -300? Which ones? What was the variance? What SF11 conditions are you using?

If you just want under 20 positions, take off the b1 knight, choose the 3 best White moves by SF11 multiPV, chose the 3 best replies to each of those, and repeat with g1 knight off. 18 positions, totally fair, no silly moves, real knight odds chess! If you want more just choose best 4 or best five for each side.
Edit: Whoops, rechecked, the mean is indeed -293, but I was selecting EPDs based on -340, so you should find the listed EPDs evaluate at around -340 centipawns. Will correct the EPD dump tomorrow am.
OK, -340 is at least within range of what I was seeing. With fixed depth searches in the range of what you were using I get evals like -390 or so, but I think fixed depth omits Contempt while movetime does not, so we're not so far apart if you add in Contempt. You were right to switch from 10 ms to 25; SF is quite weak at 10ms but strong enough at 25, the difference is huge. But I think it would be more useful to have fewer positions but no positions with ridiculous moves played, maybe only including moves in the top ten by multipv at each point for example. It doesn't seem like a simulation of knight odds if you force the players to play moves that no one over 800 rating would even consider.
Komodo rules!
Top
User avatar
Rebel
Posts: 6991
Joined: Thu Aug 18, 2011 12:04 pm

Re: Stockfish Handicap Matches

Post by Rebel »

chrisw wrote: ↑Mon Jun 22, 2020 12:47 am Updated. The test set is now mix of knight g1/b1 removed.

https://github.com/ChrisWhittington/Che ... t-odds.epd

Random sample:

Code: Select all

rnb1kbnr/ppqppppp/8/2P5/8/8/P1PPPPPP/R1BQKBNR w KQkq - 1 3
rnbqkbnr/p1pppppp/8/1p6/8/1P4P1/P1PPPP1P/RNBQKB1R w KQkq - 0 3
rnbqkbnr/2pppppp/1p6/p7/8/2N1P3/PPPP1PPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/2pppppp/8/pp6/8/4P3/PPPP1PPP/1RBQKBNR w Kkq - 0 3
rnbqkb1r/pppppppp/8/8/3P2n1/8/PPPBPPPP/R2QKBNR w KQkq - 3 3
rnbqkbnr/ppp1p1pp/3p4/5p2/8/2P4N/PP1PPPPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pp1ppppp/8/7Q/2p1P3/8/PPPP1PPP/RNB1KB1R w KQkq - 0 3
rnbqkbnr/p1p1pppp/8/1p1p4/3P4/2N5/PPP1PPPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/6p1/7p/8/6P1/PPPPPPBP/RNBQK2R w KQkq - 0 3
r1bqkbnr/p1pppppp/np6/8/7P/3P4/PPP1PPP1/RNBQKB1R w KQkq - 1 3
rnbqkbnr/p1p1pppp/3p4/1p6/7P/4P3/PPPP1PP1/R1BQKBNR w KQkq - 0 3
rnbqkbnr/1ppppp1p/p7/6p1/8/4P3/PPPPBPPP/R1BQK1NR w KQkq - 0 3
rnbqkbnr/p1pppp1p/1p6/6p1/8/PP6/2PPPPPP/RNBQKB1R w KQkq - 0 3
rnbqkbr1/pppppppp/7n/8/8/5PP1/PPPPP2P/R1BQKBNR w KQq - 1 3
r1bqkbnr/pppppppp/8/8/1n2PP2/8/PPPP2PP/R1BQKBNR w KQkq - 1 3
r1bqkbnr/ppppppp1/2n5/7p/7P/2N5/PPPPPPP1/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pppppppp/8/8/5P2/3P4/PPP1P1PP/R1BQKBNR w KQkq - 1 3
rnbqkbnr/p1p1pppp/3p4/1p6/4P3/8/PPPPQPPP/RNB1KB1R w KQkq - 0 3
rnbqkb1r/p1pppppp/5n2/1p6/2P5/P7/1P1PPPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pp1ppppp/7n/2p5/1P6/8/PBPPPPPP/R2QKBNR w KQkq - 2 3
rnbqkb1r/1ppppppp/7n/p7/P7/N7/1PPPPPPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pppppp2/7p/6p1/8/P5P1/1PPPPP1P/RNBQKB1R w KQkq - 0 3
rn1qkbnr/ppp1pppp/3pb3/8/2P5/5N2/PP1PPPPP/R1BQKB1R w KQkq - 1 3
rnbqkbnr/1ppppp1p/8/p5p1/P7/8/1PPPPPPP/RNBQKBR1 w Qkq - 0 3
rnbqkb1r/pppppp1p/5n2/6p1/2P5/7P/PP1PPPP1/R1BQKBNR w KQkq - 0 3
r1bqkbnr/ppppp1pp/n4p2/8/8/1P5P/P1PPPPP1/R1BQKBNR w KQkq - 0 3
rnb1kbnr/ppppqppp/4p3/8/8/2N4P/PPPPPPP1/R1BQKB1R w KQkq - 1 3
rnb1kbnr/pppp1ppp/4p3/8/3P3q/8/PPPBPPPP/RN1QKB1R w KQkq - 2 3
rnbqkbnr/ppp1pp1p/3p2p1/8/2PP4/8/PP2PPPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/ppp1p1pp/3p1p2/8/8/P2P4/1PP1PPPP/R1BQKBNR w KQkq - 0 3
rnbqkb1r/ppppp1pp/7n/5p2/P6P/8/1PPPPPP1/RNBQKB1R w KQkq - 0 3
rnbqkbnr/1pppppp1/8/p6p/4P3/5Q2/PPPP1PPP/R1B1KBNR w KQkq - 0 3
rnbqkbnr/p2ppppp/2p5/1p6/2P5/4P3/PP1P1PPP/RNBQKB1R w KQkq - 0 3
rnbqkbnr/pp1pp1pp/5p2/2p5/4P3/6P1/PPPP1P1P/R1BQKBNR w KQkq - 0 3
rnbqkbnr/1pppppp1/B6p/8/8/4P3/PPPP1PPP/R1BQK1NR w KQkq - 0 3
rnbqkb1r/1ppppppp/p6n/8/6P1/2P5/PP1PPP1P/RNBQKB1R w KQkq - 0 3
rnb1kbnr/pppp1ppp/8/4p3/3P3q/7N/PPP1PPPP/R1BQKB1R w KQkq - 1 3
rnbqkbnr/pp1ppp1p/2p5/6p1/3P2P1/8/PPP1PP1P/RNBQKB1R w KQkq - 0 3
r1bqkbnr/ppppppp1/n7/1B5p/4P3/8/PPPP1PPP/RNBQK2R w KQkq - 2 3
rnbqkbnr/pp1ppp1p/2p5/3P2p1/8/8/PPP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/ppp1pppp/7n/3p4/2P5/1Q6/PP1PPPPP/R1B1KBNR w KQkq - 0 3
r1bqkbnr/1ppppppp/2n5/p7/3P3P/8/PPP1PPP1/R1BQKBNR w KQkq - 1 3
rnbqk1nr/ppppppbp/8/6p1/P7/2P5/1P1PPPPP/RNBQKB1R w KQkq - 1 3
r1bqkbnr/pp1ppppp/n1p5/8/3P4/5P2/PPP1P1PP/R1BQKBNR w KQkq - 0 3
rnbqkbnr/pp1ppp1p/2p5/6p1/8/3P1P2/PPP1P1PP/R1BQKBNR w KQkq - 0 3
rnbqkbnr/pppp1p1p/4p3/6p1/2P2P2/8/PP1PP1PP/RNBQKB1R w KQkq - 0 3
r1bqkb1r/pppppppp/n4n2/8/4P3/6P1/PPPP1P1P/R1BQKBNR w KQkq - 1 3
rn1qkbnr/ppp1pppp/3pb3/8/3P4/4P3/PPP2PPP/R1BQKBNR w KQkq - 1 3
rnbqkbnr/p2ppppp/2p5/1p6/4P3/8/PPPP1PPP/R1BQKBNR w KQkq - 0 3
rnbqkbnr/1ppppp1p/p7/6pP/8/8/PPPPPPP1/R1BQKBNR w KQkq - 0 3
rnbqkbnr/ppppp2p/5pp1/8/3P4/5P2/PPP1P1PP/R1BQKBNR w KQkq - 0 3
rnbqkb1r/ppppp1pp/7n/5p2/4P3/6P1/PPPP1P1P/R1BQKBNR w KQkq - 0 3
rnbqkbnr/1ppp1ppp/4p3/p7/8/2P2N2/PP1PPPPP/R1BQKB1R w KQkq - 0 3
rnbqkbnr/pp1pp1pp/2p5/5p2/8/3P4/PPPBPPPP/RN1QKB1R w KQkq - 0 3
There is no download button for the file, can you mail it ?
90% of coding is debugging, the other 10% is writing bugs.
Top

Return to “Computer Chess Club: General Topics”