JJJ wrote:[d]rnbqkq1r/ppppp2p/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
I know this doesn't look "normal" chess, but for me, there is no normal chess, specially with any pawn handicap.
But I suggest any position with handicap of -1 to -1,5 pawn at the begining.
I think that the goal is to make handicaps as close as possible to normal chess but with a score in a given range such as you give. The examples you give are too bizarre. One very reasonable handicap in that range is to start every game with 1.e4 f6? 2.d4 Kf7?. Or 1.e4 Nc6 2.d4 Nb8?? (the first handicap is clearly larger as it forfeits castling too). But this idea got few votes in the poll. The f7 handicap is a relatively good one, but is subject to the problem that it can be analyzed rather easily before the games using computers. We don't yet have a clear idea as to what elo is needed to break even with Komodo at f7 odds.
I also like the concept of "half a knight". One game at f7 odds, then one game with computer giving knight odds but human removes f7. So every two games the human gets one extra knight, and the handicap is fairly similar in size between the odd and even numbered games.
Last edited by lkaufman on Fri Aug 07, 2015 6:21 pm, edited 1 time in total.
mjlef wrote:
Although Larry and I have been pretty successful the last roughly 2 years I have worked on Komodo, I do not think we could keep up a pace of over 100 elo a year for 3 years. We have managed a bit over 50 elo a year recently,
is that 50 just from software. and how much has hardware given you ?
mjlef wrote:
Although Larry and I have been pretty successful the last roughly 2 years I have worked on Komodo, I do not think we could keep up a pace of over 100 elo a year for 3 years. We have managed a bit over 50 elo a year recently,
is that 50 just from software. and how much has hardware given you ?
Just software, and I think Mark was being conservative.
lkaufman wrote:We don't yet have a clear idea as to what elo is needed to break even with Komodo at f7 odds.
The 2700 didn't have much trouble with Rybka 3. So I'd guess <2700. Neuman thought he'd be even with Komodo at this handicap. Again, I wouldn't place much importance on his performance in game 6, played at the end of two hot long days...
lkaufman wrote:It seems most of us underestimated the grandmaster in this match or at least underestimated the handicap. I think it's pretty obvious now that the handicap of f2 and c2 pawns was just too much for anyone or anything to give to a grandmaster in a serious game. Yet Komodo won fairly easily playing Black and giving the f7 pawn only, which is the worst pawn for Black to remove.
I think two pawn handicap is still playable against an ordinary (around 2500) grandmaster, if we are a bit more conservative about the choice of the pawns. As it was, White was not only two pawns down, but his king was weakened and his pawns were split up into three groups. Moreover he only had two of the four pawns that can control central squares. I chose this handicap out of deference to the tradition of giving the "f" pawn as a handicap, but it's just too difficult, especially if repeated game after game while the grandmaster learns each time.
When Kasparov gave two pawns to Terrence Chapman (said to be 2150 level) in a match, he removed the "a" pawn plus one other varying pawn. This is what we should have done too, although I think it was a bit unfair to play one game with both edge pawns removed, which is probably no more than the f7 handicap. I think Komodo can still offer two pawns to a grandmaster, if one is the "a" pawn and the other rotates between "b", "c", "d", and "e". These feel more like "just" a two pawn handicap with no added positional advantages on top.
Comments, anyone?
I have computed the handicaps in self-play of Komodo at ultra-fast (2 cores) with time odds, and extrapolated to 45'+15'' by analogy with the c2 and f2 pawns handicap, the only one where I tested this long time control. First, I got the values of doubling in time for Komodo (2 cores) at contempt 0, and the doubling seems indeed to be worth more than that of Houdini:
Score of Komodo 3s+0.03s vs Komodo 1.5s+0.015s:
680 - 28 - 292 [0.826] 1000
ELO difference: 271
Score of Komodo 6s+0.06s vs Komodo 3s+0.03s:
507 - 61 - 432 [0.723] 1000
ELO difference: 167
Score of Komodo 24s+0.24s vs Komodo 12s+0.12s:
385 - 51 - 564 [0.667] 1000
ELO difference: 121
Then, similarly (500 games each data point), the extrapolated handicaps to 45'+15'' on 2 cores (at faster control the handicaps are significantly lower):
Knight b1:
1170 ELO points
Pawns c2 and f2:
710 ELO points
Pawns a2 and d2:
490 ELO points
Pawns a2 and h2:
320 ELO points
Pawn f7:
510 ELO points
Pawn d7:
370 ELO points
The anomalies I get:
Two pawns c2 and f2 handicap seems fair game for a 2500 GM, it was not.
f7 pawn handicap seems unexpectedly large to me, much larger than two pawns a2 and h2 handicap, and on a par with two pawns a2 and d2 handicap.
If Komodo is 3200 FIDE ELO at 45'+15'' on multicore, then one can see the fair matches, for example d7 pawn handicap or a2 and h2 pawns handicap against Carlsen.
Thanks for running these tests! The relative size of the handicaps looks about right to me, and the 710 value for the c2/f2 handicap is consistent with my direct results if we assume Gaussian rather than Logistic distribution, as you say. I'm running similar tests myself with the aim of measuring the handicaps without having to rely much (if at all) on extrapolation, but I don't expect my results will differ too much from yours.
If you have the computer time available, could you run a couple more handicaps? Especially the Exchange (remove a1 rook, b8 knight, move rook from a8 to b8), since we will surely have matches with this one. Also "pawn and tho moves" (remove f7, play 1.e4 and it's still White's move). And perhaps a couple more of the two pawn handicaps we may still use, probably a/b, a/c, and a/e.
It is a bit strange that c2/f2 is 200 more than f7 in your results, but it seemed like a day and night difference against Neuman. But maybe with preparation and learning from experience he would do well at f7 as well.
I think that the handicaps will always be more difficult to give to humans than the engine tests show since they know to avoid unclear tactics while the engines do not. But there's nothing you can do about this.
I will go on a trip for a week, so I tried to test the handicaps you proposed now, quick 500 game matches at ultra-fast time controls with Komodo on 2 cores, extrapolated to 45'+15'':
Exchange:
540 ELO points
Pawn f7 and two moves (to e4):
600 ELO points
Pawns a2 and c2:
560 ELO points
Pawns a2 and b2:
440 ELO points
Definitely missing f7 seems very unpleasant in many circumstances, and a2 + b2 handicap seems significantly smaller than a2 + c2.
Jesse Gersenson wrote:Komodo's performance rating was 2223 in the f2/c2 match and 1996 in the knight odds match.
So if Kai's handicap values are correct, the estimated elo was 2933 for the f2/c2 match and 3166 for the knight odds match, average 3050. This is much lower than the rating lists give, but who knows what rating these engines would actually get in Elite tournaments? Randomized openings does give inflated ratings for the top engines compared to deep optimized books.
