Two Pawn Handicap

[Uri Blass]

Yes, we conceded the 6th f2/c2 game. Neuman demonstrated, very methodically, how to win against f2/c2 odds.

I may be wrong, but it looks like the final result, correctly including the conceded game, was not even posted "officially". When I looked for it, all I could find is some remarks from Larry, and the very important fact that Komodo had actually won a game, that was not even on the schedule. The GM had no chance to prepare for it, just saying. So no result, but some random success that has nothing to do with the announced event. That's what made me a little uneasy ...
People may buy this "for fun and in order to learn something"-stuff, but let's be honest and admit it is a marketing event, of course. It did not go as expected, and I would find it much better style to give this fact some room to be processed by the potential customers.
The basic "problem" (of course, there is not really one) is that Komodo cannot develop a strategy to deal with any kind of handicap situation. That's because it never does develop a strategy. The stupid humans like Carlsen, Kasparov etc. would have much better chances to beat an opponent like Neuman under the given circumstances, because they can develop a strategy. So the whole event actually highlights the general problems of computer chess instead of showing how much engines have improved. While a player whose Elo is only 200 Elo higher can potentially win in such a situation, the engine needs an absurd rating advantage because of its inflexibility to adapt to the situation.

We have no evidence that top humans have a chance to beat GM's with 2 pawns handicap.

Kasparov played against clearly weaker player and did not win easily.

Computers may be weaker than top humans with a big handicap but I think that you need queen handicap to show it and it may be interesting to have a tournament when both GM's and machines play against relatively weak players with queen handicap to see who does better.

I would like to see both Komodo and some GM in a competition like the world championship for boys under 8 when both start every game without Queen d1.
I am sure that it is possible to beat part of the players in this competition with a queen handicap but not all of them.

[Jesse Gersenson]

Chapman played in the 2015 World 50+ team championships. Got his FM title in 2013, at the age of 57! That's one motivated guy.

The variety of the odds in his Kasparov match made for interesting chess every game. Because the positions weren't repeated the deficiency of a given down-2-pawns starting position wasn't as prominent, as say in the Neuman games. That said, I quiet enjoyed watching the GM learn, and eventually solving the f2/c2 position.

a2/h2, a2/e2, a2/d2, a2/b2 are all possibilities, so is pawn and move, or pawn and two moves. I'm rereading "Paul Morphy: The Chess Champion : an Account of His Career in America and Europe : with a History of Chess and Chess Clubs and Anecdotes of Famous Players" by Frederick Milnes Edge, available for pdf download on books.google.com, to find the historical precedent for determining the odds of a given match.

Going over f7 odds games I'm surprised how often the stronger side loses.
Cecil Valentine De Vere vs Wilhelm Steinitz (1865) for example, several games played at these odds with Steinitz (born in prague) getting the worse of it:
http://www.chessgames.com/perl/chessgame?gid=1485585

[Laskos]

It seems most of us underestimated the grandmaster in this match or at least underestimated the handicap. I think it's pretty obvious now that the handicap of f2 and c2 pawns was just too much for anyone or anything to give to a grandmaster in a serious game. Yet Komodo won fairly easily playing Black and giving the f7 pawn only, which is the worst pawn for Black to remove.
I think two pawn handicap is still playable against an ordinary (around 2500) grandmaster, if we are a bit more conservative about the choice of the pawns. As it was, White was not only two pawns down, but his king was weakened and his pawns were split up into three groups. Moreover he only had two of the four pawns that can control central squares. I chose this handicap out of deference to the tradition of giving the "f" pawn as a handicap, but it's just too difficult, especially if repeated game after game while the grandmaster learns each time.
When Kasparov gave two pawns to Terrence Chapman (said to be 2150 level) in a match, he removed the "a" pawn plus one other varying pawn. This is what we should have done too, although I think it was a bit unfair to play one game with both edge pawns removed, which is probably no more than the f7 handicap. I think Komodo can still offer two pawns to a grandmaster, if one is the "a" pawn and the other rotates between "b", "c", "d", and "e". These feel more like "just" a two pawn handicap with no added positional advantages on top.
Comments, anyone?

I have computed the handicaps in self-play of Komodo at ultra-fast (2 cores) with time odds, and extrapolated to 45'+15'' by analogy with the c2 and f2 pawns handicap, the only one where I tested this long time control. First, I got the values of doubling in time for Komodo (2 cores) at contempt 0, and the doubling seems indeed to be worth more than that of Houdini:

Score of Komodo 3s+0.03s vs Komodo 1.5s+0.015s:
680 - 28 - 292 [0.826] 1000
ELO difference: 271

Score of Komodo 6s+0.06s vs Komodo 3s+0.03s:
507 - 61 - 432 [0.723] 1000
ELO difference: 167

Score of Komodo 24s+0.24s vs Komodo 12s+0.12s:
385 - 51 - 564 [0.667] 1000
ELO difference: 121

Then, similarly (500 games each data point), the extrapolated handicaps to 45'+15'' on 2 cores (at faster control the handicaps are significantly lower):

• Knight b1:
  1170 ELO points
  
  Pawns c2 and f2:
  710 ELO points
  
  Pawns a2 and d2:
  490 ELO points
  
  Pawns a2 and h2:
  320 ELO points
  
  Pawn f7:
  510 ELO points
  
  Pawn d7:
  370 ELO points

The anomalies I get:

• Two pawns c2 and f2 handicap seems fair game for a 2500 GM, it was not.
  f7 pawn handicap seems unexpectedly large to me, much larger than two pawns a2 and h2 handicap, and on a par with two pawns a2 and d2 handicap.

If Komodo is 3200 FIDE ELO at 45'+15'' on multicore, then one can see the fair matches, for example d7 pawn handicap or a2 and h2 pawns handicap against Carlsen.

[duncan]

• Knight b1:
  1170 ELO points
  
  Pawns c2 and f2:
  710 ELO points
  
  Pawns a2 and d2:
  490 ELO points
  
  Pawns a2 and h2:
  320 ELO points
  
  Pawn f7:
  510 ELO points
  
  Pawn d7:
  370 ELO points

The anomalies I get:

• Two pawns c2 and f2 handicap seems fair game for a 2500 GM, it was not.
  f7 pawn handicap seems unexpectedly large to me, much larger than two pawns a2 and h2 handicap, and on a par with two pawns a2 and d2 handicap.

If Komodo is 3200 FIDE ELO at 45'+15'' on multicore, then one can see the fair matches, for example d7 pawn handicap or a2 and h2 pawns handicap against Carlsen.

following from this since Pawns c2 and f2(710) is 340 more than Pawn d7 (370), komodo would have to gain another 340 points before having a chance against carlsen with Pawns c2 and f2. taking into account hardware and software improvements maybe 3 years. ? otoh they are not unimportant pawns , so maybe never.

[Laskos]

• Knight b1:
  1170 ELO points
  
  Pawns c2 and f2:
  710 ELO points
  
  Pawns a2 and d2:
  490 ELO points
  
  Pawns a2 and h2:
  320 ELO points
  
  Pawn f7:
  510 ELO points
  
  Pawn d7:
  370 ELO points

The anomalies I get:

• Two pawns c2 and f2 handicap seems fair game for a 2500 GM, it was not.
  f7 pawn handicap seems unexpectedly large to me, much larger than two pawns a2 and h2 handicap, and on a par with two pawns a2 and d2 handicap.

If Komodo is 3200 FIDE ELO at 45'+15'' on multicore, then one can see the fair matches, for example d7 pawn handicap or a2 and h2 pawns handicap against Carlsen.

following from this since Pawns c2 and f2(710) is 340 more than Pawn d7 (370), komodo would have to gain another 340 points before having a chance against carlsen with Pawns c2 and f2. taking into account hardware and software improvements maybe 3 years. ? otoh they are not unimportant pawns , so maybe never.

When rating (or time control or hardware) increases, the same handicap increases its ELO value. Say with Komodo 340 ELO points stronger than today, handicap of c2 and f2 increases by 240 points to 950 ELO points, so there is only little progress in playing this handicap against Carlsen. I don't know if a perfect engine would stand this handicap against Carlsen.

[mjlef]

• Knight b1:
  1170 ELO points
  
  Pawns c2 and f2:
  710 ELO points
  
  Pawns a2 and d2:
  490 ELO points
  
  Pawns a2 and h2:
  320 ELO points
  
  Pawn f7:
  510 ELO points
  
  Pawn d7:
  370 ELO points

The anomalies I get:

• Two pawns c2 and f2 handicap seems fair game for a 2500 GM, it was not.
  f7 pawn handicap seems unexpectedly large to me, much larger than two pawns a2 and h2 handicap, and on a par with two pawns a2 and d2 handicap.

If Komodo is 3200 FIDE ELO at 45'+15'' on multicore, then one can see the fair matches, for example d7 pawn handicap or a2 and h2 pawns handicap against Carlsen.

following from this since Pawns c2 and f2(710) is 340 more than Pawn d7 (370), komodo would have to gain another 340 points before having a chance against carlsen with Pawns c2 and f2. taking into account hardware and software improvements maybe 3 years. ? otoh they are not unimportant pawns , so maybe never.

Although Larry and I have been pretty successful the last roughly 2 years I have worked on Komodo, I do not think we could keep up a pace of over 100 elo a year for 3 years. We have managed a bit over 50 elo a year recently, but you never know what will happen, and we could hit an elo wall. I am not suggestion we would ever get perfection, but there has to be some elo limit based on "perfect" software. Perhaps over time we can estimate that limit, although such estimates might be wrong because we might never think of the ideas needed to advance.

[lkaufman]

It seems most of us underestimated the grandmaster in this match or at least underestimated the handicap. I think it's pretty obvious now that the handicap of f2 and c2 pawns was just too much for anyone or anything to give to a grandmaster in a serious game. Yet Komodo won fairly easily playing Black and giving the f7 pawn only, which is the worst pawn for Black to remove.
I think two pawn handicap is still playable against an ordinary (around 2500) grandmaster, if we are a bit more conservative about the choice of the pawns. As it was, White was not only two pawns down, but his king was weakened and his pawns were split up into three groups. Moreover he only had two of the four pawns that can control central squares. I chose this handicap out of deference to the tradition of giving the "f" pawn as a handicap, but it's just too difficult, especially if repeated game after game while the grandmaster learns each time.
When Kasparov gave two pawns to Terrence Chapman (said to be 2150 level) in a match, he removed the "a" pawn plus one other varying pawn. This is what we should have done too, although I think it was a bit unfair to play one game with both edge pawns removed, which is probably no more than the f7 handicap. I think Komodo can still offer two pawns to a grandmaster, if one is the "a" pawn and the other rotates between "b", "c", "d", and "e". These feel more like "just" a two pawn handicap with no added positional advantages on top.
Comments, anyone?

I have computed the handicaps in self-play of Komodo at ultra-fast (2 cores) with time odds, and extrapolated to 45'+15'' by analogy with the c2 and f2 pawns handicap, the only one where I tested this long time control. First, I got the values of doubling in time for Komodo (2 cores) at contempt 0, and the doubling seems indeed to be worth more than that of Houdini:

Score of Komodo 3s+0.03s vs Komodo 1.5s+0.015s:
680 - 28 - 292 [0.826] 1000
ELO difference: 271

Score of Komodo 6s+0.06s vs Komodo 3s+0.03s:
507 - 61 - 432 [0.723] 1000
ELO difference: 167

Score of Komodo 24s+0.24s vs Komodo 12s+0.12s:
385 - 51 - 564 [0.667] 1000
ELO difference: 121

Then, similarly (500 games each data point), the extrapolated handicaps to 45'+15'' on 2 cores (at faster control the handicaps are significantly lower):

• Knight b1:
  1170 ELO points
  
  Pawns c2 and f2:
  710 ELO points
  
  Pawns a2 and d2:
  490 ELO points
  
  Pawns a2 and h2:
  320 ELO points
  
  Pawn f7:
  510 ELO points
  
  Pawn d7:
  370 ELO points

The anomalies I get:

• Two pawns c2 and f2 handicap seems fair game for a 2500 GM, it was not.
  f7 pawn handicap seems unexpectedly large to me, much larger than two pawns a2 and h2 handicap, and on a par with two pawns a2 and d2 handicap.

If Komodo is 3200 FIDE ELO at 45'+15'' on multicore, then one can see the fair matches, for example d7 pawn handicap or a2 and h2 pawns handicap against Carlsen.

Thanks for running these tests! The relative size of the handicaps looks about right to me, and the 710 value for the c2/f2 handicap is consistent with my direct results if we assume Gaussian rather than Logistic distribution, as you say. I'm running similar tests myself with the aim of measuring the handicaps without having to rely much (if at all) on extrapolation, but I don't expect my results will differ too much from yours.
If you have the computer time available, could you run a couple more handicaps? Especially the Exchange (remove a1 rook, b8 knight, move rook from a8 to b8), since we will surely have matches with this one. Also "pawn and tho moves" (remove f7, play 1.e4 and it's still White's move). And perhaps a couple more of the two pawn handicaps we may still use, probably a/b, a/c, and a/e.
It is a bit strange that c2/f2 is 200 more than f7 in your results, but it seemed like a day and night difference against Neuman. But maybe with preparation and learning from experience he would do well at f7 as well.
I think that the handicaps will always be more difficult to give to humans than the engine tests show since they know to avoid unclear tactics while the engines do not. But there's nothing you can do about this.

[JJJ]

If c2 and f2 are 700 elo handicap, when Komodo is less good than 3200 elo ( because 2500 GM won ).

[JJJ]

[d]rnbqkrnr/ppppp2p/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1

[d]rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/BNBQKBNR w kq - 0 1

Why not try this kind of position ?

[JJJ]

[d]rnbqkq1r/ppppp2p/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1

I know this doesn't look "normal" chess, but for me, there is no normal chess, specially with any pawn handicap.

But I suggest any position with handicap of -1 to -1,5 pawn at the begining.