Komodo 10.4 vs Komodo 1 - Knight odds

[lkaufman]

Just for fun, I ran a hyper-bullet match between the latest Komodo (10.4) and the first of the series (Komodo 1) at knight odds, both using just one thread. Time limit 5" + .05", admittedly too fast to take too seriously. Randomization from a small knight odds book, from the granularity of the fast time limit, and from the new Variety feature (set to 10 in Komodo 10.4, at a likely cost of a couple elo points). For those not familiar with Komodo 1, it was not a new program, just the successor to the "Doch" series, and is rated on both the CCRL lists in the low 2900s, about equal to Rybka 2.2, the world's strongest engine in early 2007 when I joined the Rybka team. Komodo 1 would almost certainly be favored to win a match from Magnus Carlsen (or anyone else) at any practical over-the-board time limit.
Well, Komodo 1 won the match, by about 57% to 43 % or about 50 elo after 2,000 games. But considering that knight odds is generally thought to be well over a thousand elo handicap (ok, somewhat less at 5") and would normally simply be resigned between grandmasters, I find this rather remarkable. Maybe when Komodo 11 comes out we'll actually be able to win such a knight odds match from Komod 1. I wouldn't read too much into this, but it does make me feel better that I can no longer win at knight odds from Komodo at blitz (4' +2"). If any Komodo users out there have been able to win such blitz games from Komodo 10.4 (say on at least four cores, with contempt set to at least 100), let us hear from you. I suppose the top ten players can still do so, though I'm not sure.

[lkaufman]

I forgot to set Contempt for this match. So I reran a thousand games, this time with Contempt at 100, and Komodo 1 won by only 52%-48% or 14 elo. I got a very similar result for Contempt 75, suggesting that the best setting for knight odds is in the 75 to 100 range, sinc Contempt 50 and 125 each gave results around +25 elo for K1. So with any measurable progress and the right Contempt setting, Komodo 11 should be able to break even or better giving knight odds to Komodo 1 at this 5" + .05" level.

[JJJ]

Very interesting and nice to read Larry. Will you set a match against human with Knight odds anytime soon ?

If you want to run it at blitz time control, you might want to ask someone like penguingim1 aka Andrew Tang. Perhaps he would agree. Or anyone else.

[Laskos]

Just for fun, I ran a hyper-bullet match between the latest Komodo (10.4) and the first of the series (Komodo 1) at knight odds, both using just one thread. Time limit 5" + .05", admittedly too fast to take too seriously. Randomization from a small knight odds book, from the granularity of the fast time limit, and from the new Variety feature (set to 10 in Komodo 10.4, at a likely cost of a couple elo points). For those not familiar with Komodo 1, it was not a new program, just the successor to the "Doch" series, and is rated on both the CCRL lists in the low 2900s, about equal to Rybka 2.2, the world's strongest engine in early 2007 when I joined the Rybka team. Komodo 1 would almost certainly be favored to win a match from Magnus Carlsen (or anyone else) at any practical over-the-board time limit.
Well, Komodo 1 won the match, by about 57% to 43 % or about 50 elo after 2,000 games. But considering that knight odds is generally thought to be well over a thousand elo handicap (ok, somewhat less at 5") and would normally simply be resigned between grandmasters, I find this rather remarkable. Maybe when Komodo 11 comes out we'll actually be able to win such a knight odds match from Komod 1. I wouldn't read too much into this, but it does make me feel better that I can no longer win at knight odds from Komodo at blitz (4' +2"). If any Komodo users out there have been able to win such blitz games from Komodo 10.4 (say on at least four cores, with contempt set to at least 100), let us hear from you. I suppose the top ten players can still do so, though I'm not sure.

Nice! This is heavily dependent on time control. At 5''+0.05'' at top level, Knight Odds might be worth 700 ELO points. at LTC (tournament TC) might go to 1400 ELO points. But yes, for checking contempt and Komodo progress in Odds matches, it is useful. I guess Komodo progress in equal Knight-Odds matches (against weak opponent) is no more than half of Komodo progress in equal no-Odds matches (no-Odds against top engines).

[lkaufman]

Very interesting and nice to read Larry. Will you set a match against human with Knight odds anytime soon ?

If you want to run it at blitz time control, you might want to ask someone like penguingim1 aka Andrew Tang. Perhaps he would agree. Or anyone

else.

We already showed more than a year ago that at blitz (3' +2") Komodo giving knight odds was pretty even with GMs Lenderman and Mikhalevsky. But in "slow" (45' + 15") chess Komodo performed below 2000 FIDE in a dozen games. I do think that Komodo will perform above 2000 now in such matches, and I do plan to run such a match or matches fairly soon, maybe in late April or early May.

[Laskos]

We already showed more than a year ago that at blitz (3' +2") Komodo giving knight odds was pretty even with GMs Lenderman and Mikhalevsky. But in "slow" (45' + 15") chess Komodo performed below 2000 FIDE in a dozen games. I do think that Komodo will perform above 2000 now in such matches, and I do plan to run such a match or matches fairly soon, maybe in late April or early May.

The progress from Komodo 9.2 which played IM Bolzoni at Knight Odds (and lost) and Komodo 11.01 seems meager. If progress in regular chess is maybe 120 ELO points, in Knight Odds it seems no larger than 30 ELO points. Fruit 2.1 seems a suitable adversary.

At 10s + 0.1s the comparison is the following:

Code: Select all

    Program                            Score     %     Elo    +   -   Draws

  1 Komodo 11.01 (Contempt=100)    : 272.0/400  68.0    73   34  33   15.0 %
  2 Komodo 9.2   (Contempt=100)    : 255.5/400  63.9    41   32  32   18.8 %

  3 Fruit 2.1                      : 272.5/800  34.1   -57   23  23   16.9 %

At 60s + 0.6s there seems to be no progress at all:

Code: Select all

1 Komodo 9.2   (Contempt=100)       60/100:     + 52 = 16 - 32
2 Komodo 11.01 (Contempt=100)       59/100:     + 48 = 22 - 30

3 Fruit 2.1                         81/200:     + 62 = 38 - 100

Playing strength wise against humans, which compress differences, there seem to be no any change at all, and Komodo is not prepared to play Knight Odds match against someone like IM Bolzoni. The only variable here is the opening book, which can greatly influence the outcome.

Did I set the Contempt to correct values?

[lkaufman]

We already showed more than a year ago that at blitz (3' +2") Komodo giving knight odds was pretty even with GMs Lenderman and Mikhalevsky. But in "slow" (45' + 15") chess Komodo performed below 2000 FIDE in a dozen games. I do think that Komodo will perform above 2000 now in such matches, and I do plan to run such a match or matches fairly soon, maybe in late April or early May.

The progress from Komodo 9.2 which played IM Bolzoni at Knight Odds (and lost) and Komodo 11.01 seems meager. If progress in regular chess is maybe 120 ELO points, in Knight Odds it seems no larger than 30 ELO points. Fruit 2.1 seems a suitable adversary.

At 10s + 0.1s the comparison is the following:

Code: Select all

    Program                            Score     %     Elo    +   -   Draws

  1 Komodo 11.01 (Contempt=100)    : 272.0/400  68.0    73   34  33   15.0 %
  2 Komodo 9.2   (Contempt=100)    : 255.5/400  63.9    41   32  32   18.8 %

  3 Fruit 2.1                      : 272.5/800  34.1   -57   23  23   16.9 %

At 60s + 0.6s there seems to be no progress at all:

Code: Select all

1 Komodo 9.2   (Contempt=100)       60/100:     + 52 = 16 - 32
2 Komodo 11.01 (Contempt=100)       59/100:     + 48 = 22 - 30

3 Fruit 2.1                         81/200:     + 62 = 38 - 100

Playing strength wise against humans, which compress differences, there seem to be no any change at all, and Komodo is not prepared to play Knight Odds match against someone like IM Bolzoni. The only variable here is the opening book, which can greatly influence the outcome.

Did I set the Contempt to correct values?

It does appear that our progress in normal chess does not translate directly to knight odds chess. If we combine your two small samples, the net gain is about 22 elo. I would have set Comtempt a bit higher, maybe 120, but that's not a significant difference. More likely, we made some significant changes to Contempt in between those versions, which might make the ideal setting different between them, but that's not easy to determine. Also some eval changes may be counterproductive for high handicap games.
But anyway, I agree with your conclusion that Komodo is not yet strong enough to give knight odds to players in the 2200-2300 USCF (about 2100 to 2200 FIDE) range at 45' + 15" unless it also gets draw odds in each game. So instead, I think we'll try knight odds at Chess960 against such players at that time control, perhaps next week. Chess960 will be harder for the human players due to inexperience with the starting positions, which might be enough of a factor to raise Komodo's expected result from 30% or so to perhaps even chances. But this is only a guess, we have zero data on Komodo vs humans at chess960. Time to remedy this!

[JJJ]

Can't wait for it !