An Unexpected Rybka vs. Robbo result

[lkaufman]

We are currently running a private multi-round robin tournament involving three versions of Komodo, Stockfish, Rybka 3 and Robbolito, all on one core. The time limit is very fast, and Rybka and Robbo are set to half the time of the other programs to make them at least somewhat competitive. Roughly speaking, the time limit is about game in ten seconds plus 1/6 of a second per move for Komodo and Stockfish, half that for the two Rs. Without getting into how Komodo versions and Stockfish are doing, the interesting thing is that Rybka is currently 30 Elo points ahead of Robbo after around a thousand games per program!
This is quite surprising in view of the universal finding that Robbo is stronger than Rybka at blitz and bullet speeds, at least when directly matched. There are several possible explanations, or any combination of them:
1. Rybka is really stronger against unrelated programs, just weaker than Robbo on a direct matchup. However at last check, Rybka was leading even in the direct matchup here so this seems not to be the main explanation.
2. Rybka is relatively better than Robbo against Komodo. Perhaps true, but again it does not explain point 1 above.
3. This just happens to be a level that favors Rybka for some reason.
4. Rybka is stronger at very fast speeds, Robbo passes it at game/1 minute or less, and then the curve turns again at some higher level.
5. Other ideas???

So this throws into doubt the claim that Robbo is just better than Rybka, it seems to depend on the level and the opponent.

If anyone out there has an interface that allows for game in less than one minute, I would welcome Robbo vs. Rybka data at half a minute or less per game, or results for both vs. Stockfish 1.6 at such speeds.

[Michael Sherwin]

5a. Time management?

[lkaufman]

Time management is possible, but programs normally do everything proportionally, so the result should be similar to five minutes plus five seconds or one hour plus one minute in that respect.

[Dann Corbit]

Time management is possible, but programs normally do everything proportionally, so the result should be similar to five minutes plus five seconds or one hour plus one minute in that respect.

Ivanhoe gets almost all of his losses at very fast time control as losses on time. It has horrible, primitive time control for very short matches.

[lkaufman]

That's interesting, but I'm testing Robbolito, not Ivanhoe. Is there any reason to believe this would be happening to Robbolito?

[Dann Corbit]

That's interesting, but I'm testing Robbolito, not Ivanhoe. Is there any reason to believe this would be happening to Robbolito?

Sure. Ivanhoe is just the next iteration of Robbolito.
Do a web search for these keywords:
ippolit igorrit ivanhoe robbolito
and you will find source code for any/all of them as well as the explanations of the lineage from engine to engine.

[lkaufman]

I checked, and it turns out that our tester does not enforce time forfeits; it "trusts" that programs won't time forfeit. So this is not the problem. However I suppose the problem could be that Robbo simply doesn't handle small fractions of a second well for some reason. Since we are playing with increment, even though a tiny one, I would think that even playing on pure increment, with the increment close to a tenth of a second, Robbo would use this time roughly as well as it would use a one second or a five second increment, but maybe for some strange reason this is not the case. Can you tell whether there is any reason that Robbo would play much too fast with a tenth of a second but not with a full second, for example?

[Dann Corbit]

I checked, and it turns out that our tester does not enforce time forfeits; it "trusts" that programs won't time forfeit. So this is not the problem. However I suppose the problem could be that Robbo simply doesn't handle small fractions of a second well for some reason. Since we are playing with increment, even though a tiny one, I would think that even playing on pure increment, with the increment close to a tenth of a second, Robbo would use this time roughly as well as it would use a one second or a five second increment, but maybe for some strange reason this is not the case. Can you tell whether there is any reason that Robbo would play much too fast with a tenth of a second but not with a full second, for example?

Robbolito probably does not check time control often enough (so that it wastes a lot of time when it should hurry and then suddenly realizes that it has no time left at all).

This is the time allocation routine:

Code: Select all

void TimeManager (sint64 TIME, sint64 INCREMENT, int mtg)
{  
  if (mtg)
    {
      if (mtg > 25)
	mtg = 25;
      DESIRED_TIME = TIME / mtg + INCREMENT;
      ABSOLUTE_TIME =
	(TIME * mtg) / (4 * mtg - 3) - MIN (1000000, TIME / 10);
      if (mtg == 1)
	ABSOLUTE_TIME -= MIN (1000000, ABSOLUTE_TIME / 10);
      if &#40;ABSOLUTE_TIME < 1000&#41; /* porque? */
	ABSOLUTE_TIME = 1000;
    &#125;
  else
    &#123;
      ABSOLUTE_TIME = TIME / 4 - 10000;
      if &#40;ABSOLUTE_TIME < 1000&#41;
	ABSOLUTE_TIME = 1000;
      DESIRED_TIME = ABSOLUTE_TIME / 11 + INCREMENT;
    &#125;
  if &#40;DESIRED_TIME > ABSOLUTE_TIME&#41;
    DESIRED_TIME = ABSOLUTE_TIME;
  if &#40;DESIRED_TIME < 1000&#41;
    DESIRED_TIME = 1000;
  EASY_TIME = DESIRED_TIME / 4;
  BATTLE_TIME = DESIRED_TIME;
  ORDINARY_TIME = &#40;3 * DESIRED_TIME&#41; / 4;
&#125;

At first, I thought it was setting the time at too high of a lower limit. But they scale the time into millionths of a second, so setting a minimum of 1000 is really 1/1000 second, so that is not the problem. I guess that the problem is that they do not check to see if they are running out of time often enough.

If they called the above routine every time it was their turn to move and if they called the "am I out of time?" routine at every succeeding ply, I do not think that they would have severe time troubles.

As it is, however, I see time losses at fast time control in *most* games if the opponent handles time control in a sensible manner (often when ivanhoe is about to win according to the score of both engines).

[Dann Corbit]

I checked, and it turns out that our tester does not enforce time forfeits; it "trusts" that programs won't time forfeit. So this is not the problem. However I suppose the problem could be that Robbo simply doesn't handle small fractions of a second well for some reason. Since we are playing with increment, even though a tiny one, I would think that even playing on pure increment, with the increment close to a tenth of a second, Robbo would use this time roughly as well as it would use a one second or a five second increment, but maybe for some strange reason this is not the case. Can you tell whether there is any reason that Robbo would play much too fast with a tenth of a second but not with a full second, for example?

Robbolito probably does not check time control often enough (so that it wastes a lot of time when it should hurry and then suddenly realizes that it has no time left at all).

This is the time allocation routine:

Code: Select all

void TimeManager (sint64 TIME, sint64 INCREMENT, int mtg)
{  
  if (mtg)
    {
      if (mtg > 25)
	mtg = 25;
      DESIRED_TIME = TIME / mtg + INCREMENT;
      ABSOLUTE_TIME =
	(TIME * mtg) / (4 * mtg - 3) - MIN (1000000, TIME / 10);
      if (mtg == 1)
	ABSOLUTE_TIME -= MIN (1000000, ABSOLUTE_TIME / 10);
      if &#40;ABSOLUTE_TIME < 1000&#41; /* porque? */
	ABSOLUTE_TIME = 1000;
    &#125;
  else
    &#123;
      ABSOLUTE_TIME = TIME / 4 - 10000;
      if &#40;ABSOLUTE_TIME < 1000&#41;
	ABSOLUTE_TIME = 1000;
      DESIRED_TIME = ABSOLUTE_TIME / 11 + INCREMENT;
    &#125;
  if &#40;DESIRED_TIME > ABSOLUTE_TIME&#41;
    DESIRED_TIME = ABSOLUTE_TIME;
  if &#40;DESIRED_TIME < 1000&#41;
    DESIRED_TIME = 1000;
  EASY_TIME = DESIRED_TIME / 4;
  BATTLE_TIME = DESIRED_TIME;
  ORDINARY_TIME = &#40;3 * DESIRED_TIME&#41; / 4;
&#125;

At first, I thought it was setting the time at too high of a lower limit. But they scale the time into millionths of a second, so setting a minimum of 1000 is really 1/1000 second, so that is not the problem. I guess that the problem is that they do not check to see if they are running out of time often enough.

If they called the above routine every time it was their turn to move and if they called the "am I out of time?" routine at every succeeding ply, I do not think that they would have severe time troubles.

As it is, however, I see time losses at fast time control in *most* games if the opponent handles time control in a sensible manner (often when ivanhoe is about to win according to the score of both engines).

P.S.
in the above code "mtg" is "Moves To Go"

[Dann Corbit]

I checked, and it turns out that our tester does not enforce time forfeits; it "trusts" that programs won't time forfeit. So this is not the problem. However I suppose the problem could be that Robbo simply doesn't handle small fractions of a second well for some reason. Since we are playing with increment, even though a tiny one, I would think that even playing on pure increment, with the increment close to a tenth of a second, Robbo would use this time roughly as well as it would use a one second or a five second increment, but maybe for some strange reason this is not the case. Can you tell whether there is any reason that Robbo would play much too fast with a tenth of a second but not with a full second, for example?

Robbolito probably does not check time control often enough (so that it wastes a lot of time when it should hurry and then suddenly realizes that it has no time left at all).

This is the time allocation routine:

Code: Select all

void TimeManager (sint64 TIME, sint64 INCREMENT, int mtg)
{  
  if (mtg)
    {
      if (mtg > 25)
	mtg = 25;
      DESIRED_TIME = TIME / mtg + INCREMENT;
      ABSOLUTE_TIME =
	(TIME * mtg) / (4 * mtg - 3) - MIN (1000000, TIME / 10);
      if (mtg == 1)
	ABSOLUTE_TIME -= MIN (1000000, ABSOLUTE_TIME / 10);
      if &#40;ABSOLUTE_TIME < 1000&#41; /* porque? */
	ABSOLUTE_TIME = 1000;
    &#125;
  else
    &#123;
      ABSOLUTE_TIME = TIME / 4 - 10000;
      if &#40;ABSOLUTE_TIME < 1000&#41;
	ABSOLUTE_TIME = 1000;
      DESIRED_TIME = ABSOLUTE_TIME / 11 + INCREMENT;
    &#125;
  if &#40;DESIRED_TIME > ABSOLUTE_TIME&#41;
    DESIRED_TIME = ABSOLUTE_TIME;
  if &#40;DESIRED_TIME < 1000&#41;
    DESIRED_TIME = 1000;
  EASY_TIME = DESIRED_TIME / 4;
  BATTLE_TIME = DESIRED_TIME;
  ORDINARY_TIME = &#40;3 * DESIRED_TIME&#41; / 4;
&#125;

At first, I thought it was setting the time at too high of a lower limit. But they scale the time into millionths of a second, so setting a minimum of 1000 is really 1/1000 second, so that is not the problem. I guess that the problem is that they do not check to see if they are running out of time often enough.

If they called the above routine every time it was their turn to move and if they called the "am I out of time?" routine at every succeeding ply, I do not think that they would have severe time troubles.

As it is, however, I see time losses at fast time control in *most* games if the opponent handles time control in a sensible manner (often when ivanhoe is about to win according to the score of both engines).

P.S.
in the above code "mtg" is "Moves To Go"

Another possible time control problem source is the copious output. Even when the program is running out of time, it emits a huge volume of analysis data for display (e.g. 30 lines of full width analysis when there are milliseconds per move left in the time control).