Martin on the SF loss on time

[Roger Brown]

TCEC now seems to be a commercial enterprise, so it was no real surprise that they'd find a way to ensure that there would be the expected Stockfish v Komodo final.

Hello Graham,

This seems to imply that commercial events are rigged?

It would seem to me that the two strongest engines reaching a chess engine tournament final would be an expected outcome?

Unless I am missing something here?

Later.

[Graham Banks]

TCEC now seems to be a commercial enterprise, so it was no real surprise that they'd find a way to ensure that there would be the expected Stockfish v Komodo final.

Hello Graham,

This seems to imply that commercial events are rigged?

It would seem to me that the two strongest engines reaching a chess engine tournament final would be an expected outcome?

Unless I am missing something here?

Later.

Hi Roger,

I wouldn't say it is rigged, but allowing a solution to the Stockfish problem in the manner decided effectively ensures a Stockfish v Komodo final, which is what most wanted to see.

Martin was in a no-win situation and I'm not criticising him for the tough decision he had to make. He was going to be damned if he did, damned if he didn't.

However, the decision wasn't entirely his and one can't help but wonder if the commercial aspect of the event and pandering to spectator wants played a part in the decision reached.
(Did you see it is no longer the Thoresen Chess Engines Competition, but the Top Chess Engines Championship?)

Anyway, I encouraged people to accept whatever decision was reached. Martin doesn't deserve criticism, but gratitude for his TCEC involvement.

Graham.

[Uri Blass]

TCEC now seems to be a commercial enterprise, so it was no real surprise that they'd find a way to ensure that there would be the expected Stockfish v Komodo final.

If they want to ensure Stockfish v Komodo final they could decide to replay the games that stockfish lost.
It is not something sure that Stockfish is going to be in the final.

Gull3 is weaker than Stockfish but has at least small chances to promote to the final(it is now leading the tournament with 5.5 out of 8 when stockfish has only 4.5 out of 8).
All the rules that you can fix a problem of crashing and you cannot fix a problem of losing on time are not logical.

I believe that the opinion of most people(and I am not talking about stockfish fans) is that every program that lose on time because of a bug should be allowed to fix the bug immediately(at least if fixing the problem seems to be only changing a time management parameter in the original exe and not sending a new version).

In the case of stockfish it did not happen
They decided to downgrade stockfish instead of fixing the problem by changing a time management parameter(that was better) and one loss against gull was not enough to allow stockfish to downgrade the version
and even after a second loss fixing the problem was only after asking the authors of the other programs if they agree.

They could easily decide that losing on time is a serious limiting play bug and allowing fixing the problem immediately.

My opinion is that stockfish got worse treatement than the treatement that most people could give it instead of TCEC.

I do not remember losing on time earlier by a program when the bug was something that prevented the program to promote or could prevent the program to promote.
Nobody cared about the loss of ginkgo against protector in the last round of stage 1 on time because it did not prevent ginkgo to promote and in any case fixing the problem after the last round was not possible.
Do people remember more cases of losing on time(and not crashing) that could be relevant to prevent a program to promote?

[Norm Pollock]

Any forfeit loss, by time or whatever, may or may not harm the losing engine in terms of promotion. But also a forfeit may help the "winning" engine move up the table. It could possibly promote ahead of a third engine that did not get the benefit of winning by forfeit. Forfeits can set off a domino effect.

[hgm]

Indeed. Having a regularly forfeiting engine in a tournament, especially one that plays very strong Chess, makes the tournament a kind of lottery for the other participants. So it is very disrupting to keep such a forfeiting engine, and one would assume that this is the rationale behind the 'play-limiting-bug' rule. To keep it a meaningful competition to the others, you either should forfeit the defective engine from the tourney, and declare all its games played so far invalid, or allow a replacement that doesn't have the problem.

[bob]

And not setting the lag parameter to a safer value was indeed a mistake, but not one that Martin made.

Sure, it was a mistake of the Stockfish team. But it doesn't seem one worth spilling blood over, and adjusting it to a more realistic value seems a lot less drastic action then changing to a completely different version.

I agree that it would have been better not to make an exception just because it's SF. Either consider this a play limiting bug and let the SF team provide a fix as per the rules (i.e. increase the lag parameter), or let SF continue with the current version and parameter settings until the end of the stage.

Actually, I wonder why reverting to the previous version should eliminate the problem if the lag parameter is not increased at the same time. Maybe there is a technical reason for this, but the mere fact that this problem occurred only in the current stage and twice at that could just be a matter of good luck in the previous stages and of bad luck in the current stage.

I don't know how much testing time they got on the actual TCEC hardware before the stage started, but if it wasn't very much it seems a bit harsh to say "you overestimated the speed of my machine a little, and I am going to make you pay deerly for that".

It seems more a case of underestimating the time lost in stopping threads (and/or suboptimally handling this). At any rate it should have been clear that 10ms is nothing on an OS that still counts time in 1/60ths of a second. (But I would not rely on such a tiny safety margin on Linux either.)

You did understand that they changed versions prior to this stage? Switching to the new lazy-amp version which apparently exhibits this bug while old versions with normal YBW did not.

[syzygy]

You did understand that they changed versions prior to this stage? Switching to the new lazy-amp version which apparently exhibits this bug while old versions with normal YBW did not.

Yes I know, but as far as I understand the lag parameter has always been set to 10ms and time management has not really been changed.

Apparently SF loses a bit of time upon finishing a search when it waits for all threads to stop. I would think the YBWC version also stops threads before sending the best move (or it would be cheating) and loses some time on that.

Of course it might simply take the lazy-smp version a bit longer to stop the threads than the previous version, and that difference might be just enough to lose on time under the wrong circumstances...

[bob]

Indeed. Having a regularly forfeiting engine in a tournament, especially one that plays very strong Chess, makes the tournament a kind of lottery for the other participants. So it is very disrupting to keep such a forfeiting engine, and one would assume that this is the rationale behind the 'play-limiting-bug' rule. To keep it a meaningful competition to the others, you either should forfeit the defective engine from the tourney, and declare all its games played so far invalid, or allow a replacement that doesn't have the problem.

The risk I don't like is that not only does the bogus version hurt its own chances for advancement, it can hurt others as well if it is changed mid-stage, since some got to play a broken version and some will have to play a better version. This could not only cause SF to not make the next stage, but it could also change which other programs make the next stage as well, which is problematic...

[hgm]

True, butthis is why they allow the replacement only after a full cycle. Then everyone has had his chance for a free point. They should have done the same thing for a parameter change.

[lucasart]

I think SF should go with the master branch, for the final. Lazy SMP is stronger than master, but in 3h games the difference should be small:
* We do not have statistically reliable data to know the elo gain at 3h games, and we never will (because statistically reliable means tens of thousand of games, which we cannot do at this tc).
* In the absence of such data, all we have is the Occam razor: elo differences generally shrink massively between 60" games and TCEC, so there is no reason to believe that it wouldnt apply to lazy SMP. We can speculate all we want, we dont know, and never will.

To sum up, we have for the TCEC final:
* a small elo gain (probably)
* an unknown stability risk, which could cause SF to lose the final horribly, with lots of time losses.
* we gambled once, it wasnt worth it. let's not make that mistake again.

Instead, we should focus on improving and stabilizing lazy SMP, far away from the buzz and pressure of TCEC, Talkchess, SF fans… Improvements and bug fixes are happening as we speak. It's not over yet.

And when we are comfortable with it, release SF7 with lazy SMP. Then many people, especially rating lists will test it, stretching it to the limits of stability and perhaps helping uncover bugs that we missed.

Anyway, that's just my opinion. Not an official SF statement.

Patience is a virtue.