World Computer Chess Championship ?

[Laskos]

As to the "statute of limitations"... what is a reasonable period of time? 3 months? 3 years? This is a pretty serious issue, and serious crimes have a very long (or none at all) statute of limitations...

Not sure, maybe 3 months, or perhaps just until the event is over. If the protest is not registered before the event, then the protest should be due to behavior during the event, and in my mind such things should be obvious enough that it does not take years to uncover. Also, I would only want other participants to be protesting, not a situation where anyone in the world is invited to take pot shots.

To me personally, this is not very comparable to serious crimes, and should not be treated as such. Its more like a sport, and while some sports have obviously contemplated some odd retroactive anything goes type of penalties for things they don't like, most don't. If you win, you win. Protests can happen during the event by competing teams, you can have testing before, during, and immediately after events, but when the whistle blows and the testing is done its over and the winner is crowned.

-Sam

Counter examples:

(1) Ben Johnson. Gold medal revoked well after Olympics ended.

(2) NCAA football championships revoked 5 years after it was won.

There are others. This is not that uncommon, as sometimes it takes a good bit of time to investigate, sometimes evidence surfaces well after the event (strelka to start the Fruit/Rybka investigation, etc).

If taking these Ben Johnson analogies, the CVSN (sim) method would weed out the derivatives much more efficiently than the doping tests in most of the sports. I don't know what you are blabbering about the unreliability of the sim or ponder-hit methods, it is by far more efficient than anything you propose. Do you have in mind a proven false positive on sim or ponder-hits? I think it's easier to have a false negative than a false positive on sim or ponder hits.

Kai

[Eelco de Groot]

So apparently the true meaning of rule #2 is that there is an (undefined) limit on the number of ideas you are allowed to take from open-sources. It's not about copying any longer (it always was) but the volume you take of common (non-copyrightable) chess knowledge idea's found in every decent chess program is suddenly a major issue.

I don't understand where you get this dillusion from. It is almost like you are living in some parallel univers, where there is another ICGA, and another rule #2. Because in the universe where the rest of us lives ICGA rule #2 is crystal clear. You cannot take anything. What don't you understand about 'not anything' that makes you equate it to 'an undefined limit'? The only thing that is 'apparent' is that you live in a fantasy.

Don, HGM, do you read?

Yes, I read. So Bob cannot judge a code he has not looked at. Big surprise. So what is your point exactly?

Perhaps that newcomer 'programmers' who have never seen their 'own' code cannot judge if it satisfies rule #2?

Hello, I'm not following this discussion at all, I am not planning on entering now, but maybe it can be considered a bit more that the rules for the WCCC are, loosely but primarily, based on what is important in academia. Programmers are expected to care about the advancement of science, in rule No. 2 for instance, which naturally means they are expected to focus on developing -and sharing- new ideas, and not so much on integrating all that is already known, into the strongest possible program. That is something very different and which is what I would expect commercial programmers to do these days, and at which Vas Rajlich excelled. Robert Houdart is following exactly this line. Any new ideas he has he will not divulge unless he has too: the moment a new version of his program becomes available, all his new ideas are immediately obsolete and common knowledge. For Rybka this is no different, the cluster version excluded.

The problem for the ICGA is that they never really thought much about the fact that the commercial entrants couldn't care less about academic practice. I am sure Jaap and David realized it but including commercial entries made the tournament more interesting for sponsors, for the media and there was at the time money that could be asked from the commercials which paid for much of the travel expenses for others, the salary of Jaap's secretary etc. But these commercial programmers don't give a hoot about academic practice and never have. Why should they.

I am not saying any of them outright copied other programmer's code. I am sure the commercial chessprogrammers all did their utmost to develop their own ideas, and at the same time incorporate the best of what was in the common knowledge base. That was their job. Vas took it a level too far perhaps but in principle I think he did not do anything which is not done in commercial software development, the world over when he developed Rybka. Which is not the same as what the ICGA is promoting, which at the basis is science. That is something different. Maybe it is time that the ICGA reforms itself to more of a sports association, which it never was. And in my opinion it is about time to consider that retracting Vas' WCCC titles retroactively may not have been entirely fair considering that nobody expects commercial software developers to be scientists, and the ICGA could have done more, and better, in controlling participating programs.

Eelco

[bob]

As to the "statute of limitations"... what is a reasonable period of time? 3 months? 3 years? This is a pretty serious issue, and serious crimes have a very long (or none at all) statute of limitations...

Not sure, maybe 3 months, or perhaps just until the event is over. If the protest is not registered before the event, then the protest should be due to behavior during the event, and in my mind such things should be obvious enough that it does not take years to uncover. Also, I would only want other participants to be protesting, not a situation where anyone in the world is invited to take pot shots.

To me personally, this is not very comparable to serious crimes, and should not be treated as such. Its more like a sport, and while some sports have obviously contemplated some odd retroactive anything goes type of penalties for things they don't like, most don't. If you win, you win. Protests can happen during the event by competing teams, you can have testing before, during, and immediately after events, but when the whistle blows and the testing is done its over and the winner is crowned.

-Sam

Counter examples:

(1) Ben Johnson. Gold medal revoked well after Olympics ended.

(2) NCAA football championships revoked 5 years after it was won.

There are others. This is not that uncommon, as sometimes it takes a good bit of time to investigate, sometimes evidence surfaces well after the event (strelka to start the Fruit/Rybka investigation, etc).

If taking these Ben Johnson analogies, the CVSN (sim) method would weed out the derivatives much more efficiently than the doping tests in most of the sports. I don't know what you are blabbering about the unreliability of the sim or ponder-hit methods, it is by far more efficient than anything you propose. Do you have in mind a proven false positive on sim or ponder-hits? I think it's easier to have a false negative than a false positive on sim or ponder hits.

Kai

Both Don AND Adam have clearly stated that the similarity testing is a good filter. If a program passes that, then consider it most likely to be OK. That's certainly not perfectly accurate, but a good first approximation. If a program fails the similarity test, it still has not proven to be a clone, a derivative, or anything else. It is just more suspicious than the others that were tested, and it needs actual verification with traditional approaches including source comparison and such...

I don't think there is any particular evidence that suggests that more or less false positives than false negatives will occur. It is certainly obvious that some of each will happen, which is why the source comparison is still necessary.

[Laskos]

As to the "statute of limitations"... what is a reasonable period of time? 3 months? 3 years? This is a pretty serious issue, and serious crimes have a very long (or none at all) statute of limitations...

Not sure, maybe 3 months, or perhaps just until the event is over. If the protest is not registered before the event, then the protest should be due to behavior during the event, and in my mind such things should be obvious enough that it does not take years to uncover. Also, I would only want other participants to be protesting, not a situation where anyone in the world is invited to take pot shots.

To me personally, this is not very comparable to serious crimes, and should not be treated as such. Its more like a sport, and while some sports have obviously contemplated some odd retroactive anything goes type of penalties for things they don't like, most don't. If you win, you win. Protests can happen during the event by competing teams, you can have testing before, during, and immediately after events, but when the whistle blows and the testing is done its over and the winner is crowned.

-Sam

Counter examples:

(1) Ben Johnson. Gold medal revoked well after Olympics ended.

(2) NCAA football championships revoked 5 years after it was won.

There are others. This is not that uncommon, as sometimes it takes a good bit of time to investigate, sometimes evidence surfaces well after the event (strelka to start the Fruit/Rybka investigation, etc).

If taking these Ben Johnson analogies, the CVSN (sim) method would weed out the derivatives much more efficiently than the doping tests in most of the sports. I don't know what you are blabbering about the unreliability of the sim or ponder-hit methods, it is by far more efficient than anything you propose. Do you have in mind a proven false positive on sim or ponder-hits? I think it's easier to have a false negative than a false positive on sim or ponder hits.

Kai

Both Don AND Adam have clearly stated that the similarity testing is a good filter. If a program passes that, then consider it most likely to be OK.

I don't know what Adam or Don say, but this filter analogy is wrong, it's easier to find a false negative (for example copying from multiple sources) than to find a false positive. Again, do you have in mind a proven false positive on sim or ponder-hits methods? I could post a dendrogram after a dendrogram with dozens of cases with 0 shown false positives. About false negatives it's harder to be sure, but their number is not high too, the percentages and clustering is about what one expected (Hiarcses will cluster with Hiarcses and Shredders with Shredders, as one is expecting).

A much more reliable method thant the doping tests in sports, if this thread is full of sports analogies.

Kai

That's certainly not perfectly accurate, but a good first approximation. If a program fails the similarity test, it still has not proven to be a clone, a derivative, or anything else. It is just more suspicious than the others that were tested, and it needs actual verification with traditional approaches including source comparison and such...

I don't think there is any particular evidence that suggests that more or less false positives than false negatives will occur. It is certainly obvious that some of each will happen, which is why the source comparison is still necessary.

[syzygy]

Counter examples:

(1) Ben Johnson. Gold medal revoked well after Olympics ended.

Not a very good example. He was caught and disqualified during the 1988 Olympics.

His 1987 world title was revoked in 1988, though.

A quick check shows that athletics, cycling and swimming all have an 8-year statute of limitations for doping offences.

If you really want to take this analogy further, I note that doping offences are only punished when the substance in question was on the list of banned substances at the time of the event. It is crystal clear which substances are allowed and which are not.

[mcostalba]

and to treat computer chess competitions as friendly competition with clear, simple, well established rules (which in my mind does not include never-ending periods of investigation and massively retroactive penalties).

You cannot underestimate the fact that commercials do not care less of "friendly competition" they have to live out of their engines and their partecipation to this tournament is fully marketing driven. So I can unerstand why they dont' want some outsider to rain at their party: because business it's business, they invest money in this promotional activity and they want a clear return. I understand less ICGA's people that accept (more or less consciously) to act like their puppets (perhaps someone even for free )

[Rebel]

If taking these Ben Johnson analogies, the CVSN (sim) method would weed out the derivatives much more efficiently than the doping tests in most of the sports. I don't know what you are blabbering about the unreliability of the sim or ponder-hit methods, it is by far more efficient than anything you propose. Do you have in mind a proven false positive on sim or ponder-hits? I think it's easier to have a false negative than a false positive on sim or ponder hits.

And surprisingly both systems (sim & ponder hit) complement each, give the same usual suspects.

SIM - http://www.top-5000.nl/quick_guide.htm#C2

PONDER HIT - on the same page

[bob]

As to the "statute of limitations"... what is a reasonable period of time? 3 months? 3 years? This is a pretty serious issue, and serious crimes have a very long (or none at all) statute of limitations...

Not sure, maybe 3 months, or perhaps just until the event is over. If the protest is not registered before the event, then the protest should be due to behavior during the event, and in my mind such things should be obvious enough that it does not take years to uncover. Also, I would only want other participants to be protesting, not a situation where anyone in the world is invited to take pot shots.

To me personally, this is not very comparable to serious crimes, and should not be treated as such. Its more like a sport, and while some sports have obviously contemplated some odd retroactive anything goes type of penalties for things they don't like, most don't. If you win, you win. Protests can happen during the event by competing teams, you can have testing before, during, and immediately after events, but when the whistle blows and the testing is done its over and the winner is crowned.

-Sam

Counter examples:

(1) Ben Johnson. Gold medal revoked well after Olympics ended.

(2) NCAA football championships revoked 5 years after it was won.

There are others. This is not that uncommon, as sometimes it takes a good bit of time to investigate, sometimes evidence surfaces well after the event (strelka to start the Fruit/Rybka investigation, etc).

If taking these Ben Johnson analogies, the CVSN (sim) method would weed out the derivatives much more efficiently than the doping tests in most of the sports. I don't know what you are blabbering about the unreliability of the sim or ponder-hit methods, it is by far more efficient than anything you propose. Do you have in mind a proven false positive on sim or ponder-hits? I think it's easier to have a false negative than a false positive on sim or ponder hits.

Kai

Both Don AND Adam have clearly stated that the similarity testing is a good filter. If a program passes that, then consider it most likely to be OK.

I don't know what Adam or Don say, but this filter analogy is wrong, it's easier to find a false negative (for example copying from multiple sources) than to find a false positive. Again, do you have in mind a proven false positive on sim or ponder-hits methods? I could post a dendrogram after a dendrogram with dozens of cases with 0 shown false positives. About false negatives it's harder to be sure, but their number is not high too, the percentages and clustering is about what one expected (Hiarcses will cluster with Hiarcses and Shredders with Shredders, as one is expecting).

A much more reliable method thant the doping tests in sports, if this thread is full of sports analogies.

Kai

That's certainly not perfectly accurate, but a good first approximation. If a program fails the similarity test, it still has not proven to be a clone, a derivative, or anything else. It is just more suspicious than the others that were tested, and it needs actual verification with traditional approaches including source comparison and such...

I don't think there is any particular evidence that suggests that more or less false positives than false negatives will occur. It is certainly obvious that some of each will happen, which is why the source comparison is still necessary.

So you believe there is some "magic match percentage" (such as the one chosen by CSVN) that is a safe number. Anything above that is simply a clone with no investigation needed, anything below that is not?

(Hint: CSVN's number doesn't look particularly "safe" to me)...

[bob]

If taking these Ben Johnson analogies, the CVSN (sim) method would weed out the derivatives much more efficiently than the doping tests in most of the sports. I don't know what you are blabbering about the unreliability of the sim or ponder-hit methods, it is by far more efficient than anything you propose. Do you have in mind a proven false positive on sim or ponder-hits? I think it's easier to have a false negative than a false positive on sim or ponder hits.

And surprisingly both systems (sim & ponder hit) complement each, give the same usual suspects.

SIM - http://www.top-5000.nl/quick_guide.htm#C2

PONDER HIT - on the same page

Which proves what? Back in the 90's different people were trying to produce test suites that could be used to estimate the Elo of a chess engine. The idea was to run a bunch of positions, obtain time-to-solutions, and then FIT that data to the known Elo values for each program. And not surprisingly, it did not work. You can ask Don about the result when this was applied to Cray Blitz, which reported 0 seconds for each and every solution, giving a rating that was off the charts, and wildly wrong... Once there is a clear metric available, humans are very clever creatures and they will develop counter-measures to blow off such a cursory examination. Comparing source is pretty much fool-proof. Everything is an approximation.

[bob]

Counter examples:

(1) Ben Johnson. Gold medal revoked well after Olympics ended.

Not a very good example. He was caught and disqualified during the 1988 Olympics.

His 1987 world title was revoked in 1988, though.

A quick check shows that athletics, cycling and swimming all have an 8-year statute of limitations for doping offences.

If you really want to take this analogy further, I note that doping offences are only punished when the substance in question was on the list of banned substances at the time of the event. It is crystal clear which substances are allowed and which are not.

8 years covers the Rybka case. I'd think that even 5 years is enough.