ICGA Rule 2 discussion

[bob]

It sounds like you advocate "all derivatives are OK except if they come from IPPOLIT?"

To date, ippolit has not been proven to be a rybka derivative, so what makes it special here? My issue is that I believe ippolit is not original, based on looking at the source code which appears to come from hex rays or something similar. And we have no author to question.

But is it really ok to have fruit derivatives, crafty derivatives, stockfish derivatives, any other open-source derivatives?

No of course not. If a program is derived from another program and it violate's its license, it should not be allowed. Thus, in the case of open-source derivative programs, if no license violation has occured, and the program passes the similarity test, it should be allowed to participate.

Note: I only mentioned IPPOLIT as special case because it is "public domain" but is unoriginal and illegal derivative of rybka (unproven, but highly suspicious). Again, I must emphasize that we can debate about IPPOLIT all day and never come to any scientific conclusion, so lets not get idealistic. It is commonly accepted in the computer chess community that IPPOLIT is an illegal derivative, so lets not waste time about it and just move on. As long as it passes the similarity test, an IPPOLIT derivate should be fine. If not, it should not be allowed.

What about question #2. Ok to have 30 stockfish derivatives? Again, that is not something I want to do. I could run that on my cluster, for example.

30 stockfish derivatives? Again, if they pass the simtest, then they are through. I highly doubt though that they would.

So you want to use the simtest as THE test? Pass it, play, fail it, don't play? This does not do a thing for "originality".

My problem is that everybody has been arguing about how best to preserve originality, but at the same time, refusing to use the best (and only) tool we have so far in achieving this -- simtest -- and everybody keeps coming up with new arguments and new excuses everyday against it, while simultaneously not providing any alternative solution. This is ridiculous and with this attitude, WCCC will is just a pipe dream and will never regain its glory.

What people need to do right now is stop being idealistic and just go with what best we have right now. In future we can improve upon it if we can. What is important is that we get the ball rolling.

Also no one has commented on my proposal to go online-only. I think this is the only way forward.

Online-only has negative consequences. One benefit of the ICGA is face-to-face meetings, discussions, presentations, etc. Online-only doesn't offer that opportunity.

As far as "do something soon" I disagree. There is no point in doing something that starts off with flaws. The CURRENT ICGA rules have been used successfully for a long time. There's no need to rush out with something poorly thought out and do nothing more than make matters even worse.

[Dirt]

2e. Exceptions. Certain pieces of code have been developed independently and have universally become accepted and do not fall under the copying/originality rule. These include (a) endgame database probe code where the original author has either made the code public domain, or has given a participant explicit permission to use that code and data. ...

What does explicit mean here? Is being LGPL explicit enough, or does every participant who wants to use syzygy need to have a permission slip from Ronald de Man?

I think what we need are two classes of code. One is not chess related, so everyone can just use it (e.g. quicksort) if they need it.

The other case is chess related code, and the ICGA can decide on a case by case basis on whether they are allowed. If it is allowed for general use, like bitbases probably would be, it must be freely available to everyone under the same terms. If it allowed for a specific use, like perhaps Stockfish, it can only be entered once.

[bob]

That sounds like a very bad idea. It is only a matter of time before people will learn how to thwart the simtest, and then you basically give them permission to do whatever they want.

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this). That might could thwart the sim test without losing too many Elo.

There is a second similarity comparison that could be done, which is compare scores for each position. While it is also not fool-proof, it probably would not be fooled by the scenario described just above. And one run of the similarity tool can produce both the move selections and the scores.

There must be time-resistent objective rules of what is allowed. The simtest could at best be a tool used to unmask cheaters that violate the objective rules.

Exactly.

Scores are not so good. One can simply initialize a random number for the base score, so that rather than score = 0 at the top of eval, it is score = initial;. Change initial each time the program is started. Scores will vary all over the place, run to run, but since scores are relative, whether you start off at +100 or -100 makes absolutely no difference to the move actually played.

That is detectable. The scores from two runs for an engine should be highly correlated.

Is that then a requirement? What about programs that intentionally introduce randomness? I've not even seen a requirement that an engine has to use the same scores from game to game. You would notice it, sure. But what does it do to the statistical analysis?

[bob]

2e. Exceptions. Certain pieces of code have been developed independently and have universally become accepted and do not fall under the copying/originality rule. These include (a) endgame database probe code where the original author has either made the code public domain, or has given a participant explicit permission to use that code and data. ...

What does explicit mean here? Is being LGPL explicit enough, or does every participant who wants to use syzygy need to have a permission slip from Ronald de Man?

I think what we need are two classes of code. One is not chess related, so everyone can just use it (e.g. quicksort) if they need it.

The other case is chess related code, and the ICGA can decide on a case by case basis on whether they are allowed. If it is allowed for general use, like bitbases probably would be, it must be freely available to everyone under the same terms. If it allowed for a specific use, like perhaps Stockfish, it can only be entered once.

The point here was that there are SEVERAL different egtb formats. Nalimov does require Eugene's permission. He has given it to many. For someone to use it, they need his permission. GPL certainly gives that permission explicitly as part of the GPL, so long as all GPL rules are followed regarding distribution (if any).

One could argue, and I would not object, that something can't be used unless it is available for all to use (excepting opening books probably). That's yet another issue that should be addressed to close any loophole.

[Dirt]

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this)..

I would expect that getting an exact score for the second best move would cost too many Elo.

[Adam Hair]

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this)..

I would expect that getting an exact score for the second best move would cost too many Elo.

My ignorance is showing. But I think I do understand you.

[Adam Hair]

That sounds like a very bad idea. It is only a matter of time before people will learn how to thwart the simtest, and then you basically give them permission to do whatever they want.

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this). That might could thwart the sim test without losing too many Elo.

There is a second similarity comparison that could be done, which is compare scores for each position. While it is also not fool-proof, it probably would not be fooled by the scenario described just above. And one run of the similarity tool can produce both the move selections and the scores.

There must be time-resistent objective rules of what is allowed. The simtest could at best be a tool used to unmask cheaters that violate the objective rules.

Exactly.

Scores are not so good. One can simply initialize a random number for the base score, so that rather than score = 0 at the top of eval, it is score = initial;. Change initial each time the program is started. Scores will vary all over the place, run to run, but since scores are relative, whether you start off at +100 or -100 makes absolutely no difference to the move actually played.

That is detectable. The scores from two runs for an engine should be highly correlated.

Is that then a requirement? What about programs that intentionally introduce randomness? I've not even seen a requirement that an engine has to use the same scores from game to game. You would notice it, sure. But what does it do to the statistical analysis?

Is there a plausible reason to add a random offset to the score from game to game?

[bob]

That sounds like a very bad idea. It is only a matter of time before people will learn how to thwart the simtest, and then you basically give them permission to do whatever they want.

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this). That might could thwart the sim test without losing too many Elo.

There is a second similarity comparison that could be done, which is compare scores for each position. While it is also not fool-proof, it probably would not be fooled by the scenario described just above. And one run of the similarity tool can produce both the move selections and the scores.

There must be time-resistent objective rules of what is allowed. The simtest could at best be a tool used to unmask cheaters that violate the objective rules.

Exactly.

Scores are not so good. One can simply initialize a random number for the base score, so that rather than score = 0 at the top of eval, it is score = initial;. Change initial each time the program is started. Scores will vary all over the place, run to run, but since scores are relative, whether you start off at +100 or -100 makes absolutely no difference to the move actually played.

That is detectable. The scores from two runs for an engine should be highly correlated.

Is that then a requirement? What about programs that intentionally introduce randomness? I've not even seen a requirement that an engine has to use the same scores from game to game. You would notice it, sure. But what does it do to the statistical analysis?

Is there a plausible reason to add a random offset to the score from game to game?

Sure, to defeat a similarity tester. Adding random values is often used just to provide variety, or weaken an engine somewhat to make them more palatable to humans. However, I see no rule that says scores can't have some arbitrary origin rather than zero.

So far I have seen the following ideas:

(1) random score origin that I mentioned.

(2) when two moves are close to the same score choose randomly, or even always choose second best when not playing a real game.

(3) altering PSTs or other scoring terms.

(4) shift values of pieces to change scores. Or even change moves.

(5) use completely different code (evaluation or search or whatever) depending on whether you are playing a game or testing an EPD-type position.

I suppose this follows right in the mold of obfuscating depth, nodes, artificially shortening the PV, shoot, nothing says you even have to show a PV.

[bob]

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this)..

I would expect that getting an exact score for the second best move would cost too many Elo.

Invert your thinking. First move = 0.02, new best move = 0.06. Play original 0.02 move. No penalty whatsoever. That is, rather than doing a multi-PV, just wait for those opportunities where you get multiple PVs for free. Programs change to a new best move about 1 of every 6 iterations, so there is plenty of opportunity to play games there without breaking anything at all.

[hgm]

Not the strength but I think the general disgust for allowing a derivative work. I think many are not ready. Remember what happened when the CSVN allowed Rybka to participate.

Yes, I remember. But I don't think that was because it was a derivative work. It was because it was considered a stolen, illegal work. (For this discussion it is of no importance whether this is true, or not, so let's not get into that now.) I think that makes a huge difference. People rightfully get angered when theft is encouraged. I don't think it is a good idea to encourage theft.

Perhaps this is the central issue that should be answered first: do you think that illegally copied code (GPL violations, RE of commercial binaries) should be allowed to participate, when they manage to thwart the simtest? Should the ICGA just shrug, and say "Yes, we know it is stolen, but it passes the simtest, so we allow it to run it anyway. And if our member from which it was stolen doesn't like it, let him fight his own legal battle over it, that is none of our business."? Apart from the ethical aspect, I think such an attitude could get the ICGA in severe legal trouble. Aiding and abetting criminals is a crime in itself, in most countries. ICGA rules can be more specific that common law, but it can never contradict it. When you are arreigned for copyright violation "The rules of our organization do not recognize intellectual property rights" will not go down well with the judge...

Derivatives have been allowed before, (Grid Chess), and although they caused some grumbling, they did not lead to massive resigning of ICGA membership.

I think the main question of this era is: "Is Ippolyt a stolen work, or a fully legal public-domain code?". Can we afford an attitude like "if no one goes to court over it, we assume that it is legal", as a sort of easy default.

The attitude of the ICGA that this needs not be investigated, because it never participated in a WCCC is understandable from their perspective (their task is not to police the chess-programming community), but very unfortunate nevertheless. Perhaps it would be a good idea to force the ICGA's hand by entering an Ippolyt derivative in the WCCC. I don't think they could bar it just on suspicion; they would be forced to investigate it.