ICGA Rule 2 discussion

[jdart]

It sounds reasonable at first read but you are still excluding programs that are legal in every sense but don't meet ICGA's originality criteria. There are probably 2000 chess programs at least in the world, many with source available. Most won't be able to compete, should they wish to.

Mine would qualify but I am probably not going to ever enter an ICGA tourney.

And it's possible at some point someone will release a customizable engine framework that becomes widely acceptable and allows for innovation in algorithms while retaining the basic structure. That would be prohibited according to these rules.

I think ICGA has a hopeless task because they are trying to enforce a standard of originality that doesn't line up with either law or current practice (and the latter changes over time).

--Jon

[bob]

It seems there is another point to discuss here, so I'll introduce it.

The reasons why we have wanted to exclude derivatives in the past.

Originally this was to protect commercial programs, because they were worth money, and people tried to copy them, use a hex editor to make enough changes that the program looked "different" and then entered that into the older WMCCC. Back then there were no "open-source" programs.

Later, gnu chess came along and we recognized the beginning of a problem, and then there was Crafty and eventually several others. It became apparent that if we were not careful, the ICGA events would no longer be attractive to beginners, if all they got to do was play against established programs that were much stronger than they were. And we knew that many would copy existing strong programs and then make small modifications and then slip in an entry.

Rule 2 evolved over time to address this issue, we wanted to encourage NEW chess development, get NEW people involved, something that was NOT going to happen if a new person enters and gets smashed over and over by copies of the same program.

I STILL don't want to play against multiple copies of the same program. I don't want to travel a long distance, spend a significant amount of time, and play (essentially) the same program every round.

I suppose there are some that would be happy to use robolito as a starting point, and compete with that program against all the other robolito derivatives. But it doesn't seem like much of a challenge. Part of the attractiveness of writing a chess program is that it is NOT easy or trivial. It takes a lot of work. When you get it to the 1600 level on ICC, you have accomplished something that was challenging, and you can take pride in how your new creation performs. Then you can spend years improving the result, and feel a sense of pride in every step. If you start off with Robolito, most of that "investment" is missing, the sense of pride is missing, the drive to improve is missing.

That's my take on this. We should probably address the actual intent dealing with rule 2, as well as fixing rule 2 if it seems the majority of authors want to keep it, but in a more carefully-worded form.

[Adam Hair]

That sounds like a very bad idea. It is only a matter of time before people will learn how to thwart the simtest, and then you basically give them permission to do whatever they want.

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this). That might could thwart the sim test without losing too many Elo.

There is a second similarity comparison that could be done, which is compare scores for each position. While it is also not fool-proof, it probably would not be fooled by the scenario described just above. And one run of the similarity tool can produce both the move selections and the scores.

There must be time-resistent objective rules of what is allowed. The simtest could at best be a tool used to unmask cheaters that violate the objective rules.

Exactly.

Scores are not so good. One can simply initialize a random number for the base score, so that rather than score = 0 at the top of eval, it is score = initial;. Change initial each time the program is started. Scores will vary all over the place, run to run, but since scores are relative, whether you start off at +100 or -100 makes absolutely no difference to the move actually played.

That is detectable. The scores from two runs for an engine should be highly correlated.

[arjuntemurnikar]

It sounds like you advocate "all derivatives are OK except if they come from IPPOLIT?"

To date, ippolit has not been proven to be a rybka derivative, so what makes it special here? My issue is that I believe ippolit is not original, based on looking at the source code which appears to come from hex rays or something similar. And we have no author to question.

But is it really ok to have fruit derivatives, crafty derivatives, stockfish derivatives, any other open-source derivatives?

No of course not. If a program is derived from another program and it violate's its license, it should not be allowed. Thus, in the case of open-source derivative programs, if no license violation has occured, and the program passes the similarity test, it should be allowed to participate.

Note: I only mentioned IPPOLIT as special case because it is "public domain" but is unoriginal and illegal derivative of rybka (unproven, but highly suspicious). Again, I must emphasize that we can debate about IPPOLIT all day and never come to any scientific conclusion, so lets not get idealistic. It is commonly accepted in the computer chess community that IPPOLIT is an illegal derivative, so lets not waste time about it and just move on. As long as it passes the similarity test, an IPPOLIT derivate should be fine. If not, it should not be allowed.

What about question #2. Ok to have 30 stockfish derivatives? Again, that is not something I want to do. I could run that on my cluster, for example.

30 stockfish derivatives? Again, if they pass the simtest, then they are through. I highly doubt though that they would.

So you want to use the simtest as THE test? Pass it, play, fail it, don't play? This does not do a thing for "originality".

My problem is that everybody has been arguing about how best to preserve originality, but at the same time, refusing to use the best (and only) tool we have so far in achieving this -- simtest -- and everybody keeps coming up with new arguments and new excuses everyday against it, while simultaneously not providing any alternative solution. This is ridiculous and with this attitude, WCCC will is just a pipe dream and will never regain its glory.

What people need to do right now is stop being idealistic and just go with what best we have right now. In future we can improve upon it if we can. What is important is that we get the ball rolling.

Also no one has commented on my proposal to go online-only. I think this is the only way forward.

[hgm]

Also no one has commented on my proposal to go online-only. I think this is the only way forward.

I addressed it. Bob addressed it.

Since you seem to have missed that, the following summary:
1) It does not seem to contribute anything to solving the problem (see CCT vs WCCC attendance)
2) It has been tried, and was an absolute failure (0 and 1 entrant, respectively)
3) I prefer on-site over on-line events

[Rebel]

That sounds like a very bad idea. It is only a matter of time before people will learn how to thwart the simtest, and then you basically give them permission to do whatever they want.

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this). That might could thwart the sim test without losing too many Elo.

There is a second similarity comparison that could be done, which is compare scores for each position. While it is also not fool-proof, it probably would not be fooled by the scenario described just above. And one run of the similarity tool can produce both the move selections and the scores.

There must be time-resistent objective rules of what is allowed. The simtest could at best be a tool used to unmask cheaters that violate the objective rules.

Exactly.

Scores are not so good. One can simply initialize a random number for the base score, so that rather than score = 0 at the top of eval, it is score = initial;. Change initial each time the program is started. Scores will vary all over the place, run to run, but since scores are relative, whether you start off at +100 or -100 makes absolutely no difference to the move actually played.

There are some people working on that.

[hgm]

Using any kind of specific test as the official criterion for what is allowed and what not is totally unsatisfactory.

Someone intent on cheating can trivially thwart it. Just make an engine that uses a completely different evaluation when you let it search from a loaded position, rather than the initial position. Or that uses another evaluation in the first 5 moves of every game (which in the tournament would always come from its book). Or when the TC is not exactly equal or close to the TC known to be used in the tournament. Or a combination of those.

And yes, you could easily find that out by disassembling his code. And then he would just laugh in your face, and say: "So what? I passed the similarity test, didn't I? That was the only requirement. The rules do not put any restrictions on what method I use to pass it."

Besides, there can be completely valid reasons to use another eval at 1 sec/move than at 3 min/move. So you cannot even prove intent to cheat, when you would know the person did it. I could take Stockfish, make the eval TC dependent by letting it use a TC-weighted average of its own and Fairy-Max' evaluation, publish the source code to show everyone what I did, and now compete with this engine that passes the simtest. It would cost me zero Elo at the tournament TC.

[Rebel]

I would hardly call that all-inclusive yet, but it gives a starting point. I do NOT think a "public domain source" can be copied and modified and then used.

Surely it can, as long as just one entry is allowed.

Supposedly Ippolit was made "public domain" and I do NOT believe we want 10 copies of that in a chess tournament.

See above.

Free (strong) open-sources are the main reason we are in this mess, the way out is to agree on how we deal with them.

[hgm]

Free (strong) open-sources are the main reason we are in this mess, the way out is to agree on how we deal with them.

Yes, we agree about that.

But Bob is right, they should not be allowed to participate 10 times if other codes can participate only once. OTOH, it is equally unsatisfactory if they could not participate at all. If others are allowed to participate with their code once, why should that be different for open source? There is nothing intrinsically evil or criminal in opening your source, so why should there be a penalty on it?

But Bob already stated in another thread that limiting the number of derivatives of the same open-source program to just one would work for him. And, as I pointed out, it does not seem to even violate the current WCCC rule #2. So I think we already passed that point successfully.

Bob's major concern seems to be the 'right of first refusal'. If there is a conflict between two people that want to enter the same game-playing code as (part of) their program, should they somehow be treated asymmetrically, one being able to claim priority over the other? (E.g. by defining something like an 'original author', or a 'main author'.) Or is it acceptable that when, say Toga and Fruit both want to enter, Toga would be admitted and Fabien would be barred?

Apart from the fact that this is mostly hypothetical, it seems that Bob is more conservative in this respect than I would be (i.e. by stamping GPL on your code you give up all rights to it, including the right to enter it in tournaments).

[mvk]

First, here is the current ICGA Rule 2, which is regarding originality:

2. Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in their submission details. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director.

It is my understanding that Levy rewrote them for the Yokohama tournament.

When his changes were discussed, your feedback was that there was no need for change. Good to see your mind has changed. The new rules are not much better IMHO, but it is good that it is recognised that something needed to be done about them and a step was taken.

You are mis-reading. I am not trying to CHANGE rule 2. I am trying to carefully CLARIFY rule two. To get around things like Vas' comment that "yes, if I type the code then it is original". It seems wise to try to explain it as clearly and carefully as possible, so that we don't have to deal with all the nonsensical excuses when someone gets caught. If you notice, nothing I wrote has changed the originality requirement at all. I simply included the things we had already agreed was acceptable to copy (egtb, magic, rotated, etc) as another point of clarity.

I know, but better start from the current wording, and not from the troubled wording of the past.