Clone detection test

Discussion of chess software programming and technical issues.

Moderator: Ras

Mincho Georgiev
Posts: 454
Joined: Sat Apr 04, 2009 6:44 pm
Location: Bulgaria

Re: Clone detection test

Post by Mincho Georgiev »

Don wrote:
xcomponent wrote:Hi guys!
Wouldn't be more correct if the test includes the similarities in the pv as well.
A single move could misleads since the reason for picking it could be even a move ordering bug. I know that huge amount of positions could suppress these kind of possibilities, but they are still remains. I think that the test make sense, but in a perfect world, could be more correct prove if it is combined with the pv similarities and the evaluation value.
I believe the only reliable measure is the actual move played because everything else can be faked. But you cannot fake the move.

But even if it were not faked, I'm not sure the PV is very reliable. In my own program the PV changes frequently, even if the first move does not. My guess is that each successive move is increasingly unreliable as a measure of similarity. And I admit that I'm guessing here, but my sense of this is that even if it was an improvement it would be only a very minor one. And as I mentioned, it can still be faked.

But there is no need to guess, try the experiment yourself, it was very easy to construct - and see if you can produce a better measure using many moves of the PV.
Probably i will try it, it sounds interesting, but even if i do, i will consider this more like a style similarity test instead of cloning one.
User avatar
Don
Posts: 5106
Joined: Tue Apr 29, 2008 4:27 pm

Re: Clone detection test

Post by Don »

Don't get hung up on my Subject line, "Clone detection test", that was just a teaser to suck everyone in.

Larry did a similar thing recently on some forum, he made a post that implied Rybka was in trouble due to Komodo. While I would like to think that might happen someday, the title was merely a teaser, Rybka is still much stronger than Komodo.
Mincho Georgiev
Posts: 454
Joined: Sat Apr 04, 2009 6:44 pm
Location: Bulgaria

Re: Clone detection test

Post by Mincho Georgiev »

I had my suspicion on that :)
Regarding the previous posts, as a long time assembly programmer, i can say that the distance from reversing to assembler could be like from earth to the moon especially after a couple of compiler optimizations. It takes not only assembly knowledge, but a lot of free time and efforts which I in particular don't have.
diep
Posts: 1822
Joined: Thu Mar 09, 2006 11:54 pm
Location: The Netherlands

Re: Clone detection test

Post by diep »

Don wrote:
diep wrote:
Don wrote:
Dann Corbit wrote: I am beginning to think more and more that it is perhaps not a great idea to accuse someone of something very bad because his engine plays similarly to another engine. But it is also possible that some magic formula will occur that is foolproof. In any case, the idea makes me very nervous.
I think you are being overly concerned. I have seen over the years many cheaters exposed (or let's just say misunderstandings cleared up) simply because the author of a program noticed that a program in a tournament was playing like his.

For instance several years ago I am playing at the Dutch Computer Chess Championship and get an email from John Stanback - who is watching the games from the states and asks me to check into a problem, he notices that one of the program is playing just like Zarkov.

In another tournament Richard Lang is sitting across from a clone of his own program (in this case it's a perfect clone) and he notices that a program is just too similar to his even though it's disguised in a different housing (it's one of those hardware chess computers.)

In yet another tournament Bob Hyatt notices remotely that one of the contestants is playing exactly the same moves as Crafty.

As a result of these observations, the problems in each case were investigated and were brought to some kind of resolution. Just noticing the similarity itself was not considered the proof but it was good enough to start asking questions.

So just relax - if we build this tool it will be with the understanding that it's imperfect - it's just a crude measurement. Like almost any tool, it's not a bad thing but someone could grab a screwdriver and use as a weapon to hurt someone with, it doesn't mean we should not have screwdrivers.

So I believe this is a powerful test but like you I agree that it should not be used as a weapon to hit someone over the head with.

I could build a simple UCI test harness to run my test and produce a result file - anyone interesting in doing some kind of blind test?

Don
I didn't follow the thread, but what's all this idiocy man?

Just install IDA pro and look in the assembler code of an engine and you know instantly whether it's a clone.

And nothing else can prove anything unless the guy who clones is a major idiot; note majority is major idiots to copy things 100%. With source code available now it's easy to make some modifications modifying behaviour.

So just take a look in the assembler code of the engine and you know it all. Simple as that.

Thanks,
Vincent
Disassembled code looks different on different compilers and requires an expert. It's easy for you and I, but not everyone. A lot of good C programmers do not know assembler.

The similarity testers would be a tool and nothing more. It would be used in conjunction with other things, such as the disassembler.

Don
And how many 'good C programmers' is trying to find clones huh?

Let's face it, there's a lot of clones out there that copy the then best engine they have access to, and nearly 0 of those programmers ever posted a word about it, they're all hiding themselves under 100 meters of sand.

However in that shelter 100 meters deep down they complain major league about what happens and want someone else, usually me, to post the bad news.

Naum huh?

Did you take a look in its assembler already?

Every algorithm and thingie the same as rybka. Why then have an executable that at 32 bits is same speed as at 64 bits (tested by Andre van Ark) and with sureness be 50-100 elo weaker or something than rybka3?

Contractual appointments huh?

Every programmer knows it for years already and 0 have posted it so far.

So who is going to use your 'clone detection test' ?

It's again wasted time.

From the guys who post here in this thread, about half of them knows already more clone names, that officialy have not been 'uncovered'. So why do they cooperate with a clone test that actually proves nothing.

Even IF you have similarity, there has already been evidence that it is possible to train a chessprogram to *behave* like another program quite ok.

"oh and which engine is that?"

Well it's not Diep and it kicked hell out of a few other engines in a world champs one day, despite running single cpu.

There happens so much there already that has been accomplished, that it's already impossible to follow.

Is trying to copy behaviour illegal or is that also falling under 'clone law'?

As that's what todays sophisticated self learners are doing.

As for fresh engines, todays optimization techniques and testing techniques needed to produce an engine that's 2700+, they are so difficult mathematical, that no unknown person will ever manage to do so.

Anyone who just shows up and has 2700+ engine, it's a clone, stolen concept, or someone who produced it under a fake name, or someone who bought a license.

Speaking of licenses - i assume that is legal.

How legal is it that several authors buy a license, one is called rybka, another one has another name and goes under other names.

Isn't it in fact also hard fact that a lot of those engines are in face originating from the same codebase and therefore legally are the same engine?

So only 1 of them may join the world champs or?

So many seems to have bought, or illegally manufactured, their own rybka, long before the ippolit source code had been put online, that it's just sick.

Thinker huh?

As if this guy knows anything about computerchess "suddenly".

In assembler the proof already has been delivered that he tried to obfuscate it a little, but it definitely is another case of thievery. Or did it legally get licensed?

You know, all programmers , especially the C programmers, also realize this, that he can't get that strong "out of the blue sky". In assembler it's visible. If you of course obfuscate your engine and don't show the mainline, just let it play a move, of course every idiot on the street realizes why that is. It has been weakened by using smaller hashtable sizes, which youc an't set to some decent amount, say a couple of hundreds of MB, why is THAT?

Because that makes an engine a lot weaker, and entire computerchess community already knows this for a year or 15+.

Well - except for Christophe Theron then that is (i remember start of 21th century a beta version he had shipped to Jeroen to use at a tournament, which used a very tiny hashtable of a few hundreds of kilobyte also).

Deliberate, according to contract, weakened engine again?

As it may not beat rybka3 again?

You know, everyone just shuts up always. They all know what's going on, but no one talks.

So why design a test if that's not solving the core of the problem?

Vincent
sockmonkey
Posts: 588
Joined: Sun Nov 23, 2008 11:16 pm
Location: Berlin, Germany

Re: Clone detection test

Post by sockmonkey »

Would anyone mind if this thread got relocated to the forum dedicated to Engine Origins, since that's ultimately what it's about? Seems thematically appropriate. Please PM me if you have objections.

Thanks
Jeremy
User avatar
Don
Posts: 5106
Joined: Tue Apr 29, 2008 4:27 pm

Re: Clone detection test

Post by Don »

sockmonkey wrote:Would anyone mind if this thread got relocated to the forum dedicated to Engine Origins, since that's ultimately what it's about? Seems thematically appropriate. Please PM me if you have objections.

Thanks
Jeremy
I have an objection because this particular thread is of a technical nature more than political. I view that other forum as non-technical and political and it probably would not attract the more serious engineers as much as this one would.

Besides, the subject of this thread was specifically designed to be provocative, but it's really about designing instrumentation to measure the playing style of chess programs.
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: Clone detection test

Post by michiguel »

Don wrote:
sockmonkey wrote:Would anyone mind if this thread got relocated to the forum dedicated to Engine Origins, since that's ultimately what it's about? Seems thematically appropriate. Please PM me if you have objections.

Thanks
Jeremy
I have an objection because this particular thread is of a technical nature more than political. I view that other forum as non-technical and political and it probably would not attract the more serious engineers as much as this one would.

Besides, the subject of this thread was specifically designed to be provocative, but it's really about designing instrumentation to measure the playing style of chess programs.
I agree with Don.

Miguel
User avatar
Greg Strong
Posts: 388
Joined: Sun Dec 21, 2008 6:57 pm
Location: Washington, DC

Re: Clone detection test

Post by Greg Strong »

Ok, so I feel compelled to ask a couple of stupid questions...

How does this 'cloning' occur? Do the cloners have access to the source code, or do they decompile the executable and then work with the horrible mess that comes out? And what the hell's the point?
Eizenhammer

Re: Clone detection test

Post by Eizenhammer »

diep wrote: Thinker huh?
As if this guy knows anything about computerchess "suddenly".
In assembler the proof already has been delivered that he tried to obfuscate it a little, but it definitely is another case of thievery. Or did it legally get licensed?

You know, all programmers , especially the C programmers, also realize this, that he can't get that strong "out of the blue sky". In assembler it's visible. If you of course obfuscate your engine and don't show the mainline, just let it play a move, of course every idiot on the street realizes why that is. It has been weakened by using smaller hashtable sizes, which youc an't set to some decent amount, say a couple of hundreds of MB, why is THAT?

Because that makes an engine a lot weaker, and entire computerchess community already knows this for a year or 15+.
Thinker did not appear "out of the blue sky", but was quite a strong engine before any rybka was there to be copied.
User avatar
Don
Posts: 5106
Joined: Tue Apr 29, 2008 4:27 pm

Re: Clone detection test

Post by Don »

Greg Strong wrote:Ok, so I feel compelled to ask a couple of stupid questions...

How does this 'cloning' occur? Do the cloners have access to the source code, or do they decompile the executable and then work with the horrible mess that comes out? And what the hell's the point?
It's well known that many programs have been disassembled. In many cases this is not disguised, in others it is denied.

The point of doing this is to dig for ideas that you cannot come up with on your own, or would require too much effort to come up with on your own.