Similarity Detector Available

[Adam Hair]

I created a utility called similar which measures how different one chess program is from others. It does this by running 2000 position from random games and noting how often the moves agree and as output returns the percentage of moves that match.

You can get it here: http://komodochess.com/pub/similar.zip

Here is some sample output, comparing Robbolito with a few other programs:

------ RobboLito version 0.084 (time: 100 ms) ------
69.25 Houdini 1.5 w32 (time: 100 ms)
66.90 Rybka 3 (time: 100 ms)
61.70 Stockfish 1.9.1 JA 64bit (time: 100 ms)
61.35 Stockfish 1.8 JA (time: 100 ms)
59.80 Komodo64 1.2 JA (time: 100 ms)
59.15 Komodo 1.0 (time: 100 ms)
58.95 Stockfish 1.7.1 64bit (time: 100 ms)
58.95 Stockfish 1.6 64bit (time: 100 ms)
57.00 Fruit 2.3.1 (time: 100 ms)
56.20 Fruit 2.1 (time: 100 ms)


I have not tested this on windows so I'm hoping to get some feedback specific to windows.

The similar.exe is designed to run on 64 bit windows and is actually a tcl script wrapped up with a tcl runtime using tclkit technology. I am also including the "starkit" which is platform independent, but requires a tclkit runtime for your platform. It is similar to a jar file and can be taken apart and inspected and modified if you wish - assuming you know how to work with starkit's and such. google for starkit and sdx.kit for more information.

Please let me know if you find this interesting or useful. Email me at drd@mit.edu

Don

wow, amazing...
komodo and stockfish are much more similar than I would have guessed.
i was genuinely under the apparently incorrect 'assumption' that they were very different...
weird...
(i.e i thought they would certainly fail any 'clone' test...?)

You misunderstand how the test works. Only the top comparison is relevant, what you are seeing is that komodo and stockfish play equally different from robbolito, that does not mean they play like each other.

yes,
of course i misunderstand how your 'clone test' works...i'm a 'simple' cloner.

but i'm very happy you and all the other 'good old boys" understand it...
i'm confident that Graham B., Gabor S., Chris C., and Peter S., Tom G., Miguel B., Paul W., etc. etc.
(sorry i can't mention everyone here: the list is significant...)
will respond soon and clarify it all for us ignorants!...

Awesome! Instead of taking the lameass approach of pointing out
possible flaws with the similarity test ( like BB+ did at open chess),
you tore them up with ridicule and sarcasm. You are the man Norman.

[BubbaTough]

I tried to tool, it works fine for me. I doubt anyone is interested, but here are my results:

------ Hannibal 1.0a (time: 100 ms) ------
67.20 Twisted Logic 20100131x (time: 100 ms)
61.30 Stockfish 1.9.1 JA 64bit (time: 100 ms)
61.05 RobboLito 0.085e4 x64 (time: 100 ms)
57.65 Fruit 2.1 (time: 100 ms)

Hannibal is derived from Twisted Logic, so the high correlation is to be expected. I included a few other open source programs just out of curiosity.

Here are results for an old LearningLemming I had laying around:

------ LL0.41c (time: 100 ms) ------
38.65 Fruit 2.1 (time: 100 ms)
34.95 RobboLito 0.085e4 x64 (time: 100 ms)
33.55 Stockfish 1.9.1 JA 64bit (time: 100 ms)

Wow! That is errr...very different. Perhaps it would have been stronger if I had drawn more inspiration from others.


-Sam

[stevenaaus]

It does this by running 2000 position from random games and noting how often the moves agree

Don, how do you select the 2000 positions ?

Just a statistical observation (as i haven't tried it out) - But less complicated positions will generally have a "best move", which any strong engine will find. So would it be better to use 2000 hand-picked , complicated positions rather than random ones ? Or even select the exact middle move from 2000 different games.

[M ANSARI]

I created a utility called similar which measures how different one chess program is from others. It does this by running 2000 position from random games and noting how often the moves agree and as output returns the percentage of moves that match.

You can get it here: http://komodochess.com/pub/similar.zip

Here is some sample output, comparing Robbolito with a few other programs:

------ RobboLito version 0.084 (time: 100 ms) ------
69.25 Houdini 1.5 w32 (time: 100 ms)
66.90 Rybka 3 (time: 100 ms)
61.70 Stockfish 1.9.1 JA 64bit (time: 100 ms)
61.35 Stockfish 1.8 JA (time: 100 ms)
59.80 Komodo64 1.2 JA (time: 100 ms)
59.15 Komodo 1.0 (time: 100 ms)
58.95 Stockfish 1.7.1 64bit (time: 100 ms)
58.95 Stockfish 1.6 64bit (time: 100 ms)
57.00 Fruit 2.3.1 (time: 100 ms)
56.20 Fruit 2.1 (time: 100 ms)


I have not tested this on windows so I'm hoping to get some feedback specific to windows.

The similar.exe is designed to run on 64 bit windows and is actually a tcl script wrapped up with a tcl runtime using tclkit technology. I am also including the "starkit" which is platform independent, but requires a tclkit runtime for your platform. It is similar to a jar file and can be taken apart and inspected and modified if you wish - assuming you know how to work with starkit's and such. google for starkit and sdx.kit for more information.

Please let me know if you find this interesting or useful. Email me at drd@mit.edu

Don

wow, amazing...
komodo and stockfish are much more similar than I would have guessed.
i was genuinely under the apparently incorrect 'assumption' that they were very different...
weird...
(i.e i thought they would certainly fail any 'clone' test...?)

If I am not mistaken the numbers that Don is showing are relevant to how Roblito compares to Komodo and then Robolito compares to Stockfish. The numbers are not a comparison of Komodo to Stockfish.

Exactly because this is not a 1D plot, it is at least 2D. This like saying Chicago is 900 miles from NY, and Chicago is 1000 miles from Denver, so Denver is very close to NY. No!!

Miguel

yes...a 2d plot!
brilliant...hadn't thought of that.
damn that explains it all!
thanks Miguel, for your response...it's all clear to me now.

1 - AAABBBBBBB
2 - CCCBBBBBBB
3 - AAABBBBCCC

Compare sequence 1, 2, and 3.
1 to 2 is 70% similar.
1 to 3 is 70% similar
is 2 identical to 3? no, they are 40% similar.

Miguel

i get it...

stockfish/komodo/gaviotta extremely dissimilar to robbolito
r3/houdini....very similar.

yes i understand now...( )
brilliant work by you and Don...thanks?

sequence 1 = sequence 2 != sequence 3 <= anyone gives a crap...

Don is presenting something that probably took quite some time to develop. You can use it or you can ignore it and its results ... and that would be fine. Why the hell do you want to pollute the thread with crap that has nothing to do with anything. If you don't agree with the methodology used, just say so in a clear concise way ... preferably with comments on why you think his methodology might not give an accurate result. Really this board is getting polluted by childish comments!

[BKTormey]

I applaud Don and the creation of his program toward objectifying the clone discussion. His "contribution" seems a like a valid application of the Duck Test for this issue. Although it was spawned by controversial topic, it may offer value and perhaps a bit fun/interest outside of that discussion as well. Thanks Don.

[IWB]

Hello Don,

I do not fell very well with such a tool!

I played around a bit and found similarities between engines where I expected it and similarities between engines where there is no way for them of being similar.
I found some similarities of 68 to 70% (like you) between engines which have to have similarities, while I found 65% identical moves between engines were it is impossible that the engines have something in common. Sometimes 1,2,3% will decide if someone is named a "cloner" or not. This tool will be used as soon as it fits to someones goals, and not to bring truth. For years we will see comparisons which are the "truth" because "clone-tool" say so.
I consider this as an instrument of inquisition.

It is out, and as with the Litos we have to live with it.

Bye
Ingo

[Graham Banks]

Hello Don,

I do not fell very well with such a tool!

I played around a bit and found similarities between engines where I expected it and similarities between engines where there is no way for them of being similar.
I found some similarities of 68 to 70% (like you) between engines which have to have similarities, while I found 65% identical moves between engines were it is impossible that the engines have something in common. Sometimes 1,2,3% will decide if someone is named a "cloner" or not. This tool will be used as soon as it fits to someones goals, and not to bring truth. For years we will see comparisons which are the "truth" because "clone-tool" say so.
I consider this as an instrument of inquisition.

It is out, and as with the Litos we have to live with it.

Bye
Ingo

Could you please post the results that you're skeptical about or give us the names of the engines that gave skeptical results?

Cheers,
Graham.

[Damir]

so you can persecute engines and authors behind such engines of beeing cloners ?

[George Tsavdaris]

I created a utility called similar which measures how different one chess program is from others. It does this by running 2000 position from random games and noting how often the moves agree and as output returns the percentage of moves that match.

I guess the positions are a secret for avoiding future clone engines that would try to adapt and manipulate their output.
But it would be nice if there was a setting for putting your own positions.

Does the tool also compares evaluations of the engines or only the selected move?

"and noting how often the moves agree"
For different plies or one note/count per position? I mean does the tool finally divides by 2000 for getting the percentage?

I have not tested this on windows so I'm hoping to get some feedback specific to windows.

The similar.exe is designed to run on 64 bit windows

Designed to run there means designed to run better there or it means that it will run only there and not on 32bit windows?


I think this tool is more evil than one can think of ( ), since if 2 programs have a high percentage of similar output then there is a relatively strong evidence that one is a clone of the other, but if this is not true and they have a low percentage of similar output we can't say that one is not a clone of the other(because the clown author may have changed many things and programs not to be comparable any more) so we can't conclude nothing specific so we can't vindicate a suspicious engine.

So we can incriminate an engine but we can't vindicate it.


In what theory does this tool is based on?
I mean what kind of positions are these? I hope they are not best move positions right? Because then good programs would have similar selections.

I would like the positions to have many different possible good moves to play. Let's say the best 5 moves to be playable. Semi-Opening positions are good for this.
If there are 5 moves per position to choose then we have 5^2000 different combinations the engines can choose from, so the randomness is very good and the statistics at the end are strong and can be a good indicator.

[Sylwy]

so you can persecute engines and authors behind such engines of beeing cloners ?

Even:
-Fritzy;
-Shredder;
-Hiarcs;
-Junior.................................

WOW ! Great tool !!!!!! Commercial in persecution !

Thank you Don !

S