Similarity Detector Available

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
Don
Posts: 5106
Joined: Tue Apr 29, 2008 2:27 pm

Similarity Detector Available

Post by Don » Sun Dec 26, 2010 11:19 pm

I created a utility called similar which measures how different one chess program is from others. It does this by running 2000 position from random games and noting how often the moves agree and as output returns the percentage of moves that match.

You can get it here: http://komodochess.com/pub/similar.zip

Here is some sample output, comparing Robbolito with a few other programs:

------ RobboLito version 0.084 (time: 100 ms) ------
69.25 Houdini 1.5 w32 (time: 100 ms)
66.90 Rybka 3 (time: 100 ms)
61.70 Stockfish 1.9.1 JA 64bit (time: 100 ms)
61.35 Stockfish 1.8 JA (time: 100 ms)
59.80 Komodo64 1.2 JA (time: 100 ms)
59.15 Komodo 1.0 (time: 100 ms)
58.95 Stockfish 1.7.1 64bit (time: 100 ms)
58.95 Stockfish 1.6 64bit (time: 100 ms)
57.00 Fruit 2.3.1 (time: 100 ms)
56.20 Fruit 2.1 (time: 100 ms)


I have not tested this on windows so I'm hoping to get some feedback specific to windows.

The similar.exe is designed to run on 64 bit windows and is actually a tcl script wrapped up with a tcl runtime using tclkit technology. I am also including the "starkit" which is platform independent, but requires a tclkit runtime for your platform. It is similar to a jar file and can be taken apart and inspected and modified if you wish - assuming you know how to work with starkit's and such. google for starkit and sdx.kit for more information.

Please let me know if you find this interesting or useful. Email me at drd@mit.edu

Don

Engin
Posts: 880
Joined: Mon Jan 05, 2009 6:40 pm
Location: Germany
Full name: Engin Üstün

Re: clone tester available

Post by Engin » Sun Dec 26, 2010 11:46 pm

ah, yeah now we can see how similar houdini it is from Robbolito :-)

Albert Silver
Posts: 2969
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

Re: clone tester available

Post by Albert Silver » Sun Dec 26, 2010 11:56 pm

Have you tried this method comparing the default Rybka 3 to the Dynamic and Human flavors?
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

User avatar
Laskos
Posts: 10949
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: clone tester available

Post by Laskos » Mon Dec 27, 2010 12:54 am

Thanks Don, my first try, seems to work :)



C:\similar>similar.exe -r 1
------ Ivanhoe_B49jAx64 (time: 100 ms) ------
68.05 Houdini 1.5 x64 (time: 100 ms)


Now I will prepare a battery of engines :)

Kai

User avatar
kranium
Posts: 1915
Joined: Thu May 29, 2008 8:43 am

Re: clone tester available

Post by kranium » Mon Dec 27, 2010 12:54 am

Don wrote:I created a utility called similar which measures how different one chess program is from others. It does this by running 2000 position from random games and noting how often the moves agree and as output returns the percentage of moves that match.

You can get it here: http://komodochess.com/pub/similar.zip

Here is some sample output, comparing Robbolito with a few other programs:

------ RobboLito version 0.084 (time: 100 ms) ------
69.25 Houdini 1.5 w32 (time: 100 ms)
66.90 Rybka 3 (time: 100 ms)
61.70 Stockfish 1.9.1 JA 64bit (time: 100 ms)
61.35 Stockfish 1.8 JA (time: 100 ms)
59.80 Komodo64 1.2 JA (time: 100 ms)
59.15 Komodo 1.0 (time: 100 ms)
58.95 Stockfish 1.7.1 64bit (time: 100 ms)
58.95 Stockfish 1.6 64bit (time: 100 ms)
57.00 Fruit 2.3.1 (time: 100 ms)
56.20 Fruit 2.1 (time: 100 ms)


I have not tested this on windows so I'm hoping to get some feedback specific to windows.

The similar.exe is designed to run on 64 bit windows and is actually a tcl script wrapped up with a tcl runtime using tclkit technology. I am also including the "starkit" which is platform independent, but requires a tclkit runtime for your platform. It is similar to a jar file and can be taken apart and inspected and modified if you wish - assuming you know how to work with starkit's and such. google for starkit and sdx.kit for more information.

Please let me know if you find this interesting or useful. Email me at drd@mit.edu

Don
wow, amazing...
komodo and stockfish are much more similar than I would have guessed.
i was genuinely under the apparently incorrect 'assumption' that they were very different...
weird...
(i.e i thought they would certainly fail any 'clone' test...?)

BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 3:18 am

Re: clone tester available

Post by BubbaTough » Mon Dec 27, 2010 1:08 am

kranium wrote: wow, amazing...
komodo and stockfish are much more similar than I would have guessed.
Now, thanks to this cool tool, you can be specific about such claims. Are you talking level 70 similarity, or closer to 60 level similarity :). One thing that is fun about this method of comparison is it provides an objective metric of comparison...a welcome change of pace from the more wishy washy statements that are often flung around here on this topic.

-Sam

User avatar
kranium
Posts: 1915
Joined: Thu May 29, 2008 8:43 am

Re: clone tester available

Post by kranium » Mon Dec 27, 2010 1:17 am

BubbaTough wrote:
kranium wrote: wow, amazing...
komodo and stockfish are much more similar than I would have guessed.
Now, thanks to this cool tool, you can be specific about such claims. Are you talking level 70 similarity, or closer to 60 level similarity :). One thing that is fun about this method of comparison is it provides an objective metric of comparison...a welcome change of pace from the more wishy washy statements that are often flung around here on this topic.

-Sam
yes, in this case komodo seems absolutley equal to stockfish 1.8....according to CCRL....aaaargh!?

wait!
much better to leave it to the amateur user...
please test independently
(i.e. test yourself if you want reliable/unbiased results).
Last edited by kranium on Mon Dec 27, 2010 1:19 am, edited 1 time in total.

tomgdrums
Posts: 736
Joined: Wed Dec 02, 2009 8:48 am

Re: clone tester available

Post by tomgdrums » Mon Dec 27, 2010 1:19 am

kranium wrote:
Don wrote:I created a utility called similar which measures how different one chess program is from others. It does this by running 2000 position from random games and noting how often the moves agree and as output returns the percentage of moves that match.

You can get it here: http://komodochess.com/pub/similar.zip

Here is some sample output, comparing Robbolito with a few other programs:

------ RobboLito version 0.084 (time: 100 ms) ------
69.25 Houdini 1.5 w32 (time: 100 ms)
66.90 Rybka 3 (time: 100 ms)
61.70 Stockfish 1.9.1 JA 64bit (time: 100 ms)
61.35 Stockfish 1.8 JA (time: 100 ms)
59.80 Komodo64 1.2 JA (time: 100 ms)
59.15 Komodo 1.0 (time: 100 ms)
58.95 Stockfish 1.7.1 64bit (time: 100 ms)
58.95 Stockfish 1.6 64bit (time: 100 ms)
57.00 Fruit 2.3.1 (time: 100 ms)
56.20 Fruit 2.1 (time: 100 ms)


I have not tested this on windows so I'm hoping to get some feedback specific to windows.

The similar.exe is designed to run on 64 bit windows and is actually a tcl script wrapped up with a tcl runtime using tclkit technology. I am also including the "starkit" which is platform independent, but requires a tclkit runtime for your platform. It is similar to a jar file and can be taken apart and inspected and modified if you wish - assuming you know how to work with starkit's and such. google for starkit and sdx.kit for more information.

Please let me know if you find this interesting or useful. Email me at drd@mit.edu

Don
wow, amazing...
komodo and stockfish are much more similar than I would have guessed.
i was genuinely under the apparently incorrect 'assumption' that they were very different...
weird...
(i.e i thought they would certainly fail any 'clone' test...?)
If I am not mistaken the numbers that Don is showing are relevant to how Roblito compares to Komodo and then Robolito compares to Stockfish. The numbers are not a comparison of Komodo to Stockfish.

User avatar
kranium
Posts: 1915
Joined: Thu May 29, 2008 8:43 am

Re: clone tester available

Post by kranium » Mon Dec 27, 2010 1:21 am

tomgdrums wrote:
kranium wrote:
Don wrote:I created a utility called similar which measures how different one chess program is from others. It does this by running 2000 position from random games and noting how often the moves agree and as output returns the percentage of moves that match.

You can get it here: http://komodochess.com/pub/similar.zip

Here is some sample output, comparing Robbolito with a few other programs:

------ RobboLito version 0.084 (time: 100 ms) ------
69.25 Houdini 1.5 w32 (time: 100 ms)
66.90 Rybka 3 (time: 100 ms)
61.70 Stockfish 1.9.1 JA 64bit (time: 100 ms)
61.35 Stockfish 1.8 JA (time: 100 ms)
59.80 Komodo64 1.2 JA (time: 100 ms)
59.15 Komodo 1.0 (time: 100 ms)
58.95 Stockfish 1.7.1 64bit (time: 100 ms)
58.95 Stockfish 1.6 64bit (time: 100 ms)
57.00 Fruit 2.3.1 (time: 100 ms)
56.20 Fruit 2.1 (time: 100 ms)


I have not tested this on windows so I'm hoping to get some feedback specific to windows.

The similar.exe is designed to run on 64 bit windows and is actually a tcl script wrapped up with a tcl runtime using tclkit technology. I am also including the "starkit" which is platform independent, but requires a tclkit runtime for your platform. It is similar to a jar file and can be taken apart and inspected and modified if you wish - assuming you know how to work with starkit's and such. google for starkit and sdx.kit for more information.

Please let me know if you find this interesting or useful. Email me at drd@mit.edu

Don
wow, amazing...
komodo and stockfish are much more similar than I would have guessed.
i was genuinely under the apparently incorrect 'assumption' that they were very different...
weird...
(i.e i thought they would certainly fail any 'clone' test...?)
If I am not mistaken the numbers that Don is showing are relevant to how Roblito compares to Komodo and then Robolito compares to Stockfish. The numbers are not a comparison of Komodo to Stockfish.
yes of course...
it's all about whether Robolito is a clone or not...
other engines are exempt...i understand completely now.

thanx Tom!

User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: clone tester available

Post by michiguel » Mon Dec 27, 2010 1:28 am

tomgdrums wrote:
kranium wrote:
Don wrote:I created a utility called similar which measures how different one chess program is from others. It does this by running 2000 position from random games and noting how often the moves agree and as output returns the percentage of moves that match.

You can get it here: http://komodochess.com/pub/similar.zip

Here is some sample output, comparing Robbolito with a few other programs:

------ RobboLito version 0.084 (time: 100 ms) ------
69.25 Houdini 1.5 w32 (time: 100 ms)
66.90 Rybka 3 (time: 100 ms)
61.70 Stockfish 1.9.1 JA 64bit (time: 100 ms)
61.35 Stockfish 1.8 JA (time: 100 ms)
59.80 Komodo64 1.2 JA (time: 100 ms)
59.15 Komodo 1.0 (time: 100 ms)
58.95 Stockfish 1.7.1 64bit (time: 100 ms)
58.95 Stockfish 1.6 64bit (time: 100 ms)
57.00 Fruit 2.3.1 (time: 100 ms)
56.20 Fruit 2.1 (time: 100 ms)


I have not tested this on windows so I'm hoping to get some feedback specific to windows.

The similar.exe is designed to run on 64 bit windows and is actually a tcl script wrapped up with a tcl runtime using tclkit technology. I am also including the "starkit" which is platform independent, but requires a tclkit runtime for your platform. It is similar to a jar file and can be taken apart and inspected and modified if you wish - assuming you know how to work with starkit's and such. google for starkit and sdx.kit for more information.

Please let me know if you find this interesting or useful. Email me at drd@mit.edu

Don
wow, amazing...
komodo and stockfish are much more similar than I would have guessed.
i was genuinely under the apparently incorrect 'assumption' that they were very different...
weird...
(i.e i thought they would certainly fail any 'clone' test...?)
If I am not mistaken the numbers that Don is showing are relevant to how Roblito compares to Komodo and then Robolito compares to Stockfish. The numbers are not a comparison of Komodo to Stockfish.
Exactly because this is not a 1D plot, it is at least 2D. This like saying Chicago is 900 miles from NY, and Chicago is 1000 miles from Denver, so Denver is very close to NY. No!!

Miguel

Post Reply