Similarity Detector Available

[El Gringo]

Hi Don,

Spark doesn't like the similar utility. It hangs after 35%

Microsoft Windows XP [versie 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

E:\similar>similar-w32 -t spark10_32.exe
program: spark-1.0 (time: 100 ms)

0.0 percent ..................................................
2.5 percent ..................................................
5.0 percent ..................................................
7.5 percent ..................................................
10.0 percent ..................................................
12.5 percent ..................................................
15.0 percent ..................................................
17.5 percent ..................................................
20.0 percent ..................................................
22.5 percent ..................................................
25.0 percent ..................................................
27.5 percent ..................................................
30.0 percent ..................................................
32.5 percent ..................................................
35.0 percent .................................................

Best
Johan

[El Gringo]

Hi All,

Similarity from Fruit 2.1

E:\similar>similar-w32 -r 11

------ Fruit 2.1 (time: 100 ms) ------
61.55 Umko 1.1 i686 (time: 100 ms)
59.35 Equinox 0.91-x32mp (time: 100 ms)
59.00 Rybka 1.0 Beta 32-bit (time: 100 ms)
57.45 Rybka 2.2n2 mp 32-bit (time: 100 ms)
57.20 Booot 5.1.0 (time: 100 ms)
56.55 Critter 0.90 32-bit (time: 100 ms)
56.45 Komodo32 1.3 JA (time: 100 ms)
56.40 Stockfish 1.9.1 JA (time: 100 ms)
56.10 Fire 1.31 w32 (time: 100 ms)
54.40 Jonny 4.00 (time: 100 ms)
52.45 Houdini 1.5 w32 (time: 100 ms)

Umko, Equinox and Rybka 1.0 Beta are very fruity

Best
Johan

[Hood]

Hi,

may be a little bit off topic but comparasion conclusions based on statistics... ?

12% of accidents is done by drunk drivers, so ... 88% is made but not drunk

what is the conclusion ?

[Don]

Hi,

may be a little bit off topic but comparasion conclusions based on statistics... ?

12% of accidents is done by drunk drivers, so ... 88% is made but not drunk

what is the conclusion ?

You cannot draw a conclusion at all based on your example unless you know what percentage of the drivers are drunk and don't get into accidents. If 50% of the drivers are drunk, then you are safer if you get drunk before driving. If only 5% of drivers on the road are drunk, those are not good statistics. You also need to know something about the size of the sample.

I think you missed most of my posts that advised against drawing conclusions based solely on this one test. I explained exactly how the test works, and how many positions are tested. Beyond this it's up to each person to determine if the test is highly useful, slightly useful, or meaningless. You decide.

[Hood]

yes but you do not know which engine is 'drunk' ?

it is a black box with input and output.

[Don]

yes but you do not know which engine is 'drunk' ?

it is a black box with input and output.

What point are you trying to make that I have not already made?

[Hood]

There are positions in chess where only one - the strongest move exists (1) and there are the positions where are existing best moves with equal strength(2).

(1) if the engines are selecting the same best move it proves that they are similar in strength not in algorithm
to use clone tester you have to exclude such positions but for that you have to solve chess to know which move is the best

(2) if engines are selecting the same move it shows at first the equality in strength
programms can choose the same move on the differrent way so it is not prove either.

The code analysis is the only prove of the 'clone'.
Your method can only show that sth is clone of sth with some probability,
i do not know which is the factor of it - 30% but it is still not the prove.

Rgds
Hood

[Don]

There are positions in chess where only one - the strongest move exists (1) and there are the positions where are existing best moves with equal strength(2).

(1) if the engines are selecting the same best move it proves that they are similar in strength not in algorithm
to use clone tester you have to exclude such positions but for that you have to solve chess to know which move is the best

First of all, this tool is not a clone tester.

And it's already been proved that this test is relatively immune to strength. Do you remember that I tried increasing one of the programs 4x (which adds hundreds of ELO at this level) and it did not "defeat" the tester - it still thought the program was a very close match to itself and other versions of itself.

People keep coming back to this point over and over again, but it's already been show that this is not the case. Don't take my work for it, run the test! See if you can make spike look like robbolitto by running it 20x longer.

(2) if engines are selecting the same move it shows at first the equality in strength
programms can choose the same move on the differrent way so it is not prove either.

Who said it was supposed to be a proof? You are getting really annoying because you keep trying to argue against a point nobody is trying to make.

The code analysis is the only prove of the 'clone'.

Not even that is proof because different people require different thresholds of proof to decide whether it's a clone. For example to some people Rybka is an exact clone of Fruit because it has similar user interface code, but for others if even 1 byte is different they would not consider it a clone.

Your method can only show that sth is clone of sth with some probability,
i do not know which is the factor of it - 30% but it is still not the prove.

My method cannot even show that. You are not reading anything I am writing are you?

I do not even know for sure what my test shows other than how often 2 programs make the same move! That is what it shows and how you interpret that is up to you. All it does is measure something imperfectly - what it measures is not completely clear.

I do however have my own theories. I believe that it cannot tell you much about the search of the program, but it can tell you a lot about the evaluation function and playing style of a program. It doesn't take a rocket scientist to know that if two program play a lot of the same moves, they probably have a similar playing style.

Rgds
Hood

[Laskos]

There are positions in chess where only one - the strongest move exists (1) and there are the positions where are existing best moves with equal strength(2).

(1) if the engines are selecting the same best move it proves that they are similar in strength not in algorithm
to use clone tester you have to exclude such positions but for that you have to solve chess to know which move is the best

(2) if engines are selecting the same move it shows at first the equality in strength
programms can choose the same move on the differrent way so it is not prove either.

The code analysis is the only prove of the 'clone'.
Your method can only show that sth is clone of sth with some probability,
i do not know which is the factor of it - 30% but it is still not the prove.

Rgds
Hood

Try it to see, humans are good clusterers. The correlation with strength is there, but one can get rid of this. I put the programs at the same strength by adjusting time/position, the results are still clustering.

Code: Select all

C:\similar>similar -r 4
------ Houdini 1.5 x64 (time: 100 ms) ------
 73.40  Houdini 1.51x64 (time: 100 ms)
 71.65  Houdini 1.03a x64 2_CPU (time: 140 ms)
 70.20  RobboLito 0.09 x64 (time: 320 ms)
 69.70  IPPOLIT 0.080a  (time: 480 ms)
 68.80  Ivanhoe_B49jAx64 (time: 165 ms)
 64.35  Rybka 3  (time: 290 ms)
 63.65  Deep Rybka _4_x64 (time: 165 ms)
 58.85  Deep Shredder 12 x64 (time: 750 ms)


C:\similar>similar -r 6
------ IPPOLIT 0.080a  (time: 480 ms) ------
 73.75  RobboLito 0.09 x64 (time: 320 ms)
 73.50  Ivanhoe_B49jAx64 (time: 165 ms)
 73.15  Houdini 1.03a x64 2_CPU (time: 140 ms)
 69.70  Houdini 1.5 x64 (time: 100 ms)
 69.00  Houdini 1.51x64 (time: 100 ms)
 68.75  Rybka 3  (time: 290 ms)
 66.45  Deep Rybka _4_x64 (time: 165 ms)
 62.30  Deep Shredder 12 x64 (time: 750 ms)


C:\similar>similar -r 3
------ Rybka 3  (time: 290 ms) ------
 70.15  RobboLito 0.09 x64 (time: 320 ms)
 68.75  IPPOLIT 0.080a  (time: 480 ms)
 68.10  Deep Rybka _4_x64 (time: 165 ms)
 68.00  Ivanhoe_B49jAx64 (time: 165 ms)
 67.95  Houdini 1.03a x64 2_CPU (time: 140 ms)
 66.25  Houdini 1.51x64 (time: 100 ms)
 64.35  Houdini 1.5 x64 (time: 100 ms)
 61.20  Deep Shredder 12 x64 (time: 750 ms)


C:\similar>similar -r 7
------ Deep Shredder 12 x64 (time: 750 ms) ------
 62.30  IPPOLIT 0.080a  (time: 480 ms)
 62.30  Houdini 1.03a x64 2_CPU (time: 140 ms)
 61.45  RobboLito 0.09 x64 (time: 320 ms)
 61.30  Deep Rybka _4_x64 (time: 165 ms)
 61.20  Rybka 3  (time: 290 ms)
 60.55  Ivanhoe_B49jAx64 (time: 165 ms)
 59.85  Houdini 1.51x64 (time: 100 ms)
 58.85  Houdini 1.5 x64 (time: 100 ms)

Houdini 1.51 is a pure copy of Houdini 1.5, I put it to set the upper boundary of these pretty non-deterministic engines. My problem is mostly statistical, 10,000 or so positions are needed at least, and in no case the claim about cloning could be made more than circumstantially. More like a property of some engines.

Kai

ps Second problem is that some engines refuse to pass properly the test, either halting or giving stupid results.

[Don]

Houdini 1.51 is a pure copy of Houdini 1.5, I put it to set the upper boundary of these pretty non-deterministic engines. My problem is mostly statistical, 10,000 or so positions are needed at least, and in no case the claim about cloning could be made more than circumstantially. More like a property of some engines.

I actually have a version with 10,000 positions. I recently removed almost 2000 of them due to the fact that EVERY engine played the same move (out of 18 engines I tested)

I agree with you about the circumstantial evidence. The only thing this test can show you is how much stylistic similarity there is between two programs.

In fact, I think the test tells you mostly about the evaluation function of the two programs, and hardly anything about any other part of the program. However it's my belief that this is the most important and difficult part of a chess program and what separates the men from the boys.