chrisw wrote: ↑Tue Sep 03, 2019 1:26 am
Looking at your ccrl.html, massive cross table of engine results on the SimTester, I’m wondering what the similarity numbers mean. This is a list of low similarity programs in general, but one or two things jump out.
Booot6, very consistent results right across the board, it gets 25% to 30% similarity against just about all other programs.
Hiarcs14, likewise, very consistent, 30% to 35% similarity, right across the board, against everything.
They both match with other programs, maybe 1 in 3, or 1 in 4 positions, choosing the same move. Can we assume this is how genuinely independent engines behave? It’s baseline behaviour, some moves are the same, after all, these are relatively usual type chess positions, and you expect the choice of candidate moves to be not so wide.
Then there are other engines, usually scoring sim values of around 30% or so, but then, in the crosstable with Stockfish, start to move up to 50% similarity, now they choose 1 move in 2 the same. What does this mean, this move to 50%? It’s not an uncommon trend. Find your own examples.
How to disentangle all the information in that great mass of numbers ....
I hate long forum postings but I am afraid this is going to be long in order to address your points and questions properly.
Computer chess has a long history of people cloning open source code, making changes and call it their own intellectual property often a breach of copyright, GPL, EULA.
It started already in the 90's when Bob Hyatt made his Crafty open source, several programmers were exposed breaching the EULA of Crafty. And it was not always easy to detect clones, to proof a derivative work one had to look at the disassembled code.
Things got worse when in 2005 Fabien's Fruit 2.1 came out under GPL and contrary to the popular Crafty Fruit was a top engine. And many couldn't resist the temptation and Fruit 2.1 became the source of many clones and/or derivatives breaching the GPL, some even commercialized their derivative work. And it was most of time hard to unmask them as such. On the other hand there were also honest programmers who obeyed the Fruit GPL and made wonderful successors, giving credit to Fabien and releasing the source code.
A few years later a guy named Yuri Osipov did something very remarkable and disturbing at the same time. He disassembled the then leading program Rybka 1.0 and made readable source code from it, called it STRELKA and wanted to sell it to the Chess Assistant folks for distribution.
The same thing happened to Rybka 3 (in 2009), disassembled readable source code which became known as Ippolit, later Robbolito and the chess community was flooded with Rybka 3 derivatives, commercial ones included. It was a total mess.
But help was underway, about the same time (2010 if I remember correctly) Don Dailey released his SIM03 program which measured the similarity between engines by letting each engine ponder about 8238 positions and count equal moves resulting in a similarity percentage. Brilliancy is always simple.
Arrived at this point in time many engines already were exposed as derivative works via the hard way (disassembly or programmer confessions) and SIM03 just confirmed it! No more disassembling needed.
And a number of rightful questions came up such as:
1. By which
similarity percentage marker an engine is suspect to be a derivative work?
2. How reliable is the tool, can it for instance produce a false positive?
3. When an engine scores below the
similarity percentage marker does this mean the engine is an original one?
On (1), there was a big discussion at the time, some wanted the marker on 55%, others on 65%, but the majority was in favor for 60%, a close general consensus.
On (2), After 10 years no false positive has shown up. There was a discussion that a Fritz 11 version (the standard version or the deep version) produced a false positive because it showed a high similarity percentage with the CCRL ponder hit statistics and SIM03 became under pressure. As it later turned out that particular Fritz version was indeed producing an exceptional high similarity percentage by extracting the 8238 positions from SIM03 to PGN and then running them under Fritz. The image of SIM03 was safe again.
On(3), similarity testing can only proof suspect engines, never disproof. An engine can have a low similarity and can still be a derivative work without not so much elo loss.
90% of coding is debugging, the other 10% is writing bugs.