SIM new version

[Rebel]

Recently (after 5 years) I tested the current top-engines on similarity (once again) because I wanted to know 2 things:

1. if the crazy days of the Fruit cloning (2005-2010) and Rybka3 cloning (2010-2015) were over and how the situation anno 2019 is now that Stockfish is the ideal target. And so I ran SIM3 from Don Daily checking the CCRL TOP-10. And the results were amazingly good, as it seems computer chess healed itself.

#2. I also wanted to know something else, since the immense elo progress that have been made the last 5 years, is it not likely that the top-engines increase in similarity? And if that is the case then: how much. In other words, a test if SIM3 is still a reliable tool. And looking at the situation of 2019 the numbers are looking good, the top-engines nicely curdle between 45-50%. So far so good.

And then something somewhat strange happened, as last 3 engines I added the latest versions of Fire, Booot and Fizbo to the test and Fizbo showed an incredible low similarity of 24% average. That's not necessarily wrong but a bit strange it is for 3300+ elo rated engine.

And I decided to have a closer look, long story short, I found a small (but mean) bug in SIM3. In Fail-high situations it not always recorded the right move. I fixed the bug and ran the test again and the result is remarkable, softly speaking that is, view the comparison between SIM3 and SIM4. Suddenly according to the legenda Xiphos with 64% is a clone / derivative of Stockfish, even Komodo with 61% is in the red zone. High similarity between the top-engines. And of course they are not clones / derivatives of each other. Several are open source, easy to compare.

As next step I ran several engines from the Rybka3 cloning (2010-2015) period to see how they would perform with the corrected version. And suddenly they are no clones / derivatives any longer! View the comparison, similarity is about in the same range as today.

This begs the question if programmers have been wrongly accused of cloning in the past because of a bug in SIM3 ??

Page including links and download - http://rebel13.nl/home/sim.html

[Ferdy]

Interesting discovery Ed. Lets try it.

[carldaman]

Hmm, and the calendar says what?

[Sergio Martinez]

Very very interesting!

Thanks Ed.

[Ferdy]

This begs the question if programmers have been wrongly accused of cloning in the past because of a bug in SIM3 ??

Page including links and download - http://rebel13.nl/home/sim.html

I tried to log sim4.1 output here is the difference to my analyzer.
1. sim4.1 does not send ucinewgame command or restart the engine. In my analyzer engine is restarted after every position has been analyzed.

2. sim4.1 sends go depth <depth> and stop commands, this has a drawback, it might happen that after sending stop, some engines may extend more time than others and you save the bestmove sent by engine. In my analyzer I use go movetime <timems> then save the bestmove sent by engine. This is more reliable as I am not interrupting the search of the engine. But this would only work for those engines that complies the movetime command which the majority of strong engines implemented correctly. go movetime does not work for Fizbo, so I use go infinite then send stop if analysis time is reached. The way I collect the bestmove is thru the pv sent by the engine. That pv line should not have upperbound and lowerbound text in it. After the stop command the engine would send bestmove bm, but I will not use that bm, I use the first move in the pv at a time that I sent the stop command, to comply with the analysis time specified by the user.

Perhaps these differences may affect the overall results of the tests.

Using sim4.1 here are my results.

Code: Select all

sim version 4.1

  Key:

  1) Booot 6.3.1 (time: 200 ms  scale: 1.0)
  2) Deuterium v2019.2.37.37 (time: 200 ms  scale: 1.0)
  3) Ethereal 11.25 (time: 200 ms  scale: 1.0)
  4) Fire 7.1 (time: 200 ms  scale: 1.0)
  5) Laser 1.7 (time: 200 ms  scale: 1.0)
  6) Stockfish 10 (time: 20 ms  scale: 1.0)
  7) Stockfish 10 (time: 200 ms  scale: 1.0)
  8) Stockfish 10 r1 (time: 200 ms  scale: 1.0)
  9) Xiphos 0.5 (time: 200 ms  scale: 1.0)

         1     2     3     4     5     6     7     8     9
  1.  ----- 36.87 36.39 35.86 36.00 37.16 35.08 34.69 35.52
  2.  36.87 ----- 48.62 46.26 48.48 47.52 47.95 47.40 48.36
  3.  36.39 48.62 ----- 47.67 50.96 48.08 51.66 51.76 52.82
  4.  35.86 46.26 47.67 ----- 47.21 46.53 49.20 48.85 47.37
  5.  36.00 48.48 50.96 47.21 ----- 47.75 51.03 50.39 50.06
  6.  37.16 47.52 48.08 46.53 47.75 ----- 53.92 53.48 48.28
  7.  35.08 47.95 51.66 49.20 51.03 53.92 ----- 64.53 52.88
  8.  34.69 47.40 51.76 48.85 50.39 53.48 64.53 ----- 52.71
  9.  35.52 48.36 52.82 47.37 50.06 48.28 52.88 52.71 -----

I ran Stockfish 10 twice at 200ms to establish a base line. This should be the highest similarity at 64.53.
1. Stockfish 10
2. Stockfish 10 r1

Result:

Code: Select all

sim version 4.1
------ Stockfish 10 (time: 200 ms  scale: 1.0) ------
 64.53  Stockfish 10 r1 (time: 200 ms  scale: 1.0)
 53.92  Stockfish 10 (time: 20 ms  scale: 1.0)
 52.88  Xiphos 0.5 (time: 200 ms  scale: 1.0)
 51.66  Ethereal 11.25 (time: 200 ms  scale: 1.0)
 51.03  Laser 1.7 (time: 200 ms  scale: 1.0)
 49.20  Fire 7.1 (time: 200 ms  scale: 1.0)
 47.95  Deuterium v2019.2.37.37 (time: 200 ms  scale: 1.0)
 35.08  Booot 6.3.1 (time: 200 ms  scale: 1.0)

Now with this sim version we get the highest similarity of 64.53 as 100% clone, on my computer with that given analysis time and 1 thread per engine with 128 MB hash.

[cdani]

... Several are open source, easy to compare...

Even if the engine sources are different, they use more and more the same known to work techniques. I expect them to be more and more equal in the best lines that they find. So they are not clones or derivatives per se, but are partial derivatives in term of the ideas they use. Andscacs also, of course, even if I try to do everything in my own way.

[Sergio Martinez]

Hi Ed

Is there any possibility to edit the positions of the tool and add or remove positions?

Thank you

[Rebel]

Hi Ed

Is there any possibility to edit the positions of the tool and add or remove positions?

Thank you

Yes, but you need a special version of SIM3 including the source code. I am not allowed to share it.

[Sergio Martinez]

Hi Ed

Is there any possibility to edit the positions of the tool and add or remove positions?

Thank you

Yes, but you need a special version of SIM3 including the source code. I am not allowed to share it.

Ok.

Thanks.