Recently (after 5 years) I tested the current top-engines on similarity (once again) because I wanted to know 2 things:
1. if the crazy days of the Fruit cloning (2005-2010) and Rybka3 cloning (2010-2015) were over and how the situation anno 2019 is now that Stockfish is the ideal target. And so I ran SIM3 from Don Daily checking the CCRL TOP-10. And the results were amazingly good, as it seems computer chess healed itself.
#2. I also wanted to know something else, since the immense elo progress that have been made the last 5 years, is it not likely that the top-engines increase in similarity? And if that is the case then: how much. In other words, a test if SIM3 is still a reliable tool. And looking at the situation of 2019 the numbers are looking good, the top-engines nicely curdle between 45-50%. So far so good.
And then something somewhat strange happened, as last 3 engines I added the latest versions of Fire, Booot and Fizbo to the test and Fizbo showed an incredible low similarity of 24% average. That's not necessarily wrong but a bit strange it is for 3300+ elo rated engine.
And I decided to have a closer look, long story short, I found a small (but mean) bug in SIM3. In Fail-high situations it not always recorded the right move. I fixed the bug and ran the test again and the result is remarkable, softly speaking that is, view the comparison between SIM3 and SIM4. Suddenly according to the legenda Xiphos with 64% is a clone / derivative of Stockfish, even Komodo with 61% is in the red zone. High similarity between the top-engines. And of course they are not clones / derivatives of each other. Several are open source, easy to compare.
As next step I ran several engines from the Rybka3 cloning (2010-2015) period to see how they would perform with the corrected version. And suddenly they are no clones / derivatives any longer! View the comparison, similarity is about in the same range as today.
This begs the question if programmers have been wrongly accused of cloning in the past because of a bug in SIM3 ??
Page including links and download - http://rebel13.nl/home/sim.html
SIM new version
Moderators: hgm, Rebel, chrisw
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
SIM new version
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: SIM new version
Interesting discovery Ed. Lets try it.
-
- Posts: 2283
- Joined: Sat Jun 02, 2012 2:13 am
-
- Posts: 1975
- Joined: Fri Oct 05, 2012 2:35 pm
- Location: Spain
Re: SIM new version
Very very interesting!
Thanks Ed.
Thanks Ed.
Member of the CCRL Group. Write me if you want I test your engine.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: SIM new version
I tried to log sim4.1 output here is the difference to my analyzer.Rebel wrote: ↑Mon Apr 01, 2019 6:53 am This begs the question if programmers have been wrongly accused of cloning in the past because of a bug in SIM3 ??
Page including links and download - http://rebel13.nl/home/sim.html
1. sim4.1 does not send ucinewgame command or restart the engine. In my analyzer engine is restarted after every position has been analyzed.
2. sim4.1 sends go depth <depth> and stop commands, this has a drawback, it might happen that after sending stop, some engines may extend more time than others and you save the bestmove sent by engine. In my analyzer I use go movetime <timems> then save the bestmove sent by engine. This is more reliable as I am not interrupting the search of the engine. But this would only work for those engines that complies the movetime command which the majority of strong engines implemented correctly. go movetime does not work for Fizbo, so I use go infinite then send stop if analysis time is reached. The way I collect the bestmove is thru the pv sent by the engine. That pv line should not have upperbound and lowerbound text in it. After the stop command the engine would send bestmove bm, but I will not use that bm, I use the first move in the pv at a time that I sent the stop command, to comply with the analysis time specified by the user.
Perhaps these differences may affect the overall results of the tests.
Using sim4.1 here are my results.
Code: Select all
sim version 4.1
Key:
1) Booot 6.3.1 (time: 200 ms scale: 1.0)
2) Deuterium v2019.2.37.37 (time: 200 ms scale: 1.0)
3) Ethereal 11.25 (time: 200 ms scale: 1.0)
4) Fire 7.1 (time: 200 ms scale: 1.0)
5) Laser 1.7 (time: 200 ms scale: 1.0)
6) Stockfish 10 (time: 20 ms scale: 1.0)
7) Stockfish 10 (time: 200 ms scale: 1.0)
8) Stockfish 10 r1 (time: 200 ms scale: 1.0)
9) Xiphos 0.5 (time: 200 ms scale: 1.0)
1 2 3 4 5 6 7 8 9
1. ----- 36.87 36.39 35.86 36.00 37.16 35.08 34.69 35.52
2. 36.87 ----- 48.62 46.26 48.48 47.52 47.95 47.40 48.36
3. 36.39 48.62 ----- 47.67 50.96 48.08 51.66 51.76 52.82
4. 35.86 46.26 47.67 ----- 47.21 46.53 49.20 48.85 47.37
5. 36.00 48.48 50.96 47.21 ----- 47.75 51.03 50.39 50.06
6. 37.16 47.52 48.08 46.53 47.75 ----- 53.92 53.48 48.28
7. 35.08 47.95 51.66 49.20 51.03 53.92 ----- 64.53 52.88
8. 34.69 47.40 51.76 48.85 50.39 53.48 64.53 ----- 52.71
9. 35.52 48.36 52.82 47.37 50.06 48.28 52.88 52.71 -----
1. Stockfish 10
2. Stockfish 10 r1
Result:
Code: Select all
sim version 4.1
------ Stockfish 10 (time: 200 ms scale: 1.0) ------
64.53 Stockfish 10 r1 (time: 200 ms scale: 1.0)
53.92 Stockfish 10 (time: 20 ms scale: 1.0)
52.88 Xiphos 0.5 (time: 200 ms scale: 1.0)
51.66 Ethereal 11.25 (time: 200 ms scale: 1.0)
51.03 Laser 1.7 (time: 200 ms scale: 1.0)
49.20 Fire 7.1 (time: 200 ms scale: 1.0)
47.95 Deuterium v2019.2.37.37 (time: 200 ms scale: 1.0)
35.08 Booot 6.3.1 (time: 200 ms scale: 1.0)
-
- Posts: 2204
- Joined: Sat Jan 18, 2014 10:24 am
- Location: Andorra
Re: SIM new version
Even if the engine sources are different, they use more and more the same known to work techniques. I expect them to be more and more equal in the best lines that they find. So they are not clones or derivatives per se, but are partial derivatives in term of the ideas they use. Andscacs also, of course, even if I try to do everything in my own way.
Daniel José - http://www.andscacs.com
-
- Posts: 1975
- Joined: Fri Oct 05, 2012 2:35 pm
- Location: Spain
Re: SIM new version
Hi Ed
Is there any possibility to edit the positions of the tool and add or remove positions?
Thank you
Is there any possibility to edit the positions of the tool and add or remove positions?
Thank you
Member of the CCRL Group. Write me if you want I test your engine.
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Re: SIM new version
Yes, but you need a special version of SIM3 including the source code. I am not allowed to share it.Sergio Martinez wrote: ↑Tue Apr 02, 2019 9:03 am Hi Ed
Is there any possibility to edit the positions of the tool and add or remove positions?
Thank you
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 1975
- Joined: Fri Oct 05, 2012 2:35 pm
- Location: Spain
Re: SIM new version
Ok.Rebel wrote: ↑Tue Apr 02, 2019 11:32 amYes, but you need a special version of SIM3 including the source code. I am not allowed to share it.Sergio Martinez wrote: ↑Tue Apr 02, 2019 9:03 am Hi Ed
Is there any possibility to edit the positions of the tool and add or remove positions?
Thank you
Thanks.
Member of the CCRL Group. Write me if you want I test your engine.