Ethereal's 66.64% and RubiChess's 64.42% similarity with Stockfish 12
Moderators: hgm, Rebel, chrisw
-
- Posts: 30
- Joined: Thu Sep 17, 2020 9:19 pm
- Full name: William Morton
Ethereal's 66.64% and RubiChess's 64.42% similarity with Stockfish 12
I've read in the Prodeo forum the Ethereal (66.64%) and RubiChess (64.42%) similarity (1s test) with Stockfish 12. Rebel says that in his opinion the high similarity is due to the playing strength but RubiChess is about 350/400 points weaker than SF 12. what am I missing here? what's the result for Komodo Dragon?
-
- Posts: 1752
- Joined: Tue Apr 19, 2016 6:08 am
- Location: U.S.A
- Full name: Andrew Grant
Re: Ethereal's 66.64% and RubiChess's 64.42% with Stockfish 12
Probably not missing anything. Simtests are sometimes a good metric to say something might be up, sometimes not. A while back I pushed a patch which started pruning root nodes at depth 1. As a result, Ethereal's similarity tanked from 60%+ to ~30%. Not because Ethereal changed greatly, but because the sole metric that the Simtest was comparing was depth 1 search results. I would question any high similarity between engines, and take further inspection. And at the same time I would not see a low similarity is proof of a lack of cloning.
Its interesting to note the few clones that have been proven lately have had high sim scores with the cloned engine. Namely Fire has very high sim scores against Stockfish's earlier versions. Additionally, Houdini had moderate sim scores with both Stockfish and Komodo. Komodo authors claim that some of Houdini's eval features were reverse engineered from Komodo. But maybe this is a red herring; humans see patterns if they want to I suppose.
I don't have the data on hand, but are there simtests comparing early Fire versions with Gull and Roc? Would be interesting to see if the pattern holds with more examples. As an aside: Were those sims using NNUE for all 3 engines? Or maybe for Rubichess? I imagine NNUE would have high similarity at low depths,
Its interesting to note the few clones that have been proven lately have had high sim scores with the cloned engine. Namely Fire has very high sim scores against Stockfish's earlier versions. Additionally, Houdini had moderate sim scores with both Stockfish and Komodo. Komodo authors claim that some of Houdini's eval features were reverse engineered from Komodo. But maybe this is a red herring; humans see patterns if they want to I suppose.
I don't have the data on hand, but are there simtests comparing early Fire versions with Gull and Roc? Would be interesting to see if the pattern holds with more examples. As an aside: Were those sims using NNUE for all 3 engines? Or maybe for Rubichess? I imagine NNUE would have high similarity at low depths,
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
-
- Posts: 4605
- Joined: Wed Oct 01, 2008 6:33 am
- Location: Regensburg, Germany
- Full name: Guenther Simon
Re: Ethereal's 66.64% and RubiChess's 64.42% with Stockfish 12
You are missing the time control for the simtest. Ed seems a bit bored and frustrated about NNUE non-similarities...willmorton wrote: ↑Sat Mar 06, 2021 12:07 pm I've read in the Prodeo forum the Ethereal (66.64%) and RubiChess (64.42%) similarity (1s test) with Stockfish 12. Rebel says that in his opinion the high similarity is due to the playing strength but RubiChess is about 350/400 points weaker than SF 12. what am I missing here? what's the result for Komodo Dragon?
As nothing helped so far, he tried testing at much higher tc the simtest was meant for.
(higher tc means closer to same moves, if programs have reached a certain strengh and programs are much stronger then 10 years ago, the arrival of the simtest - much stronger hardware multiplies this effect)
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Ethereal's 66.64% and RubiChess's 64.42% similarity with Stockfish 12
You forgot to mention that Ethereal and Rubichess were tested with the NNUE of SF12. The research is about if the sim-test still makes sense with NNUE engines. And the conclusion is that there is no sense, for NNUE engines the sim-test is dead.willmorton wrote: ↑Sat Mar 06, 2021 12:07 pm I've read in the Prodeo forum the Ethereal (66.64%) and RubiChess (64.42%) similarity (1s test) with Stockfish 12. Rebel says that in his opinion the high similarity is due to the playing strength but RubiChess is about 350/400 points weaker than SF 12. what am I missing here? what's the result for Komodo Dragon?
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 584
- Joined: Fri Mar 30, 2018 7:20 am
- Full name: Andreas Matthies
Re: Ethereal's 66.64% and RubiChess's 64.42% similarity with Stockfish 12
Thank you for putting RubiChess (and Ethereal) in a headline together with "Similarity" and "Stockfish" not even giving a link where you found this "information" (meanwhile I found the thread myself).willmorton wrote: ↑Sat Mar 06, 2021 12:07 pm I've read in the Prodeo forum the Ethereal (66.64%) and RubiChess (64.42%) similarity (1s test) with Stockfish 12. Rebel says that in his opinion the high similarity is due to the playing strength but RubiChess is about 350/400 points weaker than SF 12. what am I missing here? what's the result for Komodo Dragon?
@Rebel: Thank you (without sarcasm this time) for clarification that this was NOT RubiChess but RubiChess playing a Stockfish net.
-
- Posts: 81
- Joined: Tue Oct 29, 2019 3:20 am
- Full name: Anthony Wilson
Re: Ethereal's 66.64% and RubiChess's 64.42% similarity with Stockfish 12
The guy obviously can't read but feeding different engines the same NNUE means you are giving them the same eval and thus testing their search similarity, doesn't it? Have you tested other engines with SF-NNUE?
I've found the thread as well: Nemorino 59.63%