SIMEX 2.1

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Rebel
Posts: 4788
Joined: Thu Aug 18, 2011 10:04 am

Re: SIMEX 2.1

Post by Rebel » Thu Sep 05, 2019 9:16 am

After inspection I made some changes to the "ccrl.htm", removed the following engines:

Laser - doesn't support the UCI depth command.
Fizbo - doesn't support the UCI depth command.
Protector - doesn't support the UCI depth command.

Unrealistic low numbers
Smarthink - can not check it since the data came from an external source.
Pedone 1.8 - looking at 1.7 and 1.8 the number is too weird to make sense for similarity testing.
Senpai 2 - same as Pedone 1.8

Booot and RofChade. Both produce indentical weird EPD output.

Code: Select all

Booot
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm Nbd2; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bd4; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Rxd4; ce -32000; acd 0;

Code: Select all

RofChade
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm d5; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bxh6; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Re2; ce -32000; acd 0;
Add (the last version) of ProDeo.

The new list - http://rebel13.nl/html/ccrl.html
90% of coding is debugging, the other 10% is writing bugs.

User avatar
Ovyron
Posts: 2833
Joined: Tue Jul 03, 2007 2:30 am

Re: SIMEX 2.1

Post by Ovyron » Thu Sep 05, 2019 10:20 am

Dann Corbit wrote:
Tue Sep 03, 2019 7:10 pm
To me, I see people trying to draw a line. But I do not clearly understand exactly what the line is supposed to mean.
The line can't be drawn.

Vasik Rajlich once said around the time Rybka 4 was released, that he could trivially produce some 20 engines that played near the strength of Rybka, but that played completely different from Rybka. He even added settings to the engine so people could change its material evaluation and do it themselves, and people like Mindbreaker managed to create settings stronger than default.

What this means is that anybody capable of improving some source code by someone else (like Robert Houdart improving Ippolit to Houdini 6 levels) is also capable of producing a stronger engine and then make changes so that its similarity with the source engine is arbitrarily low.

If no matter where you draw the line, someone can close Stockfish's source and sell it while fooling everyone because they can produce an engine that plays nothing like Stockfish, drawing the line is useless.
Great spirits have always encountered violent opposition from mediocre minds.

chrisw
Posts: 2206
Joined: Tue Apr 03, 2012 2:28 pm

Re: SIMEX 2.1

Post by chrisw » Thu Sep 05, 2019 12:00 pm

Ovyron wrote:
Thu Sep 05, 2019 10:20 am
Dann Corbit wrote:
Tue Sep 03, 2019 7:10 pm
To me, I see people trying to draw a line. But I do not clearly understand exactly what the line is supposed to mean.
The line can't be drawn.

Vasik Rajlich once said around the time Rybka 4 was released, that he could trivially produce some 20 engines that played near the strength of Rybka, but that played completely different from Rybka. He even added settings to the engine so people could change its material evaluation and do it themselves, and people like Mindbreaker managed to create settings stronger than default.

What this means is that anybody capable of improving some source code by someone else (like Robert Houdart improving Ippolit to Houdini 6 levels) is also capable of producing a stronger engine and then make changes so that its similarity with the source engine is arbitrarily low.
Well, let’s wait and see what the data has to say about that.

If no matter where you draw the line, someone can close Stockfish's source and sell it while fooling everyone because they can produce an engine that plays nothing like Stockfish, drawing the line is useless.

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: SIMEX 2.1

Post by Ferdy » Fri Sep 06, 2019 7:50 am

Rebel wrote:
Thu Sep 05, 2019 9:16 am
Booot and RofChade. Both produce indentical weird EPD output.

Code: Select all

Booot
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm Nbd2; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bd4; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Rxd4; ce -32000; acd 0;

Code: Select all

RofChade
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm d5; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bxh6; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Re2; ce -32000; acd 0;
The ce -32000 and acd 0 mean that the engine did not send score and depth info. Default score and depth are used instead.

User avatar
Rebel
Posts: 4788
Joined: Thu Aug 18, 2011 10:04 am

Re: SIMEX 2.1

Post by Rebel » Sat Sep 07, 2019 9:38 am

Ferdy wrote:
Fri Sep 06, 2019 7:50 am
Rebel wrote:
Thu Sep 05, 2019 9:16 am
Booot and RofChade. Both produce indentical weird EPD output.

Code: Select all

Booot
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm Nbd2; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bd4; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Rxd4; ce -32000; acd 0;

Code: Select all

RofChade
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm d5; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bxh6; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Re2; ce -32000; acd 0;
The ce -32000 and acd 0 mean that the engine did not send score and depth info. Default score and depth are used instead.
Yep, I noticed... those are engines that hide their mainlines for the first x plies. There are also a lot of engines that don't support the UCI depth command. I will make a list of those in the final report.
90% of coding is debugging, the other 10% is writing bugs.

chrisw
Posts: 2206
Joined: Tue Apr 03, 2012 2:28 pm

Re: SIMEX 2.1

Post by chrisw » Sun Sep 08, 2019 4:00 pm

Rebel wrote:
Thu Sep 05, 2019 9:16 am
After inspection I made some changes to the "ccrl.htm", removed the following engines:

Laser - doesn't support the UCI depth command.
Fizbo - doesn't support the UCI depth command.
Protector - doesn't support the UCI depth command.

Unrealistic low numbers
Smarthink - can not check it since the data came from an external source.
Pedone 1.8 - looking at 1.7 and 1.8 the number is too weird to make sense for similarity testing.
Senpai 2 - same as Pedone 1.8

Booot and RofChade. Both produce indentical weird EPD output.

Code: Select all

Booot
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm Nbd2; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bd4; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Rxd4; ce -32000; acd 0;

Code: Select all

RofChade
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm d5; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bxh6; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Re2; ce -32000; acd 0;
Add (the last version) of ProDeo.

The new list - http://rebel13.nl/html/ccrl.html
Quick piece of data mining on that table.
Lowest similarity is Hiarcs1 vs Merv at 29%
if you run a search on that page for '29.', '30.', '31.', '32.', '33.', '34.' you'll note the entire set is owned by Hiarcs14. No other engine gets that 'different' to any other engine until 35% similarity, when quite a few other engines kick in.
Meanwhile, the highest similarity Hiarcs 14 scores is only 37%.

This suggests to me that Hiarcs14 is 'different' (not-similar) in a different way to all the others.

From a NN Simex chart I saw recently, the 'difference' behaviour is similar, all 'NN-similarities' with AB engines are in the range 28-34%, quite similar profile to Hiarcs14.

Looking at other engines in general, it's rare for the minimum similarity to be less than 40%, and of course, the maximum similarity is stratospheric in some case.

Isn't that interesting?!

User avatar
Rebel
Posts: 4788
Joined: Thu Aug 18, 2011 10:04 am

Re: SIMEX 2.1

Post by Rebel » Sun Sep 08, 2019 6:54 pm

I think Hiarcs, Gandalf, ProDeo regarding the evaluation belong to the old-class generation before the open source revolution of Fruit, Strelka, Robbolito and currently Stockfish began and were not influenced, hence low similarity.

I added a few oldies, Ruffian, Hiarcs 13 and Shredder 8.

Seems to confirm my line of thought.

http://rebel13.nl/html/ccrl.html

On NN engines you may notice that when running on time instead of depth=1 similarity normalizes.

http://rebel13.nl/html/lc0-1000ms.html
90% of coding is debugging, the other 10% is writing bugs.

Post Reply