Page 1 of 4

Similarity tester - 2nd generation - BETA

Posted: Fri Aug 09, 2019 11:32 pm
by Rebel
I am working on the documentation of SIMEX 2.0 which is meant to be the successor of Don Dailey's famous SIM03 Similarity Tester.

SIMEX works the same way but is more user-friendly, has more features but the main advantage is that you are no longer limited to the build-in 8238 positions of SIM03 but can create your own positions using EPD, in the download 7 EPD sets are provided for demonstration purposes. SIMEX uses MEA from Ferdinand Mosca as a base.

http://rebel13.nl/misc/simex.html

The documentation is not even half-way and by this thread I take the opportunity to receive comments, ideas before an official release. The page will be frequently updated the coming days.

Re: Similarity tester - 2nd generation - BETA

Posted: Sat Aug 10, 2019 5:57 pm
by Ferdy
Is simex2 able to generate a matrix similar to sim03?
In sim03 I just use the -m option to generate the following.
Rename midgame.data to similarity.data then run

Code: Select all

sim03w32.exe -m >matrix.txt
matrix.txt

Code: Select all

sim version 3

  Key:

  1) Andscacs_0.93
  2) brainfish_2018
  3) D2019.1.36.50
  4) D2019.2.37.52
  5) Ethereal_11.25
  6) Laser_1.7
  7) Pedone_1.9
  8) Rodent_0.277
  9) RofChade_2.1
 10) RubiChess_1.5
 11) Schooner_2.0.34
 12) Stockfish_10
 13) Texel_1.07

         1     2     3     4     5     6     7     8     9    10    11    12    13
  1.  ----- 54.19 51.77 51.54 54.26 53.53 48.88 53.34 52.86 50.46 54.56 53.27 52.23
  2.  54.19 ----- 52.68 52.99 56.11 56.66 48.83 51.97 55.88 51.49 55.37 63.38 51.65
  3.  51.77 52.68 ----- 65.94 53.69 54.66 52.03 53.96 54.92 51.41 53.44 52.56 51.95
  4.  51.54 52.99 65.94 ----- 53.73 54.54 51.64 53.86 55.06 51.70 53.06 52.69 51.75
  5.  54.26 56.11 53.69 53.73 ----- 57.94 50.33 53.59 58.35 54.09 55.68 56.66 54.05
  6.  53.53 56.66 54.66 54.54 57.94 ----- 51.63 54.69 56.43 53.45 55.79 56.34 54.01
  7.  48.88 48.83 52.03 51.64 50.33 51.63 ----- 51.64 50.38 49.36 51.64 48.96 50.59
  8.  53.34 51.97 53.96 53.86 53.59 54.69 51.64 ----- 53.97 52.62 54.29 51.83 55.05
  9.  52.86 55.88 54.92 55.06 58.35 56.43 50.38 53.97 ----- 54.19 55.20 55.81 54.19
 10.  50.46 51.49 51.41 51.70 54.09 53.45 49.36 52.62 54.19 ----- 53.46 51.08 53.11
 11.  54.56 55.37 53.44 53.06 55.68 55.79 51.64 54.29 55.20 53.46 ----- 54.49 52.82
 12.  53.27 63.38 52.56 52.69 56.66 56.34 48.96 51.83 55.81 51.08 54.49 ----- 51.20
 13.  52.23 51.65 51.95 51.75 54.05 54.01 50.59 55.05 54.19 53.11 52.82 51.20 -----

I tried the midgame at 100ms, then use sim03 to generate the matrix. I need the matrix output to generate dendrogram.

Image


And this is from your html output.

Image

Deuteriums are suspects the same is true for Stockfish/Brainfish :). The dendrogram agrees with it.

Will release the sim2dendro.py later.

Re: Similarity tester - 2nd generation - BETA

Posted: Sat Aug 10, 2019 8:25 pm
by Rebel
I completed the documentation.

From the command line:

simex2 data\similarity.data >report.txt

In report.txt you get something like SIM3:

Code: Select all

Andsc asmFi brain Equin Ether Hanni Laser Stock 
----- 51.42 49.40 48.07 50.08 47.41 50.42 49.83 
51.42 ----- 60.66 49.74 53.11 48.75 53.26 61.04 
49.40 60.66 ----- 49.53 52.48 48.07 51.58 62.28 
48.07 49.74 49.53 ----- 49.74 51.29 50.19 49.71 
50.08 53.11 52.48 49.74 ----- 47.89 53.43 52.74 
47.41 48.75 48.07 51.29 47.89 ----- 49.23 48.22 
50.42 53.26 51.58 50.19 53.43 49.23 ----- 52.18 
49.83 61.04 62.28 49.71 52.74 48.22 52.18 ----- 
But maybe it's more easy when I create an output file as input for your sim2dendro.

Re: Similarity tester - 2nd generation - BETA

Posted: Sat Aug 10, 2019 8:28 pm
by Guenther
Rebel wrote: Fri Aug 09, 2019 11:32 pm I am working on the documentation of SIMEX 2.0 which is meant to be the successor of Don Dailey's famous SIM03 Similarity Tester.

SIMEX works the same way but is more user-friendly, has more features but the main advantage is that you are no longer limited to the build-in 8238 positions of SIM03 but can create your own positions using EPD, in the download 7 EPD sets are provided for demonstration purposes. SIMEX uses MEA from Ferdinand Mosca as a base.

http://rebel13.nl/misc/simex.html

The documentation is not even half-way and by this thread I take the opportunity to receive comments, ideas before an official release. The page will be frequently updated the coming days.
Thanks for this Ed and Ferdy (MEA). Hopefully I can fiddle around with this after summer with own sets of epds.

Re: Similarity tester - 2nd generation - BETA

Posted: Sun Aug 11, 2019 3:25 am
by Ferdy
Rebel wrote: Sat Aug 10, 2019 8:25 pm I completed the documentation.

From the command line:

simex2 data\similarity.data >report.txt

In report.txt you get something like SIM3:

Code: Select all

Andsc asmFi brain Equin Ether Hanni Laser Stock 
----- 51.42 49.40 48.07 50.08 47.41 50.42 49.83 
51.42 ----- 60.66 49.74 53.11 48.75 53.26 61.04 
49.40 60.66 ----- 49.53 52.48 48.07 51.58 62.28 
48.07 49.74 49.53 ----- 49.74 51.29 50.19 49.71 
50.08 53.11 52.48 49.74 ----- 47.89 53.43 52.74 
47.41 48.75 48.07 51.29 47.89 ----- 49.23 48.22 
50.42 53.26 51.58 50.19 53.43 49.23 ----- 52.18 
49.83 61.04 62.28 49.71 52.74 48.22 52.18 ----- 
But maybe it's more easy when I create an output file as input for your sim2dendro.
I think it is indeed better for specific input for sim2dendro.
Better is just a csv type formatting with or without a header. Example

engine1 name,-----,51.42,49.40, and so on
engine2 name,51.42,-----,60.66, and so on
and so on

You can just add it at the end of the report.txt so that they are just in one file. That should not be difficult to parse.

The repo is at https://github.com/fsmosca/Similarity-Dendrogram but it can only read sim3 matrix output currently.

If possible perhaps you can add in the report.txt the info on test epd and time used.

Re: Similarity tester - 2nd generation - BETA

Posted: Sun Aug 11, 2019 7:28 am
by Ovyron
Is it possible to "reverse engineer" this to make engine personalities that increase their similarity to other engines or to human players? Say, Stockfish already plays kind of similar to Laser, to maybe it'd be possible to make changes to the code that increase this similarity, so we'd get some "Laser Personality", and it'd be possible that it'd play like Laser but with a higher playing strength.

Re: Similarity tester - 2nd generation - BETA

Posted: Sun Aug 11, 2019 9:16 am
by Rebel
Ferdy wrote: Sun Aug 11, 2019 3:25 am
Rebel wrote: Sat Aug 10, 2019 8:25 pm I completed the documentation.

From the command line:

simex2 data\similarity.data >report.txt

In report.txt you get something like SIM3:

Code: Select all

Andsc asmFi brain Equin Ether Hanni Laser Stock 
----- 51.42 49.40 48.07 50.08 47.41 50.42 49.83 
51.42 ----- 60.66 49.74 53.11 48.75 53.26 61.04 
49.40 60.66 ----- 49.53 52.48 48.07 51.58 62.28 
48.07 49.74 49.53 ----- 49.74 51.29 50.19 49.71 
50.08 53.11 52.48 49.74 ----- 47.89 53.43 52.74 
47.41 48.75 48.07 51.29 47.89 ----- 49.23 48.22 
50.42 53.26 51.58 50.19 53.43 49.23 ----- 52.18 
49.83 61.04 62.28 49.71 52.74 48.22 52.18 ----- 
But maybe it's more easy when I create an output file as input for your sim2dendro.
I think it is indeed better for specific input for sim2dendro.
Better is just a csv type formatting with or without a header. Example

engine1 name,-----,51.42,49.40, and so on
engine2 name,51.42,-----,60.66, and so on
and so on

You can just add it at the end of the report.txt so that they are just in one file. That should not be difficult to parse.

The repo is at https://github.com/fsmosca/Similarity-Dendrogram but it can only read sim3 matrix output currently.

If possible perhaps you can add in the report.txt the info on test epd and time used.
Made the changes, while creating the HTML a file "dendogram.csv" is also made. Contents:

Code: Select all

epd\simex.epd,8238,100
Andscacs 0.93,-----,51.67,50.21,48.83,50.21,48.28,50.48,44.92,49.69,50.87
asmFish 2017,51.67,-----,60.21,50.07,53.00,49.06,53.38,45.74,52.27,60.65
brainfish 2018,50.21,60.21,-----,49.54,52.43,48.21,51.33,45.49,51.57,62.40
Equinox 3.30,48.83,50.07,49.54,-----,49.94,51.91,50.33,47.01,50.68,49.97
Ethereal 11.25,50.21,53.00,52.43,49.94,-----,48.22,53.40,49.03,51.66,52.74
Hannibal 1.7,48.28,49.06,48.21,51.91,48.22,-----,49.43,43.84,49.12,48.51
Laser 1.7,50.48,53.38,51.33,50.33,53.40,49.43,-----,48.21,51.21,52.31
RubiChess 1.4,44.92,45.74,45.49,47.01,49.03,43.84,48.21,-----,47.08,44.69
Schooner 2.0.34,49.69,52.27,51.57,50.68,51.66,49.12,51.21,47.08,-----,50.53
Stockfish 10,50.87,60.65,62.40,49.97,52.74,48.51,52.31,44.69,50.53,-----
Header contains used EPD, number positions, movetime.

If that's workable for you re-download SIMEX.

http://rebel13.nl/dl/simex2.7z

Re: Similarity tester - 2nd generation - BETA

Posted: Sun Aug 11, 2019 10:31 am
by Rebel
Added 8 new engines to the 2500ms statistic.

http://rebel13.nl/html/simex_2500ms_18_engines.html

Don't know what to think about the increasing orange markers. One can say that's only logical, more time (in general) means better moves.

Re: Similarity tester - 2nd generation - BETA

Posted: Sun Aug 11, 2019 1:56 pm
by Ferdy
Rebel wrote: Sun Aug 11, 2019 9:16 am
Ferdy wrote: Sun Aug 11, 2019 3:25 am
Rebel wrote: Sat Aug 10, 2019 8:25 pm I completed the documentation.

From the command line:

simex2 data\similarity.data >report.txt

In report.txt you get something like SIM3:

Code: Select all

Andsc asmFi brain Equin Ether Hanni Laser Stock 
----- 51.42 49.40 48.07 50.08 47.41 50.42 49.83 
51.42 ----- 60.66 49.74 53.11 48.75 53.26 61.04 
49.40 60.66 ----- 49.53 52.48 48.07 51.58 62.28 
48.07 49.74 49.53 ----- 49.74 51.29 50.19 49.71 
50.08 53.11 52.48 49.74 ----- 47.89 53.43 52.74 
47.41 48.75 48.07 51.29 47.89 ----- 49.23 48.22 
50.42 53.26 51.58 50.19 53.43 49.23 ----- 52.18 
49.83 61.04 62.28 49.71 52.74 48.22 52.18 ----- 
But maybe it's more easy when I create an output file as input for your sim2dendro.
I think it is indeed better for specific input for sim2dendro.
Better is just a csv type formatting with or without a header. Example

engine1 name,-----,51.42,49.40, and so on
engine2 name,51.42,-----,60.66, and so on
and so on

You can just add it at the end of the report.txt so that they are just in one file. That should not be difficult to parse.

The repo is at https://github.com/fsmosca/Similarity-Dendrogram but it can only read sim3 matrix output currently.

If possible perhaps you can add in the report.txt the info on test epd and time used.
Made the changes, while creating the HTML a file "dendogram.csv" is also made. Contents:

Code: Select all

epd\simex.epd,8238,100
Andscacs 0.93,-----,51.67,50.21,48.83,50.21,48.28,50.48,44.92,49.69,50.87
asmFish 2017,51.67,-----,60.21,50.07,53.00,49.06,53.38,45.74,52.27,60.65
brainfish 2018,50.21,60.21,-----,49.54,52.43,48.21,51.33,45.49,51.57,62.40
Equinox 3.30,48.83,50.07,49.54,-----,49.94,51.91,50.33,47.01,50.68,49.97
Ethereal 11.25,50.21,53.00,52.43,49.94,-----,48.22,53.40,49.03,51.66,52.74
Hannibal 1.7,48.28,49.06,48.21,51.91,48.22,-----,49.43,43.84,49.12,48.51
Laser 1.7,50.48,53.38,51.33,50.33,53.40,49.43,-----,48.21,51.21,52.31
RubiChess 1.4,44.92,45.74,45.49,47.01,49.03,43.84,48.21,-----,47.08,44.69
Schooner 2.0.34,49.69,52.27,51.57,50.68,51.66,49.12,51.21,47.08,-----,50.53
Stockfish 10,50.87,60.65,62.40,49.97,52.74,48.51,52.31,44.69,50.53,-----
Header contains used EPD, number positions, movetime.

If that's workable for you re-download SIMEX.

http://rebel13.nl/dl/simex2.7z
Downloaded that file an hour ago, no changes.

Code: Select all

Variety (forced nature of positions) overview

 1. 2862 (28.62%)
 2. 4468 (44.68%)
 3. 2669 (26.69%)

Positions 10000 Andsc Deute Xipho 
Andscacs 0.93   ----- 52.73 39.45 
Deuterium v2019.1.36.50                                          52.73 ----- 41.64 
Xiphos 0.5      39.45 41.64 ----- 

 1. Engine Andscacs 0.93 and Deuterium v2019.1.36.50 show a similarity of 52.73%
 2. Engine Deuterium v2019.1.36.50 and Xiphos 0.5 show a similarity of 41.64%
 3. Engine Andscacs 0.93 and Xiphos 0.5 show a similarity of 39.45%


Done...

Re: Similarity tester - 2nd generation - BETA

Posted: Sun Aug 11, 2019 4:36 pm
by Rebel
Ferdy wrote: Sun Aug 11, 2019 1:56 pm Downloaded that file an hour ago, no changes.
Check the file dendogram.csv, should be present.