Similarity tester - 2nd generation - BETA

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Rebel
Posts: 4541
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Sun Aug 11, 2019 2:42 pm

I am moving into NN engines, Lco runs fine. I am now trying Allie, it runs ok from the command line, with simex I get:

C:\Users\Gebruiker\Desktop\simex-nn>mea.exe --engine lc0\allie_04.exe --name Allie_GTX_1060 --hash 64 --threads 1 --protocol uci --epd epd\simex.epd --movetime 100 --log
epd 1 / 8238
Traceback (most recent call last):
File "mea.py", line 980, in <module>
File "mea.py", line 936, in main
File "mea.py", line 169, in run
File "mea.py", line 384, in run_uci_engine
File "site-packages\chess\__init__.py", line 517, in from_uci
ValueError: 'mo' is not in list
[5032] Failed to execute script mea

Any idea?
Everybody is unique, except me.

Ferdy
Posts: 4040
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Similarity tester - 2nd generation - BETA

Post by Ferdy » Sun Aug 11, 2019 4:09 pm

Rebel wrote:
Sun Aug 11, 2019 2:36 pm
Ferdy wrote:
Sun Aug 11, 2019 11:56 am
Downloaded that file an hour ago, no changes.
Check the file dendogram.csv, should be present.
I missed it, got it now. If you do an update, could you name it dendrogram.csv?

Ferdy
Posts: 4040
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Similarity tester - 2nd generation - BETA

Post by Ferdy » Sun Aug 11, 2019 4:27 pm

Rebel wrote:
Sun Aug 11, 2019 2:42 pm
I am moving into NN engines, Lco runs fine. I am now trying Allie, it runs ok from the command line, with simex I get:

C:\Users\Gebruiker\Desktop\simex-nn>mea.exe --engine lc0\allie_04.exe --name Allie_GTX_1060 --hash 64 --threads 1 --protocol uci --epd epd\simex.epd --movetime 100 --log
epd 1 / 8238
Traceback (most recent call last):
File "mea.py", line 980, in <module>
File "mea.py", line 936, in main
File "mea.py", line 169, in run
File "mea.py", line 384, in run_uci_engine
File "site-packages\chess\__init__.py", line 517, in from_uci
ValueError: 'mo' is not in list
[5032] Failed to execute script mea

Any idea?
This is about converting the bestmove of engine from uci move to san move format. The engine has finished searching and it sent its bestmove. If you have the log, lets see that bestmove.

line 384 of the mea.py:
https://github.com/fsmosca/Multiple-mov ... ea.py#L384

User avatar
Rebel
Posts: 4541
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Sun Aug 11, 2019 5:59 pm

Ferdy wrote:
Sun Aug 11, 2019 4:27 pm
Rebel wrote:
Sun Aug 11, 2019 2:42 pm
I am moving into NN engines, Lco runs fine. I am now trying Allie, it runs ok from the command line, with simex I get:

C:\Users\Gebruiker\Desktop\simex-nn>mea.exe --engine lc0\allie_04.exe --name Allie_GTX_1060 --hash 64 --threads 1 --protocol uci --epd epd\simex.epd --movetime 100 --log
epd 1 / 8238
Traceback (most recent call last):
File "mea.py", line 980, in <module>
File "mea.py", line 936, in main
File "mea.py", line 169, in run
File "mea.py", line 384, in run_uci_engine
File "site-packages\chess\__init__.py", line 517, in from_uci
ValueError: 'mo' is not in list
[5032] Failed to execute script mea

Any idea?
This is about converting the bestmove of engine from uci move to san move format. The engine has finished searching and it sent its bestmove. If you have the log, lets see that bestmove.

line 384 of the mea.py:
https://github.com/fsmosca/Multiple-mov ... ea.py#L384

Code: Select all

[ 2019-08-11 16:29:10,695 -     INFO ] Pos 1
[ 2019-08-11 16:29:10,695 -     INFO ] EPD: 3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - c0 "Rb3=1";
[ 2019-08-11 16:29:10,695 -     INFO ] id None
[ 2019-08-11 16:29:10,695 -     INFO ] FEN: 3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - 0 1
[ 2019-08-11 16:29:10,695 -     INFO ] Solutions: Rb3=1
[ 2019-08-11 16:29:10,695 -    DEBUG ] >> ucinewgame
[ 2019-08-11 16:29:10,695 -    DEBUG ] >> position fen 3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - 0 1
[ 2019-08-11 16:29:10,695 -    DEBUG ] >> go movetime 100
[ 2019-08-11 16:29:13,394 -    DEBUG ] << No more time and no bestmove forced=f!
[ 2019-08-11 16:29:13,409 -     INFO ] elapsed(ms) since go: 2724
Indeed, no bestmove.
Everybody is unique, except me.

User avatar
Rebel
Posts: 4541
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Mon Aug 12, 2019 2:41 pm

Ferdy wrote:
Sun Aug 11, 2019 4:09 pm
Rebel wrote:
Sun Aug 11, 2019 2:36 pm
Ferdy wrote:
Sun Aug 11, 2019 11:56 am
Downloaded that file an hour ago, no changes.
Check the file dendogram.csv, should be present.
I missed it, got it now. If you do an update, could you name it dendrogram.csv?
Yeah.
Everybody is unique, except me.

Ferdy
Posts: 4040
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Similarity tester - 2nd generation - BETA

Post by Ferdy » Mon Aug 12, 2019 3:25 pm

Can simex able to output like below?

Code: Select all

sim version 3
------ Stockfish_10 ------
 52.02  Ethereal_11.25
 51.81  Laser_1.7
 51.52  RofChade_2.1
 50.06  Gull_3
 50.02  Defenchess_2.1
 49.51  Andscacs_0.93
 49.14  Wasp_3.60
 48.93  Schooner_2.0.34
 47.73  Booot_6.3.1
 47.50  Texel_1.07
 47.04  Senpai_2.0
 46.98  Vajolet2_2.7
 46.50  Rodent_0.277
 46.30  RubiChess_1.5
 46.25  Demolito_2019-07-15
 45.87  Deuterium_v2019.1.36.50
 45.64  Arasan_21.3
 45.27  Fire_7.1
 44.16  Godel_6.0
 44.04  Hakkapeliitta_3.0
 42.95  Pedone_1.9
 42.57  Amoeba_3.0
 42.43  Cheng_4.39
 41.15  Winter_0.6
 40.31  marvin_3.3.1
I am trying to observe figures and dendrogram as there are some methods to create it, at the moment this is ward. Dendrogram tends to get weird as data increases. But it is fun to see the clustering. One use for engine authors is to select sparring partners for their engine that is on a different clusters, to be able to adapt to different style. So I am planning of ploting the dendrogram only for similar strength engine, perhaps +/- 100 rating points.

Current plot in github at the moment, all are in.

Image

User avatar
Rebel
Posts: 4541
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Mon Aug 12, 2019 4:56 pm

Tested Lc0 vs Allie, same network files, with good old SIM03

http://rebel13.nl/html/nn-500ms.html

http://rebel13.nl/html/nn-1000ms.html

I am not surprised.
Everybody is unique, except me.

Ferdy
Posts: 4040
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Similarity tester - 2nd generation - BETA

Post by Ferdy » Mon Aug 12, 2019 6:27 pm

Similarity Dendrogram v0.3 is released at https://github.com/fsmosca/Similarity-D ... m/releases
Exe file is big around 50mb due to dependent modules such as scipy, numpy and matplotlib.

Command line:

Code: Select all

similaritydendrogram.exe --input dendrogram.csv --output sim.png
Help

Code: Select all

usage: similaritydendrogram.exe [-h] --input INPUT [--output OUTPUT]
                                [--method METHOD] [--figx FIGX] [--figy FIGY]
                                [--plotx PLOTX] [--dpi DPI] [--log]

Read engine similarity matrix and output dendrogram.

optional arguments:
  -h, --help       show this help message and exit
  --input INPUT    input similarity matrix file
  --output OUTPUT  output dendrogram image file,
                   default=similarity_dendrogram.png
  --method METHOD  input distance method [ward, complete, single, average,
                   weighted, centroid, median], default=ward
  --figx FIGX      input figure x value length, default=10
  --figy FIGY      input figure y value length, default=9
  --plotx PLOTX    input plot x value length, default=None
  --dpi DPI        input image dpi, default=300
  --log            Records program logging

Similarity Dendrogram v0.3

User avatar
Rebel
Posts: 4541
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Mon Aug 12, 2019 9:00 pm

Works great!

Can I include it in the SIMEX download?
Everybody is unique, except me.

Ferdy
Posts: 4040
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Similarity tester - 2nd generation - BETA

Post by Ferdy » Mon Aug 12, 2019 9:13 pm

Rebel wrote:
Mon Aug 12, 2019 9:00 pm
Works great!

Can I include it in the SIMEX download?
Yes of course.

Post Reply