Similarity tester - 2nd generation - BETA

[Rebel]

I am moving into NN engines, Lco runs fine. I am now trying Allie, it runs ok from the command line, with simex I get:

C:\Users\Gebruiker\Desktop\simex-nn>mea.exe --engine lc0\allie_04.exe --name Allie_GTX_1060 --hash 64 --threads 1 --protocol uci --epd epd\simex.epd --movetime 100 --log
epd 1 / 8238
Traceback (most recent call last):
File "mea.py", line 980, in <module>
File "mea.py", line 936, in main
File "mea.py", line 169, in run
File "mea.py", line 384, in run_uci_engine
File "site-packages\chess\__init__.py", line 517, in from_uci
ValueError: 'mo' is not in list
[5032] Failed to execute script mea

Any idea?

[Ferdy]

Downloaded that file an hour ago, no changes.

Check the file dendogram.csv, should be present.

I missed it, got it now. If you do an update, could you name it dendrogram.csv?

[Ferdy]

I am moving into NN engines, Lco runs fine. I am now trying Allie, it runs ok from the command line, with simex I get:

C:\Users\Gebruiker\Desktop\simex-nn>mea.exe --engine lc0\allie_04.exe --name Allie_GTX_1060 --hash 64 --threads 1 --protocol uci --epd epd\simex.epd --movetime 100 --log
epd 1 / 8238
Traceback (most recent call last):
File "mea.py", line 980, in <module>
File "mea.py", line 936, in main
File "mea.py", line 169, in run
File "mea.py", line 384, in run_uci_engine
File "site-packages\chess\__init__.py", line 517, in from_uci
ValueError: 'mo' is not in list
[5032] Failed to execute script mea

Any idea?

This is about converting the bestmove of engine from uci move to san move format. The engine has finished searching and it sent its bestmove. If you have the log, lets see that bestmove.

line 384 of the mea.py:
https://github.com/fsmosca/Multiple-mov ... ea.py#L384

[Rebel]

I am moving into NN engines, Lco runs fine. I am now trying Allie, it runs ok from the command line, with simex I get:

C:\Users\Gebruiker\Desktop\simex-nn>mea.exe --engine lc0\allie_04.exe --name Allie_GTX_1060 --hash 64 --threads 1 --protocol uci --epd epd\simex.epd --movetime 100 --log
epd 1 / 8238
Traceback (most recent call last):
File "mea.py", line 980, in <module>
File "mea.py", line 936, in main
File "mea.py", line 169, in run
File "mea.py", line 384, in run_uci_engine
File "site-packages\chess\__init__.py", line 517, in from_uci
ValueError: 'mo' is not in list
[5032] Failed to execute script mea

Any idea?

This is about converting the bestmove of engine from uci move to san move format. The engine has finished searching and it sent its bestmove. If you have the log, lets see that bestmove.

line 384 of the mea.py:
https://github.com/fsmosca/Multiple-mov ... ea.py#L384

Code: Select all

[ 2019-08-11 16:29:10,695 -     INFO ] Pos 1
[ 2019-08-11 16:29:10,695 -     INFO ] EPD: 3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - c0 "Rb3=1";
[ 2019-08-11 16:29:10,695 -     INFO ] id None
[ 2019-08-11 16:29:10,695 -     INFO ] FEN: 3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - 0 1
[ 2019-08-11 16:29:10,695 -     INFO ] Solutions: Rb3=1
[ 2019-08-11 16:29:10,695 -    DEBUG ] >> ucinewgame
[ 2019-08-11 16:29:10,695 -    DEBUG ] >> position fen 3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - 0 1
[ 2019-08-11 16:29:10,695 -    DEBUG ] >> go movetime 100
[ 2019-08-11 16:29:13,394 -    DEBUG ] << No more time and no bestmove forced=f!
[ 2019-08-11 16:29:13,409 -     INFO ] elapsed(ms) since go: 2724

Indeed, no bestmove.

[Rebel]

Downloaded that file an hour ago, no changes.

Check the file dendogram.csv, should be present.

I missed it, got it now. If you do an update, could you name it dendrogram.csv?

Yeah.

[Ferdy]

Can simex able to output like below?

Code: Select all

sim version 3
------ Stockfish_10 ------
 52.02  Ethereal_11.25
 51.81  Laser_1.7
 51.52  RofChade_2.1
 50.06  Gull_3
 50.02  Defenchess_2.1
 49.51  Andscacs_0.93
 49.14  Wasp_3.60
 48.93  Schooner_2.0.34
 47.73  Booot_6.3.1
 47.50  Texel_1.07
 47.04  Senpai_2.0
 46.98  Vajolet2_2.7
 46.50  Rodent_0.277
 46.30  RubiChess_1.5
 46.25  Demolito_2019-07-15
 45.87  Deuterium_v2019.1.36.50
 45.64  Arasan_21.3
 45.27  Fire_7.1
 44.16  Godel_6.0
 44.04  Hakkapeliitta_3.0
 42.95  Pedone_1.9
 42.57  Amoeba_3.0
 42.43  Cheng_4.39
 41.15  Winter_0.6
 40.31  marvin_3.3.1

I am trying to observe figures and dendrogram as there are some methods to create it, at the moment this is ward. Dendrogram tends to get weird as data increases. But it is fun to see the clustering. One use for engine authors is to select sparring partners for their engine that is on a different clusters, to be able to adapt to different style. So I am planning of ploting the dendrogram only for similar strength engine, perhaps +/- 100 rating points.

Current plot in github at the moment, all are in.

[Rebel]

Tested Lc0 vs Allie, same network files, with good old SIM03

http://rebel13.nl/html/nn-500ms.html

http://rebel13.nl/html/nn-1000ms.html

I am not surprised.

[Ferdy]

Similarity Dendrogram v0.3 is released at https://github.com/fsmosca/Similarity-D ... m/releases
Exe file is big around 50mb due to dependent modules such as scipy, numpy and matplotlib.

Command line:

Code: Select all

similaritydendrogram.exe --input dendrogram.csv --output sim.png

Help

Code: Select all

usage: similaritydendrogram.exe [-h] --input INPUT [--output OUTPUT]
                                [--method METHOD] [--figx FIGX] [--figy FIGY]
                                [--plotx PLOTX] [--dpi DPI] [--log]

Read engine similarity matrix and output dendrogram.

optional arguments:
  -h, --help       show this help message and exit
  --input INPUT    input similarity matrix file
  --output OUTPUT  output dendrogram image file,
                   default=similarity_dendrogram.png
  --method METHOD  input distance method [ward, complete, single, average,
                   weighted, centroid, median], default=ward
  --figx FIGX      input figure x value length, default=10
  --figy FIGY      input figure y value length, default=9
  --plotx PLOTX    input plot x value length, default=None
  --dpi DPI        input image dpi, default=300
  --log            Records program logging

Similarity Dendrogram v0.3

[Rebel]

Works great!

Can I include it in the SIMEX download?

[Ferdy]

Works great!

Can I include it in the SIMEX download?

Yes of course.