Experiment 11 - Similarity between the top engines back then and now.

[Rebel]

Hi Frank,

That's a cool looking picture.

How did you do the analysis work?

Analyze EPD with Arena?

[Frank Quisinsky]

Hi Ed,

sorry for my late answer!
I am looking not often in chess fora for the moment.
I am suffering on the older chess computer fever, you understand ...
And you are one of the guilty persons!



Arena:
I don nothing with Arena since my work ended here around 2009 or so.

No, no ...
That's a grafic from the opening book project FEOBOS.
Under Shredder GUI 41.614 end positions analysed by 10 engines with 1 minutes per move, 6 cores on Intel i7 4.3Ghz. So it's nice to see if you put a template of the 41.614 postions. For FEOBOS opening book project (ended for around 14 months) we do some of such statistics under Excel. The idea is to find out the balanaced opening positions, 3 moves after ECO code ended for eng-eng testing. But with all the analyzes we can do a lot, so we try a bit under Excel:

FEOBOS:
http://www.amateurschach.de/main/_new-opening-book.htm

So 33.009 positions is the final result with Contempt = 3 (not more as 2 of 10 engines should give as eval 0,00). The topic for contempt is to find out the fast draw 3fold positions. But all this is history. Not important for your new idea.

I find your work interesting to that topic. Since a long time I lost my interest on eng-eng testing because most of the stronger engines today produced the same chess, same strengths, same weaknesses, allways the same with different names of engines. I missed a personality of the programs. Such a personality you gave Rebel (to bear the hollmarks, handwriting ... I can't find the right word in english language, sure you know what I mean) or John Stanback gave Zarkov / Wasp. That's computer chess engines for myself.

Yesterday I am reading CSS and saw the pic you hold the cup (1992).
Good, I like the pic ... again and again, you do a great job and with the years I am older and older I can more and more unterstand the criticism current computer chess devlopments got. Many people lost here interest ...

Be sure ...
I have fun all the time.
At the moment with the older chess computers.
Now 52 I have and in next month I will start my big tourney ... can be run 2 or 3 years, not important.

For you Ed ...
All the best from my site!!!
Hold your fun and ignore modern things, the time can be used better.
Hope I can write that to such a great person you are.

Friendly
Frank

[Rebel]

Thanks Frank for all your kind words, receive in return my praise for your more than excellent FEOBOS project.

Be well.

[ThatsIt]

Added latest versions of Fire, Booot, Fizbo, still sane.

Great work, THX.

Best wishes,
G.S.
(CEGT team)

[Rebel]

Added latest versions of Fire, Booot, Fizbo, still sane.

Great work, THX.

Best wishes,
G.S.
(CEGT team)

Well, there is something strange about Booot and Fizbo.

Fizbo - when I run it on a 4 core/8 threads PC at 100ms it uses the processor only for 2% instead of 13%. When I run it on 1000ms the processor usuage is 0%.

Booot - same behavior but even worse, 100ms and 1000ms processor usuage is 0%.

It probably explains the low similarity.

In any case, both are useless for the test.

[Rebel]

Found the trouble with Booot, made some changes to SIM3 and Booot became alive, Fizbo unfortunately not. Did it all over on a diiferent PC, still looking excellent.

Code: Select all

Positions  8238    Ands  Booo  Ethe  Fire  Fizb  Komo  Lase  Stoc  Xiph 
{Andscacs 0.93  } ----- 43.42 45.82 44.60 23.89 45.93 46.44 48.26 47.12 
{Booot 6.3.1    } 43.42 ----- 45.42 43.66 24.35 44.66 44.90 45.67 44.34 
{Ethereal 11.25 } 45.82 45.42 ----- 44.48 25.64 46.04 50.19 49.14 49.53 
{Fire 7.1       } 44.60 43.66 44.48 ----- 25.60 44.71 45.28 46.39 45.01 
{Fizbo 2        } 23.89 24.35 25.64 25.60 ----- 25.16 24.75 25.19 24.85 
{Komodo 9.42    } 45.93 44.66 46.04 44.71 25.16 ----- 47.37 48.40 47.33 
{Laser 1.7      } 46.44 44.90 50.19 45.28 24.75 47.37 ----- 49.15 48.51 
{Stockfish 10   } 48.26 45.67 49.14 46.39 25.19 48.40 49.15 ----- 50.62 
{Xiphos 0.5     } 47.12 44.34 49.53 45.01 24.85 47.33 48.51 50.62 ----- 

 1. Engine {Stockfish 10   } and {Xiphos 0.5     } show a similarity of 50.62%
 2. Engine {Ethereal 11.25 } and {Laser 1.7      } show a similarity of 50.19%
 3. Engine {Ethereal 11.25 } and {Xiphos 0.5     } show a similarity of 49.53%
 4. Engine {Laser 1.7      } and {Stockfish 10   } show a similarity of 49.15%
 5. Engine {Ethereal 11.25 } and {Stockfish 10   } show a similarity of 49.14%
 6. Engine {Laser 1.7      } and {Xiphos 0.5     } show a similarity of 48.51%
 7. Engine {Komodo 9.42    } and {Stockfish 10   } show a similarity of 48.40%
 8. Engine {Andscacs 0.93  } and {Stockfish 10   } show a similarity of 48.26%
 9. Engine {Komodo 9.42    } and {Laser 1.7      } show a similarity of 47.37%
10. Engine {Komodo 9.42    } and {Xiphos 0.5     } show a similarity of 47.33%
11. Engine {Andscacs 0.93  } and {Xiphos 0.5     } show a similarity of 47.12%
12. Engine {Andscacs 0.93  } and {Laser 1.7      } show a similarity of 46.44%
13. Engine {Fire 7.1       } and {Stockfish 10   } show a similarity of 46.39%
14. Engine {Ethereal 11.25 } and {Komodo 9.42    } show a similarity of 46.04%
15. Engine {Andscacs 0.93  } and {Komodo 9.42    } show a similarity of 45.93%
16. Engine {Andscacs 0.93  } and {Ethereal 11.25 } show a similarity of 45.82%
17. Engine {Booot 6.3.1    } and {Stockfish 10   } show a similarity of 45.67%
18. Engine {Booot 6.3.1    } and {Ethereal 11.25 } show a similarity of 45.42%
19. Engine {Fire 7.1       } and {Laser 1.7      } show a similarity of 45.28%
20. Engine {Fire 7.1       } and {Xiphos 0.5     } show a similarity of 45.01%
21. Engine {Booot 6.3.1    } and {Laser 1.7      } show a similarity of 44.90%
22. Engine {Fire 7.1       } and {Komodo 9.42    } show a similarity of 44.71%
23. Engine {Booot 6.3.1    } and {Komodo 9.42    } show a similarity of 44.66%
24. Engine {Andscacs 0.93  } and {Fire 7.1       } show a similarity of 44.60%
25. Engine {Ethereal 11.25 } and {Fire 7.1       } show a similarity of 44.48%
26. Engine {Booot 6.3.1    } and {Xiphos 0.5     } show a similarity of 44.34%
27. Engine {Booot 6.3.1    } and {Fire 7.1       } show a similarity of 43.66%
28. Engine {Andscacs 0.93  } and {Booot 6.3.1    } show a similarity of 43.42%
29. Engine {Ethereal 11.25 } and {Fizbo 2        } show a similarity of 25.64%
30. Engine {Fire 7.1       } and {Fizbo 2        } show a similarity of 25.60%
31. Engine {Fizbo 2        } and {Stockfish 10   } show a similarity of 25.19%
32. Engine {Fizbo 2        } and {Komodo 9.42    } show a similarity of 25.16%
33. Engine {Fizbo 2        } and {Xiphos 0.5     } show a similarity of 24.85%
34. Engine {Fizbo 2        } and {Laser 1.7      } show a similarity of 24.75%
35. Engine {Booot 6.3.1    } and {Fizbo 2        } show a similarity of 24.35%
36. Engine {Andscacs 0.93  } and {Fizbo 2        } show a similarity of 23.89%

But there is a but, more later.

[Ferdy]

Found the trouble with Booot, made some changes to SIM3 and Booot became alive, Fizbo unfortunately not.

"go movetime" and "go depth" will not work with Fizbo. But it would work using "go infinite".

Tested it on 100ms but it would not complete engine would exit. Currently testing it at 200ms, no problem so far after 6500 positions.

I use python to analyze the positions and write results to similarity.data.

[Rebel]

Found the trouble with Booot, made some changes to SIM3 and Booot became alive, Fizbo unfortunately not.

"go movetime" and "go depth" will not work with Fizbo. But it would work using "go infinite".

Tested it on 100ms but it would not complete engine would exit. Currently testing it at 200ms, no problem so far after 6500 positions.

I use python to analyze the positions and write results to similarity.data.

That's a good hint, the red. Will make it standard in the new 4.0 version. Fizbo now ok.

Code: Select all

Positions  8238   Ands  Booo  Ethe  Fire  Fizb  Komo  Lase  Stoc  Xiph 
{Andscacs 0.93 } ----- 43.42 45.82 44.60 42.21 45.93 46.44 48.26 47.12 
{Booot 6.3.1   } 43.42 ----- 45.42 43.66 39.55 44.66 44.90 45.67 44.34 
{Ethereal 11.25} 45.82 45.42 ----- 44.48 43.02 46.04 50.19 49.14 49.53 
{Fire 7.1      } 44.60 43.66 44.48 ----- 40.69 44.71 45.28 46.39 45.01 
{Fizbo 2       } 42.21 39.55 43.02 40.69 ----- 42.02 43.25 45.23 43.42 
{Komodo 9.42   } 45.93 44.66 46.04 44.71 42.02 ----- 47.37 48.40 47.33 
{Laser 1.7     } 46.44 44.90 50.19 45.28 43.25 47.37 ----- 49.15 48.51 
{Stockfish 10  } 48.26 45.67 49.14 46.39 45.23 48.40 49.15 ----- 50.62 
{Xiphos 0.5    } 47.12 44.34 49.53 45.01 43.42 47.33 48.51 50.62 ----- 

 1. Engine {Stockfish 10  } and {Xiphos 0.5    } show a similarity of 50.62%
 2. Engine {Ethereal 11.25} and {Laser 1.7     } show a similarity of 50.19%
 3. Engine {Ethereal 11.25} and {Xiphos 0.5    } show a similarity of 49.53%
 4. Engine {Laser 1.7     } and {Stockfish 10  } show a similarity of 49.15%
 5. Engine {Ethereal 11.25} and {Stockfish 10  } show a similarity of 49.14%
 6. Engine {Laser 1.7     } and {Xiphos 0.5    } show a similarity of 48.51%
 7. Engine {Komodo 9.42   } and {Stockfish 10  } show a similarity of 48.40%
 8. Engine {Andscacs 0.93 } and {Stockfish 10  } show a similarity of 48.26%
 9. Engine {Komodo 9.42   } and {Laser 1.7     } show a similarity of 47.37%
10. Engine {Komodo 9.42   } and {Xiphos 0.5    } show a similarity of 47.33%
11. Engine {Andscacs 0.93 } and {Xiphos 0.5    } show a similarity of 47.12%
12. Engine {Andscacs 0.93 } and {Laser 1.7     } show a similarity of 46.44%
13. Engine {Fire 7.1      } and {Stockfish 10  } show a similarity of 46.39%
14. Engine {Ethereal 11.25} and {Komodo 9.42   } show a similarity of 46.04%
15. Engine {Andscacs 0.93 } and {Komodo 9.42   } show a similarity of 45.93%
16. Engine {Andscacs 0.93 } and {Ethereal 11.25} show a similarity of 45.82%
17. Engine {Booot 6.3.1   } and {Stockfish 10  } show a similarity of 45.67%
18. Engine {Booot 6.3.1   } and {Ethereal 11.25} show a similarity of 45.42%
19. Engine {Fire 7.1      } and {Laser 1.7     } show a similarity of 45.28%
20. Engine {Fizbo 2       } and {Stockfish 10  } show a similarity of 45.23%
21. Engine {Fire 7.1      } and {Xiphos 0.5    } show a similarity of 45.01%
22. Engine {Booot 6.3.1   } and {Laser 1.7     } show a similarity of 44.90%
23. Engine {Fire 7.1      } and {Komodo 9.42   } show a similarity of 44.71%
24. Engine {Booot 6.3.1   } and {Komodo 9.42   } show a similarity of 44.66%
25. Engine {Andscacs 0.93 } and {Fire 7.1      } show a similarity of 44.60%
26. Engine {Ethereal 11.25} and {Fire 7.1      } show a similarity of 44.48%
27. Engine {Booot 6.3.1   } and {Xiphos 0.5    } show a similarity of 44.34%
28. Engine {Booot 6.3.1   } and {Fire 7.1      } show a similarity of 43.66%
29. Engine {Andscacs 0.93 } and {Booot 6.3.1   } show a similarity of 43.42%
30. Engine {Fizbo 2       } and {Xiphos 0.5    } show a similarity of 43.42%
31. Engine {Fizbo 2       } and {Laser 1.7     } show a similarity of 43.25%
32. Engine {Ethereal 11.25} and {Fizbo 2       } show a similarity of 43.02%
33. Engine {Andscacs 0.93 } and {Fizbo 2       } show a similarity of 42.21%
34. Engine {Fizbo 2       } and {Komodo 9.42   } show a similarity of 42.02%
35. Engine {Fire 7.1      } and {Fizbo 2       } show a similarity of 40.69%
36. Engine {Booot 6.3.1   } and {Fizbo 2       } show a similarity of 39.55%

[Rebel]

Found the trouble with Booot, made some changes to SIM3 and Booot became alive, Fizbo unfortunately not.

"go movetime" and "go depth" will not work with Fizbo. But it would work using "go infinite".

Tested it on 100ms but it would not complete engine would exit. Currently testing it at 200ms, no problem so far after 6500 positions.

I use python to analyze the positions and write results to similarity.data.

And did you get about the same results as I did in the above post for Fizbo?

[Ferdy]

Found the trouble with Booot, made some changes to SIM3 and Booot became alive, Fizbo unfortunately not.

"go movetime" and "go depth" will not work with Fizbo. But it would work using "go infinite".

Tested it on 100ms but it would not complete engine would exit. Currently testing it at 200ms, no problem so far after 6500 positions.

I use python to analyze the positions and write results to similarity.data.

And did you get about the same results as I did in the above post for Fizbo?

Still running some engines. But this is my result so far. I think my result with Fizbo is generally similar. Stockfish uses 20ms all other engines are 200ms. Most are below 50, but Stockfish version are close but below 50 (perhaps I did something wrong in the test or they are really different). Also since this is just an experiment, I mostly run the engine with hyper-threading ON, run in parallel at 6/8 threads. The dendogram below showed that these Stockfishes are in same cluster. Is the test intended for evaluation or both search and evaluation? Also surprise there are ending positions in the test.

Code: Select all

sim version 3
------ Stockfish 10 (time: 20 ms scale: 1.0) ------
 49.56  Stockfish 9 (time: 20 ms  scale: 1.0)
 48.11  Stockfish 8 (time: 20 ms  scale: 1.0)
 46.21  Rofchade 2.0 (time: 200 ms scale: 1.0)
 45.40  Fire 7.1 (time: 200 ms  scale: 1.0)
 45.35  Xiphos 0.5 (time: 200 ms scale: 1.0)
 44.78  Ethereal 11.25 (time: 200 ms scale: 1.0)
 44.68  Schooner 2.0.34 (time: 200 ms scale: 1.0)
 44.66  Senpai 2.0 (time: 200 ms scale: 1.0)
 44.12  Demolito 2018-10-29 (time: 200 ms scale: 1.0)
 43.92  Texel 1.07 (time: 200 ms scale: 1.0)
 42.74  Pedone 1.9 (time: 200 ms scale: 1.0)
 41.51  Nemorino 5.00 (time: 200 ms scale: 1.0)
 41.30  Fizbo 2 (time: 200 ms  scale: 1.0)
 37.57  Booot 6.3.1 (time: 200 ms scale: 1.0)
 34.22  Andscacs 0.95 (time: 200 ms scale: 1.0)
 28.39  Laser 1.7 (time: 200 ms scale: 1.0)

Code: Select all

sim version 3

  Key:

  1) Andscacs 0.95 (time: 200 ms scale: 1.0)
  2) Booot 6.3.1 (time: 200 ms scale: 1.0)
  3) Demolito 2018-10-29 (time: 200 ms scale: 1.0)
  4) Ethereal 11.25 (time: 200 ms scale: 1.0)
  5) Fire 7.1 (time: 200 ms  scale: 1.0)
  6) Fizbo 2 (time: 200 ms  scale: 1.0)
  7) Laser 1.7 (time: 200 ms scale: 1.0)
  8) Nemorino 5.00 (time: 200 ms scale: 1.0)
  9) Pedone 1.9 (time: 200 ms scale: 1.0)
 10) Rofchade 2.0 (time: 200 ms scale: 1.0)
 11) Schooner 2.0.34 (time: 200 ms scale: 1.0)
 12) Senpai 2.0 (time: 200 ms scale: 1.0)
 13) Stockfish 10 (time: 20 ms scale: 1.0)
 14) Stockfish 8 (time: 20 ms  scale: 1.0)
 15) Stockfish 9 (time: 20 ms  scale: 1.0)
 16) Texel 1.07 (time: 200 ms scale: 1.0)
 17) Xiphos 0.5 (time: 200 ms scale: 1.0)
 

Matrix

Code: Select all

         1     2     3     4     5     6     7     8     9    10    11    12    13    14    15    16    17
  1.  ----- 31.65 36.90 37.61 36.68 34.37 25.69 35.01 35.51 38.31 37.89 37.59 34.22 36.27 35.60 37.12 37.07
  2.  31.65 ----- 40.93 40.48 40.00 35.54 27.51 38.50 39.52 41.47 40.93 39.88 37.57 40.03 37.87 39.01 38.88
  3.  36.90 40.93 ----- 47.56 46.02 42.15 30.83 47.32 50.41 50.30 51.66 50.15 44.12 46.41 44.78 46.25 47.60
  4.  37.61 40.48 47.56 ----- 47.17 44.49 31.61 44.43 46.13 52.56 48.75 48.32 44.78 47.07 45.31 47.56 49.21
  5.  36.68 40.00 46.02 47.17 ----- 42.02 30.07 44.29 44.85 46.98 45.99 46.73 45.40 48.40 45.22 45.80 45.51
  6.  34.37 35.54 42.15 44.49 42.02 ----- 28.50 39.55 41.05 45.31 43.58 44.48 41.30 42.01 41.22 44.38 43.41
  7.  25.69 27.51 30.83 31.61 30.07 28.50 ----- 27.77 29.61 32.10 31.45 31.44 28.39 29.93 29.36 30.88 29.72
  8.  35.01 38.50 47.32 44.43 44.29 39.55 27.77 ----- 44.23 45.55 45.97 46.44 41.51 43.42 42.07 44.06 44.15
  9.  35.51 39.52 50.41 46.13 44.85 41.05 29.61 44.23 ----- 47.27 48.14 47.06 42.74 46.03 44.16 47.15 45.53
 10.  38.31 41.47 50.30 52.56 46.98 45.31 32.10 45.55 47.27 ----- 50.64 49.60 46.21 47.43 46.32 49.61 51.08
 11.  37.89 40.93 51.66 48.75 45.99 43.58 31.45 45.97 48.14 50.64 ----- 50.67 44.68 47.44 45.64 47.24 50.57
 12.  37.59 39.88 50.15 48.32 46.73 44.48 31.44 46.44 47.06 49.60 50.67 ----- 44.66 46.44 44.10 49.37 49.11
 13.  34.22 37.57 44.12 44.78 45.40 41.30 28.39 41.51 42.74 46.21 44.68 44.66 ----- 48.11 49.56 43.92 45.35
 14.  36.27 40.03 46.41 47.07 48.40 42.01 29.93 43.42 46.03 47.43 47.44 46.44 48.11 ----- 49.55 45.45 47.37
 15.  35.60 37.87 44.78 45.31 45.22 41.22 29.36 42.07 44.16 46.32 45.64 44.10 49.56 49.55 ----- 44.51 45.61
 16.  37.12 39.01 46.25 47.56 45.80 44.38 30.88 44.06 47.15 49.61 47.24 49.37 43.92 45.45 44.51 ----- 47.31
 17.  37.07 38.88 47.60 49.21 45.51 43.41 29.72 44.15 45.53 51.08 50.57 49.11 45.35 47.37 45.61 47.31 -----

Dendogram calculation source.

Code: Select all

# -*- coding: utf-8 -*-
"""
dg.py

"""

from scipy.cluster.hierarchy import dendrogram, linkage  
import matplotlib.pyplot as plt
import numpy as np


# In [1, 36], 36 is the average similarity of player 1, based from matrix
X = np.array([[1,36],  
    [2,38],
    [3,45],
    [4,45],
    [5,44],
    [6,41],
    [7,30],
    [8,42],
    [9,44],
    [10,46],
    [11,46],
    [12,45],
    [13,43],
    [14,45],
    [15,43],
    [16,44],
    [17,45],])


def main():
    # Methods [single, complete, average, weighted] and others see
    # https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.linkage.html
    linked = linkage(X, method='complete')
    
    labelList = ['Andscacs 0.95', 'Booot 6.3.1', 'Demolito 2018-10-29',
                 'Ethereal 11.25', 'Fire 7.1', 'Fizbo 2',
                 'Laser 1.7', 'Nemorino 5.00', 'Pedone 1.9', 'Rofchade 2.0',
                 'Schooner 2.0.34', 'Senpai 2.0',
                 'Stockfish 10', 'Stockfish 8', 'Stockfish 9', 
                 'Texel 1.07', 'Xiphos 0.5',]
    
    plt.figure(figsize=(9, 6))  
    dendrogram(linked,  
                orientation='right',
                labels=labelList,
                distance_sort='descending',
                leaf_rotation=0.,
                show_leaf_counts=True)
    plt.title('Hierarchical Clustering Dendrogram\nmethod=complete')
    plt.xlabel('Distance', size=12)
    plt.savefig('dendogram.png', bbox_inches='tight', dpi=200)
    plt.show()  


if __name__ == '__main__':
    main()