TCEC Simulation program

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Rebel, chrisw

sovaz1997
Posts: 261
Joined: Sun Nov 13, 2016 10:37 am

TCEC Simulation program

Post by sovaz1997 »

Hello! I created a program for the simulation of TCEC: https://github.com/sovaz1997/TCEC-Simul

Offer your ideas for improvements or optimization. The program downloads the game schedule of TCEC (in JSON format) and predicts the result based on this. You set the ratings and the quality of the simulation yourself. Also, it is possible to set the values of the advantages of white eloAdvantage and eloDraw.
Also, I want to better understand how eloDraw works in BayesElo and how it relates to the probability of a draw. Thanks! :)

In the future: export to CSV, more accuracy output.

How it's works:

Code: Select all

python tcecsimul.py -r divP.json -s 100000

Ethereal 11.38: 2976
LCZero v0.21.1-n41800: 3200
KomodoMCTS 2306.00: 3059
Komodo 2306.00: 3089
Houdini 6.03: 3100
Stockfish 19040612: 3200
AllieStein v0.3dev-n6.1: 3059
Fire 011819: 3000
Played: 132/168
Stockfish 19040612            :  98.2020 01.7980 00.0000 00.0000 00.0000 00.0000 00.0000 00.0000
LCZero v0.21.1-n41800         :  01.7980 98.0910 00.1090 00.0020 00.0000 00.0000 00.0000 00.0000
Komodo 2306.00                :  00.0000 00.1080 73.8650 19.0240 05.6360 01.3610 00.0060 00.0000
AllieStein v0.3dev-n6.1       :  00.0000 00.0020 16.9870 40.1170 27.7700 14.7220 00.4020 00.0000
KomodoMCTS 2306.00            :  00.0000 00.0000 03.1230 18.9080 34.4600 42.6180 00.8850 00.0060
Houdini 6.03                  :  00.0000 00.0010 05.9160 21.9190 31.8890 38.8420 01.4330 00.0000
Fire 011819                   :  00.0000 00.0000 00.0000 00.0300 00.2450 02.4470 91.9430 05.3350
Ethereal 11.38                :  00.0000 00.0000 00.0000 00.0000 00.0000 00.0100 05.3310 94.6590
Zevra 2 is my chess engine. Binary, source and description here: https://github.com/sovaz1997/Zevra2
Zevra v2.5 is last version of Zevra: https://github.com/sovaz1997/Zevra2/releases
Top
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: TCEC Simulation program

Post by Laskos »

sovaz1997 wrote: ↑Tue Apr 23, 2019 8:33 pm Hello! I created a program for the simulation of TCEC: https://github.com/sovaz1997/TCEC-Simul

Offer your ideas for improvements or optimization. The program downloads the game schedule of TCEC (in JSON format) and predicts the result based on this. You set the ratings and the quality of the simulation yourself. Also, it is possible to set the values of the advantages of white eloAdvantage and eloDraw.
Also, I want to better understand how eloDraw works in BayesElo and how it relates to the probability of a draw. Thanks! :)

In the future: export to CSV, more accuracy output.

How it's works:

Code: Select all

python tcecsimul.py -r divP.json -s 100000

Ethereal 11.38: 2976
LCZero v0.21.1-n41800: 3200
KomodoMCTS 2306.00: 3059
Komodo 2306.00: 3089
Houdini 6.03: 3100
Stockfish 19040612: 3200
AllieStein v0.3dev-n6.1: 3059
Fire 011819: 3000
Played: 132/168
Stockfish 19040612            :  98.2020 01.7980 00.0000 00.0000 00.0000 00.0000 00.0000 00.0000
LCZero v0.21.1-n41800         :  01.7980 98.0910 00.1090 00.0020 00.0000 00.0000 00.0000 00.0000
Komodo 2306.00                :  00.0000 00.1080 73.8650 19.0240 05.6360 01.3610 00.0060 00.0000
AllieStein v0.3dev-n6.1       :  00.0000 00.0020 16.9870 40.1170 27.7700 14.7220 00.4020 00.0000
KomodoMCTS 2306.00            :  00.0000 00.0000 03.1230 18.9080 34.4600 42.6180 00.8850 00.0060
Houdini 6.03                  :  00.0000 00.0010 05.9160 21.9190 31.8890 38.8420 01.4330 00.0000
Fire 011819                   :  00.0000 00.0000 00.0000 00.0300 00.2450 02.4470 91.9430 05.3350
Ethereal 11.38                :  00.0000 00.0000 00.0000 00.0000 00.0000 00.0100 05.3310 94.6590
Thanks, I agreed with eloAdvantage and eloDraw (350 gives draw rate between equal opponents in the 70s%), but not with the ratings. I changed them, but by now it's too late to see something, SF is too much ahead:

Code: Select all

Ethereal 11.38: 3080
LCZero v0.21.1-n41800: 3180
KomodoMCTS 2306.00: 3120
Komodo 2306.00: 3130
Houdini 6.03: 3130
Stockfish 19040612: 3200
AllieStein v0.3dev-n6.1: 3140
Fire 011819: 3100
Played: 133/168
Stockfish 19040612            :  99.9180 00.0820 00.0000 00.0000 00.0000 00.0000 00.0000 00.0000
LCZero v0.21.1-n41800         :  00.0820 99.6420 00.2640 00.0120 00.0000 00.0000 00.0000 00.0000
Komodo 2306.00                :  00.0000 00.2220 46.1090 30.6770 17.0550 05.8840 00.0530 00.0000
AllieStein v0.3dev-n6.1       :  00.0000 00.0410 31.8270 29.5020 24.9050 13.3560 00.3690 00.0000
Houdini 6.03                  :  00.0000 00.0110 16.1220 25.7270 30.7760 26.5110 00.8530 00.0000
KomodoMCTS 2306.00            :  00.0000 00.0020 05.6680 14.0450 26.9860 51.8370 01.4520 00.0100
Fire 011819                   :  00.0000 00.0000 00.0100 00.0370 00.2780 02.3890 91.9620 05.3240
Ethereal 11.38                :  00.0000 00.0000 00.0000 00.0000 00.0000 00.0230 05.3110 94.6660
Good sim, maybe I will have some ideas to adapt into it. Thanks!
Top
sovaz1997
Posts: 261
Joined: Sun Nov 13, 2016 10:37 am

Re: TCEC Simulation program

Post by sovaz1997 »

Thanks for testing :)
Now I have some updates: CSV-export and progress-bar of simulaion process. I plan to simulate the results of the CCC tournament too.

P. S. in command line you can set eloDraw and eloAdvantage (-eD, eA).
All commands for now:

Code: Select all

-h - show help
-eA <val> - set elo advantage
-eD <val> - set elo draw
-l <val> - link to JSON schedule
-r <val> - local file with engine ratings
-s <val> - number of simulations
-e <val> - export results to CSV-table
Zevra 2 is my chess engine. Binary, source and description here: https://github.com/sovaz1997/Zevra2
Zevra v2.5 is last version of Zevra: https://github.com/sovaz1997/Zevra2/releases
Top
sovaz1997
Posts: 261
Joined: Sun Nov 13, 2016 10:37 am

Re: TCEC Simulation program

Post by sovaz1997 »

I forgot about the tiebreaks. I'll do it soon. This is very important at the end of the tournament.
Zevra 2 is my chess engine. Binary, source and description here: https://github.com/sovaz1997/Zevra2
Zevra v2.5 is last version of Zevra: https://github.com/sovaz1997/Zevra2/releases
Top
sovaz1997
Posts: 261
Joined: Sun Nov 13, 2016 10:37 am

Re: TCEC Simulation program

Post by sovaz1997 »

Update simulation after game 140:

Code: Select all

Engines:
Ethereal 11.38: 2976
LCZero v0.21.1-n41800: 3200
KomodoMCTS 2306.00: 3059
Komodo 2306.00: 3089
Houdini 6.03: 3100
Stockfish 19040612: 3200
AllieStein v0.3dev-n6.1: 3079
Fire 011819: 3000

Played: 140/168

Stockfish 19040612            :  97.0900 02.9100 00.0000 00.0000 00.0000 00.0000 00.0000 00.0000
LCZero v0.21.1-n41800         :  02.9100 97.0700 00.0200 00.0000 00.0000 00.0000 00.0000 00.0000
Komodo 2306.00                :  00.0000 00.0100 74.5500 19.9200 04.3700 01.1500 00.0000 00.0000
AllieStein v0.3dev-n6.1       :  00.0000 00.0100 19.6100 59.9700 16.7300 03.6700 00.0100 00.0000
Houdini 6.03                  :  00.0000 00.0000 04.1000 12.9000 42.4000 40.3500 00.2500 00.0000
KomodoMCTS 2306.00            :  00.0000 00.0000 01.7200 07.2100 36.4700 54.2000 00.3800 00.0200
Fire 011819                   :  00.0000 00.0000 00.0000 00.0000 00.0300 00.5000 68.1500 31.3200
Ethereal 11.38                :  00.0000 00.0000 00.0000 00.0000 00.0000 00.1300 31.2100 68.6600
Program updated: fix small bug in calculation and added all tiebreaks but without crashes for now. https://github.com/sovaz1997/TCEC-Simul
P. S. The program has slowed down. It looks like this property is python. In C ++, the same algorithm worked 10-50 times faster. Can you recommend ideas for optimization, if you know.
Zevra 2 is my chess engine. Binary, source and description here: https://github.com/sovaz1997/Zevra2
Zevra v2.5 is last version of Zevra: https://github.com/sovaz1997/Zevra2/releases
Top
sovaz1997
Posts: 261
Joined: Sun Nov 13, 2016 10:37 am

Re: TCEC Simulation program

Post by sovaz1997 »

Simulation after game 144:

Code: Select all

Engines:
Ethereal 11.38: 2976
LCZero v0.21.1-n41800: 3200
KomodoMCTS 2306.00: 3059
Komodo 2306.00: 3089
Houdini 6.03: 3100
Stockfish 19040612: 3200
AllieStein v0.3dev-n6.1: 3079

Played: 144/168
Simulation processing: 100%|####################################| ETA:  0:00:00

Engine                        :  Scores  1       2       3       4       5       6       7       8
Stockfish 19040612            :  27.8784 99.7800 00.2200 00.0000 00.0000 00.0000 00.0000 00.0000 00.0000
LCZero v0.21.1-n41800         :  24.9754 00.2200 99.7700 00.0100 00.0000 00.0000 00.0000 00.0000 00.0000
Komodo 2306.00                :  21.0813 00.0000 00.0100 49.9900 25.8500 20.4000 03.7500 00.0000 00.0000
AllieStein v0.3dev-n6.1       :  20.8937 00.0000 00.0000 33.3100 42.4700 18.9000 05.3200 00.0000 00.0000
Houdini 6.03                  :  20.4955 00.0000 00.0000 15.1100 26.2000 43.1200 15.5700 00.0000 00.0000
KomodoMCTS 2306.00            :  19.9389 00.0000 00.0000 01.5800 05.4800 17.5800 75.3200 00.0400 00.0000
Fire 011819                   :  16.6185 00.0000 00.0000 00.0000 00.0000 00.0000 00.0300 74.3100 25.6600
Ethereal 11.38                :  16.1184 00.0000 00.0000 00.0000 00.0000 00.0000 00.0100 25.6500 74.3400
Link: https://github.com/sovaz1997/TCEC-Simul
Fixed bug in simulation, added crashes in tie-breaks calculation, average scores in output and CSV-output.
Zevra 2 is my chess engine. Binary, source and description here: https://github.com/sovaz1997/Zevra2
Zevra v2.5 is last version of Zevra: https://github.com/sovaz1997/Zevra2/releases
Top
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: TCEC Simulation program

Post by Laskos »

sovaz1997 wrote: ↑Thu Apr 25, 2019 9:54 am Simulation after game 144:

Code: Select all

Engines:
Ethereal 11.38: 2976
LCZero v0.21.1-n41800: 3200
KomodoMCTS 2306.00: 3059
Komodo 2306.00: 3089
Houdini 6.03: 3100
Stockfish 19040612: 3200
AllieStein v0.3dev-n6.1: 3079

Played: 144/168
Simulation processing: 100%|####################################| ETA:  0:00:00

Engine                        :  Scores  1       2       3       4       5       6       7       8
Stockfish 19040612            :  27.8784 99.7800 00.2200 00.0000 00.0000 00.0000 00.0000 00.0000 00.0000
LCZero v0.21.1-n41800         :  24.9754 00.2200 99.7700 00.0100 00.0000 00.0000 00.0000 00.0000 00.0000
Komodo 2306.00                :  21.0813 00.0000 00.0100 49.9900 25.8500 20.4000 03.7500 00.0000 00.0000
AllieStein v0.3dev-n6.1       :  20.8937 00.0000 00.0000 33.3100 42.4700 18.9000 05.3200 00.0000 00.0000
Houdini 6.03                  :  20.4955 00.0000 00.0000 15.1100 26.2000 43.1200 15.5700 00.0000 00.0000
KomodoMCTS 2306.00            :  19.9389 00.0000 00.0000 01.5800 05.4800 17.5800 75.3200 00.0400 00.0000
Fire 011819                   :  16.6185 00.0000 00.0000 00.0000 00.0000 00.0000 00.0300 74.3100 25.6600
Ethereal 11.38                :  16.1184 00.0000 00.0000 00.0000 00.0000 00.0000 00.0100 25.6500 74.3400
Link: https://github.com/sovaz1997/TCEC-Simul
Fixed bug in simulation, added crashes in tie-breaks calculation, average scores in output and CSV-output.
Thanks, when I will be back from my trip, I will check it out.
Top
sovaz1997
Posts: 261
Joined: Sun Nov 13, 2016 10:37 am

Re: TCEC Simulation program

Post by sovaz1997 »

There were some problems with the correctness of the simulation. Now, like, I fixed everything :)

After game 161 here: https://github.com/sovaz1997/TCEC-Simul
Zevra 2 is my chess engine. Binary, source and description here: https://github.com/sovaz1997/Zevra2
Zevra v2.5 is last version of Zevra: https://github.com/sovaz1997/Zevra2/releases
Top

Return to “Computer Chess Club: Tournaments and Matches”