EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
Let's begin with a 100 game match.
....
https://rebel7775.wixsite.com/rebel/emulate
Emulate - How many Games?
Moderator: Ras
-
Rebel
- Posts: 7608
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Emulate - How many Games?
90% of coding is debugging, the other 10% is writing bugs.
-
Ajedrecista
- Posts: 2270
- Joined: Wed Jul 13, 2011 9:04 pm
- Location: Madrid, Spain.
Re: Emulate - How many games?
Hello Ed:
Thank you for the reminder. The concept reminds me the fair coin and checking whether a coin is fair if we were talking about a win-lose only game and we test 'Engine A' and 'Engine A - (Copy)'.
In that case of no-draw outcome, a did a simple analysis six years ago (link) estimating a case of wins = loses. I copy some snippets like wins = loses, which is less and less probable when n grows, Prob.(wins = loses) ~ [(4^wins)/sqrt(pi*wins)] * 0.25^wins = 1/sqrt(pi*wins) and A very small value even if you would get n*Prob.(wins = loses) = n*[1/sqrt(pi*wins)] = sqrt(wins/pi) outcomes (or nearly this value) if you run n simulations of n games each, which I think they are quite on-topic with your thread. The wins = loses outcome is the most probable individually, but since this probability is so tiny with more games played or simulated, it is much easier to fall outside. OTOH, with such big number of games, the final score of (almost) every test would end close enough to 50%.
------------
I have got the same signature at Talkchess for years. One of the tools is called Minimum_number_of_games and is from early 2013. The aim of this tool was to calculate the minimum number of needed games to ensure an Elo gain with a given LOS value, which is somewhat related to your topic. Here is a sample output:
------------
Other of the tools, also from early 2013, is called Minimum_score_for_no_regression. The aim of this tool was to calculate the minimum score required for avoiding regression with a given LOS value, which is somewhat related to your topic. Here is the output for 50000 games, 50% of draw ratio and 95% confidence:
------------
Last but not least, other of the tools, from mid 2013, is called Probabilities_in_a_trinomial_distribution. The aim of this tool was to calculate the probabilities of reach certain scores when given some input data, which is greatly related to your topic. Here is some output from equal opponents (0 Elo difference), 50% of draw ratio and 100 games:
The output of this tool can be massive with more and more games. I set the limit to 50000, so I repeat with equal opponents (0 Elo difference), 50% of draw ratio and 50000 games:
Regards from Spain.
Ajedrecista.
Thank you for the reminder. The concept reminds me the fair coin and checking whether a coin is fair if we were talking about a win-lose only game and we test 'Engine A' and 'Engine A - (Copy)'.
In that case of no-draw outcome, a did a simple analysis six years ago (link) estimating a case of wins = loses. I copy some snippets like wins = loses, which is less and less probable when n grows, Prob.(wins = loses) ~ [(4^wins)/sqrt(pi*wins)] * 0.25^wins = 1/sqrt(pi*wins) and A very small value even if you would get n*Prob.(wins = loses) = n*[1/sqrt(pi*wins)] = sqrt(wins/pi) outcomes (or nearly this value) if you run n simulations of n games each, which I think they are quite on-topic with your thread. The wins = loses outcome is the most probable individually, but since this probability is so tiny with more games played or simulated, it is much easier to fall outside. OTOH, with such big number of games, the final score of (almost) every test would end close enough to 50%.
------------
I have got the same signature at Talkchess for years. One of the tools is called Minimum_number_of_games and is from early 2013. The aim of this tool was to calculate the minimum number of needed games to ensure an Elo gain with a given LOS value, which is somewhat related to your topic. Here is a sample output:
Code: Select all
[...]
_______________________________________________________________________________
Score for a wanted gain of 2.00 Elo: 50.2878 %
Sample standard deviation for 97.50 % of LOS: 0.1468 %
1.9600 sample standard deviations for 97.50 % of LOS: 0.2878 %
A LOS value of 97.50 % is equivalent to 95.00 % confidence in a two-sided test.
Minimum number of needed games: 115928 games.
_______________________________________________________________________________
If you want to include the draw ratio D, a good approximation is the next one:
Multiply the minimum number of needed games by (1 - D).
D < 1 (the used mathematical model fails with D = 1); close values to D = 1 will bring not reliable results.
_______________________________________________________________________________
[...]Other of the tools, also from early 2013, is called Minimum_score_for_no_regression. The aim of this tool was to calculate the minimum score required for avoiding regression with a given LOS value, which is somewhat related to your topic. Here is the output for 50000 games, 50% of draw ratio and 95% confidence:
Code: Select all
[...]
_______________________________________________________________________________
Theoretical minimum score for no regression: 50.3099 %
Theoretical sample standard deviation in this case: 0.1581 %
1.9600 theoretical sample standard deviations for 97.50 % of LOS: 0.3099 %
Minimum number of won points for the engine in this match: 25155.0 points.
Minimum Elo advantage, which is also the negative part of the error bar:
2.1541 Elo (for a LOS value of 97.50 %).
A LOS value of 97.50 % is equivalent to 95.00 % confidence in a two-sided test.
_______________________________________________________________________________
[...]Last but not least, other of the tools, from mid 2013, is called Probabilities_in_a_trinomial_distribution. The aim of this tool was to calculate the probabilities of reach certain scores when given some input data, which is greatly related to your topic. Here is some output from equal opponents (0 Elo difference), 50% of draw ratio and 100 games:
Code: Select all
Probabilities for a match of 100 games (rounded up to 0.0001 %):
Rating difference (rounded up to 0.01 Elo): 0.00 Elo.
Probability of a win = W ~ 25.0000 %
Probability of a draw = D ~ 50.0000 %
Probability of a lose = L ~ 25.0000 %
=================================================================================================================
Points: Accumulated (%): Probabilities (%): Approximated density function:
0.0 0.0000 0.0000
0.5 0.0000 0.0000
[...]
47.5 26.2311 4.3934 --------------------------------------------------------------
48.0 31.0364 4.8053 --------------------------------------------------------------------
48.5 36.1886 5.1521 -------------------------------------------------------------------------
49.0 41.6035 5.4150 -----------------------------------------------------------------------------
49.5 47.1826 5.5791 -------------------------------------------------------------------------------
50.0 52.8174 5.6348 --------------------------------------------------------------------------------
50.5 58.3965 5.5791 -------------------------------------------------------------------------------
51.0 63.8114 5.4150 -----------------------------------------------------------------------------
51.5 68.9636 5.1521 -------------------------------------------------------------------------
52.0 73.7689 4.8053 --------------------------------------------------------------------
52.5 78.1623 4.3934 --------------------------------------------------------------
[...]
99.5 100.0000 0.0000
100.0 100.0000 0.0000
==============================================================
SUMMARY:
Probability that the first player wins the match ~ 47.1826 %
Probability of a tied match ~ 5.6348 %
Probability that the second player wins the match ~ 47.1826 %
--------------------------------------------------------------
Prob.(first player wins) + 0.5*Prob.(tied match) ~ 50.0000 %
Prob.(second player wins) + 0.5*Prob.(tied match) ~ 50.0000 %Code: Select all
Probabilities for a match of 100 games (rounded up to 0.0001 %):
Rating difference (rounded up to 0.01 Elo): 0.00 Elo.
Probability of a win = W ~ 25.0000 %
Probability of a draw = D ~ 50.0000 %
Probability of a lose = L ~ 25.0000 %
=====================================================
Points: 100.0/100
+100 = 0 - 0
P ~ 0.0000 %
Probability of win 100.0 points out of 100: 0.0000 %
-----------------------------------------------------
Points: 99.5/100
+ 99 = 1 - 0
P ~ 0.0000 %
Probability of win 99.5 points out of 100: 0.0000 %
-----------------------------------------------------
Points: 99.0/100
+ 98 = 2 - 0
P ~ 0.0000 %
+ 99 = 0 - 1
P ~ 0.0000 %
Probability of win 99.0 points out of 100: 0.0000 %
-----------------------------------------------------
[...]
-----------------------------------------------------
Points: 50.0/100
+ 0 =100 - 0
P ~ 0.0000 %
+ 1 = 98 - 1
P ~ 0.0000 %
[...]
+ 23 = 54 - 23
P ~ 0.6781 %
+ 24 = 52 - 24
P ~ 0.8424 %
+ 25 = 50 - 25
P ~ 0.8936 %
+ 26 = 48 - 26
P ~ 0.8097 %
+ 27 = 46 - 27
P ~ 0.6264 %
[...]
+ 49 = 2 - 49
P ~ 0.0000 %
+ 50 = 0 - 50
P ~ 0.0000 %
Probability of win 50.0 points out of 100: 5.6348 %
-----------------------------------------------------
[...]
-----------------------------------------------------
Points: 0.0/100
+ 0 = 0 -100
P ~ 0.0000 %
Probability of win 0.0 points out of 100: 0.0000 %
==============================================================
SUMMARY:
Probability that the first player wins the match ~ 47.1826 %
Probability of a tied match ~ 5.6348 %
Probability that the second player wins the match ~ 47.1826 %
--------------------------------------------------------------
Prob.(first player wins) + 0.5*Prob.(tied match) ~ 50.0000 %
Prob.(second player wins) + 0.5*Prob.(tied match) ~ 50.0000 %Code: Select all
Probabilities for a match of 50000 games (rounded up to 0.0001 %):
Rating difference (rounded up to 0.01 Elo): 0.00 Elo.
Probability of a win = W ~ 25.0000 %
Probability of a draw = D ~ 50.0000 %
Probability of a lose = L ~ 25.0000 %
=================================================================================================================
Points: Accumulated (%): Probabilities (%): Approximated density function:
0.0 0.0000 0.0000
0.5 0.0000 0.0000
[...]
24950.0 26.4579 0.2066 -----------------------------------------------------------------
24950.5 26.6653 0.2074 ------------------------------------------------------------------
24951.0 26.8735 0.2082 ------------------------------------------------------------------
24951.5 27.0825 0.2090 ------------------------------------------------------------------
24952.0 27.2924 0.2098 -------------------------------------------------------------------
24952.5 27.5030 0.2106 -------------------------------------------------------------------
24953.0 27.7144 0.2114 -------------------------------------------------------------------
24953.5 27.9267 0.2122 -------------------------------------------------------------------
24954.0 28.1397 0.2130 --------------------------------------------------------------------
24954.5 28.3535 0.2138 --------------------------------------------------------------------
24955.0 28.5681 0.2146 --------------------------------------------------------------------
24955.5 28.7834 0.2153 --------------------------------------------------------------------
24956.0 28.9995 0.2161 ---------------------------------------------------------------------
24956.5 29.2164 0.2169 ---------------------------------------------------------------------
24957.0 29.4340 0.2176 ---------------------------------------------------------------------
24957.5 29.6524 0.2184 ---------------------------------------------------------------------
24958.0 29.8715 0.2191 ---------------------------------------------------------------------
24958.5 30.0913 0.2198 ----------------------------------------------------------------------
24959.0 30.3119 0.2206 ----------------------------------------------------------------------
24959.5 30.5332 0.2213 ----------------------------------------------------------------------
24960.0 30.7552 0.2220 ----------------------------------------------------------------------
24960.5 30.9779 0.2227 -----------------------------------------------------------------------
24961.0 31.2013 0.2234 -----------------------------------------------------------------------
24961.5 31.4254 0.2241 -----------------------------------------------------------------------
24962.0 31.6502 0.2248 -----------------------------------------------------------------------
24962.5 31.8756 0.2255 -----------------------------------------------------------------------
24963.0 32.1018 0.2261 ------------------------------------------------------------------------
24963.5 32.3286 0.2268 ------------------------------------------------------------------------
24964.0 32.5561 0.2275 ------------------------------------------------------------------------
24964.5 32.7842 0.2281 ------------------------------------------------------------------------
24965.0 33.0129 0.2288 -------------------------------------------------------------------------
24965.5 33.2423 0.2294 -------------------------------------------------------------------------
24966.0 33.4723 0.2300 -------------------------------------------------------------------------
24966.5 33.7030 0.2306 -------------------------------------------------------------------------
24967.0 33.9343 0.2313 -------------------------------------------------------------------------
24967.5 34.1661 0.2319 --------------------------------------------------------------------------
24968.0 34.3986 0.2325 --------------------------------------------------------------------------
24968.5 34.6316 0.2331 --------------------------------------------------------------------------
24969.0 34.8653 0.2336 --------------------------------------------------------------------------
24969.5 35.0995 0.2342 --------------------------------------------------------------------------
24970.0 35.3343 0.2348 --------------------------------------------------------------------------
24970.5 35.5696 0.2353 ---------------------------------------------------------------------------
24971.0 35.8055 0.2359 ---------------------------------------------------------------------------
24971.5 36.0420 0.2364 ---------------------------------------------------------------------------
24972.0 36.2789 0.2370 ---------------------------------------------------------------------------
24972.5 36.5164 0.2375 ---------------------------------------------------------------------------
24973.0 36.7545 0.2380 ---------------------------------------------------------------------------
24973.5 36.9930 0.2385 ----------------------------------------------------------------------------
24974.0 37.2320 0.2390 ----------------------------------------------------------------------------
24974.5 37.4715 0.2395 ----------------------------------------------------------------------------
24975.0 37.7116 0.2400 ----------------------------------------------------------------------------
24975.5 37.9520 0.2405 ----------------------------------------------------------------------------
24976.0 38.1930 0.2409 ----------------------------------------------------------------------------
24976.5 38.4344 0.2414 -----------------------------------------------------------------------------
24977.0 38.6763 0.2419 -----------------------------------------------------------------------------
24977.5 38.9186 0.2423 -----------------------------------------------------------------------------
24978.0 39.1613 0.2427 -----------------------------------------------------------------------------
24978.5 39.4044 0.2432 -----------------------------------------------------------------------------
24979.0 39.6480 0.2436 -----------------------------------------------------------------------------
24979.5 39.8920 0.2440 -----------------------------------------------------------------------------
24980.0 40.1363 0.2444 -----------------------------------------------------------------------------
24980.5 40.3811 0.2448 ------------------------------------------------------------------------------
24981.0 40.6262 0.2451 ------------------------------------------------------------------------------
24981.5 40.8717 0.2455 ------------------------------------------------------------------------------
24982.0 41.1176 0.2459 ------------------------------------------------------------------------------
24982.5 41.3638 0.2462 ------------------------------------------------------------------------------
24983.0 41.6103 0.2465 ------------------------------------------------------------------------------
24983.5 41.8572 0.2469 ------------------------------------------------------------------------------
24984.0 42.1044 0.2472 ------------------------------------------------------------------------------
24984.5 42.3519 0.2475 ------------------------------------------------------------------------------
24985.0 42.5997 0.2478 -------------------------------------------------------------------------------
24985.5 42.8478 0.2481 -------------------------------------------------------------------------------
24986.0 43.0962 0.2484 -------------------------------------------------------------------------------
24986.5 43.3449 0.2487 -------------------------------------------------------------------------------
24987.0 43.5938 0.2489 -------------------------------------------------------------------------------
24987.5 43.8430 0.2492 -------------------------------------------------------------------------------
24988.0 44.0924 0.2494 -------------------------------------------------------------------------------
24988.5 44.3421 0.2497 -------------------------------------------------------------------------------
24989.0 44.5919 0.2499 -------------------------------------------------------------------------------
24989.5 44.8420 0.2501 -------------------------------------------------------------------------------
24990.0 45.0923 0.2503 -------------------------------------------------------------------------------
24990.5 45.3428 0.2505 -------------------------------------------------------------------------------
24991.0 45.5935 0.2507 -------------------------------------------------------------------------------
24991.5 45.8444 0.2509 --------------------------------------------------------------------------------
24992.0 46.0954 0.2510 --------------------------------------------------------------------------------
24992.5 46.3466 0.2512 --------------------------------------------------------------------------------
24993.0 46.5979 0.2513 --------------------------------------------------------------------------------
24993.5 46.8494 0.2515 --------------------------------------------------------------------------------
24994.0 47.1010 0.2516 --------------------------------------------------------------------------------
24994.5 47.3527 0.2517 --------------------------------------------------------------------------------
24995.0 47.6045 0.2518 --------------------------------------------------------------------------------
24995.5 47.8564 0.2519 --------------------------------------------------------------------------------
24996.0 48.1084 0.2520 --------------------------------------------------------------------------------
24996.5 48.3604 0.2521 --------------------------------------------------------------------------------
24997.0 48.6126 0.2521 --------------------------------------------------------------------------------
24997.5 48.8647 0.2522 --------------------------------------------------------------------------------
24998.0 49.1170 0.2522 --------------------------------------------------------------------------------
24998.5 49.3692 0.2523 --------------------------------------------------------------------------------
24999.0 49.6215 0.2523 --------------------------------------------------------------------------------
24999.5 49.8738 0.2523 --------------------------------------------------------------------------------
25000.0 50.1262 0.2523 --------------------------------------------------------------------------------
25000.5 50.3785 0.2523 --------------------------------------------------------------------------------
25001.0 50.6308 0.2523 --------------------------------------------------------------------------------
25001.5 50.8830 0.2523 --------------------------------------------------------------------------------
25002.0 51.1353 0.2522 --------------------------------------------------------------------------------
25002.5 51.3874 0.2522 --------------------------------------------------------------------------------
25003.0 51.6396 0.2521 --------------------------------------------------------------------------------
25003.5 51.8916 0.2521 --------------------------------------------------------------------------------
25004.0 52.1436 0.2520 --------------------------------------------------------------------------------
25004.5 52.3955 0.2519 --------------------------------------------------------------------------------
25005.0 52.6473 0.2518 --------------------------------------------------------------------------------
25005.5 52.8990 0.2517 --------------------------------------------------------------------------------
25006.0 53.1506 0.2516 --------------------------------------------------------------------------------
25006.5 53.4021 0.2515 --------------------------------------------------------------------------------
25007.0 53.6534 0.2513 --------------------------------------------------------------------------------
25007.5 53.9046 0.2512 --------------------------------------------------------------------------------
25008.0 54.1556 0.2510 --------------------------------------------------------------------------------
25008.5 54.4065 0.2509 --------------------------------------------------------------------------------
25009.0 54.6572 0.2507 -------------------------------------------------------------------------------
25009.5 54.9077 0.2505 -------------------------------------------------------------------------------
25010.0 55.1580 0.2503 -------------------------------------------------------------------------------
25010.5 55.4081 0.2501 -------------------------------------------------------------------------------
25011.0 55.6579 0.2499 -------------------------------------------------------------------------------
25011.5 55.9076 0.2497 -------------------------------------------------------------------------------
25012.0 56.1570 0.2494 -------------------------------------------------------------------------------
25012.5 56.4062 0.2492 -------------------------------------------------------------------------------
25013.0 56.6551 0.2489 -------------------------------------------------------------------------------
25013.5 56.9038 0.2487 -------------------------------------------------------------------------------
25014.0 57.1522 0.2484 -------------------------------------------------------------------------------
25014.5 57.4003 0.2481 -------------------------------------------------------------------------------
25015.0 57.6481 0.2478 -------------------------------------------------------------------------------
25015.5 57.8956 0.2475 ------------------------------------------------------------------------------
25016.0 58.1428 0.2472 ------------------------------------------------------------------------------
25016.5 58.3897 0.2469 ------------------------------------------------------------------------------
25017.0 58.6362 0.2465 ------------------------------------------------------------------------------
25017.5 58.8824 0.2462 ------------------------------------------------------------------------------
25018.0 59.1283 0.2459 ------------------------------------------------------------------------------
25018.5 59.3738 0.2455 ------------------------------------------------------------------------------
25019.0 59.6189 0.2451 ------------------------------------------------------------------------------
25019.5 59.8637 0.2448 ------------------------------------------------------------------------------
25020.0 60.1080 0.2444 -----------------------------------------------------------------------------
25020.5 60.3520 0.2440 -----------------------------------------------------------------------------
25021.0 60.5956 0.2436 -----------------------------------------------------------------------------
25021.5 60.8387 0.2432 -----------------------------------------------------------------------------
25022.0 61.0814 0.2427 -----------------------------------------------------------------------------
25022.5 61.3237 0.2423 -----------------------------------------------------------------------------
25023.0 61.5656 0.2419 -----------------------------------------------------------------------------
25023.5 61.8070 0.2414 -----------------------------------------------------------------------------
25024.0 62.0480 0.2409 ----------------------------------------------------------------------------
25024.5 62.2884 0.2405 ----------------------------------------------------------------------------
25025.0 62.5285 0.2400 ----------------------------------------------------------------------------
25025.5 62.7680 0.2395 ----------------------------------------------------------------------------
25026.0 63.0070 0.2390 ----------------------------------------------------------------------------
25026.5 63.2455 0.2385 ----------------------------------------------------------------------------
25027.0 63.4836 0.2380 ---------------------------------------------------------------------------
25027.5 63.7211 0.2375 ---------------------------------------------------------------------------
25028.0 63.9580 0.2370 ---------------------------------------------------------------------------
25028.5 64.1945 0.2364 ---------------------------------------------------------------------------
25029.0 64.4304 0.2359 ---------------------------------------------------------------------------
25029.5 64.6657 0.2353 ---------------------------------------------------------------------------
25030.0 64.9005 0.2348 --------------------------------------------------------------------------
25030.5 65.1347 0.2342 --------------------------------------------------------------------------
25031.0 65.3684 0.2336 --------------------------------------------------------------------------
25031.5 65.6014 0.2331 --------------------------------------------------------------------------
25032.0 65.8339 0.2325 --------------------------------------------------------------------------
25032.5 66.0657 0.2319 --------------------------------------------------------------------------
25033.0 66.2970 0.2313 -------------------------------------------------------------------------
25033.5 66.5277 0.2306 -------------------------------------------------------------------------
25034.0 66.7577 0.2300 -------------------------------------------------------------------------
25034.5 66.9871 0.2294 -------------------------------------------------------------------------
25035.0 67.2158 0.2288 -------------------------------------------------------------------------
25035.5 67.4439 0.2281 ------------------------------------------------------------------------
25036.0 67.6714 0.2275 ------------------------------------------------------------------------
25036.5 67.8982 0.2268 ------------------------------------------------------------------------
25037.0 68.1244 0.2261 ------------------------------------------------------------------------
25037.5 68.3498 0.2255 -----------------------------------------------------------------------
25038.0 68.5746 0.2248 -----------------------------------------------------------------------
25038.5 68.7987 0.2241 -----------------------------------------------------------------------
25039.0 69.0221 0.2234 -----------------------------------------------------------------------
25039.5 69.2448 0.2227 -----------------------------------------------------------------------
25040.0 69.4668 0.2220 ----------------------------------------------------------------------
25040.5 69.6881 0.2213 ----------------------------------------------------------------------
25041.0 69.9087 0.2206 ----------------------------------------------------------------------
25041.5 70.1285 0.2198 ----------------------------------------------------------------------
25042.0 70.3476 0.2191 ---------------------------------------------------------------------
25042.5 70.5660 0.2184 ---------------------------------------------------------------------
25043.0 70.7836 0.2176 ---------------------------------------------------------------------
25043.5 71.0005 0.2169 ---------------------------------------------------------------------
25044.0 71.2166 0.2161 ---------------------------------------------------------------------
25044.5 71.4319 0.2153 --------------------------------------------------------------------
25045.0 71.6465 0.2146 --------------------------------------------------------------------
25045.5 71.8603 0.2138 --------------------------------------------------------------------
25046.0 72.0733 0.2130 --------------------------------------------------------------------
25046.5 72.2856 0.2122 -------------------------------------------------------------------
25047.0 72.4970 0.2114 -------------------------------------------------------------------
25047.5 72.7076 0.2106 -------------------------------------------------------------------
25048.0 72.9175 0.2098 -------------------------------------------------------------------
25048.5 73.1265 0.2090 ------------------------------------------------------------------
25049.0 73.3347 0.2082 ------------------------------------------------------------------
25049.5 73.5421 0.2074 ------------------------------------------------------------------
25050.0 73.7487 0.2066 -----------------------------------------------------------------
[...]
49999.5 100.0000 0.0000
50000.0 100.0000 0.0000
==============================================================
SUMMARY:
Probability that the first player wins the match ~ 49.8738 %
Probability of a tied match ~ 0.2523 %
Probability that the second player wins the match ~ 49.8738 %
--------------------------------------------------------------
Prob.(first player wins) + 0.5*Prob.(tied match) ~ 50.0000 %
Prob.(second player wins) + 0.5*Prob.(tied match) ~ 50.0000 %Ajedrecista.
-
Uri Blass
- Posts: 11246
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: Emulate - How many Games?
I think it may be dependent on the engine. identical deterministic engine may always get 50% against itself in a fair match with even number of games because you will always get the same game twice for every starting position.Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
Let's begin with a 100 game match.
....
https://rebel7775.wixsite.com/rebel/emulate
Things in reality may be different because the engine may be not deterministic because of some factors including using more than one thread but even in this case result with white and result with black from the same position are not independent results and I wonder if it is the case also with the random matches that EMULATE creates.
-
towforce
- Posts: 13268
- Joined: Thu Mar 09, 2006 12:57 am
- Location: Birmingham UK
- Full name: Graham Laight
Re: Emulate - How many Games?
My Suggestion
You've got a little problem: you can actually pick any number of non-drawing games you want.
Here's how I would do it:
1. Play n games
2. Discard the draws (so if you played 100 games, and 80 of them were draws, just use the 20 games which had a result)
3. From your 20 games, engine 1 (e1) should have won 10
4. You can use the binomial distribution function to calculate the probability of any other score (the binom.dist function in a spreadsheet)
5. Decide what confidence interval you're happy with (e.g. 5%)
6. For the number of games that ended in a win, work out the 5% confidence interval using the binomial distribution
7. In your binomial calculator, change the probability of an engine A win to 0.6 (or whatever level of "closeness" you want to differentiate). Are you happy that this number of games differentiates 0.5 from 0.6 well enough? If not, increase the number of games.
8. On giving the final number, don't forget to divide it by the proportion of non-drawing games (e.g. if 80% draws, the proportion of winning games is 0.2, so if 20 winning games are needed, the total number of games = 20/0.2 = 100
You've got a little problem: you can actually pick any number of non-drawing games you want.
Here's how I would do it:
1. Play n games
2. Discard the draws (so if you played 100 games, and 80 of them were draws, just use the 20 games which had a result)
3. From your 20 games, engine 1 (e1) should have won 10
4. You can use the binomial distribution function to calculate the probability of any other score (the binom.dist function in a spreadsheet)
5. Decide what confidence interval you're happy with (e.g. 5%)
6. For the number of games that ended in a win, work out the 5% confidence interval using the binomial distribution
7. In your binomial calculator, change the probability of an engine A win to 0.6 (or whatever level of "closeness" you want to differentiate). Are you happy that this number of games differentiates 0.5 from 0.6 well enough? If not, increase the number of games.
8. On giving the final number, don't forget to divide it by the proportion of non-drawing games (e.g. if 80% draws, the proportion of winning games is 0.2, so if 20 winning games are needed, the total number of games = 20/0.2 = 100
Human chess is partly about tactics and strategy, but mostly about memory
-
towforce
- Posts: 13268
- Joined: Thu Mar 09, 2006 12:57 am
- Location: Birmingham UK
- Full name: Graham Laight
Re: Emulate - How many Games?
It would be fun to derive an expression to work out the number of games needed from the draw rate, accuracy and confidence - but it would have taken some time, so I asked Gemini instead:
towforce wrote:I want to check that 2 chess engines have the same strength. The method I'm going to use is:
* play n games between the 2 engines
* discard the draws
* use the binomial distribution to calculate the probability that they are the same within a specified accuracy and confidence level
The parameters for the function are:
1. The draw rate (if 80% of the games are drawn, then the draw rate is 0.8)
2. The accuracy: the expected win rate is 0.5 (50%). If the accuracy is 0.1 then I want to be confident that the win rate is between 0.4 and 0.6
3. The confidence level: if I want to be 95% confident that the win rate is, say, between 0.4 and 0.6 then the confidence parameter would be 0.95
The result of the function will be the number of games I need to play between the two engines, including drawn games, to be confident to my specified confidence level that the two engines strength are equal within the specified accuracy.
I will need to copy the result, so give all mathematical expressions in plain text: do not use mathematical symbols.
Gemini wrote:To determine the total number of games required, we calculate the required number of decisive (non-drawn) games using standard normal approximation to the binomial distribution, and then scale up by the draw rate.
Mathematical Formulation
When testing whether two engines are evenly matched (true decisive win rate p = 0.5):
Let E be the margin of error (your accuracy parameter, e.g., 0.1).
Let Z be the critical value from the standard normal distribution corresponding to the desired confidence level C.
Let D be the draw rate (e.g., 0.8).
Let k be the number of decisive games needed.
Let n be the total number of games needed (including draws).
For a binomial proportion centered at p = 0.5 with margin of error E:
E = Z * sqrt((p * (1 - p)) / k)
Since p = 0.5, p * (1 - p) = 0.25.
Solving for k (decisive games):
k = (Z^2 * 0.25) / (E^2)
Since decisive games account for (1 - D) of all games played:
n = k / (1 - D)
Combining into a single expression:
n = (Z^2 * 0.25) / ((E^2) * (1 - D))
Common Z-Values for Confidence Levels
90% confidence: Z = 1.645
95% confidence: Z = 1.960
98% confidence: Z = 2.326
99% confidence: Z = 2.576
C# Implementation
Code: Select all
C#
using System;
public static class ChessMatchEstimator
{
/// <summary>
/// Calculates the total number of games required (including draws).
/// </summary>
/// <param name="drawRate">Expected draw rate (e.g., 0.80 for 80%)</param>
/// <param name="accuracy">Margin of error around 0.5 (e.g., 0.10 for [0.40, 0.60])</param>
/// <param name="confidenceLevel">Confidence level (e.g., 0.95 for 95%)</param>
/// <returns>Total number of games rounded up to the nearest integer</returns>
public static int CalculateTotalGames(double drawRate, double accuracy, double confidenceLevel)
{
if (drawRate < 0.0 || drawRate >= 1.0)
throw new ArgumentOutOfRangeException(nameof(drawRate), "Draw rate must be between 0.0 and 1.0.");
if (accuracy <= 0.0 || accuracy >= 0.5)
throw new ArgumentOutOfRangeException(nameof(accuracy), "Accuracy margin must be between 0.0 and 0.5.");
if (confidenceLevel <= 0.0 || confidenceLevel >= 1.0)
throw new ArgumentOutOfRangeException(nameof(confidenceLevel), "Confidence level must be between 0.0 and 1.0.");
// Compute two-tailed standard normal quantile Z
double alpha = 1.0 - confidenceLevel;
double z = MathNetInverseNormal(1.0 - (alpha / 2.0));
// Decisive games needed
double decisiveGames = (z * z * 0.25) / (accuracy * accuracy);
// Total games factoring in the draw rate
double totalGames = decisiveGames / (1.0 - drawRate);
return (int)Math.Ceiling(totalGames);
}
// Standard normal inverse CDF (probit) approximation (Acklam's algorithm)
private static double MathNetInverseNormal(double p)
{
double[] a = { -3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02,
1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00 };
double[] b = { -5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02,
6.680131188771972e+01, -1.328068155288572e+01 };
double[] c = { -7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00,
-2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00 };
double[] d = { 7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00,
3.754408661907416e+00 };
double q, r;
if (p < 0.02425)
{
q = Math.Sqrt(-2.0 * Math.Log(p));
return (((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1.0);
}
else if (p <= 0.97575)
{
q = p - 0.5;
r = q * q;
return (((((a[0]*r+a[1])*r+a[2])*r+a[3])*r+a[4])*r+a[5])*q /
(((((b[0]*r+b[1])*r+b[2])*r+b[3])*r+b[4])*r+1.0);
}
else
{
q = Math.Sqrt(-2.0 * Math.Log(1.0 - p));
return -(((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1.0);
}
}
}Gemini wrote:Worked Example
With inputs:
Draw rate = 0.80
Accuracy = 0.10
Confidence = 0.95 (Z = 1.96)
k = (1.96^2 * 0.25) / (0.10^2) = (3.8416 * 0.25) / 0.01 = 0.9604 / 0.01 = 96.04 decisive games
n = 96.04 / (1 - 0.80) = 96.04 / 0.20 = 480.2 games
Rounded up: 481 total games.
Human chess is partly about tactics and strategy, but mostly about memory
-
Rebel
- Posts: 7608
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Re: Emulate - How many Games?
The tool is a guide, not exact science. Of course the strength of an engine counts, it's therefore I introduced the draw rate input. Long time control counts also as it increases the draw rate. In practice the tool is reliable. Whenever I create a new data for NNUE I am used to play self play games in chunks of 50.000 games and hardly without exception the match scores fluctuate between 49,8% and 50.1%, mostly 50%.Uri Blass wrote: ↑Sun Aug 23, 2026 11:04 pmI think it may be dependent on the engine. identical deterministic engine may always get 50% against itself in a fair match with even number of games because you will always get the same game twice for every starting position.Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
Let's begin with a 100 game match.
....
https://rebel7775.wixsite.com/rebel/emulate
Things in reality may be different because the engine may be not deterministic because of some factors including using more than one thread but even in this case result with white and result with black from the same position are not independent results and I wonder if it is the case also with the random matches that EMULATE creates.
90% of coding is debugging, the other 10% is writing bugs.
-
towforce
- Posts: 13268
- Joined: Thu Mar 09, 2006 12:57 am
- Location: Birmingham UK
- Full name: Graham Laight
Re: Emulate - How many Games?
Rebel wrote: ↑Mon Aug 24, 2026 7:39 pmThe tool is a guide, not exact science. Of course the strength of an engine counts, it's therefore I introduced the draw rate input. Long time control counts also as it increases the draw rate. In practice the tool is reliable. Whenever I create a new data for NNUE I am used to play self play games in chunks of 50.000 games and hardly without exception the match scores fluctuate between 49,8% and 50.1%, mostly 50%.
Without knowing the draw rate, we cannot use either the binomial distribution or a normal distribution. We can do an estimate of Elo rating though:
I used the elo calculator at https://ratings.fide.com/calc.phtml?page=initial :
In taking Ed's case of 50,000 games and a score of 49.8%, that equates to a score of 24,900. I put in an average opponent rating of 3000, a score of 24,900 and a number of games of 50,000.
The calculator responded that both the rating and the number of games are invalid. Does anyone know of an online elo calculator that can handle these numbers?
Human chess is partly about tactics and strategy, but mostly about memory
-
towforce
- Posts: 13268
- Joined: Thu Mar 09, 2006 12:57 am
- Location: Birmingham UK
- Full name: Graham Laight
Re: Emulate - How many Games?
Sorry - missed this. This makes the calculation very straightforward relative to my previous answers.
You are working to 0.1%, so the score must be between 49.95% and 50.05%.
* the baseline probability is 0.5
* we want the outcome to stay between 49.95% and 50.05%. The maximum allowed drift from the true 50% centre is 0.05%, which is 0.0005 in decimal form
* for a normal distribution (which is accurate enough at this scale), capturing 95% of all outcomes requires measuring 1.96 standard errors out from the centre in both directions (the standard score, or Z-score, for 95% confidence)
* on flipping a coin many times, the spread (standard error) of the percentage gets smaller as the number of flips increases. Specifically, the spread equals the square root of (0.5 times 0.5 divided by the number of tosses)
* margin of error = 1.96 multiplied by the square root of (0.25 divided by the number of tosses)
To get the number of tosses, reverse this relationship:
* square the confidence multiplier (1.96 squared = 3.8416)
* multiply that by the coin variability (3.8416 multiplied by 0.25 = 0.9604)
* square the margin of error (0.0005 squared = 0.00000025)
* divide the coin variability by the squared margin of error (0.9604 divided by 0.00000025)
* that calculation equals 3,841,600
Of course, most games won't be a win or a loss - so the actual number of games will be 3,841,600 / (1 - draw rate).
If the draw rate is 80%, then the number of games needed to be 95% confident of getting a score between 49.95 and 50.05 will be 3,841,600 / (1 - 0.8) = 19,208,000
Human chess is partly about tactics and strategy, but mostly about memory
-
towforce
- Posts: 13268
- Joined: Thu Mar 09, 2006 12:57 am
- Location: Birmingham UK
- Full name: Graham Laight
Re: Emulate - How many Games?
Duh - each draw is worth 0.5, so they reduce the number of games needed to get an average between 49.95% and 50.05% - not increase ittowforce wrote: ↑Tue Aug 25, 2026 7:47 pm...that calculation equals 3,841,600
Of course, most games won't be a win or a loss - so the actual number of games will be 3,841,600 / (1 - draw rate).
If the draw rate is 80%, then the number of games needed to be 95% confident of getting a score between 49.95 and 50.05 will be 3,841,600 / (1 - 0.8) = 19,208,000
Could do with having another mathematician looking over my shoulder here!
Human chess is partly about tactics and strategy, but mostly about memory