Noise in ELO estimators: a quantitative approach

[mcostalba]

Yes, it should have been 100 although, as you noted, the end result doesn't change. This is my first program in Python and I was writing it in one window while I was googling for python tutorials/documentation in another Luckily Python is very easy to learn and comes very handy for such things. For me, used to C/C++, it is like to park the truck and take the bicycle, really nice.

I would like to ask interested people that keeps big log files of the test match between 2 engines (at least 10K games per test) to run them through this script and report the results, together with testing conditions. So to verify if good testing framework are distinguishable from bad ones just looking at the results of the script.

[Joerg Oster]

Hi Marco,

you know I'm running gauntlets against different opponents when testing different versions of SF. Does your tool work in such cases, too?
Here is the output of your Noise_Estimator for one of these testruns:

Code: Select all

Games: 2796 , result: [957, 902, 937]
Estimated ELO: -0.0
Noise as function of number of games:
[924.79, 285.34, 54.43, 48.36, 127.66, 200.24, 42.3, 66.6, 90.65, 30.2, 48.32, 57.41, 39.25, 54.39, 45.34, 54.39, 42.28, 42.28, 21.13, 27.18, 27.18]

This was the test for SF-edce2a8. TC 15"+0.1"

Best, Jörg.

[mcostalba]

Hi Jörg, unfortunatly at the moment it expects only two engines tests.

Here is some output from mines, always 15" per game, single thread.

Code: Select all

Games: 2377 , result: [456, 475, 1446]
Estimated ELO: -6.94963842777
Noise as function of number of games:
[11.35, 5.67, 16.8, 2.65, 3.58, 3.96, 3.96, 3.77, 0.57, 0.57, 0.19]

[gladius]

Hi Jörg, unfortunatly at the moment it expects only two engines tests.

Here is some output from mines, always 15" per game, single thread.

Code: Select all

Games: 2377 , result: [456, 475, 1446]
Estimated ELO: -6.94963842777
Noise as function of number of games:
[11.35, 5.67, 16.8, 2.65, 3.58, 3.96, 3.96, 3.77, 0.57, 0.57, 0.19]

Interesting, my noise numbers are quite a bit higher with concurrency 3. This is on a 4 core machine, all engines running single threaded. Here are the results from two runs:

Code: Select all

Games: 1699 , result: [262, 342, 1095]
Estimated ELO: -16.6897337053
Noise as function of number of games:
['217.72', '61.94', '42.47', '27.59', '29.63', '6.72', '57.93', '49.01', '56.17', '41.48', '15.70', '28.53', '33.47']

Code: Select all

Games: 1751 , result: [300, 345, 1106]
Estimated ELO: -9.03536155887
Noise as function of number of games:
['173.24', '47.80', '44.15', '57.67', '47.29', '30.85', '29.70', '18.46', '28.18', '7.44', '11.36', '25.56', '12.30']

[gladius]

And here is a 6000 game result:

Code: Select all

Games: 6063 , result: [945, 1219, 3899]
Estimated ELO: -15.9933239309
Noise as function of number of games:
['152.80', '195.34', '156.53', '42.58', '15.93', '93.26', '29.86', '23.20', '53.41', '16.39', '5.89', '9.33', '13.94', '6.22', '4.19', '19.04', '35.93', '44.54', '54.09', '82.65', '34.68', '30.58', '12.08', '13.87', '33.28', '33.59', '40.82', '13.74', '13.99', '12.54', '4.76', '3.88', '4.27', '2.60', '4.39', '5.55', '9.39', '14.05', '16.62', '20.75', '10.56', '7.69', '10.08', '8.17', '5.84', '3.06', '2.09']

[mcostalba]

And here is a 6000 game result:

They seem a bit slow to converge.

Would be interesting to see the noise figure in case you keep same conditions but without concurrency.