Finding equivalent times for different engines

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

Uri Blass
Posts: 10788
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: Finding equivalent times for different engines

Post by Uri Blass »

The main question is how much do you know about the difference between the engines.

If your previous knowledge is relatively good then I agree that it is better to reduce the influence of the first games but you should always decide to have smaller difference for later games.

I agree that it is possible to try to calculate rating based on assumptions that doubling of the time give 70 elo or something like that but it does the model more complicated when it is not clear if you get big improvement from it.

A possible improvement may be to consider draw for white as different from draw for black so you change the time by a smaller factor after draws and also have slightly different factors for win of white and win of black.

You may have something like

(1+1/sqrt(n+19)) for win of black
(1+1/sqrt(n+20)) for win of white
(1+1/sqrt(n+200)) for draw with black

you also divide by the same numbers if the stronger engine at equal time control draw with white or lose.

I do not claim that it is the best formula and you may have different numbers than 19,20 and 200

I produced the numbers 19,20 to reduce the importance of the first games if the importance of the first games is too high and the best numbers are dependent also on your apriory knowledge about the difference in strength between the programs.
User avatar
hgm
Posts: 28353
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Finding equivalent times for different engines

Post by hgm »

What you want is to determine the zero of a function (Elo vs TC) which you suspect to be smooth, but which you can sample only with noise. (If the function itself was noisy, the concept of the zero point would not be well-defined). Sampling the function very close to the zero is not necessarily the best way to determine the location of the zero. Sample points far away help to get a better fix on the slope or curvature of the function, which then helps to relate samples close to the zero (which you can average with offsets based on the slope and curvature to reduce their statistical noise).
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Finding equivalent times for different engines

Post by Sven »

I have created a second thread in the "Programming and Technical Discussions" subforum, with the intention to continue the discussion there if you like.

Sven