Search found 2029 matches

by Michel
Fri Aug 02, 2019 5:42 am
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

The problem with testing against ID-0 is after a while it is meaningless. e.g. ID-0 vs ID-600 Score of scorpio-nn1 vs scorpio-nn2: 2 - 63 - 1 [0.038] 66 Elo difference: -561.93 +/- nan ID-0 vs ID-800 Score of scorpio-nn1 vs scorpio-nn2: 4 - 77 - 1 [0.055] 82 Elo difference: -494.44 +/- 240.49 The f...
by Michel
Wed Jul 31, 2019 4:48 pm
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

The joinntdist was still running after 10 minutes before i had to stop it. I will try again later to see what kind of error bounds it produces compared to the full hessian inverse method which seem to be the better approach so far IMO. Well near its maximum the posterior is multivariate Gaussian bu...
by Michel
Wed Jul 31, 2019 6:32 am
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

The joinntdist was still running after 10 minutes before i had to stop it. I will try again later to see what kind of error bounds it produces compared to the full hessian inverse method which seem to be the better approach so far IMO. Well near its maximum the posterior is multivariate Gaussian bu...
by Michel
Tue Jul 30, 2019 9:50 pm
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

Ok I am guessing that exactdist uses the (1-dimensional) posterior for one elo assuming the other elos are exact.

And jointdist uses the true posterior. It seems to me that for more than a few players the naive (non-Monte Carlo) implementation would take a lot of memory and would be slow.
by Michel
Tue Jul 30, 2019 9:28 pm
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

Maybe these terms refer to the exact posterior distribution suitably discretized? But how would one avoid dimensional explosion? I looked in the source of BayesElo but I could not understand it. I am not an expert in Bayesian statistics, but I thought the mathematically exact methods use Monte Carlo...
by Michel
Tue Jul 30, 2019 8:25 pm
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

There are two more options for computing intervals jointdist [p] ... compute intervals from joint distribution exactdist [p] ... compute intervals assuming exact opponent Elos With exactdist Rank Name Elo + - games score oppo. draws 1 scorpio-146 415 40 40 200 56% 379 36% 2 scorpio-147 383 56 56 10...
by Michel
Tue Jul 30, 2019 5:57 pm
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

Ah...I see, I do use the same assumption as bayeselo for calculating error bars. However I do the full hessian matrix inversion when calculating likelihood of superiority (LOS). I am a bit rusty with the code since it has been too long. 827 //covariance = inverse of information matrix (negative hes...
by Michel
Tue Jul 30, 2019 5:16 pm
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

@Daniel, How do you calculate the error bars? I seem to recall BayesElo does it by default by assuming that the elos of the other engines are fixed. But then of course the error bar does not represent the variance of the absolute elo. The correct way to do it is to invert the Hessian of the log lik...
by Michel
Tue Jul 30, 2019 4:29 pm
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

@Daniel, How do you calculate the error bars? I seem to recall BayesElo does it by default by assuming that the elos of the other engines are fixed. But then of course the error bar does not represent the variance of the absolute elo. The correct way to do it is to invert the Hessian of the log like...
by Michel
Tue Jul 30, 2019 11:08 am
Forum: Computer Chess Club: Programming and Technical Discussions
Topic: best way to determine elos of a group
Replies: 51
Views: 9302

Re: best way to determine elos of a group

I am sure you are all familiar with selfplay elo graphs of lc0 are considered not so reliable ( sometimes called its "ego" due to its inflated value ). I think this is mostly because each network is being tested against the previous network -- maybe they do more but for this discussion lets stick w...