That is strange. This is the code to compute the numbers:Daniel Shawul wrote:Is that the reason why the elo I calculate from winningRate percentage sometimes do not match the one displayed on the gui with 95% confidence? For example I saw a 47% winning rate suggesting -21 elo, but only -5 elo is displayed. Also since the current selected parameters do not get all the games played (some early games are probably truncated), how are the mean and confidence intervals estimated ?Note that, in general, you cannot use CLOP to estimate the strength of the optimised program accurately. Win rates produced by CLOP are biased. Win rate over all samples is pessimistic. Local win rate tends to be optimistic. The real win rate of optimal parameters is somewhere in-between.
Code: Select all
/////////////////////////////////////////////////////////////////////////////
void MainWindow::updateWinRate(int Column, double W, double D, double L)
/////////////////////////////////////////////////////////////////////////////
{
double Total = W + D + L;
double Score = W + 0.5 * D;
double Rate = Score / Total;
double TotalVariance = W * (1.0 - Rate) * (1.0 - Rate) +
D * (0.5 - Rate) * (0.5 - Rate) +
L * (0.0 - Rate) * (0.0 - Rate);
double Margin = 1.96 * std::sqrt(TotalVariance) / Total;
const double EloMul = 400.0 / std::log(10.0);
addTableItem( 0, Column, W);
addTableItem( 1, Column, D);
addTableItem( 2, Column, L);
addTableItem( 3, Column, Total);
addTableItem( 5, Column, EloMul * pexperiment->reg.Rating(Rate + Margin));
addTableItem( 6, Column, EloMul * pexperiment->reg.Rating(Rate));
addTableItem( 7, Column, EloMul * pexperiment->reg.Rating(Rate - Margin));
addTableItem( 9, Column, Rate + Margin);
addTableItem(10, Column, Rate);
addTableItem(11, Column, Rate - Margin);
}
Code: Select all
/////////////////////////////////////////////////////////////////////////////
// Outcome model (missing multiplier for draws, should do 1 - w - l?)
/////////////////////////////////////////////////////////////////////////////
double CRegression::ResultProbability(double Rating, COutcome outcome) const
{
switch(outcome)
{
case COutcome::Win:
return CLogistic::f(Rating - DrawRating);
case COutcome::Loss:
return CLogistic::f(-Rating - DrawRating);
case COutcome::Draw:
return CLogistic::f(Rating - DrawRating) *
CLogistic::f(-Rating - DrawRating);
}
return 1.0;
}
/////////////////////////////////////////////////////////////////////////////
// Compute win rate for a rating
/////////////////////////////////////////////////////////////////////////////
double CRegression::WinRate(double Rating) const
{
double W = ResultProbability(Rating, COutcome::Win);
double L = ResultProbability(Rating, COutcome::Loss);
return W + 0.5 * (1.0 - W - L);
}
/////////////////////////////////////////////////////////////////////////////
double CRegression::Rating(double Rate) const
/////////////////////////////////////////////////////////////////////////////
{
const int Iter = 30;
double Max = 10.0;
double Min = -10.0;
double MaxRate = WinRate(Max);
double MinRate = WinRate(Min);
for (int i = Iter; --i >= 0;)
{
double Middle = (Max + Min) * 0.5;
double MiddleRate = WinRate(Middle);
if (Rate > MiddleRate)
{
Min = Middle;
MinRate = MiddleRate;
}
else
{
Max = Middle;
MaxRate = MiddleRate;
}
}
return (Max + Min) * 0.5;
}
In the next version, I will scale the ratings like in bayeselo. This may avoid confusion. But the difference you observe seems bigger than it should be. Anyway, you have all the necessary code above to understand how ratings are computed.
Rémi