I would like to CLOP-timize my piece values. So here's what I have done so far:
* modify me engine so as to understand the command setvalue (as sent by clop-cutechess-cli.py). i have 5 parameters to optimize, PNBRQ opening values (pieces have the same opening and endgame values for now and pawn endgame value is hardcoded to 100, it is the reference point).
* modified the script clop-cutechess-cli.py
* I managed to compile CLOP also, in particular clop-console and clop-gui which are the only ones useful here as I understand.
So it works, and returns "W/L/D" values. When I call the script I can specify a CPU_ID, SEED, and different piece values (amongst my 5 aforementioned parameters).
Now what do I do with that ? What value do I give to CPU_ID ? As I understand CLOP is responsible to send the SEED and the parameter values, but how do I get that to work together ?
PS: While doing this I discovered a few ridiculous bugs in my eval code, which is nice
CLOP/cutechess-cli
Moderators: hgm, Rebel, chrisw
-
- Posts: 438
- Joined: Mon Apr 24, 2006 8:06 pm
Re: CLOP/cutechess-cli
CLOP should also send the cpu id. You just have to list your cpus in the .clop file. See DummyScript.clop for an example. Maybe you missed that you have to write a .clop file in order to start your experiment.
Rémi
Rémi
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: CLOP/cutechess-cli
Thank you Remi. I've been running my CLOP experiment for over 12 hours now, and piece values are starting to converge nicely.Rémi Coulom wrote:CLOP should also send the cpu id. You just have to list your cpus in the .clop file. See DummyScript.clop for an example. Maybe you missed that you have to write a .clop file in order to start your experiment.
Rémi
* what do you think is a reasonable "stopping rule" ? is there a rule of thumb i can apply on the hessian matrix and eigen values ? do you think 10,000 games for example is enough to obtain good estimates of 5 parameters that are very influential like piece values ?
* where do I read the estimated values ? There is a tab called Max in clop-gui where I see a column "mean" for my parameters. Is that the mean since the beginning of the experiment ? (which isn't really what I want, although by a cesaro argument it would converge too) or is it really the clop estimate (which is some king of weighted mean, right?)
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: CLOP/cutechess-cli
My two cents guess regarding the pre-condition to stop would be: all eigen values are strictly negative. Does that make sense to you ?lucasart wrote:Thank you Remi. I've been running my CLOP experiment for over 12 hours now, and piece values are starting to converge nicely.Rémi Coulom wrote:CLOP should also send the cpu id. You just have to list your cpus in the .clop file. See DummyScript.clop for an example. Maybe you missed that you have to write a .clop file in order to start your experiment.
Rémi
* what do you think is a reasonable "stopping rule" ? is there a rule of thumb i can apply on the hessian matrix and eigen values ? do you think 10,000 games for example is enough to obtain good estimates of 5 parameters that are very influential like piece values ?
* where do I read the estimated values ? There is a tab called Max in clop-gui where I see a column "mean" for my parameters. Is that the mean since the beginning of the experiment ? (which isn't really what I want, although by a cesaro argument it would converge too) or is it really the clop estimate (which is some king of weighted mean, right?)
-
- Posts: 438
- Joined: Mon Apr 24, 2006 8:06 pm
Re: CLOP/cutechess-cli
negative eigenvalues is a very bad stopping rule. It may occur extremely early. It may also never happen.
CLOP will optimize forever, so the optimal stopping time depends on how close to the optimal you wish to be. If you want to be within 1 elo of the best you'll have to play many more games than if 10 Elo is good enough.
It is not very easy to estimate the expected simple regret. I don't know how to do it. You can take a look at the plots in the paper to get an idea of the expected simple regret for a given number of games, for different kinds of functions.
You can also get an idea of how uncertain parameter estimation is by looking at the weighted samples in the plot.
"Mean" is the weighted mean of all samples. I recommend it as a choice of weight values, because it is more robust than "Max". "Max" may be better in case CLOP is still sampling on the domain boundaries, though, because then boundaries create a bias. That's why I also recommend using wide intervals for parameters: it is better to let CLOP figure out by itself where it should take samples.
Rémi
CLOP will optimize forever, so the optimal stopping time depends on how close to the optimal you wish to be. If you want to be within 1 elo of the best you'll have to play many more games than if 10 Elo is good enough.
It is not very easy to estimate the expected simple regret. I don't know how to do it. You can take a look at the plots in the paper to get an idea of the expected simple regret for a given number of games, for different kinds of functions.
You can also get an idea of how uncertain parameter estimation is by looking at the weighted samples in the plot.
"Mean" is the weighted mean of all samples. I recommend it as a choice of weight values, because it is more robust than "Max". "Max" may be better in case CLOP is still sampling on the domain boundaries, though, because then boundaries create a bias. That's why I also recommend using wide intervals for parameters: it is better to let CLOP figure out by itself where it should take samples.
Rémi
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: CLOP/cutechess-cli
thank you!
CLOP is really great, it's fascinating to see the convergence slowly happening. and I would never have been able to guess the CLOP-timal values. It's interesting to see how all known theory regarding piece values is a load of BS... For example knights and bishops are now worth close to 4 pawns, not 3, rooks 6 and queens 11.5
of course this depends on other eval terms that (all other things equal) introduce a biais to piece values already and CLOP is there to help me calculate the biais correction embedded in the piece values.
CLOP is really great, it's fascinating to see the convergence slowly happening. and I would never have been able to guess the CLOP-timal values. It's interesting to see how all known theory regarding piece values is a load of BS... For example knights and bishops are now worth close to 4 pawns, not 3, rooks 6 and queens 11.5
of course this depends on other eval terms that (all other things equal) introduce a biais to piece values already and CLOP is there to help me calculate the biais correction embedded in the piece values.
-
- Posts: 134
- Joined: Mon May 16, 2011 6:58 pm
- Location: Denmark
Re: CLOP/cutechess-cli
How many games have you played?lucasart wrote:thank you!
CLOP is really great, it's fascinating to see the convergence slowly happening. and I would never have been able to guess the CLOP-timal values. It's interesting to see how all known theory regarding piece values is a load of BS... For example knights and bishops are now worth close to 4 pawns, not 3, rooks 6 and queens 11.5
of course this depends on other eval terms that (all other things equal) introduce a biais to piece values already and CLOP is there to help me calculate the biais correction embedded in the piece values.
I am trying to cloptimize an experiment version of Sungorus, now it has played 1150 games, but i know there should be much more games.
What is your experience? How many games have you played?
And to Remi, I have a question:
What does "95% UCB, ELO x, 95% LCB" and "95% UCB, winrate x, 95% LCB " mean? It is annoying me to not know what those values are
Regards
Ethan
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: CLOP/cutechess-cli
just over 10 thousand games so far (time control 6"+0.1" 2 games in // as I have a duo core processor)ethanara wrote: How many games have you played?
ethanara wrote: What is your experience?
Code: Select all
Name CLOPUCI
Script ./cutechess-cli/clop-cutechess-cli.py
IntegerParameter P 50 100
IntegerParameter N 200 500
IntegerParameter B 200 500
IntegerParameter R 300 800
IntegerParameter Q 500 1500
Processor cpu1
Processor cpu2
Replications 2
DrawElo 100
H 3
Correlations all
-
- Posts: 438
- Joined: Mon Apr 24, 2006 8:06 pm
Re: CLOP/cutechess-cli
These are bounds on the 95% confidence interval (UCB = upper confidence bound, LCB = lower confidence bound).ethanara wrote:How many games have you played?lucasart wrote:thank you!
CLOP is really great, it's fascinating to see the convergence slowly happening. and I would never have been able to guess the CLOP-timal values. It's interesting to see how all known theory regarding piece values is a load of BS... For example knights and bishops are now worth close to 4 pawns, not 3, rooks 6 and queens 11.5
of course this depends on other eval terms that (all other things equal) introduce a biais to piece values already and CLOP is there to help me calculate the biais correction embedded in the piece values.
I am trying to cloptimize an experiment version of Sungorus, now it has played 1150 games, but i know there should be much more games.
What is your experience? How many games have you played?
And to Remi, I have a question:
What does "95% UCB, ELO x, 95% LCB" and "95% UCB, winrate x, 95% LCB " mean? It is annoying me to not know what those values are
Regards
Ethan
Rémi
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: CLOP/cutechess-cli
Would you consider adding another column to show the weighted stdev of parameters. After thousands of samples, I would be curious to see how much CLOP still moves the paremeters around. Or even LCB/UCB 95% (weighted) if that is feasibleRémi Coulom wrote:"Mean" is the weighted mean of all samples. I recommend it as a choice of weight values