Trying to CLOP again

[xr_a_y]

I gave another shot to CLOP for search parameter.

My first try was about static null move (depth limit and coefficient per depth)

So my params were

Code: Select all

IntegerParameter staticNullMoveMaxDepth0 0 15
IntegerParameter staticNullMoveMaxDepth1 0 15
IntegerParameter staticNullMoveDepthCoeff0 30 300
IntegerParameter staticNullMoveDepthCoeff1 30 300

The outputed CLOP plot shows poiints everywhere, not concentrated on specific area even after many games. So I guess this is not a success.

But CLOP "Mean" output and "Win rate" seems promising (if I read them well)

95% UCB 45
Elo +17
85% LCB -9

How to interpret those data ?

[xr_a_y]

Nobody about CLOP ?

Now my session had run for more than 20000 samples, the "mean" values are still more or less centered on the experiment domain (which I understand as "there are sample points everywhere" and not in a specific area) but still somehow the announced elo and win rate are positive.

Shall I wait more ? Is it just luck and those "mean" values are indeed interesting ? Why CLOP never shows "max" values my the result table ?

[Alayan]

Terje ran a CLOP tune on a code change I did in Ethereal.

It looked good on paper.



Except all SPRT tests derived from the values suggested by CLOP failed, and quite resoundingly at that.

How the hell could there be a mean +11.7 elo perf over 26K games if it didn't manage to suggest decent values remains a complete mystery.

[xr_a_y]

And what's displayed on your mean/max tab ? Nothing on "max" also ?

[Joerg Oster]

Terje ran a CLOP tune on a code change I did in Ethereal.

It looked good on paper.



Except all SPRT tests derived from the values suggested by CLOP failed, and quite resoundingly at that.

How the hell could there be a mean +11.7 elo perf over 26K games if it didn't manage to suggest decent values remains a complete mystery.

Way too less games!

And the numbers given by CLOP are too optimistic and must be taken with care!
Which is also documented in the readme.

[Alayan]

26K games is already a significant resource investment, and more importantly, the "All" performance being high is completely abnormal.

If it isn't able to output values that perform better than the baseline, it makes no sense that all games played, including many trying random garbage values, would have such a performance.

Small sample size is not a valid explanation, the 99% LB for "All" was still +6 elo ; so there is something else going on, something very wrong. For all I know, it could be as stupid as the baseline always getting to play black.

[xr_a_y]

Code: Select all

Interpreting statistics: clop does not estimate strength accurately
-------------------------------------------------------------------
Win rates displayed in clop-gui are biased. The win rate over samples with
w(x)=1 ("Central" column) is often too optimistic. The win rate over all
samples ("All" column) is pessimistic. clop cannot estimate accurately how
strong the program is at the maximum.

Is "All" is pessimistic and "All" is already +5 or +8 elo I don't know what to think.

I'd like to know if it is ok that I have nothing in my "Max" table (I have something in "mean").