The trouble with UHO

[dkappe]

I find Pohl’s UHO openings very useful for testing. But there are some questions I have, in particular this one:

Is a test between two engines using the most common uho book predictive of their performance using a balanced book? It seems intuitively correct, but I don’t recall anyone doing an experiment on this.

[Graham Banks]

First of all, I think we need to define what the evaluation parameters are for balanced and unbalanced books.

[Graham Banks]

My conditions for a balanced opening line is that in the first 10 moves out of book, there should be at least one evaluation that is less than 0.70.

[Krzysztof Grzelak]

My conditions for a balanced opening line is that in the first 10 moves out of book, there should be at least one evaluation that is less than 0.70.

I think between 0.30 - 0.40. Weaker engines at 0.70 might not pick up anymore.

[Graham Banks]

My conditions for a balanced opening line is that in the first 10 moves out of book, there should be at least one evaluation that is less than 0.70.

I think between 0.30 - 0.40. Weaker engines at 0.70 might not pick up anymore.

Most lines in my balanced books would meet that, but there are a few outliers.

[Graham Banks]

Is a test between two engines using the most common uho book predictive of their performance using a balanced book? It seems intuitively correct, but I don’t recall anyone doing an experiment on this.

Wouldn't looking at the various rating lists give you that answer?

[dkappe]

Is a test between two engines using the most common uho book predictive of their performance using a balanced book? It seems intuitively correct, but I don’t recall anyone doing an experiment on this.

Wouldn't looking at the various rating lists give you that answer?

Most of the uho tests are bullet/private.

[lkaufman]

I find Pohl’s UHO openings very useful for testing. But there are some questions I have, in particular this one:

Is a test between two engines using the most common uho book predictive of their performance using a balanced book? It seems intuitively correct, but I don’t recall anyone doing an experiment on this.

The question needs clarification. It is extremely obvious that testing with uho books is not predictive of elo differences between unequal engines, it greatly exaggerates such differences (unless the differences are huge). I suppose you mean, is the engine that wins with UHO books also likely to win a long match against the same engine with balanced books, never mind the score? Based on all my testing, I would say the correlation (in this later sense) is high but not perfect. Especially with neural nets, some may be better trained on "normal" openings, and others on positions not likely to arise by choice. So search changes probably will benefit the same engine regardless of book, but net changes could well favor one book or the other. But then, you can also say that with long time controls and many cores and top engines, normal books will always show zero elo (roughly) since nearly every game will be a draw, in which case the question has no meaningful answer.

[pohl4711]

I find Pohl’s UHO openings very useful for testing. But there are some questions I have, in particular this one:

Is a test between two engines using the most common uho book predictive of their performance using a balanced book? It seems intuitively correct, but I don’t recall anyone doing an experiment on this.

The question needs clarification. It is extremely obvious that testing with uho books is not predictive of elo differences between unequal engines, it greatly exaggerates such differences (unless the differences are huge). I suppose you mean, is the engine that wins with UHO books also likely to win a long match against the same engine with balanced books, never mind the score? Based on all my testing, I would say the correlation (in this later sense) is high but not perfect. Especially with neural nets, some may be better trained on "normal" openings, and others on positions not likely to arise by choice. So search changes probably will benefit the same engine regardless of book, but net changes could well favor one book or the other. But then, you can also say that with long time controls and many cores and top engines, normal books will always show zero elo (roughly) since nearly every game will be a draw, in which case the question has no meaningful answer.

IMO, you said all, that can be said here. I dont think, that there is a need for more clarification. Especially because on my website, you find testruns with classical books compared with uho (and other unbalanced openings concepts) with many games and 2 different thinking-times,
3min+1sec here: https://www.sp-cc.de/anti-draw-openings.htm
5min+3sec here: https://www.sp-cc.de/uho_2022.htm

If there is any doubt, because of engine with nnue nets trained with uho, my AntiDraw openings collection offer a lot more opening concepts, containing unbalanced openings (Chess324, Drawkiller, NBC, NBSC etc.), just download the whole package and make your choice!

[pohl4711]

First of all, I think we need to define what the evaluation parameters are for balanced and unbalanced books.

Simple: Elo-spreading and draw-rate. What else? That were the parameters for me, when I built my unbalanced openings.