Moderator: Ras
When testing engines, I also think this would be the fair way to do it. Short self-generated books with a lot of positions. They don't need to be close to equal, or repeated, just varied and randomly selected from within that book.
hgm wrote: ↑Mon Jan 11, 2021 2:24 pmHow can that be? The width of the Elo curve, which can be seen as the standard deviation of the actual performance difference is 280 Elo. Pawn odds gives an advantage that corresponds to about 100 Elo. Let's say 140 to be generous. If I randomly assign an opening advantage with a standard deviation of half a Pawn (which I would consider quite unbalanced), the variances should add, and the resulting standard deviation should increase by only a factor sqrt(1 + 0.25^2) = sqrt(1.0625). It would require 6.25% more games to compensate that. With a standard deviation of the opening advantage of 1/4 Pawn it should have alread dropped to 1.6%.Michel wrote: ↑Wed Jan 06, 2021 6:56 am Replaying games with reversed colors reduces the variance of the test outcome (one should use the pentanomial model to correctly estimate this variance). So you need fewer games to reach a decision. This effect is quite substantial. Fishtest (which is the gold standard in engine testing) uses a very balanced book and there is still a 5% saving. With their previous slightly less balanced book it was 10%. With very unbalanced books it is much more.
The math is discussed here.
http://hardy.uhasselt.be/Fishtest/accou ... entity.pdf
This document is actually about comparing the trinomial and the pentanomial model, but this is the same problem.
The white advantage is about 1/6 of a Pawn. Even if I don't alternate colors, but randomly decide each game which player will have black or white, it should only require 1.7% more games to get the same accuracy.
White has much more mobility with a Knight and Bishop, so I looked at this position and gave it a shot to practice my endgameUri Blass wrote: ↑Mon Jan 11, 2021 7:50 amI doubt if strong humans are going to lose assuming they play for a draw.abgursu wrote: ↑Sun Jan 10, 2021 1:26 pmI played this when I learn chess in my childhood
Funny but there is a lot of losses happening. Komodo is the best in Non-NNUE engines but after NNUE it must be Eman or Dragon. They both won against SF but I never tested them with each other.
It may be interesting to see a human-computer match from this position.
for comp-comp games you can also use unbalanced positions like the following position
[d]4k3/pppppppp/8/8/8/8/P6P/4KBN1 w - - 0 1
Well, 30 Elo is already larger than the white advantage. I would not call that anywhere near balanced. Of course at these high draw ratios it might be better to use a very unbalanced book with reversed colors, to keep the draw ratio at 50%. But that is another issue.
My 2 cents on any chess Opening when testing engines is that as long as both engines play both sides , first with White and then the reverse it really does NOT matter for testing purpose, but most programmers prepare Opening Books specifically for their engines to get them into position suited for their playing style, just like when human GMs prepare opening to play versus other GMshgm wrote: ↑Mon Jan 11, 2021 7:03 pmWell, 30 Elo is already larger than the white advantage. I would not call that anywhere near balanced. Of course at these high draw ratios it might be better to use a very unbalanced book with reversed colors, to keep the draw ratio at 50%. But that is another issue.
The Elo model does not predict anything about the suitability of a book for testing.hgm wrote: ↑Mon Jan 11, 2021 7:03 pmWell, 30 Elo is already larger than the white advantage. I would not call that anywhere near balanced. Of course at these high draw ratios it might be better to use a very unbalanced book with reversed colors, to keep the draw ratio at 50%. But that is another issue.
Uri Blass wrote: ↑Mon Jan 11, 2021 7:50 amI doubt if strong humans are going to lose assuming they play for a draw.abgursu wrote: ↑Sun Jan 10, 2021 1:26 pmI played this when I learn chess in my childhood
Funny but there is a lot of losses happening. Komodo is the best in Non-NNUE engines but after NNUE it must be Eman or Dragon. They both won against SF but I never tested them with each other.
It may be interesting to see a human-computer match from this position.
for comp-comp games you can also use unbalanced positions like the following position
[d]4k3/pppppppp/8/8/8/8/P6P/4KBN1 w - - 0 1
