Do you test engines with or without using a book?

[abgursu]

Well nowadays I am working on a rating with Kings & Pawns Games.
[d]4k3/pppppppp/8/8/8/8/PPPPPPPP/4K3 w - - 0 1

[Graham Banks]

Well nowadays I am working on a rating with Kings & Pawns Games.
[d]4k3/pppppppp/8/8/8/8/PPPPPPPP/4K3 w - - 0 1

I get students playing this as part of my coaching.

[abgursu]

Well nowadays I am working on a rating with Kings & Pawns Games.
[d]4k3/pppppppp/8/8/8/8/PPPPPPPP/4K3 w - - 0 1

I get students playing this as part of my coaching.

I played this when I learn chess in my childhood
Funny but there is a lot of losses happening. Komodo is the best in Non-NNUE engines but after NNUE it must be Eman or Dragon. They both won against SF but I never tested them with each other.

[Chessqueen]

Well nowadays I am working on a rating with Kings & Pawns Games.
[d]4k3/pppppppp/8/8/8/8/PPPPPPPP/4K3 w - - 0 1

I get students playing this as part of my coaching.

It would be a good idea to give two knights to the White side and two Bishops to the Black side , Just by looking I believe that neither side has an advantage. But what it seems a coincidence is that both engines started exactly the same, except that Ethereal 3.Nh3 seems to be a strange move instead of waiting to move it toward the center

[pgn][Event "Computer chess game"]
[Site "DESKTOP-OFQ3C0P"]
[Date "2021.01.10"]
[Round "1"]
[White "Ethereal12.75-x64-popcnt"]
[Black "Komodo-13.3-64bit"]
[Result "1/2-1/2"]
[BlackElo "3470"]
[Time "15:17:03"]
[WhiteElo "3470"]
[TimeControl "1200+3"]
[SetUp "1"]
[FEN "2b1kb2/pppppppp/8/8/8/8/PPPPPPPP/1N2K1N1 w - - 12 1"]
[Termination "normal"]
[PlyCount "109"]


1. c4 e6 2. Nc3 b6 3. Nh3 c5 4. b3 Bb7 5. f3 h6 6. e4 g5 7. Ke2 Ke7 8. g3
Bg7 9. Nf2 d6 10. Ncd1 Bc6 11. Ne3 Bb2 12. Nd3 Bf6 13. Ng4 Bg7 14. Ne3 b5
15. h4 Bd7 16. hxg5 hxg5 17. f4 bxc4 18. Nxc4 gxf4 19. Nxf4 Bc6 20. d3 Bh8
21. Kd2 Ba1 22. Ne3 Bg7 23. Ne2 Bh6 24. Nf4 a5 25. Ke2 Bg7 26. a4 Bd4 27.
Nfg2 Kd7 28. Ne1 Kc7 29. Nf3 Bc3 30. Ng5 Be8 31. g4 Kc6 32. Nf3 d5 33. Kd1
Bf6 34. g5 Bd8 35. Ng4 Kd6 36. e5+ Ke7 37. Nf6 Bd7 38. Nxd7 Kxd7 39. Ke1
Kc6 40. Ke2 Bc7 41. Ke3 Kd7 42. Kf2 Kc6 43. Ke3 Bd8 44. Kf4 Be7 45. Kg4 Bd8
46. Nd2 Be7 47. Kh5 Kd7 48. g6 fxg6+ 49. Kxg6 Ke8 50. Nf3 Bd8 51. Kg7 Ke7
52. Nh4 Ke8 53. Nf3 Ke7 54. Nh4 Ke8 55. Nf3 {3-fold repetition} 1/2-1/2[/pgn]

[Chessqueen]

It would be a good idea to give two knights to the White side and two Bishops to the Black side , Just by looking I believe that neither side has an advantage. But what it seems a coincidence is that both engines started exactly the sam. Who do you believe won the reverse side

[pgn][Event "Computer chess game"]
[Site "DESKTOP-OFQ3C0P"]
[Date "2021.01.10"]
[Round "2"]
[White "Komodo-13.3-64bit"]
[Black "Ethereal12.75-x64-popcnt"]
[Result "*"]
[BlackElo "3470"]
[Time "16:37:21"]
[WhiteElo "3470"]
[TimeControl "1200+3"]
[SetUp "1"]
[FEN "2b1kb2/pppppppp/8/8/8/8/PPPPPPPP/1N2K1N1 w - - 12 1"]
[Termination "unterminated"]
[PlyCount "26"]

1. c4 e6 2. Nc3 b6 3. f4 f5 4. h3 Bc5 5. Nf3 Bb7 6. e3 Bb4 7. Kf2 Bxc3 8.
bxc3 Be4 9. Ke2 Ke7 10. d3 Bb7 11. g4 g6 12. Kf2 fxg4 13. hxg4 Bxf3 *[/pgn]

[Chessqueen]

It would be a good idea to give two knights to the White side and two Bishops to the Black side , Just by looking I believe that neither side has an advantage. But what it seems a coincidence is that both engines started exactly the same. Who do you believe won the reverse side. It really does NOT demonstrate or prove anything, but it shows clearly that two Bishops are NOT superior to two Knights


[pgn][Event "Computer chess game"]
[Site "DESKTOP-OFQ3C0P"]
[Date "2021.01.10"]
[Round "2"]
[White "Komodo-13.3-64bit"]
[Black "Ethereal12.75-x64-popcnt"]
[Result "*"]
[BlackElo "3400"]
[Time "16:37:21"]
[WhiteElo "3400"]
[TimeControl "1200+3"]
[SetUp "1"]
[FEN "2b1kb2/pppppppp/8/8/8/8/PPPPPPPP/1N2K1N1 w - - 12 1"]
[Termination "unterminated"]
[PlyCount "26"]


1. c4 e6 2. Nc3 b6 3. f4 f5 4. h3 Bc5 5. Nf3 Bb7 6. e3 Bb4 7. Kf2 Bxc3 8.
bxc3 Be4 9. Ke2 Ke7 10. d3 Bb7 11. g4 g6 12. Kf2 fxg4 13. hxg4 Bxf3 14.
Kxf3 h6 15. e4 c6 16. e5 d6 17. d4 b5 18. c5 dxc5 19. dxc5 Kd7 20. g5 h5
21. Kg2 Kd8 22. Kg3 Kc7 23. a3 Kc8 24. Kf2 Kd8 25. Kg3 Kc7 26. Kh3 Kc8 27.
Kh4 Kc7 28. Kg3 a5 29. Kf3 a4 30. Ke4 Kb7 31. Kd3 Kc8 32. Ke4 Kc7 33. Kf3
Kb7 34. Kg2 Kc7 35. Kf1 Kd7 36. Ke1 Kd8 37. Kd2 Ke8 38. Kd3 Kf8 39. Kd2 Ke8
40. Ke1 Kd8 41. Kd2 Kc8 42. Kd3 Kd7 43. Ke4 Kd8 44. Kd3 Kc8 45. Ke4 Kd8 46.
Kd3 Kc7 47. Ke4 Kc8 48. Kd4 Kb8 49. Ke4 Kc7 50. Kd3 Kb8 51. Ke3 Kc7 52. Ke4
{3-fold repetition} *[/pgn]

[Chessqueen]

If you show this game to a beginner and ask them what did he or she learn from this game, what do you think his answer would be? Well if he or she does not answer properly you as a teacher should tell them that that Rook immediately should look for an open file, and that the King should get out to the center and fight as well.


[pgn][Event "Computer chess game"]
[Site "DESKTOP-OFQ3C0P"]
[Date "2021.01.10"]
[Round "3"]
[White "Ethereal12.75-x64-popcnt"]
[Black "Komodo-13.3-64bit"]
[Result "1/2-1/2"]
[BlackElo "3470"]
[Time "20:10:36"]
[WhiteElo "3470"]
[TimeControl "1200+3"]
[SetUp "1"]
[FEN "r2qk2r/pppppppp/8/8/8/8/PPPPPPPP/R2QK2R w - - 0 1"]
[Termination "normal"]
[PlyCount "64"]


1. e3 f6 2. c4 c6 3. d4 d5 4. Qb3 dxc4 5. Qxc4 Qd5 6. Qxd5 cxd5 7. Kd2 Kf7
8. a4 e5 9. Rhc1 Rhc8 10. b4 Ke6 11. a5 Kd7 12. Kd3 e4+ 13. Kd2 Rxc1 14.
Rxc1 Rc8 15. Rxc8 Kxc8 16. b5 Kc7 17. Kc3 a6 18. Kb4 Kd6 19. g3 g6 20. b6
h6 21. h3 Kc6 22. Ka4 h5 23. Kb4 g5 24. Ka4 h4 25. gxh4 gxh4 26. Kb4 Kd7
27. Ka4 Ke8 28. Kb4 Kd7 29. Ka3 Kc8 30. Kb3 Kd7 31. Ka3 Kd8 32. Kb4 Kd7
{3-fold repetition} 1/2-1/2[/pgn]

[Uri Blass]

Well nowadays I am working on a rating with Kings & Pawns Games.
[d]4k3/pppppppp/8/8/8/8/PPPPPPPP/4K3 w - - 0 1

I get students playing this as part of my coaching.

I played this when I learn chess in my childhood
Funny but there is a lot of losses happening. Komodo is the best in Non-NNUE engines but after NNUE it must be Eman or Dragon. They both won against SF but I never tested them with each other.

I doubt if strong humans are going to lose assuming they play for a draw.
It may be interesting to see a human-computer match from this position.

for comp-comp games you can also use unbalanced positions like the following position
[d]4k3/pppppppp/8/8/8/8/P6P/4KBN1 w - - 0 1

[JohnW]

Well nowadays I am working on a rating with Kings & Pawns Games.
[d]4k3/pppppppp/8/8/8/8/PPPPPPPP/4K3 w - - 0 1

What does this have to do with the original question??

[hgm]

Replaying games with reversed colors reduces the variance of the test outcome (one should use the pentanomial model to correctly estimate this variance). So you need fewer games to reach a decision. This effect is quite substantial. Fishtest (which is the gold standard in engine testing) uses a very balanced book and there is still a 5% saving. With their previous slightly less balanced book it was 10%. With very unbalanced books it is much more.

The math is discussed here.

http://hardy.uhasselt.be/Fishtest/accou ... entity.pdf

This document is actually about comparing the trinomial and the pentanomial model, but this is the same problem.

How can that be? The width of the Elo curve, which can be seen as the standard deviation of the actual performance difference is 280 Elo. Pawn odds gives an advantage that corresponds to about 100 Elo. Let's say 140 to be generous. If I randomly assign an opening advantage with a standard deviation of half a Pawn (which I would consider quite unbalanced), the variances should add, and the resulting standard deviation should increase by only a factor sqrt(1 + 0.25^2) = sqrt(1.0625). It would require 6.25% more games to compensate that. With a standard deviation of the opening advantage of 1/4 Pawn it should have alread dropped to 1.6%.

The white advantage is about 1/6 of a Pawn. Even if I don't alternate colors, but randomly decide each game which player will have black or white, it should only require 1.7% more games to get the same accuracy.