Devlog of Leorik

[algerbrex]

I just finished playing a very fun game between Leorik (white) and Blunder (black) since I hadn't actually seen the two engines play a game myself before. Both engines seemed to be playing pretty well for most of the game, until move 47.

[pgn]
[Event "?"]
[Site "?"]
[Date "2022.05.31"]
[Round "?"]
[White "Leorik-2.0.2"]
[Black "Blunder 7.6.0"]
[Result "0-1"]
[ECO "C42"]
[GameDuration "00:06:25"]
[GameEndTime "2022-05-31T12:21:03.902 Central Daylight Time"]
[GameStartTime "2022-05-31T12:14:37.998 Central Daylight Time"]
[Opening "Petrov"]
[PlyCount "132"]
[TimeControl "40/120"]
[Variation "Nimzovich attack"]

1. e4 {+0.08/14 1.4s} e5 {-0.15/15 3.9s} 2. Nf3 {+0.22/15 1.2s}
Nf6 {-0.23/14 3.0s} 3. Nxe5 {+0.37/15 1.5s} d6 {-0.41/14 3.0s}
4. Nf3 {+0.18/16 1.8s} Nxe4 {-0.40/16 3.9s} 5. Nc3 {+0.30/15 1.1s}
Nxc3 {-0.19/17 3.8s} 6. dxc3 {+0.22/15 2.0s} Be7 {-0.17/17 2.9s}
7. Bc4 {+0.06/14 1.3s} O-O {-0.06/17 2.9s} 8. O-O {+0.13/15 2.2s}
c6 {-0.08/16 2.9s} 9. Bf4 {+0.09/15 1.6s} d5 {-0.11/16 3.8s}
10. Be2 {-0.03/14 2.1s} Nd7 {+0.01/14 2.9s} 11. c4 {+0.04/15 2.0s}
Nc5 {0.00/14 2.9s} 12. cxd5 {0.00/14 0.94s} cxd5 {0.00/14 2.9s}
13. c4 {+0.07/15 1.4s} Be6 {0.00/14 2.9s} 14. Rc1 {+0.07/15 1.3s}
dxc4 {+0.04/16 2.9s} 15. Bxc4 {0.00/16 1.6s} Qxd1 {0.00/18 2.9s}
16. Rfxd1 {0.00/17 1.6s} Bxc4 {+0.02/17 2.9s} 17. Rxc4 {+0.01/17 3.5s}
Rfd8 {-0.05/18 2.9s} 18. Rcd4 {0.00/16 1.4s} Ne6 {-0.02/18 2.9s}
19. Rxd8+ {-0.02/20 2.6s} Rxd8 {-0.03/20 2.9s} 20. Rxd8+ {-0.02/20 1.9s}
Bxd8 {-0.03/19 2.9s} 21. Be3 {-0.02/17 1.1s} a5 {-0.04/19 2.9s}
22. b3 {0.00/17 1.1s} Bc7 {-0.06/17 2.9s} 23. h4 {+0.09/17 1.8s}
Kf8 {-0.06/18 2.9s} 24. h5 {+0.15/19 2.8s} Nf4 {0.00/18 2.9s}
25. g4 {+0.18/19 3.4s} Ke7 {0.00/19 2.9s} 26. Kf1 {+0.10/18 3.4s}
f5 {+0.10/20 2.9s} 27. Bxf4 {+0.15/21 2.0s} Bxf4 {+0.15/21 2.9s}
28. g5 {+0.04/23 2.3s} Kd6 {+0.35/21 2.9s} 29. Ke2 {+0.02/24 2.7s}
g6 {+0.40/23 2.9s} 30. hxg6 {-0.18/25 5.9s} hxg6 {+0.29/25 2.9s}
31. Kd3 {-0.21/24 2.4s} Kc5 {+0.28/25 2.9s} 32. a4 {-0.16/27 3.4s}
Kd5 {+0.17/24 2.9s} 33. Nh4 {-0.19/26 1.7s} Bxg5 {+0.20/24 2.9s}
34. Nxg6 {-0.23/27 4.4s} Kc5 {+0.22/24 2.9s} 35. Ne5 {-0.20/26 2.0s}
b5 {+0.20/24 2.9s} 36. axb5 {-0.22/26 3.8s} Kxb5 {+0.30/25 2.9s}
37. Ke2 {-0.20/26 9.4s} Kc5 {+0.30/25 2.9s} 38. f4 {-0.20/27 4.2s}
Bd8 {+0.24/23 1.1s} 39. Ke3 {-0.17/30 4.8s} Kb5 {+0.24/26 3.8s}
40. Kd4 {-0.16/33 20s} Bb6+ {+0.27/26 3.3s} 41. Kc3 {-0.16/31 1.3s}
Bg1 {+0.24/27 3.0s} 42. Kd3 {-0.16/31 3.4s} Kc5 {+0.23/27 3.0s}
43. Nf3 {-0.16/30 2.5s} Bf2 {+0.14/30 3.0s} 44. Ne5 {-0.16/31 1.9s}
Bg3 {+0.14/29 3.0s} 45. Ke3 {-0.16/29 1.6s} Bh4 {+0.12/29 3.0s}
46. Nf3 {-0.16/28 2.3s} Bf6 {+0.12/28 3.9s} 47. Ne5 {-0.17/27 0.92s}
Bxe5 {+10.40/30 3.0s} 48. fxe5 {-8.87/32 1.6s} Kd5 {+10.60/30 3.0s}
49. e6 {-9.68/36 3.3s} Kxe6 {+10.86/31 3.0s} 50. Kd4 {-9.90/38 4.7s}
Kd6 {+10.86/30 3.0s} 51. Kd3 {-9.90/33 1.7s} Kd5 {+11.35/29 3.9s}
52. Ke3 {-18.35/34 5.0s} Ke5 {+11.35/28 3.0s} 53. Kd3 {-18.85/34 1.8s}
f4 {+11.24/25 3.8s} 54. Kc4 {-M28/33 6.7s} f3 {+19.75/28 2.9s}
55. Kd3 {-M24/32 4.7s} Kf4 {+20.92/28 4.9s} 56. Kd4 {-M24/26 1.8s}
f2 {+20.10/19 2.9s} 57. Kd5 {-M18/25 3.3s} f1=Q {+19.75/23 3.7s}
58. Kc6 {-M16/23 2.1s} Qa6+ {+19.75/20 2.8s} 59. Kd5 {-M16/24 3.2s}
Qb6 {+21.34/23 2.8s} 60. Kc4 {-M12/24 2.1s} Ke4 {+21.34/22 2.8s}
61. Kc3 {-M10/25 1.7s} Qb4+ {+21.34/22 2.8s} 62. Kc2 {-M10/26 2.2s}
Qa3 {+M9/20 2.8s} 63. b4 {-M8/24 1.4s} Kd4 {+M7/20 2.8s} 64. bxa5 {-M6/26 2.2s}
Qa2+ {+M5/21 2.8s} 65. Kd1 {-M4/27 1.7s} Kd3 {+M3/23 2.8s} 66. Ke1 {-M2/29 2.1s}
Qe2# {+M1/24 2.8s, Black mates} 0-1
[/pgn]

I was curious why 47. Ne5 was such a bad blunder, and did a little analysis; it appears it actually seems to lead to Zugzwang for white. Something I had never personally seen before in a game, engine, or human:


[fen]8/8/4k3/p3Pp2/5K2/1P6/8/8 w - - 3 50[/fen]


Taking the position and having Leorik analyze it after the game shows it clearly can spot the Zugzwang after only a couple of plies of searching, so I'm not posting this as some sort of bug report, since it was a Blunder likely due to bullet time-control I chose. Just thought it was an interesting example of null-move pruning probably going wrong at lower depths

[lithander]

I was curious why 47. Ne5 was such a bad blunder, and did a little analysis; it appears it actually seems to lead to Zugzwang for white. Something I had never personally seen before in a game, engine, or human:

Uh that's a serious blunder. With Leorik 1.0 (that does no unsafe prunings) the PV switches from f3e5 to the best move e3d3 at depth 5 already. For Leorik 2.1 it takes until depth 16 for the PV to switch to e3d3. But reaching depth 16 takes only 80ms on my machine... How fast were your time control settings exactly?

I hope there's nothing more serious going on here than just a lack of processing time. I might run a few fast games through an analysis engine to identify blunders like that and see how long it takes for the PV to switch to the correct line. Could lead to a valuable set of positions for comparing different versions of the engine. Time to depth measures just the performance but time-to-bestmove could be a useful metric to approximate playing-strength.

[algerbrex]

Uh that's a serious blunder. With Leorik 1.0 (that does no unsafe prunings) the PV switches from f3e5 to the best move e3d3 at depth 5 already. For Leorik 2.1 it takes until depth 16 for the PV to switch to e3d3. But reaching depth 16 takes only 80ms on my machine... How fast were your time control settings exactly?

Hmm, for me Leorik realizes it's mistake at depth 7:

Code: Select all

go
info depth 1 score cp 3 nodes 22 nps 2200 time 10 pv f3e5
info depth 2 score cp -1 nodes 60 nps 2142 time 28 pv f3e5 c5d5
info depth 3 score cp -15 nodes 258 nps 8322 time 31 pv f3e5 c5b4 e5d7
info depth 4 score cp -24 nodes 624 nps 18909 time 33 pv f3e5 c5d5 e5d3 f6d4
info depth 5 score cp -27 nodes 1517 nps 42138 time 36 pv f3e5 c5b4 e5c4 b4b3 c4a5
info depth 6 score cp -27 nodes 1895 nps 49868 time 38 pv f3e5 c5b4 e5c4 b4b3 c4a5 b3b4
info depth 7 score cp -19 nodes 5350 nps 127380 time 42 pv e3d3 c5b4 d3c2 f6e7 f3e5 e7d6 e5d3
info depth 8 score cp -19 nodes 7779 nps 176795 time 44 pv e3d3 c5b4 d3c2 f6e7 f3e5 e7d6 e5d3 b4a3
info depth 9 score cp -20 nodes 11110 nps 236382 time 47 pv e3d3 c5d5 f3d2 f6e7 d2c4 e7b4 d3e3 b4c5 e3d3
info depth 10 score cp -18 nodes 17116 nps 322943 time 53 pv e3d3 c5d5 f3d2 f6e7 d2c4 e7b4 d3e3 b4c5 e3d3 c5b4
info depth 11 score cp -16 nodes 26184 nps 422322 time 62 pv e3d3 c5d5 f3d2 f6e7 d2c4 e7b4 d3e3 d5e6 c4e5 b4c5 e3d3
info depth 12 score cp -17 nodes 46186 nps 584632 time 79 pv e3d3 c5b4 d3c2 f6g7 f3e5 b4c5 e5c4 a5a4 c4e3 a4b3 c2b3 g7h6
info depth 13 score cp -18 nodes 57060 nps 641123 time 89 pv e3d3 c5b4 d3c2 f6g7 f3e5 b4c5 e5c4 a5a4 c4e3 a4b3 c2b3 g7h6 b3c3
info depth 14 score cp -18 nodes 68630 nps 700306 time 98 pv e3d3 c5b4 d3c2 f6g7 f3e5 b4c5 e5c4 a5a4 c4e3 a4b3 c2b3 g7h6 b3c3 h6f4
info depth 15 score cp -18 nodes 110897 nps 866382 time 128 pv e3d3 c5b4 d3c2 f6g7 f3e5 b4c5 e5c4 a5a4 c4e3 a4b3 c2b3 g7h6 b3c3 h6f4 e3f5
info depth 16 score cp -16 nodes 199429 nps 1159470 time 172 pv e3d3 c5b4 d3c2 f6g7 f3h4 g7h6 h4f5 h6f4 f5e7 b4c5 e7f5 c5d5 c2d3 f4e5 d3e3 d5e6

I do see now I was using Leorik 2.0.2, so I'm not sure how much of a difference that makes? The time control I used for the game was 40 moves in 2 minutes, which I typically use when pitting Blunder against engines to get a feel for their style.

I hope there's nothing more serious going on here than just a lack of processing time.

I wish I would've paid more attention now, so this may just be hindsight bias, but I believe Leorik's blunder may have occurred towards the end of the 2 minutes, so it was in a bit of a time crunch perhaps? I suppose the place to start might be to investigate a bit more into Leorik's time management, which seemed to work well overall.

I hope there's nothing more serious going on here than just a lack of processing time. I might run a few fast games through an analysis engine to identify blunders like that and see how long it takes for the PV to switch to the correct line. Could lead to a valuable set of positions for comparing different versions of the engine. Time to depth measures just the performance but time-to-bestmove could be a useful metric to approximate playing-strength.

Regardless, that sounds like a good plan of action. Let me know how that goes!

[lithander]

I do see now I was using Leorik 2.0.2, so I'm not sure how much of a difference that makes? The time control I used for the game was 40 moves in 2 minutes, which I typically use when pitting Blunder against engines to get a feel for their style.
[...]
I wish I would've paid more attention now, so this may just be hindsight bias, but I believe Leorik's blunder may have occurred towards the end of the 2 minutes, so it was in a bit of a time crunch perhaps?

2 minutes per 40 moves is 3 seconds per move on average right? And when the blunder happened in move 47 the clock was just refreshed 7 moves ago so there should have been no time pressure. When you play the move f3e5 the followup is f6e5 f4e5 c5d5 <something> d5e6 until the pawn is lost for good and the position should now be evaluated around 100cp for white. It get's only worse from there and on depth ~30 even the promoted queen should appear on the radar. With quiescence search the pawn-loss should be detectable as early as depth 5! And Leorik 1.0 (which does no risky prunings) indeed finds it at depth 5!

Leorik 2.x does null-move and all kind of prunings but it also is fast enough to reach depth 30 on such a simple position within just a second. I think there's no excuse and I need to go hunt for a bug... :/

[algerbrex]

2 minutes per 40 moves is 3 seconds per move on average right? And when the blunder happened in move 47 the clock was just refreshed 7 moves ago so there should have been no time pressure.

Ah, that's silly of me, of course, there was no time pressure at move 47

Leorik 2.x does null-move and all kind of prunings but it also is fast enough to reach depth 30 on such a simple position within a second. I think there's no excuse and I need to go hunt for a bug... :/

I might take a look through your code as well, as this has made me curious too. Given everything as you said I can't think of a good reason why Leorik didn't see the right move in the game.

If it helps at all (probably not), the game was played from the start position as normal, with no opening used. So maybe it's reproducible to a degree?

[Mike Sherwin]

I've just released a "minor" new version that adds a pawn structure term to the evaluation.
https://github.com/lithander/Leorik/releases/tag/2.1

The pawn structure evaluation (including pawn hash table) turned out surprisingly simple yet effective! Or at least it feels like it's working well... I'm always struggling to judge when a feature is done enough that it's time to move on. I wish there was a an easier way to asses how much the current implementation exhausts the theoretical potential. So (@all) what's your experience with pawn structure eval terms? How much Elo did you gain from adding it in your engines?

This is probably the last version of Leorik that I'll be able to win against.

Leorik 2.1 is only about 50 Elo stronger so maybe you can still win against it?

Big improvement in playing style! Much more human like. Needs pawn storm code. This was a very interesting game!!
[pgn][Event "Computer chess game"]
[Site "DESKTOP-HFVHK2B"]
[Date "2022.06.01"]
[Round "?"]
[White "Leorik-2.1"]
[Black "Mike Sherwin"]
[Result "0-1"]
[BlackElo "2400"]
[ECO "B50"]
[Opening "Sicilian"]
[Time "12:43:07"]
[Variation "2.Nf3 d6 3.Nc3 Nf6"]
[WhiteElo "2600"]
[TimeControl "30+30"]
[Termination "normal"]
[PlyCount "145"]
[WhiteType "program"]
[BlackType "human"]

1. Nf3 {(Ng1-f3 Ng8-f6 c2-c4 c7-c5 e2-e3 e7-e6 d2-d4 c5xd4 e3xd4 d7-d5
Nb1-c3 Nb8-c6 c4xd5 Nf6xd5 Nc3xd5 Qd8xd5 a2-a3 Bf8-d6 Bc1-e3) +0.24/19 23}
c5 2. e4 {(e2-e4 Nb8-c6 Bf1-b5 a7-a6 Bb5xc6 d7xc6 O-O Ng8-f6 e4-e5 Nf6-d5
Nb1-c3 e7-e6 Nc3xd5 c6xd5 d2-d4 c5xd4 Nf3xd4 Bf8-c5 Bc1-e3) +0.21/19 18} d6
3. Nc3 {(Nb1-c3 Ng8-f6 Bf1-c4 Nb8-c6 O-O e7-e6 d2-d4 c5xd4 Nf3xd4 Bf8-e7
Nd4xc6 b7xc6 Bc1-e3 O-O Qd1-e2 d6-d5 Ra1-d1 Bc8-d7 Bc4-d3) +0.25/19 27} Nf6
4. Bb5+ {(Bf1-b5+ Bc8-d7 Bb5xd7+ Nb8xd7 O-O e7-e5 d2-d3 Ra8-c8 Bc1-e3
Bf8-e7 Ra1-c1 O-O a2-a3 Qd8-b6 b2-b3 h7-h6 Rf1-e1 Rf8-e8) +0.11/18 12} Bd7
5. Bxd7+ {(Bb5xd7+ Nb8xd7 h2-h3 e7-e6 O-O Bf8-e7 Rf1-e1 O-O d2-d3 Ra8-c8
Bc1-e3 Rf8-e8 Ra1-c1 Qd8-b6 Nc3-a4 Qb6-d8 a2-a3 Nd7-e5 Na4-c3) +0.15/19 30}
Nbxd7 6. O-O {(O-O g7-g6 d2-d3 Bf8-g7 Rf1-e1 O-O h2-h3 h7-h5 Bc1-e3 Qd8-b6
b2-b3 Ra8-d8 a2-a4 Nd7-e5 Nf3xe5 d6xe5 Ra1-c1 Rf8-e8) +0.09/18 31} g6 7. d3
{(d2-d3 Bf8-g7 h2-h3 O-O Bc1-e3 Qd8-b6 Qd1-d2 Ra8-d8 a2-a4 a7-a6 a4-a5
Qb6-b4 Rf1-b1 Nd7-e5 Nf3xe5 d6xe5 Rb1-e1 c5-c4) +0.10/18 24} Bg7 8. Be3
{(Bc1-e3 h7-h6 Qd1-d2 Qd8-c7 h2-h3 O-O-O Ra1-d1 Kc8-b8 Nc3-b5 Qc7-b6 c2-c4
Nd7-e5 Nf3xe5 d6xe5 Nb5-c3 e7-e6) +0.15/17 30} a6 9. Qd2 {(Qd1-d2 h7-h5
Ra1-e1 b7-b5 a2-a3 Ra8-c8 Be3-f4 b5-b4 a3xb4 c5xb4 Nc3-e2) +0.06/18 38} O-O
10. Bh6 {(Be3-h6 b7-b5 a2-a3 Ra8-c8 h2-h3 Rf8-e8 Ra1-c1 c5-c4 Rc1-d1 c4xd3
c2xd3 Nd7-e5 Nf3xe5 d6xe5 Qd2-g5 Qd8-d4 Bh6xg7 Kg8xg7) +0.24/18 30} b5 11.
h3 {(h2-h3 Ra8-c8 Rf1-e1 Nd7-e5 Nf3xe5 d6xe5 Qd2-g5 Bg7xh6 Qg5xh6 b5-b4
Nc3-e2 Rf8-e8 Ra1-d1 Qd8-a5 a2-a3 b4xa3 b2xa3 Qa5xa3) +0.25/18 32} Re8 12.
Rad1 {(Ra1-d1 Nd7-e5 Nf3xe5 d6xe5 Nc3-e2 e7-e6 Rf1-e1 Ra8-c8 Qd2-e3 b5-b4
a2-a3 a6-a5 Bh6xg7 Kg8xg7 a3xb4 c5xb4 c2-c4 h7-h6 Ne2-g3) +0.21/19 31} Bh8
13. Rfe1 {(Rf1-e1 Ra8-c8 Nc3-e2 Nd7-e5 Nf3xe5 d6xe5 a2-a3 e7-e6 Ne2-g3
Nf6-d7 b2-b4 Bh8-f6 c2-c4 Qd8-c7 Bh6-e3 h7-h5 Qd2-c2) +0.22/20 43} Nb6 14.
Ne2 {(Nc3-e2 Nf6-d7 c2-c3 Nd7-e5 Nf3-g5 c5-c4 d3xc4 b5xc4 Ne2-f4 Bh8-f6
Qd2-e2 e7-e6 Ng5-f3 d6-d5 Nf3xe5 Bf6xe5 e4xd5 e6xd5 a2-a3) +0.32/19 26}
Nfd7 15. c3 {(c2-c3 Nd7-e5 Nf3xe5 d6xe5 Ne2-g3 Ra8-c8 Qd2-g5 Bh8-f6 Qg5-g4
b5-b4 Rd1-d2 a6-a5 a2-a3 e7-e6 Ng3-h5 Bf6-e7 Rd2-d1 a5-a4 a3xb4) +0.29/19
22} d5 16. exd5 {(e4xd5 Nb6xd5 c3-c4 Nd5-c7 d3-d4 b5xc4 Qd2-c2 Nc7-e6 d4-d5
Ne6-c7 Bh6-g5 Bh8-f6 Qc2xc4 Nd7-b6 Qc4xc5 Nc7xd5 Bg5xf6 e7xf6) +0.46/18 21}
Nxd5 17. c4 {(c3-c4 Nd5-f6 a2-a4 Qd8-b6 b2-b3 Ra8-c8 Ne2-g3 Bh8-g7 Qd2-e3
e7-e6 Qe3-g5 Qb6-c6 Bh6xg7) +0.50/18 25} N5b6 18. a3 {(a2-a3 Qd8-c7 c4xb5
a6xb5 d3-d4 Nb6-c4 Qd2-c2 e7-e5 d4xc5 Nd7xc5 Ne2-c3 Qc7-c6 Nc3-d5 e5-e4
Nd5-b4 Qc6-f6 Nb4-d5) +0.44/17 67} Qc7 19. Nc3 {(Ne2-c3 b5xc4 d3xc4 Nd7-e5
Qd2-f4 Ne5xf3+ Qf4xf3 Ra8-c8 b2-b3 Bh8-d4 Nc3-e2 Bd4-f6 Ne2-g3 Bf6-d4
Bh6-e3 Rc8-d8 Ng3-e4) +0.46/17 20} bxc4 20. dxc4 {(d3xc4 Nd7-e5 Qd2-f4
Ne5xf3+ Qf4xf3 Ra8-d8 b2-b3 Rd8xd1 Re1xd1 Bh8-d4 Nc3-e2 Bd4-b2 Bh6-c1
Bb2xc1 Ne2xc1 Qc7-e5 Nc1-d3 Qe5-g5 a3-a4) +0.51/19 19} Rad8 21. Bf4
{(Bh6-f4 Nd7-e5 Bf4xe5 Bh8xe5 Nc3-d5 Nb6xd5 Nf3xe5 Nd5-f6 Qd2-e3 Rd8xd1
Re1xd1 h7-h5 b2-b3 Qc7-b7 Rd1-e1 Kg8-g7 a3-a4 Qb7-b4) +0.53/18 16} e5 22.
Bg5 {(Bf4-g5 f7-f6 Bg5-h4 Nd7-f8 Qd2-c2 Nf8-e6 Nc3-e4 g6-g5 Bh4-g3 Bh8-g7
Rd1xd8 Re8xd8 Qc2-e2 Ne6-f4 Bg3xf4 g5xf4 b2-b3 f6-f5 Ne4-g5) +0.38/19 39}
f6 23. Bh4 {(Bg5-h4 Nd7-f8 Qd2-e3 Nb6xc4 Qe3-e2 Nc4-b6 Qe2xa6 Nf8-e6 Rd1xd8
Re8xd8 Nc3-a4 Rd8-d6 Na4xb6 Qc7xb6 Qa6-a4) +0.37/18 24} Nf8 24. Qe2
{(Qd2-e2 Nf8-e6 Nc3-e4 Ne6-f4 Qe2-c2 g6-g5 Bh4-g3 Bh8-g7 a3-a4 a6-a5 Rd1xd8
Re8xd8 Bg3xf4 g5xf4 b2-b3 f6-f5 Ne4-g5) +0.34/17 15} Ne6 25. Ne4 {(Nc3-e4
g6-g5 Bh4-g3 Bh8-g7 Rd1xd8 Re8xd8 a3-a4 Nb6xa4 Re1-a1 Qc7-d7 b2-b3 Na4-b6
Ra1xa6 Qd7-d1+ Qe2xd1 Rd8xd1+ Kg1-h2) +0.12/17 28} g5 26. Bg3 {(Bh4-g3
Bh8-g7 b2-b3 h7-h5 Bg3-h2 h5-h4 a3-a4 Nb6-c8 Rd1xd8 Re8xd8 Re1-d1 Nc8-d6
Ne4xd6 Rd8xd6 Rd1-d5 Ne6-d4 Qe2-d3 a6-a5 Qd3-e3) +0.06/19 48} h6 27. Rxd8
{(Rd1xd8 Re8xd8 Ne4-d2 Bh8-g7 Qe2-e4 Ne6-d4 b2-b3 h6-h5 Qe4-g6 h5-h4 Bg3-h2
Nb6-d7) +0.13/17 22} Rxd8 28. b3 {(b2-b3 h6-h5 Bg3-h2 Bh8-g7 a3-a4 h5-h4
Qe2-c2 Nb6-c8 Re1-d1 Nc8-d6 Ne4xd6 Rd8xd6 Rd1-d5 Ne6-d4 Nf3xd4 c5xd4
Qc2-g6) +0.06/17 36} a5 29. Nc3 {(Ne4-c3 Bh8-g7 Qe2-e4 Qc7-d7 Re1-d1 Ne6-d4
Qe4-g6 Qd7-c7 Nf3-d2 Qc7-f7 Qg6xf7+ Kg8xf7 Nd2-e4 Nd4-e2+ Kg1-h2 Ne2xc3
Ne4xc3 Rd8xd1) +0.05/18 44} a4 30. bxa4 {(b3xa4 Ne6-d4 Qe2-d3 Bh8-g7 Nf3-d2
Nd4-c6 Nc3-d5 Qc7-f7 Nd2-e4 Nb6xd5 c4xd5 Rd8xd5 Qd3-a6 Nc6-d4 Qa6-c8+
Bg7-f8 a4-a5 Qf7-g6) +0.26/18 44} Nd4 31. Qd3 {(Qe2-d3 Bh8-g7 Nf3-d2 Nd4-c6
Nc3-d5 Qc7-f7 Nd2-e4 Nb6xd5 c4xd5 f6-f5 Ne4xc5 f5-f4 Bg3-h2 Rd8xd5 Qd3-b5
Nc6-d4 Qb5-b8+ Kg8-h7) +0.10/18 19} Qh7 32. Ne4 {(Nc3-e4 Nb6xa4 Nf3xd4
e5xd4 Qd3-d1 Qh7-d7 Qd1-h5 Bh8-g7 Qh5-g6 Qd7-e6 Re1-b1 Qe6-f7 Qg6xf7+
Kg8xf7 Rb1-b7+ Kf7-g6 Rb7-c7 Bg7-f8) +0.35/18 16} f5 33. Nxd4 {(Nf3xd4
c5xd4 Qd3-b3 Nb6-d7 Ne4-d6 f5-f4 c4-c5+ Kg8-f8 c5-c6 Nd7-c5 Qb3-b6 Nc5-e6
Nd6-b7 Rd8-e8 Qb6-b4+ Re8-e7 Bg3-h2 Qh7-c2 Nb7-a5) +0.82/19 30} cxd4 34.
Qb3 {(Qd3-b3 Nb6xa4 c4-c5+ Qh7-f7 Qb3xa4 f5xe4 c5-c6 Rd8-c8 Qa4-a5 Qf7-c4
Bg3xe5 Bh8xe5 Qa5xe5 d4-d3 c6-c7 Rc8xc7 Re1xe4 Rc7-c5) +0.63/18 20} Nc8 35.
Nd2 {(Ne4-d2 f5-f4 Bg3-h2 Qh7-f7 a4-a5 Qf7-e6 Qb3-b7 Qe6-e7 Qb7-b4 Bh8-g7
c4-c5 Nc8-a7 Qb4-c4+ Kg8-h7 Qc4-d3+ Kh7-h8 Nd2-e4 Na7-c6 a5-a6) +0.77/19
25} e4 36. c5+ {(c4-c5+ Qh7-f7 Qb3xf7+ Kg8xf7 c5-c6 d4-d3 Re1-b1 f5-f4
Rb1-b7+ Kf7-e6 Bg3-h2 Nc8-d6 Rb7-h7 Rd8-a8 Rh7xh6+ Bh8-f6 h3-h4 Nd6-f5
h4xg5) +0.86/19 32} Qf7 37. c6 {(c5-c6 Qf7xb3 Nd2xb3 f5-f4 Bg3xf4 g5xf4
Re1xe4 Nc8-d6 Re4-e6 Nd6-c4 Nb3-c5 Nc4xa3 Re6xh6 Bh8-e5 Nc5-e6 Rd8-c8
Rh6-g6+) +0.70/18 30} Qxb3 38. Nxb3 {(Nd2xb3 d4-d3 Bg3-h2 Rd8-e8 Bh2-b8
Nc8-e7 c6-c7 Re8-c8 Re1-c1 Bh8-b2 Rc1-c4 Bb2xa3 a4-a5 Ba3-d6 a5-a6 Bd6xc7
Bb8xc7 Ne7-d5 a6-a7 Nd5xc7 a7-a8Q) +0.44/21 31} f4 39. Bxf4 {(Bg3xf4 g5xf4
Re1xe4 d4-d3 c6-c7 Rd8-f8 a4-a5 Kg8-h7 Re4-e1 Rf8-f7 Re1-d1 Rf7xc7 Rd1xd3
Bh8-b2 a3-a4 Rc7-c4 a5-a6 Rc4xa4 Rd3-d7+ Kh7-g6 Nb3-c5) +0.31/21 29} gxf4
40. Rxe4 {(Re1xe4 d4-d3 c6-c7 Rd8-f8 a4-a5 Kg8-h7 Re4-e1 Rf8-f7 Re1-d1
Rf7xc7 Rd1xd3 Bh8-b2 Nb3-d2 Rc7-c6 Nd2-e4 Kh7-g6 Rd3-b3 Bb2-e5 Rb3-b5
Rc6-c1+ Kg1-h2 Be5-d4) +0.07/22 26} d3 41. c7 {(c6-c7 Rd8-f8 a4-a5 Kg8-h7
a5-a6 Rf8-f7 Re4-c4 Bh8-e5 a6-a7 Nc8xa7 c7-c8Q Na7xc8 Rc4xc8 Rf7-d7 Nb3-d2
Rd7-c7 Rc8-d8 Rc7-c3 Nd2-f3 Be5-c7 Rd8-d7+ Kh7-g6) +0.15/22 66} Rf8 42. Kf1
{(Kg1-f1 Nc8-b6 Nb3-c5 d3-d2 Kf1-e2 Bh8-c3 Nc5-e6 f4-f3+ Ke2-d1 Rf8-c8
g2xf3 Kg8-f7) +0.20/21 30} Bc3 43. Nc5 {(Nb3-c5 d3-d2 Kf1-e2 Nc8-b6 Nc5-e6
f4-f3+ Ke2-d1 Rf8-f6 g2xf3 Rf6xf3 Re4-g4+ Kg8-f7 Rg4-f4+) +0.25/21 15} d2
44. Ke2 {(Kf1-e2 Nc8-b6 Nc5-e6 f4-f3+ Ke2-d1 f3xg2 Re4-g4+ Kg8-f7 Ne6xf8
Nb6xa4 Kd1-c2 Na4-b2 Rg4-f4+ Kf7-g8 Rf4-g4+ Kg8-f7 Rg4-f4+ Kf7-g8 Rf4-g4+
Kg8-f7) 0.00/20 10} Nb6 45. Ne6 {(Nc5-e6 f4-f3+ Ke2-d1 f3xg2 Re4-g4+ Kg8-f7
Ne6xf8 Nb6xa4 Kd1-c2 Na4-b2 Rg4-f4+ Kf7-g8 Rf4-g4+ Kg8-f7 Rg4-f4+ Kf7-g8
Rf4-g4+ Kg8-f7 Rg4-f4+ Kf7-g8 Rf4-g4+) 0.00/21 30} f3+ 46. gxf3 {(g2xf3
Rf8xf3 Re4-g4+ Kg8-h7 Rg4-g1 Rf3-f6 Ne6-c5 Rf6-c6 Nc5-e4 Rc6xc7 a4-a5
Rc7-e7 Ke2-d3 d2-d1Q+ Rg1xd1 Bc3xa5 Ne4-d6 Nb6-d5 Kd3-c4 Nd5-f4) -1.14/20
17} Rxf3 47. Rg4+ {(Re4-g4+ Kg8-h8 Rg4-g1 Rf3-f6 Ne6-c5 Rf6-c6 Nc5-e4
Rc6xc7 Ne4xd2 Nb6xa4 Rg1-g4 Rc7-e7+ Rg4-e4 Re7xe4+ Nd2xe4 Bc3-b2 f2-f4
Kh8-g7 f4-f5 Na4-b6 a3-a4) -1.32/21 24} Kh8 48. Rg1 {(Rg4-g1 Rf3-f6 Ne6-c5
Rf6-c6 Nc5-e4 Rc6xc7 Ne4xd2 Nb6xa4 Rg1-g4 Rc7-e7+ Rg4-e4 Re7xe4+ Nd2xe4
Bc3-b2 f2-f4 Kh8-g7 f4-f5 Kg7-f7 Ke2-e3 Bb2-c1+ Ke3-d4 Bc1xa3) -1.34/22 29}
Rf6 49. Nc5 {(Ne6-c5 Rf6-c6 Nc5-e4 Rc6xc7 Ne4xd2 Nb6xa4 Rg1-g4 Rc7-e7+
Ke2-f1 Re7-d7 Rg4xa4 Rd7xd2 Ra4-c4 Bc3-a5 a3-a4 Kh8-g7 f2-f4 Ba5-b6 Rc4-c6
Bb6-d4 Rc6-c7+ Kg7-f6) -1.47/22 29} Rf7 50. Ne4 {(Nc5-e4 Rf7xc7 Ne4xd2
Nb6xa4 Rg1-g4 Rc7-e7+ Ke2-f1 Re7-d7 Nd2-c4 Na4-c5 f2-f4 Kh8-h7 f4-f5 Rd7-d5
Nc4-e3 Rd5-d2 a3-a4 Bc3-d4 Ne3-c4 Rd2-f2+ Kf1-g1) -1.41/21 20} Re7 51. Kd3
{(Ke2-d3 Bc3-a5 Ne4xd2 Re7-d7+ Kd3-e4 Ba5xd2 Rg1-d1 Kh8-g7 f2-f4 Rd7-e7+
Ke4-f3 Re7xc7 Rd1xd2 Rc7-c3+ Kf3-g4 Rc3xa3 Rd2-d6 Nb6xa4 f4-f5 Na4-c5
f5-f6+ Kg7-g6) -1.42/22 27} Ba5 52. c8=Q+ {(c7-c8Q+ Nb6xc8 Rg1-d1 Re7-d7+
Kd3-c2 Rd7-c7+ Kc2-b1 Nc8-b6 Ne4xd2 Rc7-c3 Nd2-e4 Rc3xa3 Rd1-g1 Nb6-d5
Rg1-g6 Ra3xh3 Rg6-a6 Ba5-b4 Kb1-c2 Rh3-h4 Kc2-d3 Rh4-h1 a4-a5 Kh8-g7
Ra6-a8) -1.89/25 40} Nxc8 53. Rd1 {(Rg1-d1 Re7-d7+ Kd3-c2 Rd7-c7+ Kc2-b1
Nc8-b6 Ne4xd2 Rc7-c3 Nd2-e4 Rc3xh3 Rd1-g1 Rh3-b3+ Kb1-a2 Rb3-f3 Rg1-g6
Kh8-h7 Rg6-g5 Rf3xf2+ Ne4xf2 h6xg5 Ka2-b3) -2.00/24 33} Rd7+ 54. Kc2
{(Kd3-c2 Rd7-c7+ Kc2-b1 Nc8-b6 Ne4xd2 Rc7-c3 Nd2-e4 Rc3xh3 Rd1-g1 Rh3-b3+
Kb1-a2 Rb3-f3 Rg1-g6 Kh8-h7 Rg6-g5 Rf3xf2+ Ne4xf2 h6xg5 Ka2-b3 Kh7-g6
Nf2-g4 Kg6-f5 Ng4-e3+) -1.95/23 26} Rc7+ 55. Kb1 {(Kc2-b1 Nc8-b6 Ne4xd2
Nb6xa4 Nd2-e4 Na4-c3+ Ne4xc3 Rc7xc3 a3-a4 Rc3xh3 Rd1-d5 Ba5-c7 Rd5-d7
Bc7-b6 f2-f4 Rh3-f3 Rd7-f7 Kh8-g8 Rf7-f6 Bb6-d4 Rf6xh6 Rf3xf4 Kb1-c2)
-2.00/23 25} Nb6 56. Nxd2 {(Ne4xd2 Nb6xa4 Nd2-e4 Na4-c3+ Ne4xc3 Rc7xc3
a3-a4 Rc3xh3 Rd1-d5 Ba5-e1 Kb1-b2 Rh3-h2 a4-a5 Rh2xf2+ Kb2-b1 Rf2-f6
Rd5-d8+ Kh8-g7 Rd8-a8 Be1-b4 a5-a6 Rf6-f1+ Kb1-c2 Rf1-f2+ Kc2-d3) -1.97/25
30} Nxa4 57. Ne4 {(Nd2-e4 Na4-c3+ Ne4xc3 Rc7xc3 a3-a4 Rc3xh3 Rd1-d5 Ba5-e1
f2-f4 Rh3-a3 Rd5-d4 Kh8-g7 Rd4-c4 Ra3-f3 Kb1-c2 Be1-g3 a4-a5 Bg3xf4 a5-a6
Rf3-f2+ Kc2-d3 Rf2-d2+ Kd3-e4 Bf4-d6) -2.13/24 62} Nc3+ 58. Nxc3 {(Ne4xc3
Rc7xc3 a3-a4 Rc3xh3 Rd1-d5 Ba5-e1 f2-f4 Rh3-a3 Rd5-d4 Kh8-g7 Rd4-e4 Be1-g3
Kb1-b2 Ra3-f3 a4-a5 Bg3xf4 a5-a6 Bf4-e3 Kb2-b3 Kg7-f7 Kb3-c4 h6-h5 Kc4-d5
Rf3-g3 Re4-c4) -2.08/25 37} Rxc3 59. a4 {(a3-a4 Rc3xh3 Rd1-d5 Ba5-e1 f2-f4
Rh3-a3 Rd5-e5 Be1-g3 Re5-e4 Kh8-g7 Kb1-b2 Ra3-f3 a4-a5 Bg3xf4 a5-a6 Bf4-e3)
-2.15/24 15} Rc5 60. Rd6 {(Rd1-d6 Rc5-h5 Rd6-a6 Ba5-b4 Kb1-c2 Rh5xh3 a4-a5
Rh3-a3 Ra6-a8+ Kh8-g7 a5-a6 Bb4-c5 Kc2-b2 Kg7-f7 f2-f3 h6-h5 Ra8-c8 Bc5-e7
Rc8-a8 h5-h4 a6-a7 h4-h3 f3-f4 h3-h2 f4-f5) -1.77/25 45} h5 61. f3 {(f2-f3
Kh8-g7 Kb1-b2 h5-h4 Rd6-e6 Rc5-f5 Re6-e4 Rf5xf3 Re4xh4 Ba5-d8 Rh4-g4+
Kg7-f7 h3-h4 Rf3-h3 Kb2-c2 Rh3-h1 Rg4-b4 Rh1-h2+ Kc2-d3 Kf7-e6 h4-h5 Rh2xh5
Rb4-b8 Bd8-f6) -1.82/24 50} Kg7 62. Kb2 {(Kb1-b2 Ba5-c7 Rd6-d3 Kg7-f6
Kb2-b3 Kf6-g5 Rd3-d7 Kg5-h4 Kb3-b4 Rc5-c1 Rd7-d5 Rc1-b1+ Kb4-c4 Rb1-a1
Kc4-b5 Ra1-a3 Rd5-c5 Bc7-f4 a4-a5 Ra3xf3 a5-a6 Rf3-b3+ Kb5-c6 Rb3xh3)
-1.92/24 28} Kf7 63. Kb3 {(Kb2-b3 Kf7-e7 Rd6-d4 Ke7-f6 Rd4-c4 Rc5-f5 f3-f4
Ba5-d2 Rc4-c6+ Kf6-e7 Rc6-c4 Ke7-e6 Rc4-e4+ Ke6-d5 Re4-e8 Bd2xf4 a4-a5
Bf4-e5 a5-a6 Rf5-f3+ Kb3-a2 Rf3xh3 a6-a7 Rh3-h2+ Ka2-b3 h5-h4) -1.86/26 22}
Ke7 64. Rd4 {(Rd6-d4 Ke7-f6 Rd4-c4 Rc5-f5 f3-f4 Ba5-d2 Rc4-c6+ Kf6-e7
Rc6-c2 Bd2-a5 Rc2-e2+ Ke7-f6 Re2-c2 Rf5xf4 Rc2-c6+ Kf6-g5 Rc6-c5+ Rf4-f5
Rc5xf5+ Kg5xf5 Kb3-c4 Kf5-f4 Kc4-b5 Ba5-e1 a4-a5 Be1-f2) -1.81/26 25} Kf6
65. Rc4 {(Rd4-c4 Rc5-f5 f3-f4 Ba5-d2 Rc4-d4 Bd2-e3 Rd4-e4 Be3-a7 Re4-b4
Ba7-f2 Rb4-c4 h5-h4 Rc4-e4 Bf2-g3 Re4-c4 Kf6-e7 Kb3-c2 Bg3xf4 a4-a5 Bf4-d6)
-1.81/27 34} Rxc4 66. Kxc4 {(Kb3xc4 Kf6-g5 Kc4-b5 Ba5-d2 Kb5-c4 Kg5-f4
Kc4-d3 Bd2-e3 Kd3-e2 Be3-b6 a4-a5 Bb6xa5 Ke2-f2 h5-h4 Kf2-g1 Kf4xf3 Kg1-h1
Ba5-e1 Kh1-g1 Be1-b4 Kg1-h1 Bb4-c5 Kh1-h2 Bc5-f2 Kh2-h1) -5.94/40 17} Kg5
67. Kb5 {(Kc4-b5 Ba5-d2 Kb5-c4 Kg5-f4 Kc4-d3 Bd2-e3 Kd3-e2 Be3-b6 h3-h4
Kf4-g3 f3-f4 Kg3xf4 Ke2-f1 Kf4-g4 Kf1-g2 Kg4xh4 Kg2-f3 Bb6-c7 Kf3-g2 Kh4-g4
Kg2-h1 Kg4-h3 Kh1-g1 Bc7-a5 Kg1-h1 Kh3-g3 Kh1-g1 Ba5-b6+ Kg1-h1 h5-h4 a4-a5
Bb6xa5 Kh1-g1 Kg3-g4) -5.97/39 30} Be1 68. Kc4 {(Kb5-c4 Kg5-f4 Kc4-d3
Kf4xf3 Kd3-c4 Kf3-g3 Kc4-c5 Kg3xh3 Kc5-b6 Be1-f2+ Kb6-b7 h5-h4 a4-a5 Kh3-g4
a5-a6 h4-h3 a6-a7 Bf2xa7 Kb7xa7 h3-h2 Ka7-b6 h2-h1Q Kb6-c5 Qh1-e4 Kc5-d6
Kg4-g3 Kd6-d7 Kg3-f3 Kd7-d6 Kf3-f2 Kd6-c5 Qe4-e5+ Kc5-c6 Kf2-f3) -9.87/40
38} Kf4 69. Kd4 {(Kc4-d4 h5-h4 Kd4-c4 Kf4-g3 Kc4-c5 Kg3xf3 Kc5-b5 Kf3-g2
a4-a5 Kg2xh3 a5-a6 Be1-f2 Kb5-c6 Kh3-g4 Kc6-b7 h4-h3 a6-a7 Bf2xa7 Kb7xa7
h3-h2 Ka7-b6 h2-h1Q Kb6-c5 Qh1-e4 Kc5-d6 Kg4-g3 Kd6-c5 Qe4-g4) -9.87/37 41}
h4 70. Kc5 {(Kd4-c5 Kf4xf3 Kc5-b5 Kf3-g2 Kb5-b6 Be1-f2+ Kb6-b7 Kg2xh3 a4-a5
Kh3-g4 a5-a6 h4-h3 a6-a7 Bf2xa7 Kb7xa7 h3-h2 Ka7-b6 h2-h1Q Kb6-c5 Kg4-f5
Kc5-d6 Qh1-d1+ Kd6-c6 Kf5-f4 Kc6-b6 Qd1-d5 Kb6-a6 Qd5-h5 Ka6-b6 Kf4-e4
Kb6-b7 Qh5-d5+ Kb7-a6 Ke4-f4 Ka6-b6 Kf4-e5 Kb6-a6 Qd5-c6+ Ka6-a5) -9.96/39
12} Kxf3 71. Kb5 {(Kc5-b5 Kf3-g2 Kb5-b6 Be1-f2+ Kb6-b7 Kg2xh3 a4-a5 Kh3-g4
a5-a6 h4-h3 a6-a7 Bf2xa7 Kb7xa7 h3-h2 Ka7-b6 h2-h1Q Kb6-c5 Kg4-f5 Kc5-d6
Qh1-d1+ Kd6-c6 Kf5-f4 Kc6-b6 Qd1-d5 Kb6-a6 Qd5-c6+ Ka6-a5 Qc6-e4 Ka5-b6
Qe4-e1 Kb6-b5 Qe1-e5+ Kb5-a4 Qe5-e4+ Ka4-b5 Qe4-d5+ Kb5-b6 Kf4-e5 Kb6-a6)
-9.96/39 47} Kg3 72. a5 {(a4-a5 Kg3xh3 a5-a6 Be1-f2 a6-a7 Bf2xa7 Kb5-a6
Ba7-g1 Ka6-b5 Kh3-g4 Kb5-a4 h4-h3 Ka4-b4 Kg4-g3 Kb4-a3 h3-h2 Ka3-a4 Bg1-f2
Ka4-b3 h2-h1Q Kb3-c3 Qh1-f3+ Kc3-b4 Qf3-c6 Kb4-b3 Bf2-b6 Kb3-b2 Bb6-d8
Kb2-a3 Bd8-e7+ Ka3-b3 Qc6-b5+ Kb3-a2) -M21/33 12 White resigns} Bxa5 73.
Kxa5 {(Kb5xa5 Kg3xh3 Ka5-b5 Kh3-g3 Kb5-c4 h4-h3 Kc4-d4 Kg3-f4 Kd4-d5 h3-h2
Kd5-e6 h2-h1Q Ke6-d6 Qh1-d1+ Kd6-e6 Qd1-d4 Ke6-e7 Kf4-e4 Ke7-f7 Qd4-d6
Kf7-g7 Ke4-f5 Kg7-h7 Kf5-f6 Kh7-g8 Qd6-g3+ Kg8-f8 Qg3-b8+) -M14/44 44 White
resigns} *
[/pgn]

[dangi12012]

Hey do you have a system setup that grades your commits to get the relative elo change yet?
So you can know exactly know how much stronger or weaker your engine is.

There are many tinkered solutions out there - but I think there is no open source yaml file that works for any git repo natively.
https://docs.github.com/en/actions/host ... ed-runners

If you are interested we can set this up for your repo!
The goal can be a normal workflow of commit etc. and asynchrounously your CI pipeline will message the strength change data for that commit.

[lithander]

Hey do you have a system setup that grades your commits to get the relative elo change yet?
So you can know exactly know how much stronger or weaker your engine is.

There are many tinkered solutions out there - but I think there is no open source yaml file that works for any git repo natively.
https://docs.github.com/en/actions/host ... ed-runners

If you are interested we can set this up for your repo!
The goal can be a normal workflow of commit etc. and asynchrounously your CI pipeline will message the strength change data for that commit.

Do you mean something like fishtest or openbench?

I always assumed my engine wouldn't be complient with openbench's way of building engines from source because it needs the .Net toolchain. And to set up something like that myself I lack the dedicated hardware that would just wait for these kind of tasks and supply a result in no time. So I'm setting the tests up on my personal computer when I'm not working or gaming at the moment.

Falsifying a small patch with the expected gain of just a few Elo takes a lot of compute, sadly.

[lithander]

Big improvement in playing style! Much more human like. Needs pawn storm code. This was a very interesting game!!

Thanks for playing the new version and glad to hear I'm making some progress in the direction of style!

A pawn storm.... is that what you did on the King Side? Moving a phalanx of pawns together? Leorik should have countered that better, you mean?

[dangi12012]

Do you mean something like fishtest or openbench?

I always assumed my engine wouldn't be complient with openbench's way of building engines from source because it needs the .Net toolchain. And to set up something like that myself I lack the dedicated hardware that would just wait for these kind of tasks and supply a result in no time. So I'm setting the tests up on my personal computer when I'm not working or gaming at the moment.

Much simpler. Yaml files provide an easy easy way to set that up. You just checkin a single file and git will start to execute you continuus integration steps against workers. These workers can be self hosted and I even have some spare machines.

If its enough to get a small standard deviation of elo per run remains to be seen - but running a tournament against itself in an 8x8 grid with 4x master and 4x commit should eliminate noise.

There has to be math already done somewhere that will give a mathemtical sound confidence interval for tournament results.