Progress on Blunder

[lithander]

But from this other test you gave, it looks like Blunder may have gained more realistically, around 30-40 Elo at very short time controls, but hopefully more at longer ones.

Here's another (and probably my last, for now^^) match result between Leorik and Blunder 8.0.0 at time controls 10s + 2s increment.

Code: Select all

Score of Leorik-2.1 vs blunder-8.0.0: 225 - 468 - 392  [0.388] 1085
...      Leorik-2.1 playing White: 133 - 212 - 198  [0.427] 543
...      Leorik-2.1 playing Black: 92 - 256 - 194  [0.349] 542
...      White vs Black: 389 - 304 - 392  [0.539] 1085
Elo difference: -79.2 +/- 16.7, LOS: 0.0 %, DrawRatio: 36.1 %

Leorik 2.1 has a CCRL 40/4 rating of 2583. If Blunder 8 is 80 Elo stronger that would put it at 2663 which is 30 Elo ahead of 7.6!

...but I'm pretty sure the ratinglist guys will tear their hair if they see me do napkin math like that!

[algerbrex]

Here's another (and probably my last, for now^^) match result between Leorik and Blunder 8.0.0 at time controls 10s + 2s increment.

Code: Select all

Score of Leorik-2.1 vs blunder-8.0.0: 225 - 468 - 392  [0.388] 1085
...      Leorik-2.1 playing White: 133 - 212 - 198  [0.427] 543
...      Leorik-2.1 playing Black: 92 - 256 - 194  [0.349] 542
...      White vs Black: 389 - 304 - 392  [0.539] 1085
Elo difference: -79.2 +/- 16.7, LOS: 0.0 %, DrawRatio: 36.1 %

Leorik 2.1 has a CCRL 40/4 rating of 2583. If Blunder 8 is 80 Elo stronger that would put it at 2663 which is 30 Elo ahead of 7.6!

...but I'm pretty sure the ratinglist guys will tear their hair if they see me do napkin math like that!

Nice Thomas! I appreciate your willingness to test this new version of Blunder. I have to say I'm mostly glad right now I seem to have gotten rid of those nasty crashes...for now.

30 Elo would be a nice gain given self-play gave around 45-50 Elo, we can't jump to too many conclusions yet

I decided to run a little gauntlet myself with engines that I believe are around Blunder 8's rating, and the results are interesting to say the least:

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw
   0 Blunder 8.0.0                 -18      21     750   47.4%   29.7%
   1 Inanis 1.0.1                   85      45     150   62.0%   36.0%
   2 GreKo 2018.02                  59      52     149   58.4%   16.1%
   3 Zahak 5.0                      44      44     151   56.3%   38.4%
   4 Blunder 7.6.0                 -28      44     151   46.0%   37.7%
   5 Nalwald 1.9                   -68      51     149   40.3%   20.1%

Now it's not done yet, and each engine plays Blunder only 200 times so that needs to be taken into account as well, but Zahak 5.0 is rated at around 2730 on the CCRL, but Inanis is rated around 2693, but according to to the above test, Inanis should be around 2750 on the CCRL. Also Nalwald is around 2663 on the CCRL, but the above test would have it around 2600 flat. The joys of testing using hyper-bullet time controls and too few games

But going off of the numbers above, 2663 doesn't seem like that bad of a ball-park estimate. Which would put Leorik 2.1.9 somewhere around ~2680 Elo on the bullet listing for CCRL!

[algerbrex]

I've made another blog post outlining my progress so far on Blunder. I've found a good bit more time to work on it recently: https://algerbrex.github.io/2022/07/07/ ... title.html

Probably the biggest development is that I'll soon need to decide if I want to add a neural network into Blunder once it breaks 2700, which was my original plan, or see if I can break 2800 using only a single-threaded search, and purely HCE (with the tuning of course), which would be my ideal scenario.

Not sure which path I prefer right now. I'm excited to work on a custom neural network, since I find machine learning projects fascinating anyway, and it'd be a good experience. But I also admire engines near the top of the CCRL at 2900-3000 Elo who have gotten there from purely HCE.

Decisions decisions

If I do go the neural network route, as I said in the post, I'd want to start with something simple and original, like 768 -> 16 -> 1, tuned using games purely from Blunder self-play.

[algerbrex]

Unfortunately, I've also seemed to discover a nasty bug in Blunder, even the dev versions.

Several times in close-to-drawn positions, Blunder has made terrible one-move blunders. In a game during one of Graham's recent tournaments, Blunder blundered a drawn game with rook and knight versus rook and rook. And in a recent game versus Zahak 5.0, Blunder found itself in this position:

[fen]1r2n3/4PB2/4P3/p3B3/P1p3k1/2P4p/1P6/7K b - - 68 105[/fen]

And chose Rb2, losing the game on the spot instead of holding the draw. And to make matters worse it proceeded to throw away its rook after giving a couple of silly checks, only to still happily report a 0 cp evaluation, even as Zahak was moving its bishop to queen it's pawn.

Thankfully I've been able to reproduce the bug...sort of. Putting this position into Blunder:

Code: Select all

position startpos moves e2e4 e7e5 b1c3 c7c5 g1f3 b8c6 f1c4 f8e7 e1g1 g8f6 d2d3 d7d6 c3d5 e8g8 c2c3 f6d5 c4d5 e7f6 c1e3 c6e7 d5b3 b7b6 h2h3 d6d5 f3h2 c8a6 h2g4 d5e4 g4f6 g7f6 d3e4 a6f1 d1g4 e7g6 a1f1 d8c7 b3d5 a8c8 f1d1 f8d8 g4h5 c5c4 h5g4 c7e7 g4h5 g6f8 h5g4 g8h8 a2a4 f8e6 d1f1 d8d6 g4h5 e6g7 h5g4 e7d7 g4h4 f6f5 f2f4 d7d8 h4d8 d6d8 f4e5 d8d7 g2g4 f5g4 f1f7 d7f7 d5f7 g4h3 e5e6 h7h5 g1h2 g7e8 e3d4 h8h7 e6e7 h7h6 e4e5 h6g5 h2h3 e8g7 d4f2 g5f4 e5e6 f4e4 f2g3 e4f5 g3f2 f5e4 f2g3 e4f5 g3h4 f5e5 h3g2 e5f4 g2h3 f4e5 h3g2 e5f4 h4f2 f4g4 f2d4 g7e8 d4e5 h5h4 f7g6 h4h3 g2h2 a7a6 e5h8 g4g5 g6f7 g5g4 h8d4 c8b8 f7g6 a6a5 h2g1 g4f4 g1h2 f4g4 g6f7 g4h4 d4f2 h4g4 f2e3 g4h4 e3f2 h4g4 f2d4 g4h4 d4e3 h4g4 e3f2 e8f6 f2d4 f6e8 f7g6 b8c8 d4b6 c8b8 h2h1 g4g3 b6g1 g3g4 g1h2 b8a8 h2e5 g4g5 g6f7 g5f5 e5g3 f5g4 h1h2 a8c8 g3e5 c8a8 h2g1 g4f5 e5g3 f5f6 g3h4 f6f5 h4f2 f5f6 f2h4 f6f5 g1h2 f5g4 h4g3 a8c8 g3e5 c8a8 h2h1 g4f5 e5d4 f5g4 h1g1 a8b8 d4e5 b8a8 g1f2 g4f5 e5d4 f5g4 d4e5 g4f5 e5d4 f5g4 f2g1 a8b8 g1h1 b8a8 f7g6 a8b8 d4a7 b8a8 a7b6 e8g7 b6d4 g7e8 g6f7 a8b8 d4a7 b8a8 a7d4 a8b8 d4e5 b8b2

Even though virtually every other engine I tested, even ones almost 1000 Elo weaker such as Rustic 3 immediately saw that Black was completely lost. Heck, even a 1000 Elo patzer like myself can see that black is lost, even after giving a couple of spite checks. But Blunder happily says the position is drawn and reports:

Code: Select all

position startpos moves e2e4 e7e5 b1c3 c7c5 g1f3 b8c6 f1c4 f8e7 e1g1 g8f6 d2d3 d7d6 c3d5 e8g8 c2c3 f6d5 c4d5 e7f6 c1e3 c6e7 d5b3 b7b6 h2h3 d6d5 f3h2 c8a6 h2g4 d5e4 g4f6 g7f6 d3e4 a6f1 d1g4 e7g6 a1f1 d8c7 b3d5 a8c8 f1d1 f8d8 g4h5 c5c4 h5g4 c7e7 g4h5 g6f8 h5g4 g8h8 a2a4 f8e6 d1f1 d8d6 g4h5 e6g7 h5g4 e7d7 g4h4 f6f5 f2f4 d7d8 h4d8 d6d8 f4e5 d8d7 g2g4 f5g4 f1f7 d7f7 d5f7 g4h3 e5e6 h7h5 g1h2 g7e8 e3d4 h8h7 e6e7 h7h6 e4e5 h6g5 h2h3 e8g7 d4f2 g5f4 e5e6 f4e4 f2g3 e4f5 g3f2 f5e4 f2g3 e4f5 g3h4 f5e5 h3g2 e5f4 g2h3 f4e5 h3g2 e5f4 h4f2 f4g4 f2d4 g7e8 d4e5 h5h4 f7g6 h4h3 g2h2 a7a6 e5h8 g4g5 g6f7 g5g4 h8d4 c8b8 f7g6 a6a5 h2g1 g4f4 g1h2 f4g4 g6f7 g4h4 d4f2 h4g4 f2e3 g4h4 e3f2 h4g4 f2d4 g4h4 d4e3 h4g4 e3f2 e8f6 f2d4 f6e8 f7g6 b8c8 d4b6 c8b8 h2h1 g4g3 b6g1 g3g4 g1h2 b8a8 h2e5 g4g5 g6f7 g5f5 e5g3 f5g4 h1h2 a8c8 g3e5 c8a8 h2g1 g4f5 e5g3 f5f6 g3h4 f6f5 h4f2 f5f6 f2h4 f6f5 g1h2 f5g4 h4g3 a8c8 g3e5 c8a8 h2h1 g4f5 e5d4 f5g4 h1g1 a8b8 d4e5 b8a8 g1f2 g4f5 e5d4 f5g4 d4e5 g4f5 e5d4 f5g4 f2g1 a8b8 g1h1 b8a8 f7g6 a8b8 d4a7 b8a8 a7b6 e8g7 b6d4 g7e8 g6f7 a8b8 d4a7 b8a8 a7d4 a8b8 d4e5 b8b2
go
info depth 1 score cp 145 nodes 17 nps 9223372036854775808 time 0 pv f7e8
info depth 2 score cp 134 nodes 43 nps 74125 time 0 pv f7e8 g4f3
info depth 3 score cp 165 nodes 185 nps 355906 time 0 pv f7e8 b2b1 h1h2 b1b2 h2g1
info depth 4 score cp 154 nodes 322 nps 598401 time 0 pv f7e8 b2b1 h1h2 b1b2 h2g1 g4f3
info depth 5 score cp 0 nodes 1870 nps 1653696 time 1 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 6 score cp 0 nodes 781 nps 3811615 time 1 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 7 score cp 0 nodes 884 nps 543866 time 2 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 8 score cp 0 nodes 1044 nps 983977 time 3 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 9 score cp 0 nodes 1457 nps 1307899 time 4 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 10 score cp 0 nodes 2228 nps 1958336 time 5 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 11 score cp 0 nodes 4118 nps 1217874 time 8 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 12 score cp 0 nodes 7331 nps 1241406 time 13 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 13 score cp 0 nodes 7987 nps 1097296 time 20 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 14 score cp 0 nodes 10949 nps 1242876 time 28 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 15 score cp 0 nodes 12198 nps 1359214 time 36 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 16 score cp 0 nodes 22653 nps 1308627 time 53 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 17 score cp 0 nodes 37398 nps 1319381 time 81 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 18 score cp 0 nodes 61703 nps 1416047 time 124 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 19 score cp 0 nodes 58088 nps 1503577 time 162 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 20 score cp 0 nodes 109012 nps 1612088 time 229 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 21 score cp 0 nodes 88613 nps 1539136 time 286 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 22 score cp 0 nodes 159964 nps 1541514 time 389 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 23 score cp 0 nodes 299437 nps 1556772 time 581 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 24 score cp 0 nodes 430191 nps 1544815 time 859 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 25 score cp 0 nodes 495562 nps 1672789 time 1155 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 26 score cp 0 nodes 432164 nps 1606339 time 1424 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2
info depth 27 score cp 0 nodes 1551239 nps 1616106 time 2383 pv f7e8 b2b1 h1h2 b1b2 h2g1 b2b1 g1f2 b1b2

I can live with Blunder making some positional Blunders. Those are slowly addressed by incrementally improving the eval, and occasionally the search. But horrible one-move blunders that even I can spot? ' Fraid not. But since it's reproducible, I should be able to hopefully track down the issue pretty quickly.

[algerbrex]

Well....the fix for the aforementioned bug is rather embarrassing.

Apparently, when doing my make/unmake and movegen refactoring for the release of Blunder 8.0.0, I somehow forgot to reset the fifty move rule counter when captures occurred. Oops One little line missing from DoMove.

I think this actually helps to explain the feeling that something was off with Blunder 8.0.0, and the draw rate when testing seemed unreasonably high, especially in self-play.

Now that that's been fixed, time to run a test to see how much Elo Blunder 8.0.0 has been missing out on because it threw away winning positions or blundered draws.

[lithander]

If I do go the neural network route, as I said in the post, I'd want to start with something simple and original, like 768 -> 16 -> 1, tuned using games purely from Blunder self-play.

Maybe as a intermediate step try to tune the HCE of Blunder with a training set exclusively derived from Blunder's own games. So forget all you learned from Zurichess' games and try to learn it again from selfplay games - starting with material only evaluation. At least that's what I'm going to try with Leorik before I transition to NNUEs...

And I will also delay looking into NNUEs until at least version 3.0 (with a lot of minor versions in between) because once you switch I fear there's no coming back to tinker with the HCE. You should really feel like you're "done" with the HCE e.g. that all your ideas that you are curious about have been tried and tested.

Now that that's been fixed, time to run a test to see how much Elo Blunder 8.0.0 has been missing out on because it threw away winning positions or blundered draws.

Leorik is scared! How much Elo was it? And are you going to release a fixed version?

[algerbrex]

Maybe as a intermediate step try to tune the HCE of Blunder with a training set exclusively derived from Blunder's own games. So forget all you learned from Zurichess' games and try to learn it again from selfplay games - starting with material only evaluation. At least that's what I'm going to try with Leorik before I transition to NNUEs...

Good idea! I like that a lot, actually.

Because if I were to start training a neural network, I suppose its data wouldn't really be original, due to using the Zurichess dataset for so much tuning. That definitely sounds like a fun challenge to try to reach the strength of the latest Blunder dev version using only purely self-trained data.

To do this, I would need to go all the way to Blunder 1.0.0, since for that version, all of the evaluation data was 100% original. All of it is likely very bad, but still original I selected basic material values and wrote some very simple PSQT myself, so the engine plays a lot like me. That would be where I would start, downloading it and playing many self-play games.

I'm actually pretty curious if I can do the sort of cyclic tuning you describe, where each tuning session I can gain strength just from the new self-play games.

And I will also delay looking into NNUEs until at least version 3.0 (with a lot of minor versions in between) because once you switch I fear there's no coming back to tinker with the HCE. You should really feel like you're "done" with the HCE e.g. that all your ideas that you are curious about have been tried and tested.

True. I don't think many people would be interested in a switch back to HCE in a new version after you've already given a taste of the power of NNUE.

I'll probably start preliminary experiments at least with NNs right now, just because I find them fascinating in their own right. Might not even be something chess-related, but still something made from scratch using a custom train, not Tensorflow or something.

But don't worry, I still have some HCE ideas on the backlog especially now that I fixed that bug. I actually have several test I need to go back and run since I'm not sure how that bug might have warped their improvement.

Leorik is scared! How much Elo was it? And are you going to release a fixed version?

Well tell Leorik not to worry too much right now after 2K games at 10+0.1s:

Code: Select all

Score of Blunder 8.2.1 vs Blunder 8.2.0: 569 - 550 - 881  [0.505] 2000
...      Blunder 8.2.1 playing White: 306 - 255 - 440  [0.525] 1001
...      Blunder 8.2.1 playing Black: 263 - 295 - 441  [0.484] 999
...      White vs Black: 601 - 518 - 881  [0.521] 2000
Elo difference: 3.3 +/- 11.4, LOS: 71.5 %, DrawRatio: 44.0 %
SPRT: llr 0.0356 (1.2%), lbound -2.94, ubound 2.94

I'm going to try testing at longer TCs as well, something like 15+0.2s or 5+0.5s, but for now the gain isn't huge. Regardless, I'm glad Blunder is no longer making such bad...er...blunders.

Anyhow, I'll definitely be releasing a bug-fixed version soon. Since the current tournament Blunder 8.0.0 is so far along, I'll probably let it finish first, since I don't want Graham to feel like he has to re-start Blunder's games just because of me but I'll release Blunder 8.2.0 before the next division tourney, which hopefully has around +50 Elo now on 8.0.0.

[algerbrex]

Quick update.

The current dev version of Blunder, 8.4.4, should be about ~50 Elo stronger than 8.0.0. Some of the strength is from a couple of bug fixes, but mainly from some search tweaks and evaluation terms that gained some Elo. The most recent addition to the search is singular extensions.

Before this, I had never gotten extensions to work in Blunder. I tried mostly things like passed pawn extensions, which seemed to help a little, but then became regressions.

The approach I took to doing singular extensions is one that I think's fairly common, as far as I can tell. Basically, in PV nodes, where we have a TT move, I check to see if it's singular by doing a recursive search on the same position but excluding the tt move, and seeing if we can get a score of ttScore - margin or better. Since I just care if ttScore - margin can be beaten, I also speed up the search by using a null window centered around ttScore - margin. Here's the gist of the code:

Code: Select all

nextDepth := depth - 1

if !isExtended &&
	depth >= SingularExtensionDepthLimit &&
	ttMove.Equal(move) &&
	isPVNode && ttHit &&
	(entry.Flag == ExactFlag || entry.Flag == BetaFlag) {

	search.Pos.UndoMove(move)
	search.RemoveHistory()

	scoreToBeat := ttScore - SingularMoveMargin
	R := 3 + depth/6

	nextBestScore := search.negamax(depth-1-R, ply+1, scoreToBeat, scoreToBeat+1, &PVLine{}, true, prevMove, move, true)

	if nextBestScore <= scoreToBeat {
		nextDepth += SingularMoveExtension
	}

	search.Pos.DoMove(move)
	search.AddHistory(search.Pos.Hash)
}

Not too complicated. The hardest part, of course, is tweaking the knobs and finding good values for the margin to use, as well as the depth reduction scheme and depth limit. I just used my gut here and guessed on the value of a little more than a pawn for the margin, 4 as the depth limit, and used the same scheme as I'm using for null-move pruning to reduce the depth.

Testing at fairly quick time controls showed it gained about 7 Elo, which isn't too bad. I suspect more would be gained at longer time controls, since hopefully Blunder should now spend more time in certain positions searching the important moves, instead of getting side tracked.

I'll probably release Blunder 8.4.4 soon, maybe after I finally wrap up that king safety paper. But I'd also like to give the testers here time to test 8.0.0, since I would like to see where it falls on the CCRL. If I do release it in the next couple of days, for any testers, feel free to focus on Blunder 8.0.0 since I'm in no rush to get the latest versions tested. Of course at the end of the day do what you want

Oh, and as a small note, I've finally decided on a bit more of a formal versions scheme instead of just incrementing the minor version number whenever I feel like it. The minor version number will be incremented when I make a strength-gaining change. Or occasionally, it might include multiple very small, related strength-gaining changes. The third number will be incremented when I make a bug fix, or non-strength-gaining change. And the major number will be incremented when I deem I've made enough strength changes that warrant a significant enough increase in strength or change, or when I add a major feature or do a major code refactoring.

[Graham Banks]

I'll probably release Blunder 8.4.4 soon, maybe after I finally wrap up that king safety paper. But I'd also like to give the testers here time to test 8.0.0, since I would like to see where it falls on the CCRL. If I do release it in the next couple of days, for any testers, feel free to focus on Blunder 8.0.0 since I'm in no rush to get the latest versions tested. Of course at the end of the day do what you want

Gauntlet starting either tomorrow or Monday.

[algerbrex]

I'll probably release Blunder 8.4.4 soon, maybe after I finally wrap up that king safety paper. But I'd also like to give the testers here time to test 8.0.0, since I would like to see where it falls on the CCRL. If I do release it in the next couple of days, for any testers, feel free to focus on Blunder 8.0.0 since I'm in no rush to get the latest versions tested. Of course at the end of the day do what you want

Gauntlet starting either tomorrow or Monday.

Cool, thanks for the update! Feel free to go ahead with that, regardless of what I do. As I said I appreciate you guys testing and I'm in no rush