Just found a 55 elo bug in SlowChess 2.1

[jonkr]

I thought this was interesting as while I often find bugs, usually the playing strength difference is small enough I can't measure it, I never expected anything in my chess engine to be broken enough to lose 55 elo (measured by 6000 15s + .15 blitz games.)

Recently I was converting Piece types with and without color to enum classes to use as parameters and return values since I've found small bugs caused by mixing up the two in the past. This lead to me looking over my history update part that decreased the history score of played moves that weren't the best move :

Code: Select all

		for (int i = 0; i < searchedQuietMoves.count; i++)
		{
			const ChessMove move = searchedQuietMoves.moves[i];
			assert(!IsCapOrPromo(move, board));
			if (move != bestMove) {
				historyTable.AdjustMove(move, movedPiece, -deltaScore, prevOppMoveIdx, prevOwnMoveIdx );
			}
		}

The movedPiece is a local variable set earlier from bestMove history bonus. So the only moves that were properly reduced were ones with the same piece type as the best move. Otherwise it would reduce moves for the wrong piece type, sometimes invalid sometimes valid.
I can't remember when I added this, it might have been originally added as far back as 1.4.

I had even added CaptureHistory in 2.1 (with same bug and no statistically significant elo gain.) Likewise I was never able to measure elo gain from history related LMR adjustments. So those were a couple clues.

I did previously gain 15 elo in 2.0 to 2.1 (measured with 8000 games) by deleting !isRoot from my LMR condition, but that wasn't a bug, more an assumption that PV focus and possibly missing alternative moves / tactics at root would be bad.

If anyone else has stories about surprisingly major bugs eventually found in a chess engine, feel free to share here.

[Alayan]

Congrats on the 50 elo gain then.

[jonkr]

Thanks. I will probably keep working and see if I can get it near the strength Xiphos 0.6 by the end of the month (able to win a 2000 game bullet match at least once.) Then release version 2.2 whether or not the goal is reached.

Does make me start to think my evaluation & eval tuning might have some worthwhile ideas. Although still far behind the best and I doubt any similarly major mistakes are still lurking, just lack of search tuning in general. I will try to do a more detailed eval/tuning write-up with 2.2

[silentshark]

If anyone else has stories about surprisingly major bugs eventually found in a chess engine, feel free to share here.

I continue to find bugs, but as you say sometimes fixing them gives little improvement, which is weird.

I had a horrible bug a while back in my SEE, which meant the SEE was pretty random in what it did. Fixing that added some ELO, but I was surprised at how little.

[RubiChess]

You are a monster jonkr, still finding so many Elo lying around.
All the "bugs" I found lately lost Elo while fixing them

[silentshark]

You are a monster jonkr, still finding so many Elo lying around.
All the "bugs" I found lately lost Elo while fixing them

Yeah, I know the feeling. I've got plenty of bugs, but just not the +50 ELO flavoured ones

[mvanthoor]

Converting the pieces to an enum is a good idea to avoid mixing them up with squares. I've thought about it by doing this with pieces and squares, but using enums as indexes and converting back and forth between the enum and the integer value is incredibly annoying in Rust. Therefore I'm probably only go an make an enum for the Piece type, because it only has 6 possible values (KQRNBp) instead of 64.

I now have alias definitions such as:

type Square = usize;
type Piece = usize;

They're interchangeable. You'll see. After I convert everything to enums, Rust will release edition 2021, and support type differentiating by aliases so Square and Piece are not the same type anymore even though under water, they are.

[jonkr]

So far I haven't felt the need to have a separate enum for square but not bad idea if I keep surprising myself with bugs. I guess my refactor was more making sure I actually used functions with the enum returns to feed into functions with enum parameters, and had consistent naming, since I had enums before. For the piece types I had a bug somewhere from comparison board.sqs[src] == PAWN since the sq array contains color info too only matched with WHITE_PAWN, so that inspired some refactoring which luckily caught an even bigger bug.

I think screwing up history is one of the very few things that can cause such a large Elo loss (at least with current style of chess programs that have aggressive Late Move Reductions and Late Move Pruning based on move ordering.) In general if I find a bug I fix it and don't even try to measure Elo.

I've kept my test machine running on various tweaks, for the first 4 days there was no clear improvement only confusion and was getting frustrating as usual. But I just found a change that has been +8 elo at 8000 games (4000 self-play, 4000 against other programs). I usually make several changes at once for a test, my reasoning being that lot of changes if reasonable will be about 0 elo difference and this increases my chance of finding something measurable with my very limited testing. The part that I think was main elo gain was using the SEE result of non-pawn quiet moves in the LMR calculation. I already had this computed since it's part of computing a tactical score for each move that I currently use to adjust LMR and LMP (tactics being mate threats, threatening undefended pieces or higher value pieces, forks/skewers, advanced passed pawn push, moves that escape or defend threats, cover pawn promo square, stuff like that.) That system was +14 elo last I measured, but that might be have been because of my broken history Early on before I was doing modern LMR/LMP it was around +25 elo, but that was probably because any increase in selectivity was helpful.

The SlowChess code in general still contains a lot of questionable search and eval term ideas but as it's gotten stronger I feel a bit better that there are some surviving ideas that are not outright bad.

[lucasart]

Converting the pieces to an enum is a good idea to avoid mixing them up with squares. I've thought about it by doing this with pieces and squares, but using enums as indexes and converting back and forth between the enum and the integer value is incredibly annoying in Rust. Therefore I'm probably only go an make an enum for the Piece type, because it only has 6 possible values (KQRNBp) instead of 64.

I now have alias definitions such as:

type Square = usize;
type Piece = usize;

They're interchangeable. You'll see. After I convert everything to enums, Rust will release edition 2021, and support type differentiating by aliases so Square and Piece are not the same type anymore even though under water, they are.

I tried Rust, briefly, after looking at some Rust evangelist video on Youtube. I just wanted to have enum for color, piece, square, file, rank, square:

• you can't even enumerate like in C. You have to write the values of each square one by one. What's the point of an enum that can't even enumerate ?
• then I trued to operate on them. You can't. It's unsafe, of course, what if the result of the operation is not a legal enum value. Ok, so I tried to cast them, can't do that either. Right, so I tried to enable operators on them. Still not.

That's where I gave up on Rust.

So Rust is safe, because it doesn't allow you to do anything, so that you can't do anything dangerous. No data races, if you can't share memory across threads, no risk of going out of bouds of an enum if you can't do any operations on it, etc.

At least C++ allows you to escape that type safety non-sense when you need it. You can use (unsafe) C-style enum, or type-safe C++ enum, but you can still define operators on them (and assert there to check bounds).

Time wasted fighting against the language to do anything vs. time avoided debugging: I'm not convinved.

Then I tried Go, which is much nicer. But when I saw the size of statically linked Go executable, I gave up on go, and went back to C.

[mar]

Time wasted fighting against the language to do anything vs. time avoided debugging: I'm not convinved.

Absolutely, why bother with safety at language level when we have ASAN (and other sanitizers)?
There's always a price to pay (performance and/or friction).

From what I understand, Rust compiles even slower than C++, so I'm not sold at all. Slow compile times are #1 reason to look for an alternative,
at least for me.

Naturally for a chess program, compile speed doesn't matter much and C is an excellent choice I think.
Actually anything that gives you native performance.

As for 50 elo bugs: never heard of anything like that related to strong engines (we measure improvements, right?).
Can happen at lower levels for sure, but I don't believe in fairy tales.
From my experience, typically any bugfix was no measurable elo gain or barely anything significant at all. 50 elo is huge. (nullmove itself is like what, 70?)