Wow, I am pretty dense sometimes. As I was testing Pedantic in order to release version 0.3, I noticed that its Elo rating was almost 150 points higher when I enabled adjudications in cute chess!! I first thought to myself, that maybe everyone else's engine which I've been testing against just had better end game logic or maybe used egtb. My initial self-play test indicated that Pedantic had gained +184 elo in this release:
Code: Select all
SELF-PLAY ELO
Score of Pedantic 0.3 vs Pedantic 0.2: 573 - 88 - 339 [0.743] 1000
... Pedantic 0.3 playing White: 309 - 39 - 152 [0.770] 500
... Pedantic 0.3 playing Black: 264 - 49 - 187 [0.715] 500
... White vs Black: 358 - 303 - 339 [0.527] 1000
Elo difference: 184.0 +/- 18.4, LOS: 100.0 %, DrawRatio: 33.9 %
I'm not going to lie, but this had felt like such a long tough slog of a release since many enhancements I tried either yielded none or negative gains (ex. counter move history, history + gravity, enhanced root node move ordering, ...). Even some of the features that gained Elo didn't yield everything that I had hoped for. However, when I started reviewing the PGNs I noticed Pedantic was losing end-games that even I could have won. Additionally, it was making blunders while trying to force mate even when search had already revealed a line that forced mate-in-N. But this was something particular to the 0.3 release since Pedantic 0.2 looked like an endgame genius in comparison. Going back through my change log for this release, one of the first things I did early was implement changes that would speed up search. One of those changes was that I no longer clear the transposition table after every search. This seems to contribute significantly to search efficiency, at least until the end-game. Also, looking at blundered games in analysis mode, Pedantic's analysis and suggested moves seemed spot on. This all added up to me as a misunderstanding of the dynamics of the transposition table. As an experiment I decided to clear the transposition table after every search once I approached the end-game and voilà!! Pedantic immediately gained about 130 Elo!!!
Code: Select all
SELF-PLY ELO
Score of Pedantic 0.3 vs Pedantic 0.2: 385 - 28 - 87 [0.857] 500
... Pedantic 0.3 playing White: 200 - 12 - 38 [0.876] 250
... Pedantic 0.3 playing Black: 185 - 16 - 49 [0.838] 250
... White vs Black: 216 - 197 - 87 [0.519] 500
Elo difference: 311.1 +/- 35.2, LOS: 100.0 %, DrawRatio: 17.4 %
While I still don't have a clear understanding why clearing the TT is having such a pronounced affect, I am going to leave this "hack" in for now and release. I am running a gauntlet now with no adjudications that should give me a better idea of where Pedantic 0.3 stands in terms of playing strength. I should be able to finally release Pedantic 0.3 in the next day or so.
Never before did I spent so much time on a single idea without having anything testable. If it fails it will be my biggest disappointment so far but meanwhile with each additional evening spent on some tiny component of the increasingly complex pipeline I get more giddy and excited.
