Unstable results with little blitzer

[Rebel]

Not sure if this addresses the trouble your are facing but have you done a reliability test on your testing environment? A good test is to do a self-play match at fixed depth. It's output should be 100% identical, an exact 50% result, identical games, scores and even nodes.

Unfortunately this won't happen when using an opening book, at least not with the attribute "exact" ...

Hence one should use predefined openings with reversed colors.

The OP reminded me to check my own system again and shucks it was NOT ok. After some puzzling I found a change in my last commercial (Rebel 12) which for some reason did not clear the TT during a new game command. And thus incidentally it changed the move ordering because it found a best-move in the TT of the previous game. And a different move ordering guarantees different moves. Now everything is back to normal.

In a testing environment where volume is used to weed out randomness it's odd introducing randomness yourself.

[Desperado]

I have some problems with the stabillity of my results when I am using little blitzer. I run 10.000 games divided on 5 opponents (40 moves 8 sec). When I tested the same version I once got a score of 44.4% and the next time got a score of 37.8%. This is way to big diference considering the number of games. I have had this problem before with little blitzer, but when I test with arena my results are allways within an exepteced error bar. Does anyone have any suggestion if I might be doing something wrong during testing? I test by using openingbooks, and as far as I can see there are not may equal games. Is there a tool for removing equal games from a PGN file?

Regards
Jacob

Hello, Jacob,

beside all things pointed out already in this thread,
you may check if the reload behaviour is the same under your favourite
user interfaces. Well, i am not sure, but i think LB is reloading the engines
for every game, but what Arena is doing, i really do not know.

Even if you re-init all your data, there is no guarantee that all the other
engines are doing so too, so that may produce noise if engines are not
reloaded.

regards, Michael