Can principal variation search be worth no Elo?

[amanjpro]

Hey Christian,

I just want to let you know that you can never say your search parameters are totally optimized.

Whenever you add a term anywhere in the engine, you probably can retune every param and expect elo from at least some of them.

What I mean is stop worrying about squeezing every last one of elo from a feature, and start making the engine more knowledgeable and bug free. And retune and tune as you go.

Zahak started as a very badly tuned engine, and now it is rated north of 3000 (at least the dev version is). I just kept tuning it over and over again, and fun thing is that you are never done as there is always something that you can juice.

My two cents

[algerbrex]

...

Damnit. Unfortunately, the saga continues.

It bothered me that I was no longer checking if the time was up before saving an entry into the TT because I was quite sure not doing so would harm TT performance. And it seems my suspicions were correct. Very much so. Not checking if the time was up before saving a TT entry caused a ~170 Elo drop in performance, so neglecting to do that just isn't an option.

Of course, adding back in the code to check if the time is up now causes PVS to fail again...even though enhancing the TT performance should enhance PVS. So I'm currently back to square one, and thinking about why PVS hates Blunder so much

[algerbrex]

Hey Christian,

I just want to let you know that you can never say your search parameters are totally optimized.

Whenever you add a term anywhere in the engine, you probably can retune every param and expect elo from at least some of them.

What I mean is stop worrying about squeezing every last one of elo from a feature, and start making the engine more knowledgeable and bug free. And retune and tune as you go.

Zahak started as a very badly tuned engine, and now it is rated north of 3000 (at least the dev version is). I just kept tuning it over and over again, and fun thing is that you are never done as there is always something that you can juice.

My two cents

Thanks Amanj,

What you're saying makes a lot of sense. I always want to make sure each feature I add to my engine is bug-free, and this process usually seems to correlate with maximizing a feature's Elo gain; hence why I and others talk about doing the latter.

But I agree, it doesn't make much sense to endless tune a certain feature when there's often so much synergy between various features. I'll keep that in mind going forward. Especially since I have a working evaluation tuner for Bluner (albeit very slow), and there seem to be many good programs out here to automate search parameters.

[amanjpro]

Hey Christian,

I just want to let you know that you can never say your search parameters are totally optimized.

Whenever you add a term anywhere in the engine, you probably can retune every param and expect elo from at least some of them.

What I mean is stop worrying about squeezing every last one of elo from a feature, and start making the engine more knowledgeable and bug free. And retune and tune as you go.

Zahak started as a very badly tuned engine, and now it is rated north of 3000 (at least the dev version is). I just kept tuning it over and over again, and fun thing is that you are never done as there is always something that you can juice.

My two cents

Thanks Amanj,

What you're saying makes a lot of sense. I always want to make sure each feature I add to my engine is bug-free, and this process usually seems to correlate with maximizing a feature's Elo gain; hence why I and others talk about doing the latter.

But I agree, it doesn't make much sense to endless tune a certain feature when there's often so much synergy between various features. I'll keep that in mind going forward. Especially since I have a working evaluation tuner for Bluner (albeit very slow), and there seem to be many good programs out here to automate search parameters.

What is highlighted is not really necessarily true... when I added NNUE, my elo gain was 0, why? I thought I have a bug, but I didn't
I disabled half of the pruning of the search, elo gain went up, disabled a quarter, the elo gain went higher
until I found the only pruning that didn't play well with NNUE....

I didn't have bugs, I had incompatible search and evaluation... So, you may be able to implement a bug free NMP, and still get less elo than someone else, because your other search features are not fit for it

[algerbrex]

Hey Christian,

I just want to let you know that you can never say your search parameters are totally optimized.

Whenever you add a term anywhere in the engine, you probably can retune every param and expect elo from at least some of them.

What I mean is stop worrying about squeezing every last one of elo from a feature, and start making the engine more knowledgeable and bug free. And retune and tune as you go.

Zahak started as a very badly tuned engine, and now it is rated north of 3000 (at least the dev version is). I just kept tuning it over and over again, and fun thing is that you are never done as there is always something that you can juice.

My two cents

Thanks Amanj,

What you're saying makes a lot of sense. I always want to make sure each feature I add to my engine is bug-free, and this process usually seems to correlate with maximizing a feature's Elo gain; hence why I and others talk about doing the latter.

But I agree, it doesn't make much sense to endless tune a certain feature when there's often so much synergy between various features. I'll keep that in mind going forward. Especially since I have a working evaluation tuner for Bluner (albeit very slow), and there seem to be many good programs out here to automate search parameters.

What is highlighted is not really necessarily true... when I added NNUE, my elo gain was 0, why? I thought I have a bug, but I didn't
I disabled half of the pruning of the search, elo gain went up, disabled a quarter, the elo gain went higher
until I found the only pruning that didn't play well with NNUE....

I didn't have bugs, I had incompatible search and evaluation... So, you may be able to implement a bug free NMP, and still get less elo than someone else, because your other search features are not fit for it

That's an interesting example, thanks. My point was just that usually from my experience when implementing the basics of a chess engine, not getting the expected Elo from a certain feature was because I had a bug somewhere.

If you go back and look at the first version of Blunder I released (https://github.com/algerbrex/Blunder---Pre-Release), I estimated it was around ~1400 Elo, which, given its feature set, meant there were bugs somewhere in the code. So ever since then I've been diligent about making sure I maximize Elo by ensuring my code was bug-free.

But I do agree with you, especially given where I'm at right now. I'm confident I've rooted out possible bugs in my search code, and my implementation of PVS also looks bug-free. And several times I thought I fixed the "bug," only to realize I had broken some other part of my code. So beyond what I've already done, I do think it's just a waste of my time to continue searching for a PVS bug in my code that isn't there.

I suppose my only concerns with going forward without PVS would mostly be ones of personal annoyance since I'd like to know what parts of Blunder are incompatible with PVS and why. Although my other concern would be correctly differentiating between PV and non-PV nodes while searching since many basic pruning techniques rely on knowing whether or not a node is a PV node. Any suggestions on this?

Thanks again for the advice, much appreciated, as I'm still comparatively quite new to chess programming, and I always like being correct if I'm going wrong in my thinking.

[amanjpro]

Well, a pv-node == beta-alpha != 1
else it is not pv-node

https://github.com/amanjpro/zahak/blob/ ... ch.go#L224

[algerbrex]

Well, a pv-node == beta-alpha != 1
else it is not pv-node

https://github.com/amanjpro/zahak/blob/ ... ch.go#L224

Right. But does that hold true even when not using PVS? I wasn't sure if it did.

[amanjpro]

Well, a pv-node == beta-alpha != 1
else it is not pv-node

https://github.com/amanjpro/zahak/blob/ ... ch.go#L224

Right. But does that hold true even when not using PVS? I wasn't sure if it did.

Oh, that... I don't know really... I did pvs really early that I don't know much without it hhhh

Sorry for that

[algerbrex]

Well, a pv-node == beta-alpha != 1
else it is not pv-node

https://github.com/amanjpro/zahak/blob/ ... ch.go#L224

Right. But does that hold true even when not using PVS? I wasn't sure if it did.

Oh, that... I don't know really... I did pvs really early that I don't know much without it hhhh

Sorry for that

That's ok! Thanks again for the advice. I'll figure something out with it

[mvanthoor]

Damnit. Unfortunately, the saga continues.

It bothered me that I was no longer checking if the time was up before saving an entry into the TT because I was quite sure not doing so would harm TT performance.

You save entries into the TT in alpha-beta, not in iterative deepening (I hope). I don't check if time is up before saving an entry into the TT; I check this at the beginning of alpha-beta. The problem you now have is this:

- You do alpha-bèta
- After the move loop, you write into the TT
- Somewhere along the line, time's up
- And then the recursive alpha-beta unwinds
- But when you should be saving to the TT you don't, because the time is up.

If you are saving into the TT in the bèta cutoff (beta-flag) and after the move loop (alpha-flag), then you're missing all the alpha-flag saves. If you are saving only once (like me, with fail-soft), you miss a huge number of TT-writes.

You shouldn't be checking if the time is up before you write to the TT. You need to allocate something like 50-200ms (depending on how fast your engine is) of headroom. So if you calculate a time slice of 800ms for a move, you subtract 100ms. Then you can spend the slice of 700ms for searching, and then you have 100ms to unwind alpha-beta (AND write to the TT) when time is up.

And it seems my suspicions were correct. Very much so. Not checking if the time was up before saving a TT entry caused a ~170 Elo drop in performance, so neglecting to do that just isn't an option.

Strange. I'll do a time-up check before writing to the TT then, and see if my engine increases 170 Elo in playing strength. I still think you're doing something strange here. You _SHOULD_ check if you've broken iterative deepening because of time-up, because you will have to ignore that run and send the previously saved best move to the GUI.

Of course, adding back in the code to check if the time is up now causes PVS to fail again...even though enhancing the TT performance should enhance PVS. So I'm currently back to square one, and thinking about why PVS hates Blunder so much

Couldn't you try my search and move ordering code again, and see if this works with PVS; including the time-up check in the same spot as were it is in my code?