Plain alphabeta versus pvs+aspiration

[Bas]

What is going on here??? Let me explain below what puzzles me...

Since my engine is early stages, I pitted it against older engines like SOS and Ruffian. SOS soon stopped being a challenge, but Ruffian is tougher. I suspect both SOS and Ruffian use 4 cores too, judging by their nps, so I am against 5x the nps. At a point I could beat Rufian in 100 games by +220 elo. Then I went from plain alphabeta without aspiration to pvs with aspiration. As soon as I did that my engine no longer seemed to outsearch Ruffian. I checked, rechecked, but no. Just standard asp+pvs+LMR procedure. Then I created two versions one pvs+aspiration the other one the good old plain ab version, everything else equal. I mean everything. Then in a tournament with SOS/Ruffian, it was confirmed what I saw earlier: the ab version is a Ruffian/SOS killer by 360 elo AND THEN gets totally killed itsself by the pvs version. Check this out:

-----------------Tao-ab-----------------
Tao-ab - Ruffian 1.0.5 : 9,0/10 9-1-0 (1110111111) 90% +382
Tao-ab - SOS 5.1 for Arena : 8,0/9 8-1-0 (111110111 ) 89% +363
Tao-ab - Tao-pvs : 3,5/9 2-4-3 (0==00=011 ) 39% -78
-----------------Tao-pvs-----------------
Tao-pvs - Ruffian 1.0.5 : 7,0/10 6-2-2 (1101=1=011) 70% +147
Tao-pvs - SOS 5.1 for Arena : 8,5/10 7-0-3 (=11==11111) 85% +301
Tao-pvs - Tao-ab : 5,5/9 4-2-3 (1==11=100 ) 61% +78

Any explanation why alphabeta makes a better Ruffian/SOS killer than pvs? I checked quite some hard testpositions with both versions analyzing parallel and up till a certain point it's hard to say which one is better. Until...high depths are reached, then pvs wins bigtime. Could it even be that pvs is counterproductive in blitz? And why does pvs suddenly starts to get way better at 17 ply or so, could it be hash related, relatively full hash favors pvs? Strange results...!

Best regards
Bas

[Aleks Peshkov]

Sounds like 30 years old post. https://www.chessprogramming.org/Sequen ... Ratio_Test

[hgm]

The point is that the result of a 10-game match has such a large statistical error that it is practically meaningless. So the answer to your question is simply: 'luck'. The ab version is not a better SOS/Ruffian killer, but just had more luck while doing it in your tests.

[Bas]

T

he point is that the result of a 10-game match has such a large statistical error that it is practically meaningless. So the answer to your question is simply: 'luck'. The ab version is not a better SOS/Ruffian killer, but just had more luck while doing it in your tests.

Yeah I get it...but sheeshs you guys have become statistics jockeys..! Anyway I just learnt that it is possible to play games parallel, one per core, that makes the new reality a bit less ugly.

Bas

[syzygy]

T

he point is that the result of a 10-game match has such a large statistical error that it is practically meaningless. So the answer to your question is simply: 'luck'. The ab version is not a better SOS/Ruffian killer, but just had more luck while doing it in your tests.

Yeah I get it...but sheeshs you guys have become statistics jockeys..!

Not a great way to react to people answering your question.

[flok]

T

he point is that the result of a 10-game match has such a large statistical error that it is practically meaningless. So the answer to your question is simply: 'luck'. The ab version is not a better SOS/Ruffian killer, but just had more luck while doing it in your tests.

Yeah I get it...but sheeshs you guys have become statistics jockeys..!

Not a great way to react to people answering your question.

Valt wel mee toch?

[Steve Maughan]

Hi Bas,

For Juggernaut, I find aspiration windows almost impossible to get working and show any meaningful gain in strength. They also add instability to the search for endgame positions with a mating sequence. Try pvs without aspiration — that's what I'm using. With simple pvs, 99% of nodes will have zero window anyway.

— Steve

[Bas]

T

he point is that the result of a 10-game match has such a large statistical error that it is practically meaningless. So the answer to your question is simply: 'luck'. The ab version is not a better SOS/Ruffian killer, but just had more luck while doing it in your tests.

Yeah I get it...but sheeshs you guys have become statistics jockeys..!

Not a great way to react to people answering your question.

Valt wel mee toch?

Volgens mij ook, geen moment gedacht dat HGM er zich aan zou storen. Persoonlijk stoor ik me meer aan mensen die voor anderen willen denken.

Ciao,

Bas

[Bas]

Hi Steve,

That is interesting, to be honest I have been sceptical for a long time too. I have used the same method as you in my previous engine (ahem, 30 years ago, cough), pvs without aspiration. But right now I am not so sure anymore. I have compared pvs+aspiration with ab in a blitz match and done some parallel comparisons. Which is NOT the same as with/without aspiration of course. Up till a certain depth it makes very little difference, but at higher depths pvs+asp leaves ab totally in the dust, almost no exceptions, just wait long enough. I can't explain why pvs totally beats ab after a certain depth, and not before.

And after that I have seen some stats that were an eye opener for me. Really, optically, watching games I could have sworn ab was more convincing than pvs, but after a bunch more games:

-----------------Tao-ab-----------------
Tao-ab - Ruffian 1.0.5 : 45,0/61 38-9-14 (111011111111=0=1110111=1=1010101===1=100=111==0=1=1111=111111) 74% +182
Tao-ab - SOS 5.1 for Arena : 51,0/60 49-7-4 (1111101111101111111111=1==1101011111111101=11111101111110111 ) 85% +301
Tao-ab - Tao-pvs : 19,0/60 8-30-22 (0==00=0110==1100==0=10=0===0=1=00000000=000==00000===0==1010 ) 32% -131
-----------------Tao-pvs-----------------
Tao-pvs - Ruffian 1.0.5 : 45,0/60 39-9-12 (1101=1=01111111=1011=11==01==11011=1=01111111011101==1101111 ) 75% +191
Tao-pvs - SOS 5.1 for Arena : 50,5/61 48-8-5 (=11==1111111100111111111111111101=111101101111111=11100110111) 83% +275
Tao-pvs - Tao-ab : 41,0/60 30-8-22 (1==11=1001==0011==1=01=1===1=0=11111111=111==11111===1==0101 ) 68% +131

It seems pvs+aspiration is a bit better after all, at one point pvs was better on all fronts, but that point has passed

Bas

Hi Bas,

For Juggernaut, I find aspiration windows almost impossible to get working and show any meaningful gain in strength. They also add instability to the search for endgame positions with a mating sequence. Try pvs without aspiration — that's what I'm using. With simple pvs, 99% of nodes will have zero window anyway.

— Steve