Singular Extensions

[Don]

Hi,

for the example of stockfish implementation of singular extension I suspected a better result in faster time controls. Stockfish is extending at 8 remaining depth for non pv and 6 remaining depth for pv nodes. In fast time controls I expected most relevant hash entries are not overwritten. In long time controls it should get more and more random if using the same hash size.

Greetings Volker

I finally stopped the test last night, error bar was down to +/- 8, difference was +18 Elo. Not insignificant, but also not in line what claims I had seen on freechess. One person there claimed +100 or so which would be remarkable for any change.

When I free up some CPU time I will test the algorithm on Komodo and report the benefit (or lack thereof.)

[Daniel Shawul]

Well i expected a better result at longer tc like 10+10 which btw is too long to test at for anyone unless you have a cluster.
So the results we see here are disappointing and that is a big understatement compared to previous hype.
I honestly did not expect stockfish singular to do that bad especially at 5 + 5.
I think I just got back 7-8 elo (that is 1% increase in 3000games against 11 opponents, played twice for confirmation) by doing
recapture extensions at depthleft >= 4 which I left out in the past..

Well we now know what to expect of them. It may very well be we have different holes to fill in elsewhere but I agree with Bob that
there are more reasonable ways to deal with those issues.

[Don]

For those interested in the singular extension idea, I'm running a test using my laptop which is not doing anything useful. So it will take many days to get significant results, but this is Komodo's implementation of the singular hash table move vs the identical version of the program without the singular move.

The results should of course be taken with a grain of salt as there are so many things that can make this not so easy to compare. One might imagine that the program was tuned with these extensions on for example.
Or it could very well be that this algorithm is strongly affected by what other extensions you use.


However, I clearly remember that we got an immediate and large ELO increase without any further turning of other parts of the program. We did not adjust LMR or anything else. We may have adjusted the program since then of course.

Here are some basic details of how I do this in Komodo: This is done in PV nodes only, there must be a hash table move available, not generated from IID, I use 0.80 pawn margin with a zero window search reduced always by 4 ply and the test is only done when the depth remaining is 5 or greater. I don't take into consideration whether the hash table score is a bound nor do I consider the depth of the hash table entry. I noticed that stockfish looks at some of these issues but I don't.

Code: Select all

  RANK      ELO     +/-     Tme/Gme  Tot Gms  PLAYER
-------  -------  -----  ----------  -------  ----------------
     1    3000.0  115.0     103.710       37  komodo 1.2
     2    2962.8  115.0     102.730       37  komodo 1.2-noSing

Since the sample is only 37 games, I seriously doubt we will see a 40 ELO gain but it seems like I remember it being something like 20 or 30 ELO, it was a long time ago when I implemented it and things might have changed since this so the results will be interesting to me also.

If I don't get too distracted and the results are interesting, I will add a foreign player to the mix later.

Hi,

for the example of stockfish implementation of singular extension I suspected a better result in faster time controls. Stockfish is extending at 8 remaining depth for non pv and 6 remaining depth for pv nodes. In fast time controls I expected most relevant hash entries are not overwritten. In long time controls it should get more and more random if using the same hash size.

Greetings Volker

I finally stopped the test last night, error bar was down to +/- 8, difference was +18 Elo. Not insignificant, but also not in line what claims I had seen on freechess. One person there claimed +100 or so which would be remarkable for any change.

When I free up some CPU time I will test the algorithm on Komodo and report the benefit (or lack thereof.)

[Don]

I forgot to mention that these games are hyper blitz speed. I'm running 4 simultaneous games on a 2 core laptop at 60 seconds + 1 second Fischer time control.

And here is an update:

Code: Select all

  RANK      ELO     +/-     Tme/Gme  Tot Gms  PLAYER
-------  -------  -----  ----------  -------  ----------------
     1    3000.0   84.6     106.010       68  komodo 1.2
     2    2974.6   84.6     105.078       68  komodo 1.2-noSing

[Daniel Shawul]

What time control are you using ?

Here are some basic details of how I do this in Komodo: This is done in PV nodes only, there must be a hash table move available, not generated from IID, I use 0.80 pawn margin with a zero window search reduced always by 4 ply and the test is only done when the depth remaining is 5 or greater. I don't take into consideration whether the hash table score is a bound nor do I consider the depth of the hash table entry. I noticed that stockfish looks at some of these issues but I don't.

I think yours is as constrained as it gets.At PV nodes only makes it a hand full of nodes to test for singularity. And 0.8 pawn margin, I think that is too big. That will not consider many good positional moves. Most will be captures and devastating positional moves better than any other move. The 20-30 elo advantage is more believable especially if we consider everyone does different forms of reductions and extensions.

IIRC Zappa used to use these singular extensions and had a capability to display singular moves in its PV. When I watched it play most of the time it was captures.

[Ralph Stoesser]

Well i expected a better result at longer tc like 10+10 which btw is too long to test at for anyone unless you have a cluster.
So the results we see here are disappointing and that is a big understatement compared to previous hype.

Yes, but nevertheless, the Elo gain from 5+5 results compared to 10+10 results is remarkable. Because the difference is rather huge, it would be interesting to see some 60+60 results for comparison.

[Don]

What time control are you using ?

I answered that in my late and last post

Here are some basic details of how I do this in Komodo: This is done in PV nodes only, there must be a hash table move available, not generated from IID, I use 0.80 pawn margin with a zero window search reduced always by 4 ply and the test is only done when the depth remaining is 5 or greater. I don't take into consideration whether the hash table score is a bound nor do I consider the depth of the hash table entry. I noticed that stockfish looks at some of these issues but I don't.

I think yours is as constrained as it gets.At PV nodes only makes it a hand full of nodes to test for singularity. And 0.8 pawn margin, I think that is too big. That will not consider many good positional moves. Most will be captures and devastating positional moves better than any other move. The 20-30 elo advantage is more believable especially if we consider everyone does different forms of reductions and extensions.

Of course we tested many variations of this. I remember that the exact value of the margin made little difference in the ELO and so we wanted to error on the conservative side since we don't check too many things. So you are right about this not picking up moves of passed pawns for instance or other strong positional moves that might be singular. On the other hand, this is just a trade-off - the impact on the speed of the search is less when you use a big margin.

IIRC Zappa used to use these singular extensions and had a capability to display singular moves in its PV. When I watched it play most of the time it was captures.

[Daniel Shawul]

Stockfish 1.8 averages 21+ depth on my pc at 5 + 5. Singularity tests are done for depth >=8 or 6 so that is deep enough. This is also the case for smp search,LMR and other staff you use depthleft as a criterion to not do certain things below a horizon. If you do the test at reasonable TC to get the desired effect , there is no other reason why it would suddenly pick up momentum. That is why the rating lists at 40/4 and 40/40 of CEGT almost always map to one another. Did Stockfish show significant improvement there ? If so I would be more inclined to beleive you

[Daniel Shawul]

CEGT List

40/4

Code: Select all

Stockfish 1.8 x64 1CPU 	3128 	16 	16 	1300 	67.0% 	3005 	35.6%
Stockfish 1.7.1 x64 1CPU 	3099 	11 	11 	2340 	62.0% 	3014 	36.2%

40/20

Code: Select all

Stockfish 1.8 x64 1CPU 	3090 	20 	20 	650 	62.8% 	2999 	45.8%
Stockfish 1.7.1 x64 1CPU 	3084 	17 	17 	875 	61.6% 	3002 	47.8%

I don't know if the same base is used for the lists , but if it was then infact stockfish seems stronger in blitz not longer tc. Even if that was not the case, the relative ranking almost remains the same.
So the singular extensions or anything else for that matter is not suddenly giving it a push.

[Don]

Stockfish 1.8 averages 21+ depth on my pc at 5 + 5. Singularity tests are done for depth >=8 or 6 so that is deep enough. This is also the case for smp search,LMR and other staff you use depthleft as a criterion to not do certain things below a horizon. If you do the test at reasonable TC to get the desired effect , there is no other reason why it would suddenly pick up momentum. That is why the rating lists at 40/4 and 40/40 of CEGT almost always map to one another. Did Stockfish show significant improvement there ? If so I would be more inclined to beleive you

I'm not sure who you are addressing. If it's me, my belief is that SE helps more with depth - I have no proof to offer up however and it's only a hunch on my part.