Increasing returns of tablebases with TC

[Laskos]

I took Stockfish 9 in 4 versions: not enabled with Syzygy TBs, enabled with 4-men, enabled with 5-men, enabled with 6-men. The Syzygy bases are all on a fast SSD. To have high sensitivity to tablebases, I used my older 3-mover EPD book built from this opening position
[D]1nn1k1n1/4p3/8/8/8/PPPPPPPP/8/4K3 w - - 0 1

This endgame variant which I called "Peasant Revolt" is described here:
http://talkchess.com/forum/viewtopic.ph ... =0&t=65065

It's a borderline variant having about half Black wins and half draws, at various time controls, with various engines. Again, it is very sensitive to tablebases, exaggerating their advantages compared to regular chess. I played a round-robin with these 4 Stockfishes at two different time controls:

10s + 0.1s

Code: Select all

Rank Name                          ELO     +/-   Games   Score   Draws
   1 SF_6_men                       48      12    1500     57%     52%
   2 SF_5_men                       18      12    1500     53%     51%
   3 SF                            -32      12    1500     45%     54%
   4 SF_4_men                      -33      12    1500     45%     54%
Finished match

60s + 0.6s

Code: Select all

Rank Name                          ELO     +/-   Games   Score   Draws
   1 SF_6_men                       68      12    1500     60%     54%
   2 SF_5_men                       16      12    1500     52%     56%
   3 SF_4_men                      -39      11    1500     44%     59%
   4 SF                            -43      12    1500     44%     56%
Finished match

As we can see, the total advantage of 6-men bases is 80 ELO points at 10s + 0.1s and 111 ELO points at 60s + 0.6s. So, an increasing return at 6x time control by 31 ELO points. The error margins shown are trinomial 2SD error margins, the correct pentanomial error margins are 1.7-1.8 times smaller. So, all in all, increasing return here from tablebases is 31 +/- 10 ELO points with 6x time control, a highly significant result. Especially pronounced is the increased return with time control of 6-men compared to 5-men.

[Rodolfo Leoni]

I find this test very interesting. 111 ELOs are a lot and if it's true the fact of the spread growing with longer TCs (or with faster hardware as well) we can imagine there's an improvement curve leading to an amazing difference with 60 mins + 60 secs TC or 40 mins/40 moves TC.

Margin between 5 men and 6 men is 30 ELOs with ultrafast TC and it grows up to 52 ELOs with longer TCs. It'd be nice to compare with 7 men too, if someone has them and is willing to try.

[Uri Blass]

I find this test very interesting. 111 ELOs are a lot and if it's true the fact of the spread growing with longer TCs (or with faster hardware as well) we can imagine there's an improvement curve leading to an amazing difference with 60 mins + 60 secs TC or 40 mins/40 moves TC.

Margin between 5 men and 6 men is 30 ELOs with ultrafast TC and it grows up to 52 ELOs with longer TCs. It'd be nice to compare with 7 men too, if someone has them and is willing to try.

Interpolation is wrong.

common sense suggest that at VLTC tablebases are going to give less elo because it is impossible to improve perfect chess.

[Ovyron]

it is impossible to improve perfect chess.

I disagree.

Specially against an opponent that is not perfect, tablebases may not see the difference between giving away a Bishop or keeping it because both options are drawn with perfect play, but keeping the Bishop would have allowed the perfect side to win, so it's an improvement on perfect chess.

[Toadofsky]

If it's not too much trouble I'd like to see some intermediate data points to better understand the shape of the improvement curve...

[Laskos]

If it's not too much trouble I'd like to see some intermediate data points to better understand the shape of the improvement curve...

I think it's better to have more extreme points (longer time control), not intermediate. I managed to get one at 240s + 2.4s, but only in 300 games each engine in RR:

240s + 2.4s

Code: Select all

Rank Name                          ELO     +/-   Games   Score   Draws
   1 SF_6_men                       70      27     300     60%     55%
   2 SF_5_men                       19      27     300     53%     53%
   3 SF_4_men                      -40      26     300     44%     57%
   4 SF                            -49      26     300     43%     56%
Finished match

At 24x time control, the difference seems even larger now, 119 Elo points, but error margins are too large to have a conclusive result compared to 60s + 0.6s (at 6x time control), where I got 111 Elo points.

[Nordlandia]

Kai Laskos: consider redo this with following modifications:

Instead of

Rank Name
1 SF_6_men
2 SF_5_men
3 SF_4_men
4 SF

Do the following
SF_5_men_6_men_adj | Stockfish is probing 5-men during search plus automatic 6-men adjudication
SF_4_men_5_men_adj | Stockfish is probing 4-men during search plus automatic 5-men adjudication
SF_3_men_4_men_adj | Stockfish is probing 3-men during search plus automatic 4-men adjudication

Or alternatively comparing probing 5-men during search + 6-men adj vs probing 6-men during search. This can be done by adjusting probe limit to 5 pieces for one side. Assuming sf selfplay.

Latest cutechess gui - https://www.sendspace.com/file/eqal7v

[hgm]

I am not sure what to make of these results. How can EGTs help to win a position as exotic as this? Is it that it provides the knowledge which KNNKP positions are won, and that the initial material balance is such that most of the games with good play end up in KNNKP?

Stockfish is unlikely to contain specific knowledge on KNNKP (for which there do exist fairly simple rules). If without EGT it just converts randomly to KNNKP, it is likely to bungle a lot of games that it could have won by converting selectively. That would translate to a lot of Elo. If at faster TC less accurate play would lead to many games being decided before getting to KNNKP, either as a certain win or a certain draw, it would explain why the EGT have less effect there.

When you split out the final positions of the drawn games by material composition, what do you get?

I am not sure how much of this would carry over to games starting from the standard opening position. Although I do expect the number of games that are trivially decided long before any end-game is reached (and on which the use of EGT thus doesn't have any effect) to increase with less accurate play at fast TC. The EGT can only have any effect if the position is still approximately equal (say +/- 1 Pawn) in the late end-game stage.

[guyhaw]

Excellent experiment: bigger time-budget ==> EGT-access earlier in the search and greater nodes/sec.

In TCEC11.P, game 66.2, move 64w, Stockfish consulted the EGT 1,274,028,324 times @ nearly 4m EGT access/sec.

Has anyone seen bigger numbers than these!

g

[Nordlandia]

400 games à la 2m+1.2 just finished.