Natural TB

[bob]

Good job!

Positive: code simplification. And we don't need DTZ tables (reduce by half the size of the download for users)

That is why Scorpio bitbases have only WDL. 6-men are just 48 GBs. Compare that to syg-bulky-gyz

Once you reach a won ending according to WDL, how do you actually WIN the position rather than aimlessly moving around until the 50 move rule ends the game???

[bob]

You could simply check (ss+1)->tbScore and if, for instance it is a TB loss then you can safely return a win, without continuing searching, so although you didn't TB probe at the beginning of the search(), it does not take a lot, in case of a winning position, to early cut-off the search, thanks to the TB score stored in the stack of the inner nodes.

Does this scheme seems reasonable?

You would have to be lucky to play the winning move first.

If you are in a winning position you don't have to be lucky, any move will do.

Think again.

Not in general case, but in many cases yes, for instance in KRvK any move will do.

And also in other cases, engine is usually good in finding the best moves early on already by itself.

What about "any move" that now pushes the mate beyond the 50 move rule? WDL won't save you because of the assumption that 50 move counter = 0 when you probe. So you make what WOULD be a winning move, were the 50 move counter == 0, but it isn't and you draw instead...

Unless I misunderstood your explanation... And yes, I have seen MANY programs that can't win KR vs K without tables, much less with WDLs that tend to fall all over the 50 move rule blindly...

[bob]

Did a little test ,played with 5men-syzygy
First one is with Marco last source
second is with Joseph last source

I see no difference in game play!

Core i7 2670QM 2cores, Blitz 5m 0

SugaR SE B7 4c - Stockfish 040616 64 POPCNTB 4 15.5 - 14.5 +1/=29/-0 51.67%
SugaR SE B7 4c - Stockfish 040616 64 POPCNTB2 15.5 - 14.5 +1/=29/-0 51.67%

JP.

With a grand total of 30 games?

[syzygy]

That's just fine. If that terminal node is searched, the search may always (independently of the values of alpha and beta) return the value of the terminal node: TB win.

Yeah, it's fine, but the problem is the nodes that stop being searched because they get detected and saved into hash at a depth greater than the next iteration, which guarantees that entry will be used (and not searched).

Agreed, storing the TB result as an exact value in hash with an increased depth is no longer correct if TBWIN is returned as the "exact heuristic value" of the TB position at a depth insufficient to find mate. At the increased depth, mate might be found and the heuristic value would then be different.

The exact bound isn't necessary either, it just makes the entries more difficult to replace, which is not really what you want when trying to resolve the shortest mate ASAP.

If storing an exact score in the TT has repercussions for replacement strategies, then that is something to be taken into account as well, yes.

Still, if TBWIN falls within the alpha-beta window, then there really seems to be no reason not to store TBWIN as an exact value, because that's what you do if the value is within the window.

If TBWIN < alpha, then TBWIN may be stored as upper bound if that interacts better with replacement strategies, just like MATE_in_16 may be stored as upper bound for a certain depth if alpha = MATE_in_14. (Obviously a MATE_in_16 returned by a non-buggy search will in reality be a lower bound on the "real" score of the position searched at infinite depth. But in the framework of alpha-beta it is really an upper bound at the depth at which it was returned by the search.)

[syzygy]

If TBWIN < alpha, then TBWIN may be stored as upper bound if that interacts better with replacement strategies, just like MATE_in_16 may be stored as upper bound for a certain depth if alpha = MATE_in_14. (Obviously a MATE_in_16 returned by a non-buggy search will in reality be a lower bound on the "real" score of the position searched at infinite depth. But in the framework of alpha-beta it is really an upper bound at the depth at which it was returned by the search.)

Oops, that MATE_in_16 returned by a non-buggy search is in reality a lower bound is of course false.

If you search with, say, alpha = +MATE_in_14, beta = +INF, a correct search may return +MATE_in_16 even if there is no mate at all. This is because MATE_in_16 is already so "bad" (i.e. < alpha) that the search did not need to check other moves that may have refuted the mate completely.

So in this sense a TBWIN directly returned by probe_wdl() is different, because it really does mean that the position is a TB win. Still, alpha-beta normally does not have this kind of knowledge and does not need it; it may treat TBWIN as an upper bound if alpha > TBWIN.

[Laskos]

First they seriously deteriorate the 5-men play at the root: on randomly occurring in games 5-men wins Natural fails to convert about 5% of them. On 5-men draws, it loses about 0.4% of them. On MES1258 varied endgame testsuite I got a SPRT stop for (-4,1):

BTW did you know that current syzygy when root is in TB completely disables the probing in inner nodes?

So what you have tested is vanilla SF (with DTZ filtering at root) vs natural.

I have done some important improvments to Natural TB, so interested people please pull again from github the updated version.

Thanks.

Done. Test match will start soon.

Could somebody upload a win x64 binary? I am getting so bad results with the new Natural, that I am suspicious of my compile. It seems worse on hard 5-men at root than no TB at all!

[Joerg Oster]

Thanks Joerg for testing it

You're welcome.

After now 2200 games of 3000 played, NaturalTB seems to be slightly weaker.
After about 400 games it settled to -4 elo, and this never changed noticeably since then.

Conditions: time control 15"+0.2", 16 MB Hash, 12moves opening book, 5men syzygy on SSD.

And here is the end result. Pretty close to make any judgement.
Much more games are needed, it seems.

Code: Select all

Score of SF-Natural2 vs SF-Master: 426 - 451 - 2123  [0.496] 3000

cutechess-cli command:

Code: Select all

./cutechess-cli.sh -openings file=12moves30k.pgn plies=24 order=random -repeat -rounds 3000 -draw movenumber=50 movecount=8 score=8 -concurrency 3 -ratinginterval 20 -wait 4 -pgnout ./SF-Natural2-15.pgn -engine name=SF-Natural2 cmd=./sf-natural2 -engine name=SF-Master cmd=./sf-master-tbs -each proto=uci option.Hash=16 option.SyzygyPath=/home/joster/syzygy5men/syzygy tc=15+0.2

[Laskos]

First they seriously deteriorate the 5-men play at the root: on randomly occurring in games 5-men wins Natural fails to convert about 5% of them. On 5-men draws, it loses about 0.4% of them. On MES1258 varied endgame testsuite I got a SPRT stop for (-4,1):

BTW did you know that current syzygy when root is in TB completely disables the probing in inner nodes?

So what you have tested is vanilla SF (with DTZ filtering at root) vs natural.

I have done some important improvments to Natural TB, so interested people please pull again from github the updated version.

Thanks.

Done. Test match will start soon.

Could somebody upload a win x64 binary? I am getting so bad results with the new Natural, that I am suspicious of my compile. It seems worse on hard 5-men at root than no TB at all!

If my binary isn't broken, then there is something wrong with this new code.

5men_suite 0.1s/ position:

SF No TB
385/434

SF Natural TB
431/434

Seems fine, right? But although it sees almost all solutions, it fails to convert in-games when fed with the same suite as openings:

SF Natural TB vs SF Syzygy: quick SPRT(-5,1) stop (tc 0.1s/move)

Score of SF NAT vs SF Syzygy: 136 - 234 - 287 [0.425] 657
ELO difference: -52.21 +/- 19.98
SPRT: llr -2.96, lbound -2.94, ubound 2.94 - H0 was accepted
Finished match

With Hard 5-men positions at root, Natural loses heavily not only to Syzygy, but to no TB too. It seems it performs fine on 5-men suites but fails to convert in-games. Again, if my binary isn't broken.

[mcostalba]

And here is the end result. Pretty close to make any judgement.

Thanks Joerg.

Very much appreciated!

[IQ]

I have modified current SF+sygyzy in a way that I have called "Natural TB"..... hiding mates, unnatural sacrifices ... (cursed win and losses at the moment are ignored)

Please tell me this is a bad dream. The notion of "unnatural" play is completely misguided even as i will argue in human terms. Yes, people complain about "unnatural" play, strange engine scores and wonder why a computer chess engine does not find the shortest mates. BUT, who are those people? They are most certainly not strong chess players who even have an iota of tournament experience. To be frank here, these people are usually not even club players but more sunday morning puzzle sovlers, who's only exposure to real chess is the bi-weekly chess puzzle they want to solve. Only thing they understand even less than chess is how modern engines work and they are not a one-size fits all tool. What they think is "natural" or "unnatural" bears very little correspondence to what club players think, which in turn is diffirent from what elite players think and even in those groups there are hugely divergent opinions about what is natural or not:
1) Trading down into a won ending, instead of looking for the shortest mate. Every chess player from club-level upwards does this! It is an essential skill - why is the same for SF+SYZYGY now considered "unnatural" is completely beyond me. To the contrary, even if a strong player sees a complicated mate in 12, he would still trade down to a known winning endgame position - the risk would be to high to miss something. If generally a "natural" TB SF would now play mate in 67 instead of trading into a won ending - it would be unnatural from a human standpoint. Yes there will be some positions where even I would agree that a mate in 4 is in realm of human play instead of trading into a 47 move endgame win - but as a general rule it fails. Also i find it hard to fault a 3200+ player for knowing more winning endgame positions than a <1500 chess-puzzle afficionado.

2) Ignoring cursed wins, draws, by relying on the engine?! Seriously - not an option! I wont even comment on this. Even more troubling is that Marco wants to test this in self-play instead of thinking it through. There are just two many TB positions for this to be foolproof, even if it passes some of the endgame testsuites some use for self-play (which this "natural" TB does not even do at the moment)

3) I am all for commenting the syzygy code and catching a bug here and there (Even if my impression at the moment is that Marco is introducing more than he fixes). But isn't this "natural play" against everything SF stands for? It complicates the code, it has no value for strength in self-play - which is/was the holy grail for SF development, it makes it even harder to debug, it relies on search to play those cursed wins perfectly, it distracts from the golden opportunity to include TB in SFs self play framework to get rid of a lot of endgame knowledge and tune better for middle game play.....

Please tell me I am about to wake up and this is all a bad dream and SF is about strength of play, simple readable code, and is not out to win a completely misguided beauty contest. For those who want to solve mates and endgame studies please look at gustav, SF Matefinder, FinalGen (great for > 6 p endgame positions)...... just use the right tool and do not sodomize SFs Syzygy code. At least make it optional so i can turn this "natural" crap off immediatly.