7-men Syzygy attempt

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Isaac
Posts: 265
Joined: Sat Feb 22, 2014 7:37 pm

Re: 7-men Syzygy attempt

Post by Isaac » Sat May 12, 2018 12:00 pm

Ozymandias wrote:
Sat May 12, 2018 9:44 am
OK, what you meant is that playing with a time control as opposed to fixed depth lets both engines search deeper in endgame positions. That is indeed true and fixing depth indeed artificially lowers the level of endgame play of both engines.
I think you're again missing Ronald's point.

In addition, it seems to me that you set this fixed depth limit to have a clear picture of the benefits of TB. If that was your goal, the method you employed to achieve it is flawed. Ronald pointed this out several times.

I suggest rereading what he wrote in his last answers here.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Sat May 12, 2018 12:15 pm

Isaac wrote:
Sat May 12, 2018 12:00 pm
Ozymandias wrote:
Sat May 12, 2018 9:44 am
OK, what you meant is that playing with a time control as opposed to fixed depth lets both engines search deeper in endgame positions. That is indeed true and fixing depth indeed artificially lowers the level of endgame play of both engines.
I think you're again missing Ronald's point.
But now you're quoting my text ;-)

But indeed my point is that increased search depth is one of the advantages of TBs.

Ozymandias is right that using a time control instead of fixed depth increases depth in endgames whether you use TBs or not. But that was not my point.

Ozymandias' point is also an important reason why testing at fixed depth is flawed. If you fix depth to just 15 ply or so all top engines will be absolutely terrible at playing endgames. So I should have included this in my initial list.

So there are at least three reasons why fixed-depth tests are not useful for estimating the Elo gain from TBs:
- fixed depth hides the speed penalty (lower nps) incurred by probing TBs;
- fixed depth hides the benefit of reaching larger depths in late middlegame and endgame when using TBs; and
- fixed depth artificially cripples endgame play of top engines, which gives too much weight to the advantage of having perfect knowledge close to the root.

Isaac
Posts: 265
Joined: Sat Feb 22, 2014 7:37 pm

Re: 7-men Syzygy attempt

Post by Isaac » Sat May 12, 2018 12:25 pm

I indeed messed up with the quoting part.
Now your points are, I hope, crystal clear.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Sat May 12, 2018 12:33 pm

I just noticed a problem:
https://lichess.org/analysis/standard/6 ... /1B6/KQ6_w_-_-

The best move is Qg6+ with DTZ 604. Then Kf8 (only move) with DTZ 603. But then Qg7+ has DTZ 104??

Interestingly, after Ke8 the best move is Nc7+ with DTZ 600, which seems correct again. But something is wrong in between.

Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.

User avatar
Ozymandias
Posts: 1137
Joined: Sun Oct 25, 2009 12:30 am

Re: 7-men Syzygy attempt

Post by Ozymandias » Sat May 12, 2018 1:03 pm

syzygy wrote:
Sat May 12, 2018 9:56 am
Ozymandias wrote:
Sat May 12, 2018 9:44 am
syzygy wrote:
Sat May 12, 2018 9:31 am
Testing TBs at fixed depth will not give a complete picture. On the one hand, they take time to probe. On the other hand, they allow the search to go deeper. By fixing the depth you eliminate these two important components from the equation.
What do you mean by "time to probe"? CPU throttle? There isn't.
Time to probe is simply time to probe. It takes many cpu cycles to probe a position, and data may have to be read from SSD.
Not shooting for the obvious, but everything takes time, aren't those CPU cycles equivalent to the ones needed to reach the same depth without TBs?
syzygy wrote:
Sat May 12, 2018 9:56 am
Ozymandias wrote:
Sat May 12, 2018 9:44 am
As the test shows, more depth equals less of a gap, if you increase said depth you'll hardly register any differences at all.
No, your test artificially limits the depth reached by the engine playing with TBs. At the same time, it artificially removes the speed penalty of probing.
Limiting depth is less of an intrusion than limiting time (more on that later), and both the engines playing with TBs, as well as the one without, were depth limited. I think that's fair testing.

What's "the speed penalty of probing"? When the HDD doesn't get in the way, I've noticed none.
syzygy wrote:
Sat May 12, 2018 9:56 am
Ozymandias wrote:
Sat May 12, 2018 9:44 am
Just because it's the more chosen time control, it doesn't mean you should jump onboard blindly.
Playing with fixed depth is simply not a time control at all.
Only in the sense that the TC isn't fixed, total time per game is variable because the engine requires more or less time, depending on how difficult the positions involved turn out to be. For engine testing, it makes sense to externally influence the engine, imposing a deadline, but looking at TBs performance is different. Let me explain with an example, I played two quick matches between asmFish 9 at depth 12, and asmFish 9 with a TC of 25 moves per second (equivalent total time on that PC); in one of them, asmFish 9 25/1 used 6-men and in the other, no TBs. The difference in engine's strength was between the margin of error, does that mean that TBs only add a few Elo points? No, it means that when the engine has to manage time allocation, it saves more than enough for the "ending" (past move 53) and that doubling time or probing TBs will have little effect.

noobpwnftw
Posts: 362
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Sat May 12, 2018 1:05 pm

syzygy wrote:
Sat May 12, 2018 12:33 pm
I just noticed a problem:
https://lichess.org/analysis/standard/6 ... /1B6/KQ6_w_-_-

The best move is Qg6+ with DTZ 604. Then Kf8 (only move) with DTZ 603. But then Qg7+ has DTZ 104??

Interestingly, after Ke8 the best move is Nc7+ with DTZ 600, which seems correct again. But something is wrong in between.

Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.
Just checked that Cfish gives the same results on my local table, rebuilding to see what happens.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Sat May 12, 2018 1:13 pm

syzygy wrote:
Sat May 12, 2018 12:33 pm
Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.
It does correct itself if you play Qb3+:
Be1 (562), Qc4 (561), Kb1 (222 ??), Qb3+ (103 ??), Kb2 (558).

The problem is the position after Qb3+ probing as having DTZ=103.
The position after Kb1 is not in the half-sided DTZ table, so probing that position causes a 1-ply round of probing, which would have given the correct result if the position after Qb3+ had returned the correct value.

noobpwnftw
Posts: 362
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Sat May 12, 2018 1:21 pm

syzygy wrote:
Sat May 12, 2018 1:13 pm
syzygy wrote:
Sat May 12, 2018 12:33 pm
Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.
It does correct itself if you play Qb3+:
Be1 (562), Qc4 (561), Kb1 (222 ??), Qb3+ (103 ??), Kb2 (558).

The problem is the position after Qb3+ probing as having DTZ=103.
The position after Kb1 is not in the half-sided DTZ table, so probing that position causes a 1-ply round of probing, which would have given the correct result if the position after Qb3+ had returned the correct value.
Then it looks like a probing problem of the site?

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Sat May 12, 2018 1:23 pm

syzygy wrote:
Sat May 12, 2018 1:13 pm
The problem is the position after Qb3+ probing as having DTZ=103.
DTZ=102 to be precise.

The first problem is caused by the position after Ke8 incorrectly returning DTZ=102.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Sat May 12, 2018 2:00 pm

noobpwnftw wrote:
Sat May 12, 2018 1:21 pm
syzygy wrote:
Sat May 12, 2018 1:13 pm
syzygy wrote:
Sat May 12, 2018 12:33 pm
Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.
It does correct itself if you play Qb3+:
Be1 (562), Qc4 (561), Kb1 (222 ??), Qb3+ (103 ??), Kb2 (558).

The problem is the position after Qb3+ probing as having DTZ=103.
The position after Kb1 is not in the half-sided DTZ table, so probing that position causes a 1-ply round of probing, which would have given the correct result if the position after Qb3+ had returned the correct value.
Then it looks like a probing problem of the site?
If Cfish showed the same problem, then the problem is probably in the table.

The WDL table seems to be OK and the DTZ table seems to be OK for most positions. But several positions have the wrong DTZ value.

There also problems with non-cursed wins:
https://lichess.org/analysis/standard/8 ... 5/2n5/8_b-_-
DTZ=17 for Qe3, then DTZ=7

Post Reply