The history of Syzygy tablebases

[Nordlandia]

Systems with sufficient enough ram like tcec for instance, benefit alot from already cached egtb and is much faster to reach than probing it from the ssd.

Current algorithm work great for systems with sufficient ram installed, for systems w/o it will eventually compromise windows stability.

One idea for syzygy is analysis mode. In analysis mode syzygy functions similarly to nalimov - ram is unaffected.

[hgm]

Sounds like nonsense. Syzygy doesn't dedicate any RAM to its use; it purely relies on what the OS gives it. If that would go at the expense of other tasks that are dependent on the RAM, and now cannot get it because it was all given away to some low-priority task that did not even request it, the OS would utterly suck.

[Laskos]

Sounds like nonsense. Syzygy doesn't dedicate any RAM to its use; it purely relies on what the OS gives it. If that would go at the expense of other tasks that are dependent on the RAM, and now cannot get it because it was all given away to some low-priority task that did not even request it, the OS would utterly suck.

Sure not nonsense to me. I am using WIndows 8.1, I don't know how to tell OS not to mess with idle processes I use later, and are often high RAM consumers, which if cached, take forever to load again. Sure, I am not a disciplined PC user, I reboot my PC once a week-a-month, and re-use some idle tasks after awhile.

[hgm]

Sounds like Windows 8.1 has an enormous memory leak.

I still don't see how it would make any difference between using Nalimov and Syzygy, though. In Nalimov everything that is not explicitly cached in the engine would have to come from disk too, and if the OS keeps that in its disk cache, it would eat the memory just the same.

This is not something that can be ascribed to the EGT format, btw. You can just as easily write probing code that uses an internal cache of limited size for Syzygy.

[Nordlandia]

GT format, btw. You can just as easily write probing code that uses an internal cache of limited size for Syzygy.

If that cache can be rewritten throughout the game based on relevance, that sounds like a good idea. E.g limted cache at move 40 might not be as useful at move 50. Sorting internal cache by priority & relevance.

[Nordlandia]

Here is the concept visualized.

[hgm]

But I still don't see how that can help. Everything that is in the internal cache will have to be read through the OS from the disk, and the OS will then cache that too. The RAM used by the internal cache is just a duplicat. Even if you flush the internal cache, to make space for new data, it would not inspire the OS to flush that same data from the system cache.

The problem is that the OS uses the 25% you depict in any way it wants. Applications don't have any say in that. So if the OS prefers to use it for storing a duplicate of what is in your internal Syzygy cache, at the expense of code memory for waiting other applications, you are dead. The OS has no idea that you are caching all that data internally. It just know that it is something you have been reading. The only thing to prevent that is not read any data at all.

[tpoppins]

Systems with sufficient enough ram like tcec

Current algorithm work great for systems with sufficient ram installed

Sufficient ram, sufficient ram -- could you be more vague, please?

Sounds like Windows 8.1 has an enormous memory leak.

You nailed it. That's what it amounts to in effect, and the same goes for Win7. I haven't seen how Windoze caching performs under Win10 (had it replaced with Win7 within 24 hours on the only box that came with it) but I doubt it's any better.

Last year I looked for a solution to RAM filling up to 99% during extended analysis sessions with 6-men Syzygy and didn't find one. Programs that are supposed to limit Windoze cache, registry tweaks -- none of them worked. Perhaps they used to at some point and then some Windoze update broke them.

The problem is particularly evident on the two of my boxes that have "only" 32GB of RAM. My main desktop has 48 and it's much less pronounced. In fact, I wasn't aware of the issue until I got one of the 32GB ones; if I had noticed any slowdown I must have put it down to some other factor.

[syzygy]

"Does not eat RAM during analysis"... Isn't that the same thing as saying "Fails to profit from RAM that otherwise would be idle, and just leaves that go to waste"?

Not really. Suppose you have two programs: one does tons of disk thrashing if it doesn't have 8GB of RAM available, the other doesn't:

Code: Select all

       4GB    8GB
A    2400   2450
B    1800   2480

I would say "B plays poorly with less than 8GB of RAM". I would never say "A isn't taking as good advantage of the extra RAM".

Compared to Syzygy TBs, Nalimov needs 17x as much RAM to get anywhere near a decent performance. Nalimov devours RAM, it just doesn't show up as part of the process's shared memory in the task manager.

To get rid of disk thrashing, disable the page file. Problem solved.

[Nordlandia]

Page file advice is good solution but keeping windows oscillating between 97-98% may inflict compromised windows stability.