Using large pages - periodic reboot?

[gordonr]

Hi,

I've just recently started using large pages in Windows 11 Pro, with 64 GB RAM, for relatively big hash tables and long analysis. From an old Houdini 6 online manual, I read today: "Very often large memory pages will only be available shortly after booting Windows. After a while the Windows memory becomes too fragmented for large page allocation, and Houdini will fall back to standard memory page usage.". Is this still applicable nowadays with other engines? Should I reboot every so often?

Thanks

[Modern Times]

I may be wrong, but I think once they are allocated you don't lose them. There is some ambiguity in that wording. Maybe someone else knows for sure.

[gordonr]

I may be wrong, but I think once they are allocated you don't lose them. There is some ambiguity in that wording. Maybe someone else knows for sure.

Thanks

I think the setting remains but for how long "large pages" can be efficiently used without reboots, I'm not sure. I've since found the following link:

https://mahdytech.com/large-pages-how-when/#the-whynot

It's not chess specific but it does highlight a potential issue. I'm going to monitor performance in combination with doing some reboots periodically.

[chrisw]

How much do large pages help chess engines?
Why?
How?

[Modern Times]

I may be wrong, but I think once they are allocated you don't lose them. There is some ambiguity in that wording. Maybe someone else knows for sure.

Thanks

I think the setting remains but for how long "large pages" can be efficiently used without reboots, I'm not sure. I've since found the following link:

https://mahdytech.com/large-pages-how-when/#the-whynot

It's not chess specific but it does highlight a potential issue. I'm going to monitor performance in combination with doing some reboots periodically.

I think you only potentially have an issue if you close the program using the large pages and then start it up again later. I guess in the absence of more concrete information you just have to see.

[Jouni]

Benefit is free 10% speedup because RAM is allocated for chess engine only. Normally windows can use same memory occasionally and slow down! But I have noticed You must start SF immediately after boot. If You stop or do something else You lose them soon. So benefit is limited.

[gordonr]

How much do large pages help chess engines?
Why?
How?

Admittedly I've never done any formal testing but the initial appearance based on nodes per second, etc looked to be a very small gain and didn't look any worse than without large pages. And I know nodes per second isn't necessarily a reliable metric in terms of overall benefit. So all I have is a crude assessment so far.

From an old Houdini 6 manual: "Houdini Pro will use so-called large memory pages if they are provided by the operating system. Depending on the hash table size the speed gain may be between 5% and 15%"

[kasinp]

Wise memory optimizer https://www.wisecleaner.com/wise-memory-optimizer.html will defrag your memory in less than a minute.
Have been using it for this purpose for years.

Cheers,
Peter

[gordonr]

Wise memory optimizer https://www.wisecleaner.com/wise-memory-optimizer.html will defrag your memory in less than a minute.
Have been using it for this purpose for years.

Cheers,
Peter

Excellent, thanks! I hadn't considered this but will now.

And thanks everyone for input

[RubiChess]

How much do large pages help chess engines?

Running depth 26 bench in my engine on a Windows machine with permission to use large pages enabled and using 1024MB hash tables.
1. Usage of large pages disabled by UCI option:

Code: Select all

RubiChess-20221102_x86-64-avx2.exe -option "Allow Large Pages" false -option Hash 1024 bench -depth 26
Time  : 57754
Nodes : 122640389
NPS   : 2123478

2. Usage of large pages enabled:

Code: Select all

RubiChess-20221102_x86-64-avx2.exe -option Hash 1024 bench -depth 26
Time  : 49201
Nodes : 122640389
NPS   : 2492605

So we have 17% more speed under these conditions.
With 256MB hash and a depth 24 bench I measured only 5% more speed with large pages enabled compared to disabled, so the advantage depends on the conditions.