Solaris: Disabling Out Of The Box (OOB) Large Page Support

Starting with the release of Solaris 10 1/06 (aka Solaris 10 Update 1), large page OOB feature turns on MPSS (Multiple Page Size Support) automatically for applications' data (heap) and text (libraries).

One obvious advantage of this large page OOB feature is that it improves the performance of user land applications by reducing the wastage of CPU cycles in serving iTLB and dTLB misses. For example, if the heap size of a process is 256M, on a Niagara (UltraSPARC-T1) box it will be mapped on to a single 256M page. On a system that doesn't support large pages, it will be mapped on to 32,768 8K pages. Now imagine having all the words of a story on a single large page versus having the words spread into 32,500+ small pages. Which one do you prefer?

However large page OOB feature may have negative impact on some applications - eg., application may crash due to some wrong assumption(s) about the page size {by the application} or there could be an increase in virtual memory consumption due to the way the data and libraries are mapped on to larger pages.

Fortunately Solaris provides a bunch of /etc/system tunables to enable/disable large page OOB support.

/etc/system tunables to disable large page OOB feature

• set exec_lpg_disable = 1
  
  This parameter prevents large pages from being used when the kernel is allocating memory for processes being executed. These constitute the memory needed for a processes' text/data/bss.
  
  
• set use_brk_lpg = 0
  
  This parameter prevents large pages from being used for heap. To enable large pages for heap, set the value of this parameter to 1 or remove this parameter from /etc/system completely.
  
  Note:
  brk() is the kernel routine that is called whenever a user level application invokes malloc().
  
  
• set use_stk_lpg = 0
  
  This parameter disables the large pages for stack. Set it to 1 to retain the default functionality.
  
  
• set use_zmap_lpg = 0
  
  This variable controls the size of anonymous (anon) pages.
  
  
• set use_text_pgsz4m = 0
  
  This tunable disables the default use of 4M text pages on UltraSPARC-III/III+/IV/IV+/T1 platforms.
  
  
• set use_text_pgsz64k = 0
  
  This tunable disables the default use of 64K text pages on UltraSPARC-T1 (Niagara) platform.
  
  
• set use_initdata_pgsz64k = 0
  
  This tunable disables the default use of 64K data pages on UltraSPARC-T1 (Niagara) platform.

Tuning off large page OOB support for heap/stack/anon pages on-the-fly

Setting /etc/system parameters require the system to be rebooted to enable/disable large page OOB support. However it is possible to set the desired page size for heap/stack/anon pages dynamically as shown below. Note that the system goes back to the default behavior when it is rebooted. Depending on the need to turn off large page support, use mdb or /etc/system parameters at your discretion.

To turn off large page support for heap, stack and anon pages dynamically, set the following under mdb -kw:

• use_brk_lpg/W 0 (heap)
  
• use_stk_lpg/W 0 (stack)
  
• use_zmap_lpg/W 0 (anon)

Note:
Java sets its own page size with memctl() interface - so, the /etc/system changes won't impact Java at all. Consider using the JVM option -XX:LargePageSizeInBytes=pagesize[K|M] to set the desired page size for Java process mappings.

How to check whether disabling large page support is really helping?

Compare the outputs of the following {along with application specific data} before and after changes:

• vmstat 2 50 - look under free and id columns
  
• trapstat -T 5 5 - check %time column
  
• mdb -k and then ::memstat

How to set the maximum large page size?

Run pagesize -a to get the list of supported page sizes for your platform. Then set the page size of your choice as shown below.

% mdb -kw
Loading modules: [ unix krtld genunix specfs dtrace ufs sd ip sctp usba random fcp fctl nca lofs ssd logindmux ptm cpc sppp crypto nfs ipc ]
> auto_lpg_maxszc/W <hex_value>

where:
hex_value = { 0x0 for 8K,
              0x1 for 64K,
              0x2 for 512K,
              0x3 for 4M,
              0x4 for 32M and
              0x5 for 256M }

How to check the maximum page size in use?

Here is an example from a Niagara box (T2000):

% pagesize -a
8192
65536
4194304
268435456


% mdb -kw
Loading modules: [ unix krtld genunix specfs dtrace ufs sd ip sctp usba random fcp fctl nca lofs ssd logindmux ptm cpc sppp crypto nfs ipc ]
> auto_lpg_maxszc/X
auto_lpg_maxszc:
auto_lpg_maxszc:5
> ::quit

See Also:
6287398 vm 1.5 dumps core with -d64

Acknowledgements:
Sang-Suan Sam Gam
___________________
Technorati tags:
Solaris | OpenSolaris | Sun | Performance