Large page support for instructions (text) in Solaris 10 1/06

Solaris 10 Update 1 (aka Solaris 10 1/06) is now available for download, at Sun downloads web site. One of the notable features of this major release is the large page support for instructions. It is a known fact that dTLB/dTSB, iTLB/iTSB miss rates is a bottleneck for very large applications. The system has to spend some time serving the virtual-to-physical memory translations. These translations are expensive, and the time spent in these translations are counted in usr% (CPU).

Until the release of Solaris 9, 8K is the only supported page size on Solaris. Solaris 9 introduced Multiple Page Size Support (MPSS), for the applications to use any of the possible hardware address translation sizes supported by the system (use pagesize -a to get the supported page sizes). However on Solaris 9, MPSS for user applications is provided only for anonymous memory eg., heap, stack, ISM, MAP_ANON mappings.

Solaris 10 introduced dynamic TSB support, which helps reducing the dTSB miss rate considerably. However, on Solaris 10, user iTLB misses are still a major bottleneck. In Solaris, user text segments are implemented with memory mapped files, and can potentially be mapped with large pages.

VMPSS - Large Pages for Executables and Files

Now with Solaris 10 Update 1, Sun introduced MPSS for executables (text/instructions) and files, which is referred as MPSS for Vnodes or simply VMPSS. VMPSS extended MPSS support to regular file mappings in order to reduce iTLB misses for text, dTLB misses for initialized data, and also to reduce dTLB misses in general for other memory mapped files. Due to VMPSS, large applications may get large pages automatically, for text and initialized data segments, based on the segment size and the page size support capabilities of the underlying hardware. So if you have a large application running on Solaris 10 Update 1, the application might already be enjoying the benefits of VMPSS, and hence you may observe noticeable improvement in the run-time performance of the application. To check if the application is taking advantage of VMPSS, run pmap -sx <pid>, and observe the page sizes under Pgsz column of pmap output.

Default page sizes, and tunables on different platforms

UltraSPARC II

No large pages are used by default on US-II systems. To enable the default use of 64K or 4M pages for text, add the following lines in /etc/system {and reboot the machine once}:

        set use_text_pgsz64k 1
        set use_text_pgsz4m 1

To enable the default use of 64K pages for data, add set use_initdata_pgsz64k 1 as well, to /etc/system

UltraSPARC III/III+/IV/IV+

4M is the only large page size supported by default for text mappings. If the application exhibits performance regression with this default behavior, the default use of 4M text pages can be disabled by adding the following line in /etc/system:

        set use_text_pgsz4m 0

No large pages are used by default for initialized data segments, on these platforms.

UltraSPARC T1 (aka Niagara)

64K and 4M page sizes for text, and 64K page sizes for initialized data are used by default.

In case of regressions, add the following lines in /etc/system.

To disable the defalt use of 64K or 4M text pages:

        set use_text_pgsz64k 0
        set use_text_pgsz4m 0

To disable the default use of 64K data pages:
        set use_initdata_pgsz64k 0

You can find more information in Denis Sheahan's blog post: UltraSPARC T1 large page projects in Solaris

x86

No large pages are used by default on x86. The default use of 2M text pages on PAE machines or 4M pages on non-PAE machines can be enabled with the following setting in /etc/system:

        set use_text_largepages 1

----
All the above mentioned {VMPSS} design policies should result in the use of large pages when they are most expected to help performance, and limit potential regressions from large pages when they are unlikely to improve performance.

Acknowledgements:
Aleksandr Guzovskiy, Sun Microsystems

Technorati tags
Solaris | OpenSolaris | Sun