Linux Scroll

Just another WordPress.com site

Category Archives: solaris

Solaris install patch

Download patch to /tmp

Login to console

sc> console

Login to system.

Swift to System Maintenance Mode.

# init S
****    SYSCON CHANGED TO /dev/console  ****
# svc.startd: The system is coming down for administration.  Please wait.
Nov  5 05:26:43 server3 syslogd: going down on signal 15
svc.startd: Killing user processes: done.
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
SINGLE USER MODE

Root password for system maintenance (control-d to bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode

Nov  5 05:27:03 su: 'su root' succeeded for root on /dev/console

    Authorized uses only. All activity may be monitored and reported.

You have new mail.
# 

Install patch:

# patchadd 120753-06
Validating patches...

Loading patches installed on the system...

Done!

Loading patches requested to install.

Done!

Checking patches that you specified for installation.

Done!


Approved patches will be installed in this order:

120753-06


Checking installed patches...
Verifying sufficient filesystem capacity (dry run method)...
Installing patch packages...

Patch 120753-06 has been successfully installed.
See /var/sadm/patch/120753-06/log for details

Patch packages installed:
  SUNWlibm
  SUNWlibmsr

Verify patch

# patchadd -p | grep 120753-06
Patch: 120753-06 Obsoletes: Requires: Incompatibles: Packages: SUNWlibmsr SUNWlibm
# shutdown -y -i6 -g0
Advertisements

dladm

# dladm show-dev
vsw0            link: unknown   speed: 1000  Mbps       duplex: full
vsw1            link: unknown   speed: 1000  Mbps       duplex: full
vsw2            link: unknown   speed: 1000  Mbps       duplex: full
vsw3            link: unknown   speed: 1000  Mbps       duplex: full
e1000g0         link: up        speed: 1000  Mbps       duplex: full
e1000g1         link: up        speed: 1000  Mbps       duplex: full
e1000g2         link: up        speed: 1000  Mbps       duplex: full
e1000g3         link: up        speed: 1000  Mbps       duplex: full

The account should be locked for interactive login.

# passwd -N username
# passwd -s username
username NL

Solaris user account password never expired.

# passwd -x -1 username

Core Dump Management on the Solaris OS

Abstract: Abnormal termination of a process will trigger a core dump file. A core dump file is very helpful to programmers or support engineers for determining the root cause of abnormal termination, because it provides invaluable information about the runtime status at crash time. This article provides information about core dumps, as well as features and analysis tools in the Solaris Operating System that can be used to manage core dumps.

Types of Core Dumps: Process and System

A core dump is a file that records the contents of a process along with other useful information, such as the processor register’s value. There are two types of core dumps: system core dumps and process core dumps. They differ in many aspects, such as the manner in which they are created and the method used to analyze them.
Cause of Process Core Dumps

When an application process receives a specific signal and terminates, the system generates a core dump and stops the process. In most cases, the signal leading to the application crash is SIGSEGV or SIGBUS.

SIGSEGV indicates that the application is accessing an invalid memory address. This situation often occurs in C/C++ programs if there are code errors in pointer manipulation.

On the Solaris OS, you can use the libumem(3LIB) library as the user-mode memory allocator instead of libc. The libumem library can help find memory leaks, buffer overflows, attempts to use freed data, and many other memory allocation errors. Also, its memory allocator is very fast and scalable with multithreaded applications.

SIGBUS indicates that the application is accessing a memory address that does not conform to CPU memory alignment rules. This usually happens to a system with an UltraSPARC processor. Systems with x86/x64 CPUs can handle unaligned memory addresses, but there is a performance impact.

The Sun Studio C/C++ compiler has the -xmemalign option, which can be used to adjust the behavior of the UltraSPARC CPU when there are unaligned memory addresses that can be determined at compile time. The -xmemalign option causes the compiler to generate additional load/store instructions for unaligned memory access. However, the -xmemalign option cannot handle unaligned memory access during runtime. If unaligned memory access happens during runtime, the developer needs to change the source code.

There are other signals whose default disposition is to create a core dump, for example, SIGFPE, which indicates a floating point exception. The Signal(3HEAD) man page provides more details.
How to Manage a Process Core Dump

The Solaris OS attempts to create up to three core dump files for each abnormally terminated process. One of the core dump files, which is called the per-process core file, is located in the current directory. Another core dump file, which is called the global core file, is created in the system-wide location. If the process is running in a local zone, a third core file is created in the global zone’s location.

You can use the coreadm(1M) command to manage the core dumps. All the settings are saved in the /etc/coreadm.conf configuration file.

Below is a typical scenario, which shows the current system configuration for core dumps:

-bash-3.00# coreadm
global core file pattern:
global core file content: default
init core file pattern: core
init core file content: default
global core dumps: disabled
per-process core dumps: enabled
global setid core dumps: disabled
per-process setid core dumps: disabled
global core dump logging: disabled

In the previous output:

The global core dumps: disabled line indicates no global core dump will be generated.
The per-process core dumps: enabled line indicates a per-process core dump will be generated for each abnormal process.
The init core file pattern line indicates the contents will be gathered from the live process to the per-process core dump.

You can also use the coreadm command to control the core dump file name:

-bash-3.00# coreadm -i core.%f.%p

This command causes the per-process core file name to be appended with the program file name (%f) and the runtime process ID (%p). A core dump file will be generated in the current working directory of the process.

-bash-3.00# coreadm -g /globalcore/core.%f.%p -e global

By default, the global core dump is disabled. You need to use the coreadm command with the -e global option to enable it. The -g option causes the command to append the program name (%f) and the runtime process ID (%p) to the core file name.

As indicated previously, coreadm can specify the parts of the process that will be saved to the core file. Previously, when you performed a post-mortem analysis, you needed to obtain all the specific versions of the dependent libraries and runtime modules, because the core dump file does not contain this text information. It is quite a headache for programmers to recreate the environment from the original machine.

With the default configuration, the Solaris OS applies the “default” pattern to each process core dump, which means the process core dump contains stack, heap, text, shared memory (SHM), intimate shared memory (ISM), and dynamic intimate shared memory (DISM) information, plus other information. The text part of the process core dump also contains a partial symbol table (dynsm), which will help you get a readable stack trace directly from one core file without any other boring dependent libraries. If the dynsm is insufficient, you can use coreadm to include all symbol tables, as follows:

-bash-3.00# coreadm -G all -I all

This previous command makes both the global core file (-G) and the per-process core file (-I) contain all the parts of the process.

Here’s how to use coreadm to verify the changes:

-bash-3.00# coreadm
global core file pattern: /globalcore/core.%f.%p
global core file content: all
init core file pattern: core.%f.%p
init core file content: all
global core dumps: enabled
per-process core dumps: enabled
global setid core dumps: disabled
per-process setid core dumps: disabled
global core dump logging: disabled

# coreadm
global core file pattern: /var/core/core_%n_%f_%u_%g_%t_%p
global core file content: default
init core file pattern: core
init core file content: default
global core dumps: enabled
per-process core dumps: enabled
global setid core dumps: enabled
per-process setid core dumps: enabled
global core dump logging: enabled

The coreadm command is used to edit the configuration file of the coreadm service, which is managed by the Service Management Facility (SMF) with this service identifier: svc:/system/coreadm:default.
How to Create a Process Core Dump Manually

The Solaris OS provides the gcore(1) command in case you need to create a core dump manually for a live process for analysis purposes:

-bash-3.00# echo $$
2770
-bash-3.00# gcore $$
gcore: core.2770 dumped

The live process ID is appended automatically to the name of the generated core dump. In the previous example, the process of the current shell is dumped and its process ID is 2770.

Note: There are other constraints you need take into account while generating the core dump, for example, the write permissions on the destination directory, the existence of the destination directory, the file system mount option, and process resource limitation. For resource limitation information, refer to the man pages for setrlimit(2) and ulimit(1).

Another useful tool called AppCrash is available. It automatically collects diagnostic and debugging information when any application crashes under the Solaris OS. This article does not address its usage. For more information on using AppCrash, refer to Greg Nakhimovsky’s blog.
How to Analyze a Process Core Dump File

There are lots of tools in the Solaris OS for analyzing core dump files: dbx(1), mdb(1), and pstack(1). The most convenient method is to use the pstack tool to determine the process stack. This tool helps show multithreaded programs as well:

-bash-3.00# pstack core.2580 | more
core ‘core.2580’ of 2580: java_vm
—————– lwp# 1 / thread# 1 ——————–
fef40a27 read (b, 804280c, 1)
feb11ba8 __1cDhpiEread6FipvI_I_ (b, 804280c, 1) + a8
feb11aef JVM_Read (b, 804280c, 1) + 2f
fe77045e ???????? (80685b8, 8042864, 22)

feb1d55c jni_CallStaticVoidMethod (80685b8, 8069020, 80e8355,
0) + 14c
080516c2 main (2, 8047168, 8047174) + 50c
08050daa ???????? (2, 80472cc, 80472d4, 0, 80472d5, 8047301)
—————– lwp# 2 / thread# 2 ——————–
fef40d27 lwp_cond_wait (8067ae8, 8067ad0, fb3a9c08, 0)
fef2de3f _lwp_cond_timedwait (8067ae8, 8067ad0, fb3a9c50) + 35

fef3fc32 _thr_setup (fef82400) + 4e
fef3ff20 _lwp_start (fef82400, 0, 0, fb3a9ff8, fef3ff20,
fef82400)
—————– lwp# 3 / thread# 3 ——————–
fef40d27 lwp_cond_wait (8116588, 8116570, 0, 0)
feab737c __1cCosHSolarisFEventEpark6M_v_ (8116548) + 4c

In general, if the program’s symbol table is not stripped and its runtime stack trace is available, you can expect almost 50 percent of the problems to be resolved.

dbx is a free source-level debug tool provided by Sun Studio software. Sun Studio software includes free, optimizing C, C++, and Fortran compilers that can be used on both the Solaris OS and Linux. dbx not only helps you inspect the state of your program, but it also collects the program performance data. Here is a typical scenario for analyzing the core file using dbx. For more details on dbx, please refer to the document called Sun Studio 11: Debugging a Program With dbx.

-bash-3.00# /opt/SUNWspro/bin/dbx tServer core
For information about new features see ‘help changes’
To remove this message, put ‘dbxenv suppress_startup_message 7.5’
in your .dbxrc
Reading tServer
core file header read successfully
Reading ld.so.1
Reading libpthread.so.1
Reading librt.so.1
Reading libsocket.so.1
Reading libnsl.so.1
Reading libc.so.1
Reading libthread.so.1
Reading libCrun.so.1
Reading libm.so.1
Reading libkstat.so.1
t@1 (l@1) program terminated by signal SEGV (no mapping at
the fault address)
0xffffffff7ce3ce90: strcmp+0x0014: ldub [%i1], %i5
Current function is txnAtomMatchRqst
177 && strcmp(pMsg->inHeader.msgVer, “01” == 0)) {
(dbx) threads ** show all the threads
o> t@1 a l@1 ?() signal SIGSEGV in strcmp()
t@2 b l@2 tTimerThread() LWP suspended in __pollsys()
(dbx) thread -info t@1 ** show the thread information

Thread t@1 (0xffffffff7a500000) at priority 0
state: active on l@1
base function: 0x0: 0x0000000000000000() stack:
0xffffffff80000000[8388608]
flags: (none)
masked signals: (none)
Currently active in strcmp
(dbx) where ** show the thread stack
current thread: t@1
[1] strcmp(0x100263d63, 0x0, 0xac, 0x0, 0x30, 0x31), at
0xffffffff7ce3ce90
=>[2] tAtomMatchRqst(), line 177 in “tAtomMatchRqst.c”
[3] tFlow(), line 96 in “tFlow.c”
[4] tServer(rqst = 0x1001e6c58), line 73 in “tServer.c”
[5] _tsvcdsp(0x1700, 0x0, 0x10004ca60, 0x1001e55c0, 0x0,
0x1001d9440), at 0xffffffff7e15d138
[6] _trunserver(0x1001e3844, 0x1001da958, 0x0,
0xffffffff7e3525c8, 0x1400, 0x1001ee400), at 0xffffffff7e180ea0
[7] _tstartserver(0x0, 0xffffffff7ffff568, 0x1001bcc38,
0x1001d9440, 0x0, 0x0), at 0xffffffff7e15be28
[8] main(0xf, 0xffffffff7ffff568, 0xffffffff7ffff5e8, 0x0,
0x0, 0x100000000), at 0x1000099ec
(dbx) quit
-bash-3.00#

From the previous example, you can use dbx to determine the abnormal thread, which is marked with “o,” and its root cause by showing the source code. Of course, this will not happen unless you provide the application source code and add debug information during the compile phase.

If you are familiar with assembly language and hardware specifications, you can use mdb to debug the core file, because mdb is a low-level debugging utility for both programs and the Solaris OS.
Cause of System Core Dumps

There are lots of reasons why the Solaris OS might crash and produce a core dump. Not only software problems, such as like drivers and programs, but also hardware errors can induce a system core dump.
How a System Core Dump Is Created

When detecting whether the integrity of data was corrupted or whether a fatal error in hardware occurred, the Solaris OS invokes panic(). The panic() routine interrupts all processes as if the OS is suspended. Then it generates a system core dump, which is a copy of OS in the memory, and saves it to the dump device. After a crash, the OS use savecore(1) to retrieve the core dump from the dump device to the savecore directory during the next boot. The savecore routine generates two files. One file is unix., which is an OS symbol table list, and the other is vmcore., which is the core dump data file. By default, the dump device is a swap disk partition and the savcore directory is set to /var/crash/. The trailing in the file names is an integer that grows every time savecore runs.
How to Manage a System Core Dump

You can use dumpadm(1M) to manage dump devices and the savecore directory:

-bash-3.00# dumpadm -d /dump -s /savecore
Dump content: kernel pages
Dump device: /dump (dedicated)
Savecore directory: /savecore
Savecore enabled: yes

To verify this or see the current configuration, you can run only dumpadm:

-bash-3.00# dumpadm
Dump content: kernel pages
Dump device: /dump (dedicated)
Savecore directory: /savecore
Savecore enabled: yes

You can also use dumpadm to set the dump content and enable savecore(1) operation during the boot.

All the configuration information is saved in the /etc/dumpadm.conf configure file. The system crash dump service is also managed by SMF with this service identifier: svc:/system/dumpadm:default.
How to Create a System Core Dump Manually

In some cases, you need to save a core dump manually to take a snapshot of the live system. In the Solaris OS, there are several means you can use. For example, you can use reboot -d to force the generation of a core dump with reboot. Or you can use savecore -L to create a live OS core dump. If you want to use savecore(1M) to create a live core dump, you must use dumpadm to set a non-swap device as the dump device, because live core dumps take a swap device as a part of virtual memory, which is to be dumped.

Sometimes, the system will hang without crashing. If you are using a Sun UltraSPARC processor-based machine, you can press Stop-A to run in OpenBoot PROM (OBP) mode, and then use the sync OBP command to force a crash core dump.

On x86 platforms, there is no corresponding OBP unit. However, you can use kmdb(1M). To use kmdb to create a core dump, you need load its module during system booting.

Here are the steps for the Solaris 10 1/06 OS or later.

Edit the /boot/grub/menu.lst file and append the -k string to the initrd line, as follows:

title Solaris 10 11/06 s10x_u3wos_10 X86
root (hd0,1,a)
kernel /platform/i86pc/multiboot -k
module /platform/i86pc/boot_archive

This will make the OS boot with kmdb.
Then restart the machine manually.

Alternatively, if you are using the Solaris 10 GA OS, just enter b -k when you see the Select (B)oot or (I)nterpreter: system prompt during the system boot stage.

After performing these steps, press F1-A to break the system to kmdb. This action must be performed in console mode, because kmdb suspends the system and GUI applications. If you are using a desktop system, the Solaris OS will fail to switch to console mode and your desktop will appear to hang. However kmdb is running and you can still type commands.

$<systemdump

The systemdump command generates the core dump file for you. The dump device and savecore directory for this operation are still constrained by dumpadm.

Sometimes, the system will hang without any response even when you use kmdb or OBP. In this case, use the "deadman timer." The deadman timer allows the OS to force a kernel panic in the event of a system hang. This feature is available on x86 and SPARC systems. Add the following line to /etc/system and reboot so the deadman timer will be enabled.

set snooping=1

The enabled deadman timer will perform a level 15 interrupt once a second. It will check whether the kernel lbolt variable is updated. If the deadman timer notices that the lbolt variable has not been incremented for a period of time (the default is 50 seconds), it will cause a panic. The period of time can be configured in /etc/system. The following example makes the deadman timer wait 120 seconds for the lbolt variable update:

set snoop_interval=120000000

Solaris Dynamic Tracing (DTrace) was introduced with the release of Solaris 10 OS. DTrace allows you to understand and explore applications or the operating system. DTrace contains a feature called Anonymous Tracing. It provides device driver developers with a way to debug and trace system activity that occurs during the system boot. If the Solaris OS hangs, you can use this feature to generate a core dump and catch other information you want. For the information on using DTrace, refer to the DTrace HowTo Guide and the Solaris Dynamic Tracing Guide.
How to Analyze a System Core Dump File

This article cannot provide solutions for fixing a system core dump, because such an analysis requires much low-level computing knowledge of the OS kernel and also of the hardware. However here are some basic guidelines for your reference:

Check the system console and the /var/adm/messages file, because they contain valuable information for identifying the problem that the system encountered.
Use the strings(1) command to process the core dump file. This command prints out the ASCII strings in any binary file, including a core dump file. You need to look at these ASCII strings.
Check the error you encounter on the SunSolve web site, or use the free Solaris Crash Analysis Tool (CAT) to help you investigate further.

Switching between ALOM and ILOM

ALOM -> ILOM

sc> userclimode YOURUSERNAME default
sc> logout

Login again

ILOM -> ALOM

-> set /SP/users/YOURUSERNAME cli_mode=alom
-> exit

Login again

Solaris LDom boot failsafe

Howto boot into failsafe mode.

Disable auto-boot:

# ldm set-variable auto-boot?=false ServerName 

Stop server:

# ldm stop-domain -f  ServerName
LDom  stopped

Start server:

# ldm start-domain  ServerName
LDom  started

Check the status:

# ldm list

Login to server:

# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'. 
Connecting to console "ServerName" in group "ServerName" ....
Press ~? for control options .. 
{0} ok boot -F failsafe
Boot device: /virtual-devices@100/channel-devices@200/disk@0 File and args: -F failsafe
SunOS Release 5.10 Version Generic_137137-09 64-bit
Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Configuring devices.
Searching for installed OS instances... 
Solaris 10 10/08 s10s_u6wos_07b SPARC was found on /dev/dsk/c0d0s0.
Do you wish to have it mounted read-write on /a? [y,n,?] yes
mounting /dev/dsk/c0d0s0 on /a 
Starting shell.
# 
.
.
# reboot
Terminated
syncing file systems... done
rebooting...
Resetting...

Enable auto boot:

# ldm set-variable auto-boot?=true ServerName 

Solaris kernel memory usage

Use mdb memstat:

# echo ::memstat | mdb -k
> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    1645756             12857   80%
Anon                       144767              1130    7%
Exec and libs               14327               111    1%
Page cache                  45537               355    2%
Free (cachelist)            30303               236    1%
Free (freelist)            177857              1389    9%

Total                     2058547             16082
Physical                  2037015             15914

Check kernel memory:

# echo ::kmastat | mdb -k
cache                        buf    buf    buf    memory     alloc alloc
name                        size in use  total    in use   succeed  fail
------------------------- ------ ------ ------ --------- --------- -----
kmem_magazine_1               16   8170  83312   1343488  39227967     0
kmem_magazine_3               32   7692  40132   1294336   3508851     0
kmem_magazine_7               64   3350  11303    729088   1877738     0
kmem_magazine_15             128  36334  37296   4849664   3589181     0
kmem_magazine_31             256   5918   5921   1564672   3058714     0
kmem_magazine_47             384   4294   7287   2842624   1520151     0
kmem_magazine_63             512    688    750    409600    210174     0
kmem_magazine_95             768   2816   2820   2310144    250631     0
kmem_magazine_143           1152  10585  10591  12394496   1619915     0
kmem_slab_cache               56 204439 232000  13107200  33012727     0
kmem_bufctl_cache             24 488696 524433  12673024  38182097     0
kmem_bufctl_audit_cache      128      0      0         0         0     0
kmem_va_8192                8192 165134 174592 1430257664   1393380     0
kmem_va_16384              16384  34934  41584 681312256    439536     0
kmem_va_24576              24576    122    150   3932160     20362     0
kmem_va_32768              32768    149    168   5505024     18797     0
kmem_va_40960              40960     54     84   3670016     23481     0
kmem_va_49152              49152     12     25   1310720     15449     0
kmem_va_57344              57344    101   3932 257687552   1033903     0
kmem_va_65536              65536     15     24   1572864     13509     0
kmem_alloc_8                   8 788217 896994   7225344 2741847016     0
kmem_alloc_16                 16  52376  56388    909312 4203420892     0
kmem_alloc_24                 24  86260  92547   2236416 904647777     0
kmem_alloc_32                 32  41541 131318   4235264 876801297     0
kmem_alloc_40                 40 100594 169099   6823936 468268828     0
kmem_alloc_48                 48  70987  96161   4661248 3674285699     0
kmem_alloc_56                 56  64066 102660   5799936 3969357818     0
kmem_alloc_64                 64  47860 143510   9256960 1806781066     0
kmem_alloc_80                 80 245069 287951  23355392 3088177948     0
kmem_alloc_96                 96 525784 646380  63037440 2338400564     0
kmem_alloc_112               112 271069 331632  37732352 1686107907     0
kmem_alloc_128               128 109243 136899  17801216 3406621021     0
kmem_alloc_160               160  54123 121100  19841024 1681095961     0
kmem_alloc_192               192  39922  50064   9764864 3775804778     0
kmem_alloc_224               224  10650 113652  25862144 2967560349     0
kmem_alloc_256               256  25932 239196  63209472 3031078054     0
kmem_alloc_320               320    966   2000    655360 2086227831     0
kmem_alloc_384               384   1976   2373    925696 526682329     0
kmem_alloc_448               448    210    270    122880  44813368     0
kmem_alloc_512               512    936   8610   4702208 722407205     0
kmem_alloc_640               640    333    432    294912  60336405     0
kmem_alloc_768               768     69    220    180224 721623266     0
kmem_alloc_896               896     53     99     90112   3246426     0
kmem_alloc_1152             1152   1373   1666   1949696 4100490601     0
kmem_alloc_1344             1344     31     66     90112 126147033     0
kmem_alloc_1600             1600     49     90    147456   1844172     0
kmem_alloc_2048             2048    340    600   1228800 1210486775     0
kmem_alloc_2688             2688    268    360    983040 224442841     0
kmem_alloc_4096             4096    288    400   1638400 2526752643     0
kmem_alloc_8192             8192  43018  45503 372760576 1846257716     0
kmem_alloc_12288           12288     22     34    417792    240028     0
kmem_alloc_16384           16384    159    175   2867200   1061014     0
kmem_alloc_24576           24576     71     81   1990656      3827     0
kmem_alloc_32768           32768    523    555  18186240   7099798     0
streams_mblk                  64   8537  16256   1048576 2934955604     0
streams_dblk_16              128    167   1197    155648 2261565887     0
streams_dblk_80              192    631   3696    720896 2147139417     0
streams_dblk_144             256      2    620    163840 707372521     0
streams_dblk_208             320    265   1150    376832 495915977     0
streams_dblk_272             384      0    273    106496 812959285     0
streams_dblk_336             448      0    648    294912 2008234167     0
streams_dblk_528             640      0    864    589824 1588500597     0
streams_dblk_1040           1152      7    875   1024000 2727568647     0
streams_dblk_1488           1600      0    125    204800 474833022     0
streams_dblk_1936           2048      2    648   1327104 1528413562     0
streams_dblk_2576           2688      0    504   1376256 408130651     0
streams_dblk_3920           4032      0     36    147456  37951781     0
streams_dblk_8192            112     16    819    106496 821525039     0
streams_dblk_12112         12224      0     14    172032  19674127     0
streams_dblk_16384           112      0    189     24576  63851267     0
streams_dblk_20304         20416      0     38    778240   2157872     0
streams_dblk_24576           112      0    126     16384    245799     0
streams_dblk_28496         28608      0     12    344064     46802     0
streams_dblk_32768           112      0    126     16384    285249     0
streams_dblk_36688         36800      0      6    221184   1150850     0
streams_dblk_40960           112      0     63      8192      2732     0
streams_dblk_44880         44992      0      4    180224      4401     0
streams_dblk_49152           112      0     63      8192      3295     0
streams_dblk_53072         53184      0     46   2449408 193975650     0
streams_dblk_57344           112      0     63      8192         5     0
streams_dblk_61264         61376      0     10    614400        78     0
streams_dblk_65536           112      0    126     16384     72298     0
streams_dblk_69456         69568      0      0         0         0     0
streams_dblk_73728           112      0      0         0         0     0
streams_dblk_esb             112      0    441     57344  42958079     0
streams_fthdr                264      0      0         0         0     0
streams_ftblk                232      0      0         0         0     0
multidata                    248      0      0         0         0     0
multidata_pdslab            7112      0      0         0         0     0
multidata_pattbl              32      0      0         0         0     0
log_cons_cache                48     13    169      8192    255953     0
taskq_ent_cache               56   2263  18850   1064960  66488929     0
taskq_cache                  216     83    111     24576       101     0
id32_cache                    32      0      0         0         0     0
bp_map_8192                 8192      0     32    262144 190876942     0
bp_map_16384               16384      0     16    262144    246173     0
bp_map_24576               24576      0     10    262144     75024     0
bp_map_32768               32768      0      8    262144     27126     0
bp_map_40960               40960      0      6    262144     13794     0
bp_map_49152               49152      0      5    262144     10154     0
bp_map_57344               57344      0      8    524288      6759     0
bp_map_65536               65536      0      4    262144      5196     0
memseg_cache                 112      0      0         0         0     0
mod_hash_entries              24   2513   7797    188416  19863924     0
ipp_mod                      304      0      0         0         0     0
ipp_action                   368      0      0         0         0     0
ipp_packet                    64      0      0         0         0     0
mmuctxdom_cache              248      4     31      8192         4     0
sfmmuid_cache                576    239    420    245760 1574595493     0
sfmmu_tsbinfo_cache           64    250    508     32768 2765086445     0
sfmmu_tsb8k_cache           8192      0      0         0         0     0
sfmmu_tsb_lgrp0_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp1_cache       8192    128    182   1490944 1256525791     0
sfmmu_tsb_lgrp2_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp3_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp4_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp5_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp6_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp7_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp8_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp9_cache       8192      0      0         0         0     0
sfmmu_tsb_lgrp10_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp11_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp12_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp13_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp14_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp15_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp16_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp17_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp18_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp19_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp20_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp21_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp22_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp23_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp24_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp25_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp26_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp27_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp28_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp29_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp30_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp31_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp32_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp33_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp34_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp35_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp36_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp37_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp38_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp39_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp40_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp41_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp42_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp43_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp44_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp45_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp46_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp47_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp48_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp49_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp50_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp51_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp52_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp53_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp54_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp55_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp56_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp57_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp58_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp59_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp60_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp61_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp62_cache      8192      0      0         0         0     0
sfmmu_tsb_lgrp63_cache      8192      0      0         0         0     0
sfmmu8_cache                 312 214247 287534  90595328 693189563     0
sfmmu1_cache                  88  94122 108008   9617408 158391685     0
pa_hment_cache                64    246    508     32768 1245245949     0
ism_blk_cache                272      0      0         0         0     0
ism_ment_cache                32      0      0         0         0     0
seg_cache                     72  10939  19662   1425408 1968456830     0
dev_info_node_cache          520    186    210    114688       543     0
segkmem_ppa_4096            4096      0     12     49152      1108     0
segkp_8192                  8192    234    304   2490368 223700253     0
segkp_16384                16384      0      0         0         0     0
segkp_24576                24576      0      0         0         0     0
segkp_32768                32768   1146   1212  39714816   2494843     0
segkp_40960                40960      0      0         0         0     0
umem_np_8192                8192      0     32    262144     88991     0
umem_np_16384              16384      0     16    262144     54420     0
umem_np_24576              24576      0     20    524288     21177     0
umem_np_32768              32768      0     16    524288     33240     0
umem_np_40960              40960      0      0         0         0     0
umem_np_49152              49152      0      0         0         0     0
umem_np_57344              57344      0      0         0         0     0
umem_np_65536              65536      0     16   1048576     29464     0
thread_cache                 808    704   1593   1449984 1446459461     0
wbuf32_cache                 512    465    525    286720 998616193     0
wbuf64_cache                1024    234    441    516096  70069593     0
lwp_cache                    912    699    824    843776 112510355     0
turnstile_cache               64   1112   2667    172032 1515314717     0
tslabel_cache                 48      2    169      8192         2     0
cred_cache                   232    156    315     73728   8450365     0
rctl_cache                    48   3154   4563    221184 4109081266     0
rctl_val_cache                64   6499   8763    565248  65234488     0
task_cache                   128     89    189     24576    268700     0
cyclic_id_cache               64      8    127      8192       125     0
dnlc_space_cache              24  61630  62376   1507328 141815198     0
vn_cache                     240 290675 302219  79863808 1884943311     0
vsk_anchor_cache              40      8    203      8192         8     0
file_cache                    56  18652  19140   1081344 3768551339     0
stream_head_cache            400    417    920    376832 1244269412     0
queue_cache                  656    753   1428    974848 1371822627     0
syncq_cache                  160     17    100     16384     72142     0
qband_cache                   64      0      0         0         0     0
linkinfo_cache                48     12    169      8192        12     0
ciputctrl_cache              512      0      0         0         0     0
serializer_cache              64     47    508     32768    483385     0
as_cache                     224    238    432     98304 1577798941     0
marker_cache                 128      0    126     16384 510692697     0
anon_cache                    48 542542 1754220  85032960 3959556280     0
anonmap_cache                 64   6833  12065    778240  72954937     0
segvn_cache                  152  10939  19981   3088384 4291619139     0
flk_edges                     48      0    169      8192       641     0
fdb_cache                    104      0      0         0         0     0
timer_cache                  136      1     59      8192       110     0
vmu_bound_cache               32   2806   5334    172032    111326     0
vmu_object_cache              32   2004   3810    122880     74403     0
physio_buf_cache             248      0     64     16384    171945     0
fnode_cache                  176     10     84     16384     75559     0
pipe_cache                   320     75    200     65536 580905012     0
snode_cache                  152    228    371     57344 330064579     0
ufs_inode_cache              368  48604 167024  62193664 201563474     0
directio_buf_cache           272      0     58     16384     82595     0
lufs_save                     24      0    339      8192 146288546     0
lufs_bufs                    256      0     62     16384 156399452     0
lufs_mapentry_cache          112   1308  20520   2334720 1226045397     0
px0_dvma_8192               8192    606   1020   8355840 663820611     0
mpt0_cache                   520      3    600    327680 824542780     0
dv_node_cache                184    576    616    114688       717     0
clnt_clts_endpnt_cache        88      0      0         0         0     0
md_stripe_parent              96      0    672     65536 760179691     0
md_stripe_child              312      0    624    196608 783697057     0
md_mirror_parent             184      0    616    114688 408975161     0
md_mirror_child              304      0    598    188416 773839865     0
md_mirror_wow              16440      0      8    139264    741322     0
kcf_sreq_cache                48      0      0         0         0     0
kcf_areq_cache               288      0      0         0         0     0
kcf_context_cache            112      0      0         0         0     0
ip_minor_arena_sa_1            1     22    192       192  21374355     0
ip_minor_arena_la_1            1    136    448       448  75309467     0
ipcl_conn_cache              504     29     90     49152  30239160     0
ipcl_tcpconn_cache          1704    170    756   1376256  56712780     0
ire_cache                    368     61    132     49152  41463451     0
rt_entry                     168      6     48      8192        27     0
radix_mask                    32      2    254      8192         2     0
radix_node                   120      4     67      8192         4     0
ipsec_actions                 72      1    113      8192    464885     0
ipsec_selectors               80      1    101      8192         1     0
ipsec_policy                  72      6    113      8192         6     0
ipsec_info                   352      0     46     16384    578065     0
tcp_timercache                88    706   4876    434176    718347     0
tcp_sack_info_cache           80    131   2525    204800  50542953     0
tcp_iphc_cache               120    167   2412    294912  52999055     0
squeue_cache                 136     11     42      8192        11     0
sctp_conn_cache             2320      1      7     16384         1     0
sctp_faddr_cache             176      0      0         0         0     0
sctp_set_cache                24      0      0         0         0     0
sctp_ftsn_set_cache           16      0      0         0         0     0
ire_gw_secattr_cache          32      0      0         0         0     0
sctpsock                     616      0      0         0         0     0
sctp_assoc                    64      0      0         0         0     0
sdpsock                      560      0      0         0         0     0
socktpi_cache                456    142    323    155648  78708637     0
socktpi_unix_cache           456     54    170     81920    548862     0
audit_buffer                 152     27   2544    393216 3504394090     0
audit_proc                    40    242    609     24576 1007891894     0
mac_impl_cache               400      4     20      8192         5     0
dls_cache                    160      6    100     16384     22098     0
soft_ring_cache              176      0      0         0         0     0
dls_vlan_cache                72      4    113      8192         5     0
dls_link_cache               368      4     22      8192         5     0
dld_ctl_1                      1      0      0         0         0     0
dld_str_cache                272      6     29      8192     22699     0
udp_cache                    440     18     72     32768  30161726     0
process_cache               3104    241    335   1097728 1192196130     0
exacct_object_cache           40      0      0         0         0     0
kssl_cache                  1560      0      0         0         0     0
px1_dvma_8192               8192      2    120    983040  66339974     0
tl_cache                     432     66    252    114688    549103     0
keysock_1                      1      0     64        64         4     0
fctl_cache                   112      0     72      8192      7373     0
spdsock_1                      1      0     64        64       160     0
rds_alloc_cache               88      0      0         0         0     0
namefs_inodes_1                1    115    192       192       759     0
port_cache                    80      2    101      8192         3     0
fp1_cache                    744      1     30     24576      2818     0
fp0_cache                    744      1     30     24576      5882     0
fcp1_cache                  1184      0     78     98304  58502824     0
fcp0_cache                  1184      0    390    491520 211805246     0
Hex0x600219af648_minor_1       1      0      0         0         0     0
Hex0x60021a07588_minor_1       1      0      0         0         0     0
zio_cache                    688      0  46409  34562048 1175991936     0
zio_buf_512                  512 400326 425340 232292352  59791614     0
zio_data_buf_512             512   6513   6570   3588096  30716909     0
zio_buf_1024                1024    663   1008   1032192   8672159     0
zio_data_buf_1024           1024   7446   7472   7651328   9563926     0
zio_buf_1536                1536    169    205    335872    719039     0
zio_data_buf_1536           1536   7082   7085  11608064   4376957     0
zio_buf_2048                2048    102    136    278528   2889564     0
zio_data_buf_2048           2048   8781   8792  18006016   6413469     0
zio_buf_2560                2560     81    120    327680    365792     0
zio_data_buf_2560           2560   7213   7221  19718144   4749716     0
zio_buf_3072                3072     43     96    294912   1999892     0
zio_data_buf_3072           3072   3946   3968  12189696   8134523     0
zio_buf_3584                3584     26     81    294912  12239796     0
zio_data_buf_3584           3584   6887   6894  25100288   5799032     0
zio_buf_4096                4096   1011   1028   4210688  13792317     0
zio_data_buf_4096           4096   2840   2844  11649024   3450301     0
zio_buf_5120                5120     44     64    327680   3619408     0
zio_data_buf_5120           5120   5168   5176  26501120   6845058     0
zio_buf_6144                6144     23     64    393216   3136385     0
zio_data_buf_6144           6144   6941   6952  42713088  12837273     0
zio_buf_7168                7168     11     32    229376   2639282     0
zio_data_buf_7168           7168   5400   5408  38764544   6102065     0
zio_buf_8192                8192     12     25    204800   6182570     0
zio_data_buf_8192           8192   3913   3925  32153600  10175613     0
zio_buf_10240              10240      7     24    245760   3965922     0
zio_data_buf_10240         10240  10431  10436 106864640  12118756     0
zio_buf_12288              12288      9     24    294912   3029262     0
zio_data_buf_12288         12288   3834   3844  47235072  16120860     0
zio_buf_14336              14336      4    264   3784704  94951867     0
zio_data_buf_14336         14336   2323   2332  33431552  11814561     0
zio_buf_16384              16384  41245  41310 676823040   7030707     0
zio_data_buf_16384         16384   3530   3534  57901056  11350710     0
zio_buf_20480              20480      3     18    368640   2416959     0
zio_data_buf_20480         20480   1739   1754  35921920  26011849     0
zio_buf_24576              24576      2     14    344064   2079896     0
zio_data_buf_24576         24576   1837   1847  45391872  20163242     0
zio_buf_28672              28672      2     14    401408   1404523     0
zio_data_buf_28672         28672    508    524  15024128  19216076     0
zio_buf_32768              32768      1     13    425984   1329593     0
zio_data_buf_32768         32768    574    586  19202048  19004319     0
zio_buf_40960              40960      5     16    655360   1797911     0
zio_data_buf_40960         40960    314    320  13107200  37060133     0
zio_buf_49152              49152      0     12    589824   1313545     0
zio_data_buf_49152         49152   1779   1785  87736320  37354075     0
zio_buf_57344              57344      5     18   1032192   1062974     0
zio_data_buf_57344         57344    128    132   7569408  36366171     0
zio_buf_65536              65536    162    176  11534336   1017050     0
zio_data_buf_65536         65536    221    222  14548992  36461419     0
zio_buf_73728              73728      2     15   1105920    755826     0
zio_data_buf_73728         73728    129    130   9584640  36220912     0
zio_buf_81920              81920      0     11    901120    912543     0
zio_data_buf_81920         81920    220    222  18186240  35187315     0
zio_buf_90112              90112      0     11    991232    719217     0
zio_data_buf_90112         90112    105    108   9732096  34534945     0
zio_buf_98304              98304      0      9    884736    682368     0
zio_data_buf_98304         98304    729    733  72056832  26383116     0
zio_buf_106496            106496      1     10   1064960    498701     0
zio_data_buf_106496       106496     28     34   3620864  16737086     0
zio_buf_114688            114688      3     13   1490944    632616     0
zio_data_buf_114688       114688     35     39   4472832  15734312     0
zio_buf_122880            122880      2     12   1474560    531399     0
zio_data_buf_122880       122880     95    101  12410880  15705002     0
zio_buf_131072            131072      4     18   2359296  12874591     0
zio_data_buf_131072       131072  71400  71423 771620864 235118431     0
dmu_buf_impl_t               192 615880 760410 148316160 290973470     0
dnode_t                      656 660056 766512 523272192   8251517     0
arc_buf_hdr_t                168 415299 422016  72024064 745888908     0
arc_buf_t                     40 308114 356265  14376960 756621645     0
zil_lwb_cache                200      1    120     24576  11809125     0
zfs_znode_cache              200  78544 130600  26746880 240675842     0
pty_map                       56      6    145      8192      2176     0
Hex0x30003056dc8_minor_1       1      0      0         0         0     0
dtrace_state_cache         32768      0     12    393216        75     0
md_raid_parent               120      0      0         0         0     0
md_raid_child               1040      0      0         0         0     0
md_raid_cbufs                376      0      0         0         0     0
md_trans_parent               80      0      0         0         0     0
md_trans_child               248      0      0         0         0     0
authkern_cache                72      0      0         0         0     0
authloopback_cache            72      0      0         0         0     0
authdes_cache_handle          80      0      0         0         0     0
rnode_cache                  656      0      0         0         0     0
nfs_access_cache              56      0      0         0         0     0
client_handle_cache           32      0      0         0         0     0
rnode4_cache                 968      0      0         0         0     0
svnode_cache                  40      0      0         0         0     0
nfs4_access_cache             56      0      0         0         0     0
client_handle4_cache          32      0      0         0         0     0
nfs4_ace4vals_cache           48      0      0         0         0     0
nfs4_ace4_list_cache         264      0      0         0         0     0
NFS_idmap_cache               48      0      0         0         0     0
crypto_session_cache         104      0      0         0         0     0
sppptun_map                  432      0      0         0         0     0
aggr_port_cache             1064      0      0         0         0     0
aggr_grp_cache               688      0      0         0         0     0
fcsm_job_cache               104      0      0         0         0     0
fcip1_cache                  744      0      0         0         0     0
fcip1_sendup_cache            24      0      0         0         0     0
fcsm1_cmd_cache              728      0      0         0         0     0
fcip0_cache                  744      0      0         0         0     0
fcip0_sendup_cache            24      0      0         0         0     0
fcsm0_cmd_cache              728      0      0         0         0     0
md_softpart_parent            88      0      0         0         0     0
md_softpart_child            304      0      0         0         0     0
------------------------- ------ ------ ------ --------- --------- -----
Total [static]                                   2531328 1040019393     0
Total [hat_memload]                             90595328 693189563     0
Total [kmem_msb]                                53518336 126058146     0
Total [kmem_va]                                2385248256   2958417     0
Total [kmem_default]                           2739216384 3598957859     0
Total [bp_map]                                   2359296 191261168     0
Total [kmem_tsb_lgrp1]                           1490944 1256525791     0
Total [hat_memload1]                             9617408 158391685     0
Total [segkmem_ppa]                                49152      1108     0
Total [umem_np]                                  2621440    227292     0
Total [segkp]                                   42205184 226195096     0
Total [zio_buf]                                1635262464 807828132     0
Total [px0_dvma]                                 8355840 663820611     0
Total [ip_minor_arena_sa]                            192  21374355     0
Total [ip_minor_arena_la]                            448  75309467     0
Total [px1_dvma]                                  983040  66339974     0
Total [keysock]                                       64         4     0
Total [spdsock]                                       64       160     0
Total [namefs_inodes]                                192       759     0
------------------------- ------ ------ ------ --------- --------- -----

vmem                         memory     memory    memory     alloc alloc
name                         in use      total    import   succeed  fail
------------------------- --------- ---------- --------- --------- -----
heap                      1101801971712 4398046511104         0  26202061     0
    vmem_metadata          75390976   75759616  75759616      9010     0
        vmem_seg           70631424   70631424  70631424      8617     0
        vmem_hash           4149888    4161536   4161536       473     0
        vmem_vmem            542880     632816    598016       219     0
    static                  2596864    2596864   2596864       635     0
        static_alloc          57344      57344     57344         7     0
    hat_memload            90595328   90595328  90595328     19871     0
    kstat                    420160     425984    360448     13573     0
    kmem_metadata          73490432   97779712  97779712     96286     0
        kmem_msb           53518336   53518336  53518336     92392     0
        kmem_cache         13541704   16670720  16670720       464     0
        kmem_hash           3272704    3301376   3301376      8255     0
    kmem_log               33636448   33652736  33652736         6     0
    kmem_firewall_va              0          0         0         0     0
        kmem_firewall             0          0         0         0     0
    mod_sysfile                 462       8192      8192        12     0
    kmem_oversize         165525117  166477824 166477824  22977155     0
    kmem_va               2399125504 2399125504 2399125504    138025     0
        kmem_default      2739216384 2740805632 2740805632   3071081     0
    little_endian            202144     229376    229376     68839     0
    big_endian             17601836   18653184  18653184     69510     0
    bp_map                  2359296    2359296   2359296     76946     0
    ksyms                   1915492    1982464   1982464       287     0
    ctf                      251443     278528    278528       286     0
    kmem_tsb               12582912   12582912  12582912   3439496     0
        kmem_tsb_lgrp0            0          0         0         0     0
        kmem_tsb_lgrp1      7208960   12582912  12582912  50117907 3436400
        kmem_tsb_lgrp2            0          0         0         0     0
        kmem_tsb_lgrp3            0          0         0         0     0
        kmem_tsb_lgrp4            0          0         0         0     0
        kmem_tsb_lgrp5            0          0         0         0     0
        kmem_tsb_lgrp6            0          0         0         0     0
        kmem_tsb_lgrp7            0          0         0         0     0
        kmem_tsb_lgrp8            0          0         0         0     0
        kmem_tsb_lgrp9            0          0         0         0     0
        kmem_tsb_lgrp10           0          0         0         0     0
        kmem_tsb_lgrp11           0          0         0         0     0
        kmem_tsb_lgrp12           0          0         0         0     0
        kmem_tsb_lgrp13           0          0         0         0     0
        kmem_tsb_lgrp14           0          0         0         0     0
        kmem_tsb_lgrp15           0          0         0         0     0
        kmem_tsb_lgrp16           0          0         0         0     0
        kmem_tsb_lgrp17           0          0         0         0     0
        kmem_tsb_lgrp18           0          0         0         0     0
        kmem_tsb_lgrp19           0          0         0         0     0
        kmem_tsb_lgrp20           0          0         0         0     0
        kmem_tsb_lgrp21           0          0         0         0     0
        kmem_tsb_lgrp22           0          0         0         0     0
        kmem_tsb_lgrp23           0          0         0         0     0
        kmem_tsb_lgrp24           0          0         0         0     0
        kmem_tsb_lgrp25           0          0         0         0     0
        kmem_tsb_lgrp26           0          0         0         0     0
        kmem_tsb_lgrp27           0          0         0         0     0
        kmem_tsb_lgrp28           0          0         0         0     0
        kmem_tsb_lgrp29           0          0         0         0     0
        kmem_tsb_lgrp30           0          0         0         0     0
        kmem_tsb_lgrp31           0          0         0         0     0
        kmem_tsb_lgrp32           0          0         0         0     0
        kmem_tsb_lgrp33           0          0         0         0     0
        kmem_tsb_lgrp34           0          0         0         0     0
        kmem_tsb_lgrp35           0          0         0         0     0
        kmem_tsb_lgrp36           0          0         0         0     0
        kmem_tsb_lgrp37           0          0         0         0     0
        kmem_tsb_lgrp38           0          0         0         0     0
        kmem_tsb_lgrp39           0          0         0         0     0
        kmem_tsb_lgrp40           0          0         0         0     0
        kmem_tsb_lgrp41           0          0         0         0     0
        kmem_tsb_lgrp42           0          0         0         0     0
        kmem_tsb_lgrp43           0          0         0         0     0
        kmem_tsb_lgrp44           0          0         0         0     0
        kmem_tsb_lgrp45           0          0         0         0     0
        kmem_tsb_lgrp46           0          0         0         0     0
        kmem_tsb_lgrp47           0          0         0         0     0
        kmem_tsb_lgrp48           0          0         0         0     0
        kmem_tsb_lgrp49           0          0         0         0     0
        kmem_tsb_lgrp50           0          0         0         0     0
        kmem_tsb_lgrp51           0          0         0         0     0
        kmem_tsb_lgrp52           0          0         0         0     0
        kmem_tsb_lgrp53           0          0         0         0     0
        kmem_tsb_lgrp54           0          0         0         0     0
        kmem_tsb_lgrp55           0          0         0         0     0
        kmem_tsb_lgrp56           0          0         0         0     0
        kmem_tsb_lgrp57           0          0         0         0     0
        kmem_tsb_lgrp58           0          0         0         0     0
        kmem_tsb_lgrp59           0          0         0         0     0
        kmem_tsb_lgrp60           0          0         0         0     0
        kmem_tsb_lgrp61           0          0         0         0     0
        kmem_tsb_lgrp62           0          0         0         0     0
        kmem_tsb_lgrp63           0          0         0         0     0
    hat_memload1            9617408    9617408   9617408      1239     0
    kcage_arena                 440       8192      8192        11     0
    segkmem_ppa               49152      49152     49152         3     0
    umem_np                 2752512    2752512   2752512      1747     0
heap32                      6750208  134217728         0     86070     0
    promplat                      0          0         0     90146     0
    id32                          0          0         0         0     0
    module_data             2498843    6619136   6356992       397     0
    trapstat                      0          0         0         0     0
heaptext                   41033728  134217728         0       130     0
    module_text             8923260    9125888   7471104       285     0
logminor_space                   29     262137         0    255359     0
taskq_id_arena                   38 2147483647         0        56     0
heap_lp                   876609536 1098974756864         0      1108     0
    kmem_lp               876609536  876609536 876609536       242   866
segkp                      42205184 2147483648         0      4117     0
zio                       10225197056 16863617024         0  30287435     0
    zio_buf               10225197056 10225197056 10225197056  30287435     0
rctl_ids                         36      32767         0        36     0
zoneid_space                      0       9998         0         0     0
taskid_space                     89     999999         0    261386     0
pool_ids                          0     999998         0         0     0
contracts                        95 2147483646         0    212029     0
regspec                    12025856 5368709120         0       259     0
px0_dvma                   22708224 1040187392         0  47308512     0
ip_minor_arena_sa               192     262140         0         3     0
ip_minor_arena_la               448 4294705152         0        12     0
dls_minor_arena                  25 4294967295         0        25     0
dld_ctl                           0 4294967295         0         0     0
px1_dvma                    5701632 1040187392         0  99910938     0
tl_minor_space                   66     262138         0    482392     0
keysock                          64 4294967295         0         1     0
spdsock                          64 4294967295         0         1     0
namefs_inodes                   192      65536         0         3     0
Hex0x600219af648_minor            0 4294967294         0         0     0
Hex0x60021a07588_minor            0 4294967294         0         0     0
ptms_minor                        6         16         0      2162     0
devfsadm_event_channel            1        101         0         1     0
devfsadm_event_channel            1          2         0         1     0
syseventconfd_event_channel         0        101         0         0     0
syseventconfd_event_channel         1          2         0         1     0
syseventd_channel                59        101         0       745     6
syseventd_channel                 1          2         0         1     0
Hex0x30003056dc8_minor            0 4294967294         0         0     0
dtrace                        58859 4294967295         0 521452795     0
dtrace_minor                      0 4294967293         0        58     0
heaptext_holesrc_16          434176    2097152         0        25     0
    heaptext_hole_16         408472     434176    434176        56     0
heaptext_holesrc_15          581632    2097152         0        19     0
    heaptext_hole_15         569948     581632    581632        50     0
heaptext_holesrc_14          450560    2097152         0        17     0
    heaptext_hole_14         424036     450560    450560        46     0
heaptext_holesrc_13          409600    2097152         0        14     0
    heaptext_hole_13         381616     409600    409600        40     0
semids                           17        128         0   1325334     0
shmids                           43        128         0      8306     0
msqids                            0        128         0         0     0
crypto                            0         16         0      9286     0
ibcm_local_sid                    0 4294967295         0         0     0
ibcm_ip_sid                       0      65535         0         0     0
logdmux_minor                     0        256         0         0     0
sppptun_minor                     0         16         0         0     0
module_text_holesrc         1351680    4194304         0        27     0
    heaptext_hole_0         1322784    1351680   1351680        59     0
heaptext_holesrc_12           24576    2097152         0         1     0
    heaptext_hole_12          21020      24576     24576         1     0
------------------------- --------- ---------- --------- --------- -----

Increase solaris zone memory capping

Check zone resources:

# /bin/prctl -n zone.max-swap `pgrep -z appzone init`
process: 10371: /sbin/init
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
zone.max-swap
        privileged      44.8GB      -   deny                                 -
        system          16.0EB    max   deny                                 -
# /bin/prctl -n zone.max-locked-memory `pgrep -z appzone init`
process: 10371: /sbin/init
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
zone.max-locked-memory
        system          16.0EB    max   deny                                 -

To use memory caps on a server or inside a zone, the rcapadm utility needs to be run with the ā€œ-Eā€ (enable memory caps) option to enable memory caps:

# rcapstat -z 1 1
rcapstat: rcapd is not active
# rcapadm
rcapadm: resource caps not configured
# rcapadm -E

Temporary increase zone.max-swap resource:

# /bin/prctl -n zone.max-swap -r -v 90G `pgrep -z appzone init`

Temporary increase zone.max-locked-memory resource:

# /bin/prctl -n zone.max-locked-memory -r -v 90G `pgrep -z appzone init`

Change the physical memory capping:

# rcapadm -z appzone -m 90G

Permanent setting:

# zonecfg -z appzone

    zonecfg:appzone> select capped-memory
    zonecfg:zone:capped-memory> set physical=90G
    zonecfg:zone:capped-memory> set swap=90G
    zonecfg:zone:capped-memory> set locked=90G
    zonecfg:zone:capped-memory> end
    zonecfg:zone:> commit

After zone reboot, the setting will make permanent.
Check zone setting:

# zonecfg -z appzone info

capped-memory:
        physical: 90G
        [swap: 90G]
        [locked: 90G]

Solaris install vmware tools

# mkdir /root

# cd /cdrom/vmwaretools

# cp vmware-solaris-tools.tar.gz /root/

# cd /root/

# ls

# gunzip -d vmware-solaris-tools.tar.gz

# ls -l

# tar -xvf vmware-solaris-tools.tar

# cd vmware-tools-distrib/

# ls

# ./vmware-install.pl