[Solaris] Cleanup all IPC resources
Run the following wrapper script with root user privileges.
for i in `ipcs -a | awk '{ print $2 }'`
do
ipcrm -m $i 2> /dev/null
ipcrm -q $i 2> /dev/null
ipcrm -s $i 2> /dev/null
done
[Java, WebLogic] Find the process id (pid) of a WebLogic managed server instance
Run the following as the user who owns the process, or with root user privileges.
/usr/java/bin/jps -v | grep <WLS_server_name> | awk '{ print $1 }'
I think this tip is applicable on all supported platforms.
eg.,
Finding the pid of a managed server, bi_server1.
# /usr/java/bin/jps -v | grep bi_server1 | awk '{ print $1 }'
18659
# pargs 18659 | grep weblogic.Name
argv[7]: -Dweblogic.Name=bi_server1
[Oracle Database] Make Oracle ignore hints
Set the following hidden parameter.
_optimizer_ignore_hints=TRUE
(in general, Oracle does not recommend playing with hidden parameters. Check with Oracle support when in doubt).
[Oracle Database] Data Pump Export in a RAC environment fails with ORA-31693, ORA-31617, ORA-19505, ORA-27037 errors
eg.,
ORA-31693: Table data object "<SCHEMA>"."<TABLE>":"P_1147" failed to load/unload and is being skipped due to error:
ORA-31617: unable to open dump file "<FILE>" for write
ORA-19505: failed to identify file "<FILE>"
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3
Workaround:
Add CLUSTER=N to the list of existing expdp options.
[Solaris, ZFS] Check the current ARC size and its breakdown
kstat -m zfs | grep size (any user) - OR -
echo ::arc | mdb -k | grep size (root user)
echo ::memstat | mdb -k (root user)
eg.,
# echo ::arc | mdb -k | grep size
size = 259391 MB
buf_size = 3218 MB
data_size = 249309 MB
other_size = 6863 MB
l2_hdr_size = 0 MB
# kstat -m zfs | grep size
buf_size 3375105344
data_size 261419672320
l2_hdr_size 0
other_size 7197048560
size 271991826224
# echo ::memstat | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 14646326 114424 4%
ZFS File Data 31948806 249600 10%
Anon 24660113 192657 7%
Exec and libs 8912 69 0%
Page cache 126052 984 0%
Free (cachelist) 24517 191 0%
Free (freelist) 263965754 2062232 79%
Total 335380480 2620160
[Fusion Middleware] Disable Fusion Middleware Diagnostic Framework (DFW) Dump Sampling
The Diagnostic Framework in FMW 11g environments detect, diagnose and resolve critical errors such as uncaught exceptions, deadlocked threads and out of memory errors. It is enabled by default.
Though DFW is supposed to diagnose and fix some of the issues transparently, due to the inevitable bugs in [all kinds of] software and misconfigurations, sometimes DFW itself may become a major issue. For instance, there is a bug that reported very high system CPU time on a SPARC server where FMW 11g was running. Per the bug description, the system CPU utilization spikes every minute exactly at 00s of a minute, CPU utilization goes down within few seconds - but the pattern persists and the spiky behavior returns within a minute. Another symptom was the sudden drop in available swap space from tens of giga bytes to a few mega bytes when the CPU spike occurs. Upon close examination, it was found out that DFW in FMW is forking tens of jstack processes to collect the thread dumps from an equal number of java processes running in that FMW environment, causing the sudden spike in CPU (each process is busy gathering thread dumps at the same time) and a steep drop in swap space (each jstack process forked a jmap process. both jstack and jmap processes consume some virtual memory just like any other process). All this happened because DFW thought it found a critical issue, and it wasn't noticed or addressed by anyone including the administrators (DFW couldn't fix this particular issue on its own) - so, it kept gathering the diagnostic data continuously. In this example, DFW did the right thing but the diagnostic data collection frequency was too short - only one minute, that diminished the value of DFW and made it a liability. In such dire situations, probably it is best to disable the dump sampling feature of Diagnostic Framework tentatively while the underlying original issue is being fixed in that application environment. It can be enabled again when the critical issue was fixed, and no longer an issue.
Steps to disable Fusion Middleware Diagnostic Framework (DFW) Dump Sampling: (courtesy: Shashidhara Varamballi)
Method (1) Using WLST:
- run wlst.sh
- connect to the AdminServer
- execute command:
enableDumpSampling (enable=0, server='<server_name>')
Method (2) Manual editing of config file:
- Edit $DOMAIN_HOME/config/fmwconfig/servers/<server_name>/dfw_config.xml
- Change the "enabled" attribute from "true" to "false".
eg.,<dumpSampling enabled="false"> - Change the "useExternalCommands" attribute from "true" to "false".
eg.,<threadDump useExternalCommands="false"/> - Save the changes
SEE ALSO:
Fusion Middleware Diagnostics weblog
[Solaris 11] Virtual-to-physical link (NIC) mapping
Check the output of /sbin/dladm show-phys (any user). By default, only those physical links that are available on the running system are displayed. Option -P shows the physical device and attributes of all physical links.
eg.,
$ /sbin/dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 1000 full ixgbe0
net5 Infiniband down 0 unknown ibp2
net1 Ethernet up 1000 full ixgbe1
net6 Infiniband down 0 unknown ibp3
net4 Ethernet up 10 full usbecm2
$ /sbin/dladm show-phys -P
LINK DEVICE MEDIA FLAGS
net8 ibp1 Infiniband r----
net0 ixgbe0 Ethernet -----
net7 ibp0 Infiniband r----
net3 ixgbe3 Ethernet r----
net5 ibp2 Infiniband -----
net1 ixgbe1 Ethernet -----
net6 ibp3 Infiniband -----
net4 usbecm2 Ethernet -----
net2 vsw0 Ethernet r----
