Blast from the Past : The Weekend Playlist #3

The 80s contd.,

The 80s witnessed the rise of fine talent - so, it is only fitting to dedicate another complete playlist for the 80s. Here it is. Enjoy. Earlier playlists can be accessed from the following locations:

    Blast from the Past : The Weekend Playlist #2 (80s)
    Blast from the Past : The Weekend Playlist #1 (50s, 60s and 70s)

Audio-Visual material courtesy: YouTube

1. Aerosmith - Dude (Looks Like a Lady) (1987) Featured in Robin Williams' Mrs. Doubtfire.	2. Kool & the Gang - Celebration (1980) San Francisco Bay Area Star 101.3 audience must be hating this one.
3. Wham - Wake Me Up Before You Go-Go (1984) That was George Michael before his solo career.	4. Toto - Rosanna (1982) Grammy winner
5. Club Nouveau - Lean on Me (1987) Cover version. Original by Bill Withers in 1972.	6. Tom Petty - Free Fallin' (1989) Enjoy
7. Kenny Loggins - Footloose (1984) Of course, it was featured in Kevin Bacon's Footloose	8. Simple Minds - Don't You (Forget About Me) (1985) This is my brother, Vishu's, pick.
9. Fun Boy Three - Our Lips Are Sealed (1983) Another cover. Original by The Go-Go's just two years earlier.	10. Men without Hats - The Safety Dance (1983) S s s s A a a a F f f f E e e e T t t t Y y y y

Read More

Measuring Network Bandwidth Using iperf

iperf is a simple, open source tool to measure the network bandwidth. It can test TCP or UDP throughput. Tools like iperf are useful to check the performance of a network real quick, by comparing the achieved bandwidth with the expectation. The example in this blog post is from a Solaris system, but the instructions and testing methodology are applicable on all supported platforms including Linux.

Download the source code from iperf's home page, and build the iperf binary. Those running Solaris 10 or later, can download the pre-built binary (file size: 245K) from this location to give it a quick try (right click and "Save Link As .." or similar option).

Testing methodology:

iperf's network performance measurements are based on the client-server communication model - hence requires establishing both a server and a client. The same iperf binary can be used to run the process in server and client modes.

1. Start iperf in server mode

   iperf -s -i <interval>

   Option -s or --server starts the process in server mode. -i or --interval is the sampling interval in seconds.

2. Start iperf in client mode, and test the network connection between client and the server with arbitrary data transfers.

   
   iperf -n <bytes> -i <interval> -c <ServerIP>
   

   Option -c or --client starts the process in client mode. Option -n or --bytes specify the number of bytes to transmit in bytes, KB (use suffix K) or MB (use suffix M). -i or --interval is the sampling interval in seconds. The last option is the IP address or the hostname of the server to connect to. By default, client connects to the server using TCP. -u or --udp switches to UDP.

3. Check the network link speed on server and client, and compare the throughput achieved.

Check the man page out for the full list of options supported by iperf in client and server modes.

Here is a simple demonstration.

On server node:

iperfserv% dladm show-phys net0
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      igb0

iperfserv% ifconfig net0 | grep inet
        inet 10.129.193.63 netmask ffffff00 broadcast 10.129.193.255

iperfserv% ./iperf -v
iperf version 3.0-BETA5 (28 March 2013)SunOS iperfserv 5.11 11.1 sun4v sparc sun4v


iperfserv% ./iperf -s -i 1
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

On client node:

client% dladm show-phys net0
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      igb0

client% ifconfig net0 | grep inet
        inet 10.129.193.151 netmask ffffff00 broadcast 10.129.193.255

client% ./iperf  -n 1024M  -i 1 -c 10.129.193.63
Connecting to host 10.129.193.63, port 5201
[  4] local 10.129.193.151 port 63507 connected to 10.129.193.63 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   105 MBytes   875 Mbits/sec
[  4]   1.01-2.02   sec   112 MBytes   934 Mbits/sec
[  4]   2.02-3.00   sec   110 MBytes   934 Mbits/sec
   [...]
[  4]   8.02-9.01   sec   110 MBytes   933 Mbits/sec
[  4]   9.01-9.27   sec  30.0 MBytes   934 Mbits/sec
[ ID] Interval           Transfer     Bandwidth
      Sent
[  4]   0.00-9.27   sec  1.00 GBytes   927 Mbits/sec
      Received
[  4]   0.00-9.27   sec  1.00 GBytes   927 Mbits/sec

iperf Done.

At the same time, somewhat similar messages are written to stdout on the server node.

iperfserv% ./iperf  -s -i 1
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.129.193.151, port 33457
[  5] local 10.129.193.63 port 5201 connected to 10.129.193.151 port 63507
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   104 MBytes   874 Mbits/sec
[  5]   1.00-2.00   sec   111 MBytes   934 Mbits/sec
[  5]   2.00-3.00   sec   111 MBytes   934 Mbits/sec
   [...]
[ ID] Interval           Transfer     Bandwidth
      Sent
[  5]   0.00-9.28   sec  1.00 GBytes   927 Mbits/sec
      Received
[  5]   0.00-9.28   sec  1.00 GBytes   927 Mbits/sec
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

The link speed is specified in Mbps (megabit per second). In the above example, the network link is operating at 1000 Mbps speed, and the achieved bandwidth is 927 Mbps, which is 92.7% of the advertised bandwidth.

Notes:

• It is not necessary to execute iperf in client and server modes as root or privileged user
• In server mode, iperf uses port 5201 by default. It can be changed to something else using -p or --port option
• Restart iperf server after each client test to get reliable, consistent results
• Using iperf is just one of many ways to measure the network bandwidth. There are other tools such as uperf, ttcp, netperf, bwping, udpmon, tcpmon, .. just to name a few. Research and pick the one that best suits your requirement.

Read More

Blast from the Past : The Weekend Playlist #2

The 80s

The quality of music steadily improved over the decades in mid-1900s, and eventually peaked in the 80s. Some may disagree, but in my opinion, the 80s were easily one of the best decades for music in the United States. The decade witnessed the emergence of many successful artists who delivered solid hits, that are still relevant and part of many pop culture references today. The launch of MTV in 1981 upped the ante to produce interesting videos in an effort to increase the global outreach.

In this iteration, let's focus on the decade of eighties. The following playlist has some random songs from the 80s in no particular order. The previous playlist can be accessed from this location:

    Blast from the Past : The Weekend Playlist #1 (50s, 60s and 70s)

Audio-Visual material courtesy: YouTube

1. INXS - Need You Tonight (1987) Love the guitar riff.	2. Queen - I Want to Break Free (1984) Featured in some of the Coke commercials in mid-2000s.
3. John Farnham - You're the Voice (1986) Wasn't so popular in US, I believe. Great song nevertheless.	4. Phil Collins - Another Day in Paradise (1989) Grammy winner.
5. Journey - Don't Stop Believin' (1981) Very popular in pop culture.	6. Tears for Fears - Shout (1984) #1 on the Billboard Hot 100 chart for 3 weeks.
7. Fine Young Cannibals - She Drives Me Crazy (1989) Enjoy!	8. Mr Mister - Kyrie (1985) Has 80s feel all over it.
9. Dream Academy - Life In A Northern Town (1985) Expect to be hit with something unexpected @00:00:52s. Featured in one of the episodes of King of the Hill.	10. Robert Palmer - Addicted To Love (1986) Who wouldn't like models in uniform fiddling with guitars and musical instruments.

Read More

Things to Consider when Planning the Redo logs for Oracle Database

Very basic and generic discussion from the performance point of view. Customers still have to do their due diligence in understanding redo logs, and how they work in Oracle database, before finalizing redo log configuration for their deployments.

• size them properly
  • log writer writes to a single redo log file until either it is full or a manual log switch is requested
        Oracle supports multiplexed redo logs for availability, but this behavior of writing to a file until it is full or a log switch happens, still hold
  • if the transactions generate a lot of redo before a database commit, consider large sizes in tens of gigabytes for redo logs
  • if not sized properly, it leads to unnecessary log switches, which in turn increase checkpoint activity resulting in unnecessary slow down of the database operations
        two redo logs each with at least 5G in size might be a good start. observe the log switches, checkpoints and increase (or decrease, though there is no performance benefit) the file size accordingly
  
  
• do not mix redo logs with the rest of the database or anything else
  • in a normal functioning database, most of the time, log writer simply writes redo entries sequentially to redo logs
  • any slow down in writing the redo data to logs hurt the performance of the database
  • best not to share the disks/volumes on which redo logs are hosted, with anything else
        set of disks, volumes exclusive to redo logs, that is
  
  
• ensure that the underlying disks or I/O medium used to store the redo logs are fast, optimally configured and can sustain the amount of I/O bandwidth needed to write the redo entries to the redo logs
      if those requirements are not met, it could lead to 'log file sync' waits, which will slow down the database transactions
  
  
• redo logs on non-volatile flash storage may have performance benefits over the traditional hard disk drives
  • check this blog post out, Redo logs on F40 PCIe Cards, for related discussion (keywords: 4K block size for redo logs, block alignment)

Read More

Blast from the Past : The Weekend Playlist #1

Music and Movies - the two powerful forms of entertainment, are very subjective. In general, there is no good or bad music, and there are no good or bad movies. Their success depend on a combination of various factors such as the mood we are in, our biases, memories, brand (eg., Pixar in movies; The Beatles in music, although I never came across any palatable original work from this boy-band group), derivation from other successful work -- remixes (Fatboy Slim), mashups (DJs), mockery (spoof movies, comedy skits), influence/herd mentality (Bieber, Miley on Billboard top singles), celebrity endorsements, timing of their release (cliched superhero summer tent-poles), their competition (someone will emerge as a winner even when the actual content and its presentation is equally weak), marketing push (Kelly Clarkson, The Avengers movie), viral status (Gangnam Style), cult following (Office Space), audience desperation, events that evoke feelings such as happiness, smugness, sympathy (Heath Ledger's death helping Nolan's Batman series) and pity (MJ's "This is it"), luck, the artists, crew and of course, the actual content. In short, if something was "well liked", it does not necessarily mean that it was really "well liked" -- multiple factors surrounding that piece of work must have properly aligned and helped in one way or the other, to make it more successful.

Now that the obvious disclaimer is out of the way, I think I can openly list out some of the stuff that I think is worth listening to. If you don't like or really hate something that you see here, too bad I guess - just deal with it. :-)

To kick start this series, I chose a handful of oldies from 50s, 60s and 70s. (yes, I intend to publish a few more playlists down the line, unfortunately). Here it goes in no particular order. I really like those shorter 2+ minute durations. Audio-Visual material courtesy: YouTube.

1. BJ Thomas - Raindrops Keep Fallin' on My Head (1969) Featured in the movie, Butch Cassidy and the Sundance Kid.	2. Sonny & Cher - I Got You Babe (1965) Prominently featured in Groundhog Day.
3. Stealers Wheel - Stuck in the Middle with You (1972) Featured in the movie Reservoir Dogs.	4. Donovan - Jennifer Juniper (1968) Featured in one of the episodes of The Simpsons.
5. Barry McGuire - Eve of Destruction (1965)	6. Billy Joe Royal - Down in the Boondocks (1965)
7. Link Wray - Rumble (1958) Sounds unreal - way ahead of its time. Featured in Tarantino's Pulp Fiction.	8. America - A Horse with No Name (1972) Earned bad song reputation for some of its lines.
9. Ella Fitzgerald & Louis Armstrong - Let's Call the Whole Thing Off (1959) Featured in one of the episodes of The Simpsons animated series.	10. The Tokens - The Lion Sleeps Tonight (1961) I guess this one too got some bad rap.

Before concluding: I do not know much about composing music, or playing musical instruments of any kind. Still I attempted composing a few instrumental tracks with the help of software. It's been a fun exercise so far. Listen to those amateur tracks @ icompositions | giri04 webpage.

Read More

[Script] Breakdown of Oracle SGA into Solaris Locality Groups

Goal: for a given process, find out how the SGA was allocated in different locality groups on a system running Solaris operating system.

Download the shell script, sga_in_lgrp.sh. The script accepts any Oracle database process id as input, and prints out the memory allocated in each locality group.

Usage: ./sga_in_lgrp.sh <pid>

eg.,

# prstat -p 12820

   PID USERNAME  SIZE   RSS STATE   PRI NICE      TIME  CPU PROCESS/NLWP
 12820 oracle     32G   32G sleep    60  -20   0:00:16 0.0% oracle/2

# ./sga_in_lgrp.sh 12820

Number of Locality Groups (lgrp): 4
------------------------------------

lgroup 1 :   8.56 GB
lgroup 2 :   6.56 GB
lgroup 3 :   6.81 GB
lgroup 4 :  10.07 GB

Total allocated memory:  32.00 GB

For those who wants to have a quick look at the source code, here it is.

# cat sga_in_lgrp.sh

#!/bin/bash

# check the argument count
if [ $# -lt 1 ]
then
        echo "usage: ./sga_in_lgrp.sh <oracle pid>"
        exit 1
fi

# find the number of locality groups
lgrp_count=$(kstat -l lgrp | tail -1 | awk -F':' '{ print $2 }')
echo "\nNumber of Locality Groups (lgrp): $lgrp_count"
echo "------------------------------------\n"

# save the ism output using pmap
pmap -sL $1 | grep ism | sort -k5 > /tmp/tmp_pmap_$1

# calculate the total amount of memory allocated in each lgroup
for i in `seq 1 $lgrp_count`
do
        echo -n "lgroup $i : "
        grep "$i   \[" /tmp/tmp_pmap_$1 | awk '{ print $2 }' | sed 's/K//g' | awk '{ sum+=$1} END {printf ("%6.2f GB\n", sum/(1024*1024))}'
done

echo
echo -n "Total allocated memory: "
awk '{ print $2 }' /tmp/tmp_pmap_$1 | sed 's/K//g' | awk '{ sum+=$1} END {printf ("%6.2f GB\n\n", sum/(1024*1024))}'

rm /tmp/tmp_pmap_$1

Like many things in life, there will always be a better or simpler way to achieve this. If you find one, do not fret over this approach. Please share, if possible.

Read More

Miscellaneous Tips: Solaris, Oracle Database, Java, FMW

[Solaris] Cleanup all IPC resources

Run the following wrapper script with root user privileges.

for i in `ipcs -a | awk '{ print $2 }'`
do
 ipcrm -m $i 2> /dev/null
 ipcrm -q $i 2> /dev/null
 ipcrm -s $i 2> /dev/null
done

---

[Java, WebLogic] Find the process id (pid) of a WebLogic managed server instance

Run the following as the user who owns the process, or with root user privileges.

/usr/java/bin/jps -v | grep <WLS_server_name> | awk '{ print $1 }'

I think this tip is applicable on all supported platforms.

eg.,
Finding the pid of a managed server, bi_server1.

# /usr/java/bin/jps -v | grep bi_server1 | awk '{ print $1 }'
18659

# pargs 18659 | grep weblogic.Name
argv[7]: -Dweblogic.Name=bi_server1

---

[Oracle Database] Make Oracle ignore hints

Set the following hidden parameter.

_optimizer_ignore_hints=TRUE

(in general, Oracle does not recommend playing with hidden parameters. Check with Oracle support when in doubt).

---

[Oracle Database] Data Pump Export in a RAC environment fails with ORA-31693, ORA-31617, ORA-19505, ORA-27037 errors

eg.,

ORA-31693: Table data object "<SCHEMA>"."<TABLE>":"P_1147" failed to load/unload and is being skipped due to error:
ORA-31617: unable to open dump file "<FILE>" for write
ORA-19505: failed to identify file "<FILE>"
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3

Workaround:

Add CLUSTER=N to the list of existing expdp options.

---

[Solaris, ZFS] Check the current ARC size and its breakdown

kstat -m zfs | grep size   (any user)  - OR -
echo ::arc | mdb -k | grep size  (root user)

echo ::memstat | mdb -k  (root user)


eg.,

# echo ::arc | mdb -k | grep size
size                      =       259391 MB
buf_size                  =         3218 MB
data_size                 =       249309 MB
other_size                =         6863 MB
l2_hdr_size               =            0 MB

# kstat -m zfs | grep size
        buf_size                        3375105344
        data_size                       261419672320
        l2_hdr_size                     0
        other_size                      7197048560
        size                            271991826224

# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   14646326            114424    4%
ZFS File Data            31948806            249600   10%
Anon                     24660113            192657    7%
Exec and libs                8912                69    0%
Page cache                 126052               984    0%
Free (cachelist)            24517               191    0%
Free (freelist)         263965754           2062232   79%
Total                   335380480           2620160

---

[Fusion Middleware] Disable Fusion Middleware Diagnostic Framework (DFW) Dump Sampling

The Diagnostic Framework in FMW 11g environments detect, diagnose and resolve critical errors such as uncaught exceptions, deadlocked threads and out of memory errors. It is enabled by default.

Though DFW is supposed to diagnose and fix some of the issues transparently, due to the inevitable bugs in [all kinds of] software and misconfigurations, sometimes DFW itself may become a major issue. For instance, there is a bug that reported very high system CPU time on a SPARC server where FMW 11g was running. Per the bug description, the system CPU utilization spikes every minute exactly at 00s of a minute, CPU utilization goes down within few seconds - but the pattern persists and the spiky behavior returns within a minute. Another symptom was the sudden drop in available swap space from tens of giga bytes to a few mega bytes when the CPU spike occurs. Upon close examination, it was found out that DFW in FMW is forking tens of jstack processes to collect the thread dumps from an equal number of java processes running in that FMW environment, causing the sudden spike in CPU (each process is busy gathering thread dumps at the same time) and a steep drop in swap space (each jstack process forked a jmap process. both jstack and jmap processes consume some virtual memory just like any other process). All this happened because DFW thought it found a critical issue, and it wasn't noticed or addressed by anyone including the administrators (DFW couldn't fix this particular issue on its own) - so, it kept gathering the diagnostic data continuously. In this example, DFW did the right thing but the diagnostic data collection frequency was too short - only one minute, that diminished the value of DFW and made it a liability. In such dire situations, probably it is best to disable the dump sampling feature of Diagnostic Framework tentatively while the underlying original issue is being fixed in that application environment. It can be enabled again when the critical issue was fixed, and no longer an issue.

Steps to disable Fusion Middleware Diagnostic Framework (DFW) Dump Sampling: (courtesy: Shashidhara Varamballi)

Method (1) Using WLST:

1. run wlst.sh
2. connect to the AdminServer
3. execute command: enableDumpSampling (enable=0, server='<server_name>')

Method (2) Manual editing of config file:

1. Edit $DOMAIN_HOME/config/fmwconfig/servers/<server_name>/dfw_config.xml
2. Change the "enabled" attribute from "true" to "false".
   eg.,
   <dumpSampling enabled="false">
3. Change the "useExternalCommands" attribute from "true" to "false".
   eg.,
   <threadDump useExternalCommands="false"/>
4. Save the changes

--

SEE ALSO:

Fusion Middleware Diagnostics weblog

---

[Solaris 11] Virtual-to-physical link (NIC) mapping

Check the output of /sbin/dladm show-phys (any user). By default, only those physical links that are available on the running system are displayed. Option -P shows the physical device and attributes of all physical links.

eg.,

$ /sbin/dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      ixgbe0
net5              Infiniband           down       0      unknown   ibp2
net1              Ethernet             up         1000   full      ixgbe1
net6              Infiniband           down       0      unknown   ibp3
net4              Ethernet             up         10     full      usbecm2

$ /sbin/dladm show-phys -P
LINK              DEVICE       MEDIA                FLAGS
net8              ibp1         Infiniband           r----
net0              ixgbe0       Ethernet             -----
net7              ibp0         Infiniband           r----
net3              ixgbe3       Ethernet             r----
net5              ibp2         Infiniband           -----
net1              ixgbe1       Ethernet             -----
net6              ibp3         Infiniband           -----
net4              usbecm2      Ethernet             -----
net2              vsw0         Ethernet             r----

Read More