Oracle: Snapshots and AWR report

Starting with Oracle 10g database management system, Oracle offers a set of scripts to extract performance statistics from Automatic Workload Repository (AWR) to generate a human readable report. This report will be a starting point in finding {and fixing them sometimes, almost with no additional effort} the bottlenecks in the database system.

Oracle automatically generates snapshots of the performance data once every hour and stores the statistics in the workload repository. While diagnosing some issues, it might be necessary to take several snapshots manually. create_snapshot procedure can be used to create a database snapshot manually, as shown below:

    % sqlplus / as sysdba
    SQL> exec dbms_workload_repository.create_snapshot();

Note down the date and time of all such snapshots; and use the corresponding snap IDs while generating the AWR report for a certain interval.

How to generate an AWR report?

Simply run the awrrpti.sql as shown below. The questions from the script are pretty straight forward to answer.

    sqlplus / as sysdba
    SQL> @$ORACLE_HOME/rdbms/admin/awrrpti.sql

How to interpret an AWR report?

Here are some useful resources (note that statspack interpretation is still applicable to AWR):

• Statspack 101: Interpreting Your Statspack Report by Troy Campano
  
• Interpreting the Statspack report at AskTom.com

For more information:

• Oracle® Database Performance Tuning Guide - Automatic Workload Repository
  
• Automatic Workload Repository (AWR) in Oracle Database 10g

__________
Technorati tags:
Oracle | Performance

Read More

Solaris/SPARC: dbx style regs (registers) output

Recently one of our partners requested for some help in capturing the values of all the registers at the time of a process crash. Their idea is to get all the information that is necessary for a crash dump analysis, programmatically.

It is no surprise to hear such an idea, as majority of the customers may not be interested in installing and running a long list of commands under a debugger whenever there is a process crash. If the application can gather as much information as it can about the state of the process (context of the process in OS terminology) before it goes down, customers can simply send that information back to the software vendor along with their bug report(s).

Since it is a native application, I gave them the following C code that tries to print the values of all global (g0 - g7) and output (o0 - o7) registers of SPARC, just the way dbx does with regs command. Also it prints the values of program counter (PC) and next program counter (nPC). Note that this program is not complete - it does not print the values of local (l0 - l7), input (i0 - i7) registers. Also the values of multiply/divide register (y), processor state register (psr) and the single-precision & double-precision floating-point registers (f0 - f31) are missing. I will try to enhance this code later to include all the missing pieces. However this sample program gave them some idea on how to proceed with their actual plan.

Programming example:

The following example demonstrates the process crash with SIGSEGV in 32- and 64-bit code. Correctness of this code can be verified by comparing the output of the C program with the output of regs command in dbx environment.

% cat regs.c
#include <sys/types.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <limits.h>
#include <ucontext.h>

void signal_handler (int signo, siginfo_t *si, void *data)
{
        ucontext_t *uc;
        u_int           pc;
        u_int           sp;
        gregset_t      *regs;
        u_int           tspage;
        struct frame   *fp;
        int             i;

        uc = (ucontext_t *) data;

        if (signo = SIGSEGV)
        {
                fprintf(stdout, "Caught signal SEGV\n");

                fprintf(stderr, "\n*** SIGNAL TRAPPED: SIGNAL %d ***\n", signo);
                fprintf(stderr, "si_signo = %d\n", si->si_signo);
                fprintf(stderr, "si_code  = %d\n", si->si_code);
                fprintf(stderr, "si_errno = %d\n", si->si_errno);
                fprintf(stderr, "si_addr  = %p\n", si->si_addr);
                fprintf(stderr, "si_trapno= %p\n", si->si_trapno);
                fprintf(stderr, "si_pc    = %p\n\n", si->si_pc);

                fprintf(stderr, "stack info:\nsp: 0x%#016p  size: %#016x        flags: %d\n\n",
                        uc->uc_stack.ss_sp, uc->uc_stack.ss_size, uc->uc_stack.ss_flags);

                fprintf(stderr, "\ng0-g1        0x0000000000000000  0x%#016p", uc->uc_mcontext.gregs[REG_G1]);
                fprintf(stderr, "\ng2-g3        0x%#016p  0x%#016p", uc->uc_mcontext.gregs[REG_G2], uc->uc_mcontext.gregs[REG_G3]);
                fprintf(stderr, "\ng4-g5        0x%#016p  0x%#016p", uc->uc_mcontext.gregs[REG_G4], uc->uc_mcontext.gregs[REG_G5]);
                fprintf(stderr, "\ng6-g7        0x%#016p  0x%#016p", uc->uc_mcontext.gregs[REG_G6], uc->uc_mcontext.gregs[REG_G7]);
                fprintf(stderr, "\no0-o1        0x%#016p  0x%#016p", uc->uc_mcontext.gregs[REG_O0], uc->uc_mcontext.gregs[REG_O1]);
                fprintf(stderr, "\no2-o3        0x%#016p  0x%#016p", uc->uc_mcontext.gregs[REG_O2], uc->uc_mcontext.gregs[REG_O3]);
                fprintf(stderr, "\no4-o5        0x%#016p  0x%#016p", uc->uc_mcontext.gregs[REG_O4], uc->uc_mcontext.gregs[REG_O5]);
                fprintf(stderr, "\no6-o7        0x%#016p  0x%#016p", uc->uc_mcontext.gregs[REG_O6], uc->uc_mcontext.gregs[REG_O7]);
                fprintf(stderr, "\npc   0x%#016p",  uc->uc_mcontext.gregs[REG_PC]);
                fprintf(stderr, "\nnpc  0x%#016p",  uc->uc_mcontext.gregs[REG_nPC]);
                printf("\n");

                exit(1);
        }
        else
        {
                        fprintf(stdout, "default handler\n");
        }
}

int main (void)
{
        char *a;
        struct sigaction sa, osa;
        unsigned int b = ULONG_MAX;

        sa.sa_flags = SA_ONSTACK | SA_RESTART | SA_SIGINFO;
        sa.sa_sigaction = signal_handler;
        sigaction(SIGSEGV, &sa, &osa);

        strcat(a, "dummy");
        printf("\a = %s", *a);

        return b;
}

32-bit code:

% /opt/SUNWspro/prod/bin/cc -o regs regs.c

%./regs
Caught signal SEGV

*** SIGNAL TRAPPED: SIGNAL 11 ***
si_signo = 11
si_code  = 1
si_errno = 0
si_addr  = ffffffff
si_trapno= 0
si_pc    = 0

stack info:
sp: 0x00000000ff400000  size: 0x00000000800000  flags: 0

g0-g1   0x0000000000000000  0x00000000ff2b0ba0
g2-g3   0x0000000000000000  0x0000000000000000
g4-g5   0x0000000000000000  0x0000000000000000
g6-g7   0x0000000000000000  0x00000000ff3a2000
o0-o1   0x00000000ffffffff  0x00000000fffffaf0
o2-o3   0x00000000ffffffff  0x00000000ff3704f8
o4-o5   0x0000000000000003  0x00000000ff3704fc
o6-o7   0x00000000ffbfec88  0x00000000ff316130
pc      0x00000000ff2b0bb8
npc     0x00000000ff2b0bbc
Segmentation Fault (core dumped)

% /opt/SUNWspro/prod/bin/dbx regs core
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.5' in your .dbxrc
Reading regs
core file header read successfully
Reading ld.so.1
Reading libc.so.1
program terminated by signal SEGV (no mapping at the fault address)
0xffffffffffffffff:     

(dbx) up
0xff2c075c: _exithandle+0x003c: call     %l7

(dbx) up
0xff2af080: exit+0x0004:        call     _PROCEDURE_LINKAGE_TABLE_+0xd8 [PLT]   ! 0xff36935c

(dbx) up
0x00010e1c: signal_handler+0x0194:      call     exit [PLT]     ! 0x21114

(dbx) up
0xff33fec8: __sighndlr+0x000c:  call     %i3

(dbx) up
0xff2b0bb8: strlen+0x0018:      ldub     [%o2], %o1

(dbx) where
  [1] 0x6d6d7900(0xffbfe728, 0x1084, 0x0, 0x0, 0x4, 0x0), at 0x6d6d7900
  [2] _exithandle(0xff36db80, 0xff3a6475, 0xff36cbc0, 0x0, 0xff3a2000, 0xffbfe950), at 0xff2c075c
  [3] exit(0x1, 0x21258, 0x0, 0x21276, 0xff368284, 0x1), at 0xff2af080
  [4] signal_handler(0xb, 0xffbfec08, 0xffbfe950, 0x0, 0xff3a2000, 0xffbfe950), at 0x10e1c
  [5] __sighndlr(0xb, 0xffbfec08, 0xffbfe950, 0x10c88, 0x0, 0x1), at 0xff33fec8
  ---- called from signal handler with signal 11 (SIGSEGV) ------
=>[6] strlen(0xffffffff, 0xfffffaf0, 0xffffffff, 0xff3704f8, 0x3, 0xff3704fc), at 0xff2b0bb8
  [7] _ndoprnt(0x110d2, 0xffbff9c4, 0xffbff241, 0xffffffff, 0x0, 0x0), at 0xff316130
  [8] printf(0x110cc, 0x21258, 0x0, 0x21276, 0xff368284, 0x0), at 0xff3182dc
  [9] main(0x1, 0xffbffa8c, 0xffbffa94, 0x21000, 0xff3a00c0, 0xff3a0100), at 0x10e90

(dbx) regs
current frame:  [6]
g0-g1    0x00000000 0x00000000 0x00000000 0xff2b0ba0
g2-g3    0x00000000 0x00000000 0x00000000 0x00000000
g4-g5    0x00000000 0x00000000 0x00000000 0x00000000
g6-g7    0x00000000 0x00000000 0x00000000 0xff3a2000
o0-o1    0x00000000 0xffffffff 0x00000000 0xfffffaf0
o2-o3    0x00000000 0xffffffff 0x00000000 0xff3704f8
o4-o5    0x00000000 0x00000003 0x00000000 0xff3704fc
o6-o7    0x00000000 0xffbfec88 0x00000000 0xff316130
l0-l1    0x00000000 0x00000073 0x00000000 0x00000000
l2-l3    0x00000000 0x00000000 0x00000000 0x00001000
l4-l5    0x00000000 0x00000000 0x00000000 0x00000000
l6-l7    0x00000000 0xff36bf69 0x00000000 0xff3708f8
i0-i1    0x00000000 0x000110d2 0x00000000 0xffbff9c4
i2-i3    0x00000000 0xffbff241 0x00000000 0xffffffff
i4-i5    0x00000000 0x00000000 0x00000000 0x00000000
i6-i7    0x00000000 0xffbff918 0x00000000 0xff3182dc
y        0x00000000 0x00000000
ccr      0x00000000 0x00000000
pc       0x00000000 0xff2b0bb8:strlen+0x18      ldub     [%o2], %o1
npc      0x00000000 0xff2b0bbc:strlen+0x1c      tst      %o1

64-bit code:

% /opt/SUNWspro/prod/bin/cc -o regs64 -xarch=v9 regs.c

%./regs64
Caught signal SEGV

*** SIGNAL TRAPPED: SIGNAL 11 ***
si_signo = 11
si_code  = 1
si_errno = 0
si_addr  = ffffffffffffffff
si_trapno= 0
si_pc    = 0

stack info:
sp: 0xffffffff7f800000  size: 0x00000000800000  flags: 0

g0-g1   0x0000000000000000  0xffffffff7f239940
g2-g3   0x0000000000000000  0x0000000000000000
g4-g5   0xfffffffffffffad8  0x0000000000000202
g6-g7   0x0000000000000000  0xffffffff7f402000
o0-o1   0xffffffffffffffff  0x0000000000000053
o2-o3   0xffffffffffffffff  0x0000000000000000
o4-o5   0x0000000000000003  0x0000000000000053
o6-o7   0xffffffff7fffe171  0xffffffff7f2a2b54
pc      0xffffffff7f239958
npc     0xffffffff7f23995c
Segmentation Fault (core dumped)

% /opt/SUNWspro/prod/bin/dbx regs64 core
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.5' in your .dbxrc
Reading regs64
core file header read successfully
Reading ld.so.1
Reading libc.so.1
program terminated by signal SEGV (no mapping at the fault address)
0xffffffff7f249240: _exithandle+0x0044: call     %l6

(dbx) up
0xffffffff7f237d10: exit+0x0004:        call     _PROCEDURE_LINKAGE_TABLE_+0x200 [PLT]  ! 0xffffffff7f3e6700

(dbx) up
0x0000000100000b3c: signal_handler+0x01ac:      call     exit [PLT]     ! 0x100100fa0

(dbx) up
0xffffffff7f2cd0bc: __sighndlr+0x000c:  call     %i3

(dbx) up
0xffffffff7f239958: strlen+0x0018:      ldub     [%o2], %o1

(dbx) where
  [1] _exithandle(0xffffffff7f3eff00, 0xffffffff7fffe170, 0xffffffff7f3eef40, 0xd, 0x0, 0xffffffff7fffe590), at 0xffffffff7f249240

  [2] exit(0x1, 0x2000, 0x0, 0x1001012e4, 0xffffffff7f3e4000, 0x1), at 0xffffffff7f237d10
  [3] signal_handler(0xb, 0xffffffff7fffe870, 0xffffffff7fffe590, 0xd, 0x0, 0xffffffff7fffe590), at 0x100000b3c
  [4] __sighndlr(0xb, 0xffffffff7fffe870, 0xffffffff7fffe590, 0x100000990, 0x0, 0xa), at 0xffffffff7f2cd0bc
  ---- called from signal handler with signal 11 (SIGSEGV) ------
=>[5] strlen(0xffffffffffffffff, 0x53, 0xffffffffffffffff, 0x0, 0x3, 0x53), at 0xffffffff7f239958
  [6] _ndoprnt(0x100000e46, 0xffffffff7ffff878, 0xffffffff7f2a0de4, 0xffffffff7fffefc9, 0xffffffffffffffff, 0x100000e45), at 0xffffffff7f2a2b54

  [7] printf(0x100000e40, 0x2000, 0x0, 0x1001012e4, 0xffffffff7f3e4000, 0x0), at 0xffffffff7f2a4de0
  [8] main(0x1, 0xffffffff7ffff9b8, 0xffffffff7ffff9c8, 0x0, 0xffffffffffffffff, 0xffffffff7f400100), at 0x100000bc4

(dbx) regs
current frame:  [5]
g0-g1    0x0000000000000000 0xffffffff7f239940
g2-g3    0x0000000000000000 0x0000000000000000
g4-g5    0xfffffffffffffad8 0x0000000000000202
g6-g7    0x0000000000000000 0xffffffff7f402000
o0-o1    0xffffffffffffffff 0x0000000000000053
o2-o3    0xffffffffffffffff 0x0000000000000000
o4-o5    0x0000000000000003 0x0000000000000053
o6-o7    0xffffffff7fffe171 0xffffffff7f2a2b54
l0-l1    0x0000000000000000 0x0000000000000073
l2-l3    0x0000000000000000 0x0000000000000000
l4-l5    0x0000000000001000 0x0000000000000000
l6-l7    0x0000000000000000 0xffffffff7f3ede21
i0-i1    0x0000000100000e46 0xffffffff7ffff878
i2-i3    0xffffffff7f2a0de4 0xffffffff7fffefc9
i4-i5    0xffffffffffffffff 0x0000000100000e45
i6-i7    0xffffffff7fffef41 0xffffffff7f2a4de0
y        0x0000000000000000
ccr      0x0000000000000000
pc       0xffffffff7f239958:strlen+0x18 ldub     [%o2], %o1
npc      0xffffffff7f23995c:strlen+0x1c tst      %o1

Note:
You might have noticed that the value of global register, g0, is hard coded to 0. The primary reason being the first global register (g0 on SPARC) is the assembly language equivalent of /dev/null. No matter what you try to put into this register, the value always remain zero.

To DO://

1. Enhance the above code to include input, local, floating-point registers. Also print the actual instructions of program counter (PC) and next program counter (nPC)
   
2. Post similar code for x86/x64 architecture

__________
Technorati tags:
Solaris | SPARC | dbx

Read More

My favorite fictional characters

Although majority of my acquaintances make fun of me watching TV-Y/TV-Y7 television rating cartoon shows, I still prefer watching simple, short, clean and colorful cartoon shows over long, overly-hyped yet insipid sagas on celluloid. Thought I'd introduce some of my favorite fictitious characters from cartoon shows and comic strips - so, here they are: (courtesy: wikipedia.org)

Garfield
Garfield comic strip


Tom & Jerry
Tom and Jerry


Stewie Griffin (Stewie)
Family Guy


Blooregard Q. Kazoo (Bloo)
Foster's Home for Imaginary Friends


Bender Bending Rodríguez (Bender)
Futurama


Bugs Bunny
Looney Tunes


Pointy haired boss (PHB)
Dilbert comic strip


Norville Shaggy Rogers & Scoobert Scooby-Doo (Shaggy & Scooby)
Scooby-Doo, Where Are You?


Mandy
The Grim Adventures of Billy and Mandy
(definitely not one of my favorite shows)

Read More

Rest in Peace

 
                          Seshamma Mandalika (1922-2006)

We will miss you, Grandma. May your soul rest in peace.

Read More

Solaris 10/Oracle: Fixing ORA-27102: out of memory Error

Symptom:

As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an ORA-27102: out of memory error message. The system had enough free memory to serve the needs of Oracle.

SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument

Diagnosis

$ oerr ORA 27102
27102, 00000, "out of memory"
// *Cause: Out of memory
// *Action: Consult the trace file for details

Not so helpful. Let's look the alert log for some clues.

% tail -2 alert.log
WARNING: EINVAL creating segment of size 0x000000028a006000
fix shm parameters in /etc/system or equivalent

Oracle is trying to create a 10G shared memory segment (depends on SGA/PGA sizes), but operating system (Solaris in this example) responded with an invalid argument (EINVAL) error message. There is a little hint about setting shm parameters in /etc/system.

Prior to Solaris 10, shmsys:shminfo_shmmax parameter has to be set in /etc/system with maximum memory segment value that can be created. 8M is the default value on Solaris 9 and prior versions; where as 1/4th of the physical memory is the default on Solaris 10 and later. On a Solaris 10 (or later) system, it can be verified as shown below:

% prtconf | grep Mem
Memory size: 32760 Megabytes

% id -p
uid=59008(oracle) gid=10001(dba) projid=3(default)

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      7.84GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Now it is clear that the system is using the default value of 8G in this scenario, where as the application (Oracle) is trying to create a memory segment (10G) larger than 8G. Hence the failure.

So, the solution is to configure the system with a value large enough for the shared segment being created, so Oracle succeeds in starting up the database instance.

On Solaris 9 and prior releases, it can be done by adding the following line to /etc/system, followed by a reboot for the system to pick up the new value.

set shminfo_shmmax = 0x000000028a006000

However shminfo_shmmax parameter was obsoleted with the release of Solaris 10; and Sun doesn't recommend setting this parameter in /etc/system even though it works as expected.

On Solaris 10 and later, this value can be changed dynamically on a per project basis with the help of resource control facilities . This is how we do it on Solaris 10 and later:

% prctl -n project.max-shm-memory -r -v 10G -i project 3

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      10.0GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Note that changes made with the prctl command on a running system are temporary, and will be lost when the system is rebooted. To make the changes permanent, create a project with projadd command and associate it with the user account as shown below:

% projadd -p 102  -c 'eBS benchmark' -U oracle -G dba  -K 'project.max-shm-memory=(privileged,10G,deny)' OASB
% usermod -K project=OASB oracle

Finally make sure the project is created with projects -l or cat /etc/project commands.

% projects -l
...
...
OASB
        projid : 102
        comment: "eBS benchmark"
        users  : oracle
        groups : dba
        attribs: project.max-shm-memory=(privileged,10737418240,deny)

% cat /etc/project
...
...
OASB:102:eBS benchmark:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)

With these changes, Oracle would start the database up normally.

SQL> startup
ORACLE instance started.

Total System Global Area 1.0905E+10 bytes
Fixed Size                  1316080 bytes
Variable Size            4429966096 bytes
Database Buffers         6442450944 bytes
Redo Buffers               31457280 bytes
Database mounted.
Database opened.

Related information:

1. What's New in Solaris System Tuning in the Solaris 10 Release?
2. Resource Controls (overview)
3. System Setup Recommendations for Solaris 8 and Solaris 9
4. Man page of prctl(1)
5. Man page of projadd

---

Addendum : Oracle RAC settings

Anonymous Bob suggested the following settings for Oracle RAC in the form of a comment for the benefit of others who run into similar issue(s) when running Oracle RAC. I'm pasting the comment as is (Disclaimer: I have not verified these settings):

Thanks for a great explanation, I would like to add one comment that will help those with an Oracle RAC installation. Modifying the default project covers oracle processes great and is all that is needed for a single instance DB. In RAC however, the CRS process starts the DB and it is a root owned process and root does not use the default project. To fix ORA-27102 issue for RAC I added the following lines to an init script that runs before the init.crs script fires.

# Recommended Oracle RAC system params
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536

# For root processes like crsd
prctl -n project.max-shm-memory -r -v 8G -i project system
prctl -n project.max-shm-ids -r -v 512 -i project system

# For oracle processes like sqlplus
prctl -n project.max-shm-memory -r -v 8G -i project default
prctl -n project.max-shm-ids -r -v 512 -i project default

So simple yet it took me a week working with Oracle and SUN to come up with that answer...Hope that helps someone out.

Bob
# posted by Blogger Bob : 6:48 AM, April 25, 2008

_____________
Technorati tags:
Solaris | Open Solaris | Oracle | troubleshooting

Read More

Solaris: Workaround to stdio's 255 open file descriptors limitation

A quick web search with key words Solaris stdio open file descriptors results in numerous references to stdio's limitation of 255 open file descriptors on Solaris.

#1085341: 32-bit stdio routines should support file descriptors >255, a 14 year old RFE explains the problem and the bug report links to handful of other bugs which are some how related to stdio's open file descriptors limitation.

Now the good news: the wait is finally over. Sun Microsystems finally made a fix/workaround available to the community in the form of a new library called extendedFILE. If you are running Solaris Express (SX) 06/06 or later builds, your system already has the workaround. You just need to enable it to get around the 255 open file descriptors problem with stdio's API. This workaround will be part of the Update 3 release of Solaris 10, which is due in October 2006. There are some plans to backport this workaround to Solaris 9 and 10. However there is no clear timeline for completion of this backport, at the moment.

The workaround does not require any source code changes or re-compilation of the objects. You just need to increase the file descriptor limit using limit or ulimit commands; and pre-load /usr/lib/extendedFILE.so.1 before running the application.

However applications fiddling with _file field of FILE structure may not work. This is because when extendedFILE library is pre-loaded, descriptors > 255 will be stored in an auxiliary location and a fake descriptor will be stored in the FILE structure's _file field. In fact, accessing _file field was long discouraged; and to discourage non-standard practices even further _file has been renamed to _magic starting with SX 06/06. So, applications which access _file directly rather than with fileno() function, may encounter compilation errors starting with S10 U3. This step is necessary to ensure that the source code is in a clean state, so the resulting object code is not vulnerable to data corruption during run-time.

The following example shows the failure and the steps to workaround the issue. Note that with the extendedfile library pre-loaded, the process can open upto 65532 files excluding stdin, out and err.

* Test case (simple C program tries to open 65536 files):

 % cat files.c
 #include <stdio.h>
 #define NoOfFILES 65536

 int main()
 {
         char filename[10];
         FILE *fds[NoOfFILES];
         int i;
 
         for (i = 0; i < NoOfFILES; ++i)
         {
                 sprintf (filename, "%d.log", i);
                 fds[i] = fopen(filename, "w");
 
                 if (fds[i] == NULL)
                 {
                         printf("\n** Number of open files = %d. fopen() failed with error:  ", i);
                         perror("");
                         exit(1);
                 }
                 else
                 {
                         fprintf (fds[i], "some string");
                 }
 
         }
         return (0);
 }

 % cc -o files files.c

* Re-producing the failure:

 % limit
 cputime         unlimited
 filesize        unlimited
 datasize        unlimited
 stacksize       8192 kbytes
 coredumpsize    unlimited
 descriptors     256
 memorysize      unlimited

 % uname -a
 SunOS sunfire4 5.11 snv_42 sun4u sparc SUNW,Sun-Fire-280R

 %./files

 ** Number of open files = 253. fopen() failed with error: Too many open files

* Showcasing the Solaris workaround:

 % limit descriptors 5000

 % limit
 cputime         unlimited
 filesize        unlimited
 datasize        unlimited
 stacksize       8192 kbytes
 coredumpsize    unlimited
 descriptors     5000
 memorysize      unlimited

 % setenv LD_PRELOAD_32 /usr/lib/extendedFILE.so.1

 % ./files

 ** Number of open files = 4996. fopen() failed with error: Too many open files

 % limit descriptors 65536

 % limit
 cputime         unlimited
 filesize        unlimited
 datasize        unlimited
 stacksize       8192 kbytes
 coredumpsize    unlimited
 descriptors     65536
 memorysize      unlimited

 %./files

 ** Number of open files = 65532. fopen() failed with error: Too many open files

(Note that descriptor 0, 1 and 2 will be used by stdin, stdout and stderr)

 % ls -l 65531.log
 -rw-rw-r--   1 gmandali ccuser        11 Aug  9 12:35 65531.log

For further information about the extendedFILE library and for the extensions to fopen, fdopen, popen, .. please have a look at the new and updated man pages:

extendedFILE
enable_extended_FILE_stdio
fopen
fdopen

__________
Technorati tags:
Sun | Solaris | OpenSolaris

Read More

Java Troubleshooting & Diagnostic Guide

This nice 100+ page document is freely downloadable from Sun web site at: http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf.

The document has copious information about tools like jinfo, jmap, jstack, jconsole, jps, jstat, Heap Analysis Tool (HAT), Heap Profiler (HPROF), jdb, dbx; and discusses about fatal error handling, deadlock detection, memory leak diagnosis, diagnosing crashes in native code, compiled code, VM and HotSpot compiler threads etc., in detail. Must have document for majority of Java developers.

__________
Technorati tags:
Sun | java | troubleshooting

Read More