Solaris: Workaround to stdio's 255 open file descriptors limitation

A quick web search with key words Solaris stdio open file descriptors results in numerous references to stdio's limitation of 255 open file descriptors on Solaris.

#1085341: 32-bit stdio routines should support file descriptors >255, a 14 year old RFE explains the problem and the bug report links to handful of other bugs which are some how related to stdio's open file descriptors limitation.

Now the good news: the wait is finally over. Sun Microsystems finally made a fix/workaround available to the community in the form of a new library called extendedFILE. If you are running Solaris Express (SX) 06/06 or later builds, your system already has the workaround. You just need to enable it to get around the 255 open file descriptors problem with stdio's API. This workaround will be part of the Update 3 release of Solaris 10, which is due in October 2006. There are some plans to backport this workaround to Solaris 9 and 10. However there is no clear timeline for completion of this backport, at the moment.

The workaround does not require any source code changes or re-compilation of the objects. You just need to increase the file descriptor limit using limit or ulimit commands; and pre-load /usr/lib/extendedFILE.so.1 before running the application.

However applications fiddling with _file field of FILE structure may not work. This is because when extendedFILE library is pre-loaded, descriptors > 255 will be stored in an auxiliary location and a fake descriptor will be stored in the FILE structure's _file field. In fact, accessing _file field was long discouraged; and to discourage non-standard practices even further _file has been renamed to _magic starting with SX 06/06. So, applications which access _file directly rather than with fileno() function, may encounter compilation errors starting with S10 U3. This step is necessary to ensure that the source code is in a clean state, so the resulting object code is not vulnerable to data corruption during run-time.

The following example shows the failure and the steps to workaround the issue. Note that with the extendedfile library pre-loaded, the process can open upto 65532 files excluding stdin, out and err.

* Test case (simple C program tries to open 65536 files):

 % cat files.c
 #include <stdio.h>
 #define NoOfFILES 65536

 int main()
 {
         char filename[10];
         FILE *fds[NoOfFILES];
         int i;
 
         for (i = 0; i < NoOfFILES; ++i)
         {
                 sprintf (filename, "%d.log", i);
                 fds[i] = fopen(filename, "w");
 
                 if (fds[i] == NULL)
                 {
                         printf("\n** Number of open files = %d. fopen() failed with error:  ", i);
                         perror("");
                         exit(1);
                 }
                 else
                 {
                         fprintf (fds[i], "some string");
                 }
 
         }
         return (0);
 }

 % cc -o files files.c

* Re-producing the failure:

 % limit
 cputime         unlimited
 filesize        unlimited
 datasize        unlimited
 stacksize       8192 kbytes
 coredumpsize    unlimited
 descriptors     256
 memorysize      unlimited

 % uname -a
 SunOS sunfire4 5.11 snv_42 sun4u sparc SUNW,Sun-Fire-280R

 %./files

 ** Number of open files = 253. fopen() failed with error: Too many open files

* Showcasing the Solaris workaround:

 % limit descriptors 5000

 % limit
 cputime         unlimited
 filesize        unlimited
 datasize        unlimited
 stacksize       8192 kbytes
 coredumpsize    unlimited
 descriptors     5000
 memorysize      unlimited

 % setenv LD_PRELOAD_32 /usr/lib/extendedFILE.so.1

 % ./files

 ** Number of open files = 4996. fopen() failed with error: Too many open files

 % limit descriptors 65536

 % limit
 cputime         unlimited
 filesize        unlimited
 datasize        unlimited
 stacksize       8192 kbytes
 coredumpsize    unlimited
 descriptors     65536
 memorysize      unlimited

 %./files

 ** Number of open files = 65532. fopen() failed with error: Too many open files

(Note that descriptor 0, 1 and 2 will be used by stdin, stdout and stderr)

 % ls -l 65531.log
 -rw-rw-r--   1 gmandali ccuser        11 Aug  9 12:35 65531.log

For further information about the extendedFILE library and for the extensions to fopen, fdopen, popen, .. please have a look at the new and updated man pages:

extendedFILE
enable_extended_FILE_stdio
fopen
fdopen

__________
Technorati tags:
Sun | Solaris | OpenSolaris

Read More

Java Troubleshooting & Diagnostic Guide

This nice 100+ page document is freely downloadable from Sun web site at: http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf.

The document has copious information about tools like jinfo, jmap, jstack, jconsole, jps, jstat, Heap Analysis Tool (HAT), Heap Profiler (HPROF), jdb, dbx; and discusses about fatal error handling, deadlock detection, memory leak diagnosis, diagnosing crashes in native code, compiled code, VM and HotSpot compiler threads etc., in detail. Must have document for majority of Java developers.

__________
Technorati tags:
Sun | java | troubleshooting

Read More

Siebel Analytics 7.8.4 support for Solaris 10

In May 2006, Oracle announced Siebel CRM 7.8's support for Solaris 10. It is Analytics' turn now. Solaris 10 is an officially certified platform for Siebel Analytics 7.8.4.

Check Siebel Support Web site for the following revised SRSP document:

System Requirements & Supported Platforms for the Siebel Business Analytics Platform
Version 7.8.4, Rev. B July 2006

Related post:
Siebel CRM 7.8 certified on Solaris 10

______________
Technorati tag: Solaris | Siebel | Analytics

Read More

Ubuntu 6.06: setting up the root password

For some reason, Ubuntu installer won't let you choose password for root user during the installation of the operating environment (OE). Then how do we run commands which need root privileges? The answer is simple: by using sudo.

From what I read/understand, the user that we create during the installation of OE will have sudo privileges to run root only commands. But I am pretty sure that it is inconvenient to prepend the string sudo before all such commands. So, it is better to login as root user rather than the normal user, when a large number of "root only" commands/tools/utilities needs to be run. The first step is to set the password for the root user. Use the sudo privileges of the default user to run passwd root command. Enter the password string of the default normal user when it prompts for password with an useful hint: enter your non-root user password. Type in the password chosen for the root user, when you are prompted to enter the new UNIX password with another useful hint: enter new password for root. Re-type the chosen password one more time in the next step, and you are done.

 % sudo passwd root
 Password: (enter your non-root user password) <- default normal user's password
 Enter new UNIX password:    (enter new password for root) <- choose a password for the root user. It is different 
                 from the one you typed in previous step -- and of course the default normal user's
                 password is not going to change after this step.
 Retype new UNIX password:   (re-enter new password for root)
 passwd: password updated successfully

Perhaps the steps are the same for earlier versions of Ubuntu - but this is the very first time I have ever got my hands on Ubuntu distro.
_______________
Technorati tags: Ubuntu | Linux

Read More

Sun Studio: symbol collisions revisited

Even though my blog entry C/C++: global const variables, symbol collisions & symbolic scoping and article Reducing Symbol Scope with Sun Studio C/C++ discuss about techniques to avoid symbol collisions in great length with examples, apparently they failed to provide a simple way to detect if the anomalous behavior is due to a symbol conflict. This blog entry tries to fill that gap with the help of Sun Studio debugger dbx and Solaris' link-edit tracing facility.

Let us start with the obligatory example first.

 % cat dummy.c
 #include <stdio.h>

 void compare (int x, int y)
 {
         printf("\ndummy.c: compare(int, int)");
 }

 void Comparable (int a, int b)
 {
         printf("\nNext line should print \"dummy.c: compare(int, int)\". It you see something 
                   else, it must be a symbol collision.");
         compare (a, b);
 }

 % cc -G -o libdummy.so dummy.c

 % cat thirdparty.c
 #include <stdio.h>

 int compare (char x, char y)
 {
         printf("\nthirdparty.c: compare(char, char)");
         return (0);
 }

 % cc -G -o lib3rdparty.so thirdparty.c
 % export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

 % cat test.c
 void Comparable(int, int);

 int main()
 {
         Comparable(1, 2);
         return (0);
 }

 % cc -o test -l3rdparty -ldummy test.c

 % ./test

 Next line should print "dummy.c: compare(int, int)". It you see something else, it must be a symbol collision.
 thirdparty.c: compare(char, char)

1) Using dbx to detect symbol collision

 % dbx test
 For information about new features see `help changes'
 To remove this message, put `dbxenv suppress_startup_message 7.5' in your .dbxrc
 Reading test
 Reading ld.so.1
 Reading lib3rdparty.so
 Reading libdummy.so
 Reading libc.so.1

 (dbx) stop in compare
 More than one identifier 'compare'.
 Select one of the following:
  0) Cancel
  1) `lib3rdparty.so`compare
  2) `libdummy.so`compare
  a) All
 > a
 dbx: warning: 'compare' has no debugger info -- will trigger on first instruction
 dbx: warning: 'compare' has no debugger info -- will trigger on first instruction
 Will create handlers for all 2 hits
 (2) stop in compare
 (3) stop in compare

 (dbx) run
 Running: test
 (process id 4404)

 Next line should print "dummy.c: compare(int, int)". It you see something else, it must be a symbol collision.stopped in compare at 0xd27a0240
 0xd27a0240: compare       :     pushl    %ebp

 (dbx) where
 =>[1] compare(0x1, 0x2), at 0xd27a0240
   [2] Comparable(0x1, 0x2), at 0xd27702d9
   [3] main(0x1, 0x8047324, 0x804732c), at 0x80506f8

proc -map shows the list of objects loaded with addresses.

 (dbx) proc -map
 Loadobject mappings for Process ID: 4404
 0x08050000 /export/home/techno/C/test
 0xd27a0000 /export/home/techno/C/lib3rdparty.so
 0xd2770000 /export/home/techno/C/libdummy.so
 0xd2690000 /lib/libc.so.1
 0xd27c8000 /lib/ld.so.1 [LM_ID_LDSO]

From the above mappings, it appears that the symbol compare was bound to the definition in lib3rdparty.so. However it should really be bound to the definition in libdummy.so.

Other way to check the mappings is by getting the process id with proc -pid command, and using pmap <pid> command in dbx environment.

 (dbx) proc -pid
 4404

 (dbx) pmap 4404
 4404:   /export/home/techno/C/test
 08046000       8K rwx--    [ stack ]
 08050000       4K r-x--  /export/home/techno/C/test
 08060000       4K rwx--  /export/home/techno/C/test
 D261E000     384K rw---    [ anon ]
 D2680000      24K rwx--    [ anon ]
 D2690000     764K r-x--  /lib/libc.so.1
 D275F000      24K rw---  /lib/libc.so.1
 D2765000       8K rw---  /lib/libc.so.1
 D2770000       4K r-x--  /export/home/techno/C/libdummy.so
 D2780000       4K rwx--  /export/home/techno/C/libdummy.so
 D2790000       4K rwx--    [ anon ]
 D27A0000       4K r-x--   /export/home/techno/C/lib3rdparty.so
 D27B0000       4K rwx--  /export/home/techno/C/lib3rdparty.so
 D27BA000       4K rwxs-    [ anon ]
 D27C8000     140K r-x--  /lib/ld.so.1
 D27FB000       4K rwx--  /lib/ld.so.1
 D27FC000       8K rwx--  /lib/ld.so.1
  total      1396K

The following outputs in dbx env. confirm that the symbol compare was actually resolved in lib3rdparty.so rather than libdummy.so.

 (dbx) which compare
 `lib3rdparty.so`compare

which command prints the full qualification of a given name.
 

(dbx) scopes
 Function   compare
 File       "(unknown OF)"
 Loadobject /export/home/techno/C/lib3rdparty.so

The scopes command prints a list of active scopes.

 (dbx) status
 *(2) stop in compare
  (3) stop in compare

The status prints the stop breakpoints in effect. In the above output we do not know which compare belong to which library. -s option of status will show the information we need.

 (dbx) status -s
  stop in `lib3rdparty.so`#compare
  stop in `libdummy.so`#compare

 2) Using Solaris' link-edit tracing to detect symbol collision

Setting the run-time linker (ld.so.1) env variable LD_DEBUG to the token bindings causes the run-time linker to show the binding of a symbol reference to a symbol definition. Usually the output from run-time linker will be overwhelming if the application is big with a handful of symbols. By default, all this output will be displayed on stdout; so, it might be a little inconvenient to read all the output on stdout. However the tracing data from run-time linker can be redirected to a file using LD_DEBUG_OUTPUT env variable as shown below.

eg.,

 % export LD_DEBUG=bindings
 % export LD_DEBUG_OUTPUT=/tmp/bindings.log

 % ./test
 Next line should print "dummy.c: compare(int, int)". It you see something else, it must be a symbol collision.
 thirdparty.c: compare(char, char)

 % grep compare /tmp/bindings.log.0443*
 /tmp/bindings.log.04436:04436: binding file=./libdummy.so to file=./lib3rdparty.so: symbol `compare'

Observe that the call to symbol (function or method) compare in libdummy.so was resolved in lib3rdparty.so. It confirms the symbol conflict. This problem arises when the symbols are left in default global scope. Use elfdump tool to check the scope and attribute of the symbol of interest (compare in this example).

 % /usr/ccs/bin/elfdump -sN.symtab libdummy.so | grep compare
       [40]  0x00000280 0x00000027  FUNC GLOB  D    0 .text       compare

 % /usr/ccs/bin/elfdump -sN.symtab lib3rdparty.so | grep compare
       [41]  0x00000240 0x00000036  FUNC GLOB  D    0 .text       compare

Symbols with global default scope are always vulnerable to symbol conflicts, as seen in the above example. The primary reason being any one can interpose upon these global default symbols. To prevent others from interposing on the symbols that are completely useless outside the defining component (executables or shared objects), make them protected with Sun Studio's -xldscope=symbolic compiler option or __symbolic attribute. Note that these language extensions were introduced in Sun Studio 8. Refer to Reducing Symbol Scope with Sun Studio C/C++ article for detailed information about global, symbolic scopes and also about the linker scoping language extension. If you are using any older compiler, use -Bsymbolic linker option to build your executables and shared libraries there by making the symbols protected.

eg.,

 % cc -G -o libdummy.so -xldscope=symbolic dummy.c

 % /usr/ccs/bin/elfdump -sN.symtab libdummy.so | grep compare
       [40]  0x00000280 0x00000027  FUNC GLOB  P    0 .text       compare

 % cc -G -o lib3rdparty.so -xldscope=symbolic thirdparty.c

 % /usr/ccs/bin/elfdump -sN.symtab lib3rdparty.so | grep compare
       [41]  0x00000240 0x00000036  FUNC GLOB  P    0 .text       compare

 % cc -o test -l3rdparty -ldummy test.c

 % ./test

 Next line should print "dummy.c: compare(int, int)". It you see something else, it must be a symbol collision.
 dummy.c: compare(int, int)

Note that the compiler option -xldscope=symbolic and the linker option -Bsymbolic make all the symbols protected - so, it is not possible for others to interpose upon the symbols of such libraries or executables. For some reason if you have to leave some symbols in global default scope, use __symbolic specifier for those symbols which must be protected from interposers.

eg.,

 % cat dummy.c
 #include <stdio.h>

 __symbolic void compare (int x, int y)
 {
         printf("\ndummy.c: compare(int, int)");
 }

 void Comparable (int a, int b)
 {
         printf("\nNext line should print \"dummy.c: compare(int, int)\". It you see something 
                   else, it must be a symbol collision.");
         compare (a, b);
 }

 % cc -G -o libdummy_.so dummy.c

 % /usr/ccs/bin/elfdump -sN.symtab libdummy_.so | grep -i compar
       [40]  0x00000280 0x00000027  FUNC GLOB  P    0 .text       compare
       [45]  0x000002b0 0x00000037  FUNC GLOB  D    0 .text       Comparable

Observe that the symbol compare is protected (although global in scope); and hence cannot be overridden by another global symbol with name compare. However Comparable can still be overridden by a different implementation, due to its global default scope.

Just to make sure that the symbol was bound to the definition in the right library, you can check the mappings, scope etc., again with dbx. Link-edit tracing won't show any binding information when it gets resolved within the same library.
_______________
Technorati tags: Sun Studio | Solaris | linker | Programming

Read More

Siebel CRM 7.8 certified on Solaris 10

It is official now - Siebel CRM 7.8 supports Solaris 10. This information can be found in the revised system requirements and supported platforms document at Siebel support web web site. Just look for the document with title:

Siebel System Requirements and Supported Platforms
Business Applications Version 7.8, Rev. L May 2006
______________
Technorati tag: Sun | Solaris | Siebel

Read More

JavaOne, Downtown San Francisco & Angel Island

There has been a lot of activity lately. I'd like to share some of my experiences from JavaOne 2006; and briefly touch the other {fun} activity as well.

[05/17/06 and 05/18/06] JavaOne, Moscone Center, San Francisco

Although I have been to Moscone to do booth duty to demonstrate our team's performance portal project back in 2002 Sun Network event, this is my first real JavaOne experience. I am glad I was there to see the power of Java. The turnout was very good with thousands of developers flocked from all over. I've got a chance to attend few technical/birds-of-a-feather (BOF)/general sessions (Thanks George).

May 17:

General session - Lies, damn lies and Java by BEA folks

The main focus of this talk is to reiterate that Java is not going to be another COBOL in the imminent future; and to assure that still there is scope for lot of innovation around Java technologies. BEA dedicated most of the time to introduce the features of their next version of WebLogic application server.

BOF - Solving the Mysteries of the Java Technology Class Loader

This presentation introduced the Java class loader along with its evolution, and explained when a class gets loaded and how. There is a simple example to write a custom class loader as well.

May 18:

Discussion - Java Technology, AJAX, Web 2.0 and SOA

A discussion about these buzzing technologies, among a panel of experts. It wasn't very interesting for me. I guess many audience felt the same - the overflowing room was full 10 minutes before the scheduled time, and then half-full within 10 minutes once the discussion was started.

Technical session - New Compiler Optimizations in the Java HotSpot Virtual Machine

An interesting session about the compiler optimizations introduced in the forth coming JDK (6.0, code named Mustang). Most of the benchmark results with the new compiler optimizations are very impressive. It appears that some work is in progress to merge client and server HotSpot VMs into a single VM (may not make it to Java SE 6.0 though).

Technical session - Secure Coding Antipatterns: Avoiding Vulnerabilities

Another interesting session even though designers and architects are the targeted audience. There are some real world {source code} examples from various JDK versions, explaining why it is vulnerable to have code like that and offered some suggestions to make it secure against hacking attacks. Pretty interesting stuff.

_______________

In my opinion:

* the folks who approve these sessions should focus on quality of the presentations and demos, over quantity -- there are too many

* it is better to have shorter sessions with 30 minute duration esp. those that sessions that explain something that exists already for quite some time -- I find it hard to sit straight for a 50+ min presentation {and even the presenters are running out of material, anyway)

* better not overlap too many sessions at the same time -- the knowledge transfer should commensurate with the $$s paid for the conference.

* there shouldn't be any break in the schedule -- don't know why the 5-7PM slots are left unfilled

* the whole event can be squeezed into two days, unless there is a lot of new/unknown stuff that needs to be evangelized

* finally there should be some easy way to fill out the feedback forms over the web (are we still in 90s to fill out paper forms?) -- it will be effective in saving the paper and the associated printing costs, as well as getting proper feedback from the audience (because they are not forced to fill out the form within the 30+ min session).

Some photographs from the JavaOne conference:

 
Howard Street, Downtown SF                      Howard Street, Downtown SF

 
Moscone Center, South Block - Pavilion          Moscone Center, North Block - Auditoriums

 
Rest area between South & North blocks          Way to auditoriums

 
Sun territory                                   Solaris/OpenSolaris booths

 
Project looking glass (3D desktop) demo         Oracle booth

 
BEA - think liquid                               Slot car racing challenge

[05/13/06 and 05/14/06] Downtown San Francisco

Even though I have been in Bay Area for nearly 4 years, I never got a chance to roam around the streets of downtown San Francisco. I've read many exciting reviews about Lombard street, the crooked street in Frisco located between Hyde and Leavenworth; and thought it is worth give a shot.

[Off-topic] On my way to downtown, I narrowly escaped a collision with an SUV on the ramp towards I-280 North. People who wish to share the road with others should really learn about the importance of signals; and those who fail to understand the importance of safety of others on the road, should be packaged in a nicely wrapped box without air holes and sent to the deepest forest in Amazon (just like Garfield mailing Nermal to Abu Dhabi whenever he gets irked). That's a different story anyway.

Contrary to majority of people's opinion, it wasn't a very exciting experience for me on Lombard crooked street. The exuberance was quickly vanished, as the drive on this street is very short, and lasted only for 90 seconds. Besides, it is not too challenging (many reviewers on web found it challenging). Unless being chased by someone at 50+ mph speed (like we see in movies), any person with little driving experience can easily maneuver on this popular crooked street. However the descent at Leavenworth followed by an acclivity on the same street towards Coit tower was really entertaining. The scary part is that I could hardly see oncoming vehicles atop the downhill waiting at the stop signs.

As you can see in the following photographs, even driving in the other busy streets in the vicinity of downtown along the rails of cable car is fun.

 
SF skyline from Bay Bridge exit                 SF skyline from Bay Bridge exit

 
Familiar traffic pattern for bay areans on 101  Lombard, Crooked street

 
Lombard, Crooked street                         Descent at Leavenworth

 
Driving over cable car tracks                   Driving along the cable car tracks

 
Driving along the cable car tracks              Sharp descent

 
An acclivity in downtown SF vicinity            Descent followed by an acclivity

 
Transamerica Pyramid in the middle             Bay bridge connecting downtown SF and Oakland

 
Somewhere in downtown San Francisco             Testing the camera while moving @ 65+ mph

I realized two things from these photographs: (1) my digital camera is able to capture the images sharply, even when I was moving at 65+ mph speed (2) my car needs a thorough wash - automatic car wash is not good enough

[05/10/06] Hiking in the Angel Island

As part of the team building event, our group at work with nearly 15 people went for hiking in the Angle Island, which is somewhere in the middle of San Francisco bay. Luckily it was sunny, but not too hot; and the hike wasn't much stressful. The view of San Francisco downtown from the peak of Mt. Livermore is awesome - but I forgot my camera at home, as usual.

As I spend most of my time at our partner site, it is an opportunity for me to introduce myself to the other people in the group whom I never met before, and to spend some time with my colleagues expatiating various topics from photography to profile feedback optimization. Spent some time playing Frisbee in the afternoon while we were waiting for the ferry to arrive.

Overall it was fun. Glad I didn't miss that - now I know more people in our group.

______________
Technorati tag: JavaOne | Sun | San Francisco | Lombard Street

Read More