Last friday I was assigned to look into an issue where the application is not able write into files, once it is up for more than one week. It is a 32-bit application running on Solaris (SPARC platform) and the error message says,
A little background on stdio's fopen():
A common known problem (perhaps a "fact") is that when the 32-bit
However if the program uses
The following C program illustrates the limitation of the number of open files with
How to resolve this issue:
Make it a 64-bit binary; it will allow the program to have 65536 open files
As we can see,
Note:
To use 64-bits, the processor and the OS must have support for 64-bit binaries
Since I cannot re-compile the code at customer site, the option of creating 64-bit binaries has been ruled out.
A closer look at the output of
So, to find the actual number of calls to
Interestingly the interposer caught only two calls to
This observation made my job little simple. To resolve this particular customer issue, the application just needs to reserve low numbered file descriptors <= 255 for use by
How to reserve the file descriptors?
Use file control function,
In fact, database management systems like Oracle, Sybase, Informix addressed the
As changing the application code is not feasible (and not possible), this can be done very easily with an interposing library, with interfaces to
Interposing code for
The same functionality has to be replicated for the other interfaces:
_______________________
Because of issues like this, it is recommended to use
References and suggested reading:
too many open files
. With little effort, we came to know that all those errors are due to the calls to fopen()
, from the application.A little background on stdio's fopen():
fopen()
is part of stdio
API. For a 32-bit application, a stdio
library FILE
structure represents the underlying file descriptor as an unsigned char
(8 bits), limiting the range of file descriptors which can be opened as FILE's to 0-255 inclusive.A common known problem (perhaps a "fact") is that when the 32-bit
stdio
is used in large applications on Solaris, the 255 limit for the number of open files, is frequently reached. File descriptors are allocated by the operating system starting at 0, and are then allocated in numerical order. Descriptors 0, 1, and 2 are opened for every process as stdin
, stdout
and stderr
at startup.open()
system call can also be used to open files from a C program. Both open
and fopen
use file descriptors which are taken from the total number of file descriptors allowed by the environment. Also the system allocates descriptors from the same pool of file descriptors, for calls to popen(), socket(), accept()
and any other system call that returns a descriptor. That is, the same pool of file descriptors will be shared by various system calls like fopen, open, popen, accept, socket
. So, if the application has numerous calls to these functions, and assuming if they are not closed immediately, it is very likely that a call to fopen
may fail even before it reaches the 253 (266 - 3 = 253) file descriptors, that it can have them open as permitted by the OS.However if the program uses
open/popen/socket/accept
exclusively, then the program will be able to open as many files/pipes/sockets/connections as the current soft limit allows. The soft limit defines how many files a process can open. There are actually two environmental limits. The soft limit and the hard limit. The soft limit is the number of files a process can open by default. The hard limit is the maximum number of files a process can open if it increases the soft limit.The following C program illustrates the limitation of the number of open files with
fopen()
:
% cat fopen.c
#include <stdio.h>
#include <errno.h>
#define MAXFOPEN 275
int main() {
FILE *fps[MAXFOPEN];
char fname[15];
int i, j;
/*
* Test total number of fopen()'s which can be completed
*/
for (i = 0; i < MAXFOPEN; i++) {
sprintf(fname, "fopen_%d", i);
if ((fps[i] = fopen(fname, "w+")) == NULL) {
perror("fopen fails");
break;
}
}
printf("fopen() completes: %d\n", i);
/*
* Close the file descriptors
*/
for (j =0; j < i; j++) {
if (fclose(fps[j]) == EOF) {
perror("fclose failed");
}
}
return (0);
}
% cc -o fopen fopen.c
% file fopen
fopen: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required,
dynamically linked, not stripped
% ./fopen
fopen failed: Too many open files
fopen() completes: 253
How to resolve this issue:
Make it a 64-bit binary; it will allow the program to have 65536 open files
% cc -xarch=v9 -o fopen fopen.c
% file fopen
fopen: ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not
stripped
% ./fopen
fopen() completes: 275
As we can see,
fopen()
was able to overcome the 253 open files limitation with 64-bit executable.Note:
To use 64-bits, the processor and the OS must have support for 64-bit binaries
Since I cannot re-compile the code at customer site, the option of creating 64-bit binaries has been ruled out.
A closer look at the output of
lsof
(LiSt of Open Files), gave me a clue that the most of the open files are actually TCP sockets/connections.
% grep TCP openfiles.log
app 6913 giri 11u IPv4 0x30013e4d3c0 0t0 TCP *:49152 (LISTEN)
app 6913 giri 12u IPv4 0x30011722200 0t0 TCP *:49153 (LISTEN)
app 6913 giri 13u IPv4 0x300126e8680 0t0 TCP *:49154 (LISTEN)
app 6913 giri 14u IPv4 0x30010faf300 0t0 TCP *:49155 (LISTEN)
app 6913 giri 15u IPv4 0x300082c2d00 0t0 TCP *:49156 (LISTEN)
app 6913 giri 16u IPv4 0x30011e3a180 0t0 TCP *:49157 (LISTEN)
app 6913 giri 17u IPv4 0x30018e36700 0t0 TCP *:1571 (LISTEN)
app 6913 giri 18u IPv4 0x30009bad900 0t0 TCP *:49158 (LISTEN)
app 6913 giri 43u IPv4 0x3001aa0a700 0t0 TCP as7:44232->as7:49156 (ESTABLISHED)
app 6913 giri 46u IPv4 0x30011cff800 0t0 TCP as7:1571->as3:27025 (ESTABLISHED)
app 6913 giri 49u IPv4 0x3000ce48d40 0t0 TCP as7:1571->as3:27026 (ESTABLISHED)
app 6913 giri 51u IPv4 0x300199d3980 0t722051 TCP as7:44238->repo:1521 (ESTABLISHED)
app 6913 giri 52u IPv4 0x30014d40c40 0t793865 TCP as7:44239->repo:1521 (ESTABLISHED)
app 6913 giri 55u IPv4 0x300197db340 0t0 TCP as7:1571->as3:27027 (ESTABLISHED)
app 6913 giri 56u IPv4 0x30011b5f800 0t675177 TCP as7:44243->repo:1521 (ESTABLISHED)
app 6913 giri 57u IPv4 0x30012853880 0t0 TCP as7:1571->as3:27028 (ESTABLISHED)
app 6913 giri 58u IPv4 0x30011d94d00 0t723190 TCP as7:44244->repo:1521 (ESTABLISHED)
app 6913 giri 62u IPv4 0x30016d5b240 0t0 TCP as7:1571->as3:27029 (ESTABLISHED)
app 6913 giri 63u IPv4 0x3001126d9c0 0t575246 TCP as7:44247->repo:1521 (ESTABLISHED)
app 6913 giri 64u IPv4 0x3000a825900 0t0 TCP as7:1571->as3:27030 (ESTABLISHED)
...
...
app 6913 giri 250u IPv4 0x300139899c0 0t0 TCP as7:1571->as3:27076 (ESTABLISHED)
app 6913 giri 251u IPv4 0x30017fc4700 0t0 TCP as7:1571->as3:27077 (ESTABLISHED)
app 6913 giri 252u IPv4 0x30011c3b900 0t403370 TCP as7:44390->repo:1521 (ESTABLISHED)
app 6913 giri 253u IPv4 0x3000cd32c40 0t445290 TCP as7:44391->repo:1521 (ESTABLISHED)
app 6913 giri 257u IPv4 0x30017f640c0 0t0 TCP as7:1571->as3:27078 (ESTABLISHED)
app 6913 giri 258u IPv4 0x300141f1280 0t0 TCP as7:1571->as3:27079 (ESTABLISHED)
So, to find the actual number of calls to
fopen()
, I have created a simple interposing library with only one interface that interposes on actual fopen()
function.
% cat logfopen.c
#include <dlfcn.h>
#include <stdio.h>
#include <stdarg.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/errno.h>
#include <thread.h>
#include <synch.h>
#include <fcntl.h>
FILE *fopen(const char *filename, const char *mode) {
FILE *fd;
static void * (*func)();
if(!func) {
func = (void *(*)()) dlsym(RTLD_NEXT, "fopen");
if (func == NULL) {
(void) fprintf(stderr, "dlopen(): %s\n", dlerror());
return(0);
}
}
fd = func(filename, mode);
if (fd != NULL) {
fprintf(stderr, "\nfopen(): fd = %d filename = %s mode = %s",
fileno(fd), filename, mode);
} else {
fprintf(stderr, "\nfopen() failed; returned NULL. Tried to open %s
with mode: %s", filename, mode);
}
return (fd);
}
Interestingly the interposer caught only two calls to
fopen()
, during a 10 min real world simulation run of the application. This was confirmed by running truss
tool.
% grep fopen stderrout.log
fopen(): fd = 32 filename = /export/home/oracle/network/names/.sdns.ora mode = r
/export/home/C/liblogfopen.so:fopen+0x3c
fopen(): fd = 32 filename = /export/home/oracle/network/admin/tnsnames.ora mode = r
/export/home/C/liblogfopen.so:fopen+0x3c
% grep fopen truss.log
6913/14@14: -> libc:fopen(0xe48f6178, 0xe48f627c, 0x1, 0x61)
6913/14@14: <- libc:fopen() = 0
6913/14@14: -> libc:fopen(0xe48f8ad0, 0xe48f8bd4, 0x0, 0x61)
6913/14@14: <- libc:fopen() = 0xfdae884c
This observation made my job little simple. To resolve this particular customer issue, the application just needs to reserve low numbered file descriptors <= 255 for use by
fopen()
.How to reserve the file descriptors?
Use file control function,
fcntl()
to return lowest file descriptor greater than or equal to 256, that is not already associated with an open file. fcntl()
takes the OS assigned file descriptor and returns a new file descriptor greater than or equal to the value passed as 3rd argument. ie., once fcntl()
successfully returns a new file descriptor, we will have two file descriptors pointing to the same open file. Since our intention is to make, as many file descriptors available as possible for fopen() to succeed, and since we don't need two file descriptors, we can close the OS assigned file descriptor safely.In fact, database management systems like Oracle, Sybase, Informix addressed the
fopen
issue by employing this technique of reserving the low numbered file descriptors for exclusive use by stdio
routines. As changing the application code is not feasible (and not possible), this can be done very easily with an interposing library, with interfaces to
open(), popen(), socket(), accept()
. (A brief introduction to library interposition, is available at: Solaris: hijacking a function call (interposing)). The interfaces of the interposing library catches all calls to open(), popen(), socket(), accept()
etc., even before the actual implementation receives the call, and duplicates the file descriptors with the help of fcntl()
function, to get a new file descriptor that is > 256, and returns the OS assigned file descriptor, to the pool of available file descriptors ie., to the OS.Interposing code for
socket()
:
% cat fopenfix.c
#include <stdio.h>
#include <dlfcn.h>
#include <ucontext.h>
#include <sys/types.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
int socket(int domain, int type, int protocol) {
int sd, newsd = -1;
static void * (*func)();
if(!func) {
func = (void *(*)()) dlsym(RTLD_NEXT, "socket");
}
sd = (int) func(domain, type, protocol);
if (sd != NULL) {
if (sd < 256) {
newsd = (int) fcntl(sd, F_DUPFD, 256);
if (newsd == -1) {
fprintf(stderr, "\nfcntl() failed. Cannot return %d to OS", sd);
return (sd);
} else {
close (sd);
return (newsd);
} // if-else
} else {
return (sd);
}
} else {
return (sd);
}
}
The same functionality has to be replicated for the other interfaces:
int open(const char *path, int oflag, ...);After preloading the interposing library, all the file descriptors of
int accept(int s, struct sockaddr *addr, void *addrlen);
...
open, popen, socket, accept
etc., were mapped to the descriptors > 256, leaving enough room for fopen()
to have nearly 253 open files._______________________
Because of issues like this, it is recommended to use
open()
and its family of functions, for file handling instead of stdio
's fopen()
and its subordinates. It is a good practice for the developers to think about problems like this and handle them properly (just like Oracle/Informix/.. did), during the early stages of the design/development of the application.References and suggested reading: