Migrating Linux Applications to Solaris

If you are a developer looking for some guidance on porting Linux applications to Solaris operating system (running on x86/x64/SPARC architectures), here is a short but useful porting guide for you:

Porting to the Solaris™ OS: A guide for Linux developers

__________
Technorati tags:
Linux | Solaris | OpenSolaris | Porting

Read More

Upgrading Oracle E-Business Suite 11.5.10 to CU2

Some notes I collected from the exercise of upgrading Oracle E-Business Suite 11.5.10 to 11.5.10 CU2^*, on Sun Solaris:

1. [Middle-tier] Apply patch 4318672 on all application server nodes
   
   
   • Enable maintenance mode by running adadmin.
     
     
   • Apply patch 4318672 with the help of adpatch tool.
     
     
   • Carefully read README.txt of 4318672, and run Technology Stack Validation tool in both middle- and db-tiers.
     
     Note:
     
     
     • Set PARALLEL_MAX_SERVERS parameter to the default value returned by show parameter cpu_count. For example, if the database is running on a SunFire T2000 server, set PARALLEL_MAX_SERVERS to 32.
       
       
     • [DB-tier] When prompted for applications context file, supply the output of echo $CONTEXT_FILE environment variable.
       
       
     • After running Technology Stack Validation tool, the generated html reports from middle- and db-tiers should have an 'ALLPASS' status. Otherwise, fix the errors and re-run the TSV tool until it achieves ALLPASS status.
   
   
2. [Middle-tier] Apply patch 4229931 on all application server nodes
   
   
   • Apply patch 4229931 with the help of adpatch tool.
     
     Location of adpatch log file: $APPL_TOP/admin/<DBInstanceName>/log/adrelink.log.
   
   
3. [Concurrent Processing node] Apply patch 4297568
   
   
   • Enable maintenance mode by running adadmin tool, if concurrent managers are not running on the same machine where all application server processes are running; then apply patch 4297568 by following the special instructions posted in README.txt file.
     
     Note:
     
     • Do NOT use adpatch tool to apply this patch, 4297568. Read the instructions of README.txt.
       
       
     • Without this patch, 4297568, CU2 maintenance patch, 3480000, may fail with error messages like:

       ld: fatal: file  /ebiz/visappl/sht/11.5.0/lib/ilog/5.1/libcplex8.a: 
       open failed: No such file or directory
       ld: fatal: File processing errors. No output written to /ebiz/visappl/demoappl/mst/11.5.0/bin/MSTCPP

       See Metalink document 311635.1 for more.
       
       
     • Some executables need to be re-linked as part of this step. Re-linking can be done in two ways:
       
       
       1. Using adadmin tool.
          
          Navigation: Maintain Applications Files -> Relink Applications Programs. This step re-links all the executables of the application.
          
          
       2. To re-link one or more applications' executable programs, run adrelink from the operating system shell:
          
          Syntax:
          adrelink force=y "<product> <module>"
          
          eg., To re-link MSO: MSONEW, run:
          
          adrelink force=y "MSO MSONEW"
          
          Shutdown all concurrent managers before running adrelink.
     
     
     
   • Disable maintenance mode by running adadmin, if concurrent managers are not running on the same machine where all application server processes are running.
   
   
4. [Middle-tier] Apply release 11.5.10.2 maintenance pack, 3480000
   
   
   • Apply patchset, 4712852, with the help of adpatch tool.
     
     
   • Apply release 11.5.10.2 maintenance pack, 3480000, with the help of adpatch tool. Spare some time to read Metalink document 316365.1, before applying the maintenance pack.
     
     Note:
     
     
     • Stop all application server services including concurrent managers before running adpatch on 3480000 (CU2 maintenance pack). Failure to do so may result in the following error(s):

       ATTENTION: All workers either have failed or are waiting:
               FAILED: file afcmgr.odf on worker 1
       ATTENTION: please fix the above failed worker(s) so the manager can continue

       When such an error occurs, do the following:
       
       
       • Open up another shell and shutdown all services by running adstpall stop apps/apps command.
         
         
       • Run: adctrl
         
         • Check: 1) Show Worker Status.
           
         • 2) Tell worker to restart a failed job
         
         After this step adpatch resumes automatically.
         
         
       • adworker log file(s) default location: $APPL_TOP/admin/<DBInstanceName>/log.
     
     
   • Troubleshooting:
     
     
     • ERROR: file azstrctall.ldt on worker 1
       
       Follow the instructions of Metalink document '331284.1 Applying 3480000 Patch --- azstrctall.ldt failed --- ORA-20001: The specified api_code does not exist'.
       
       
     • FAILED: file AsgRunInst.class on worker 1
       
       Follow the instructions of Metalink document 270156.1. In short, skip the failed thread with an hidden option {of adctrl} : 8) Skip & Restart for worker thread 1. Run the failed thread manually, later.
   
   
   
5. [Middle-tier] Disable maintenance mode by running adadmin

_________
^* CU2 = Consolidated Update 2.

Reference Document:
Metalink document 316366.1 11.5.10 Oracle E-Business Suite Consolidated Update 2 (CU2).

Technorati tags:
Oracle | Oracle Applications | E-Business Suite

Read More

Oracle Apps on T2000: ORA-04020 during Autoinvoice

The goal of this brief blog post is to provide a quick solution to all Sun-Oracle customers who may run into a deadlock when a handful of threads are running Autoinvoice Master Program (RAXMTR) on SunFire CoolThreads server, T2000.

If you are not happy with Autoinvoice performance, or if you feel like the AI program is crawling^* when multiple threads are handling the submitted jobs, check the trace files generated in udump directory for a pattern like 'ORA-04020: deadlock detected while trying to lock object ...'. If there is a deadlock, you may see something like:

*** ACTION NAME:(Concurrent Request) 2007-03-20 17:55:25.238
*** MODULE NAME:(RAXMTR) 2007-03-20 17:55:25.238
*** SERVICE NAME:(SYS$USERS) 2007-03-20 17:55:25.238
*** SESSION ID:(2469.90) 2007-03-20 17:55:25.238

A deadlock among DDL and parse locks is detected.
This deadlock is usually due to user errors in
the design of an application or from issuing a set
of concurrent statements which can cause a deadlock.
This should not be reported to Oracle Support.
The following information may aid in finding
the errors which cause the deadlock:

ORA-04020: deadlock detected while trying to lock object AR.RA_INTERFACE_LINES_ALL

--------------------------------------------------------
object   waiting  waiting       blocking blocking
handle   session     lock mode   session     lock mode
--------  -------- -------- ----  -------- -------- ----
457bf9a78  479d813f0 46dddbd10    X  47fd9ce08 468973ee0    S
46561f560  47fd9ce08 46894f520    X  479d813f0 46dc4edf8    X
--------------------------------------------------------
...
...

Setting profile option AR: Autoinvoice Gather Statistics to Yes or leaving it to the default value of NULL is one of the reasons for this deadlock. When the profile option is set to either Yes or NULL, by default it runs Autoinvoice Gather Statistics every time Autoinvoice program is run.

Running Autoinvoice Gather Statistics process is too expensive; and it is not necessary to run this process as part of the Autoinvoice process. Hence Autoinvoice Gather Statistics can be safely turned off as part of a normal Autoinvoice processing exercise.

To implement this, simply set the profile option AR: Autoinvoice Gather Statistics to No. Navigation: System Administrator responsibility -> Profile -> System.

Perhaps this solution is applicable to all other platforms where this deadlock issue shows up with quite a few threads running Autoinvoice (RAXMTR). I'm not sure - I never tried it.
________

^*Run the following SQL to get the list of Autoinvoice request IDs, request date along with the actual start and completion times.

SELECT REQUEST_ID,
TO_CHAR (REQUEST_DATE, 'yyyymmddhh24mi') "REQUEST DATE",
TO_CHAR (ACTUAL_START_DATE, 'yyyymmddhh24mi') "START DATE",
TO_CHAR (ACTUAL_COMPLETION_DATE, 'yyyymmddhh24mi') "COMPLETION DATE"
FROM FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID IN
   (SELECT REQUEST_ID
           FROM APPLSYS.FND_CONCURRENT_REQUESTS FCR, APPLSYS.FND_CONCURRENT_PROGRAMS FCP
           WHERE FCP.CONCURRENT_PROGRAM_ID = FCR.CONCURRENT_PROGRAM_ID
           AND FCP.CONCURRENT_PROGRAM_NAME = 'RAXTRX'
           AND ACTUAL_COMPLETION_DATE > <desired_date>);

________
UltraSPARC-T1 | Cool Threads | T2000 | Oracle Applications | E-Business Suite

Read More

Speeding up Oracle Applications/Payroll processing on Niagara servers

Sun Fire Cool Threads Oracle Configuration Notes says: Long running single threaded Oracle batch jobs may run longer on Sun Fire CoolThreads server, which is true esp. with Oracle Applications (E-Business Suite)/Payroll jobs running on Sun Fire T1000/T2000 servers.

Assuming only one thread is processing all payroll jobs on a 8 core 1.2 GHz T2000, the thread should be getting 300 million cycles per second (Here's the math: 1.2 GHz per core = 1200 MHz per core = 1200 MHz/4 threads per core = 300 MHz per thread per core). So, the performance of a single thread on this server will be something like the performance of a single thread of an US-II based server. Fortunately we have 32 such threads on 8 core T1000/T2000 servers; and it is possible to submit multiple payroll jobs by updating the value of parameter name 'THREADS' in table pay_action_parameters.

Here is how you do it from the CLI:

% sqlplus apps/apps

To retrieve the current value of 'THREADS' parameter:

SQL> select parameter_value
from pay_action_parameters
where parameter_name = 'THREADS';

To set the value for the number of payroll threads:

SQL> update pay_action_parameters
set parameter_value = <number_of_threads>
where parameter_name = 'THREADS';

SQL> commit;

Experiment by increasing/decreasing the number of payroll threads until you are happy with the overall processing time with decent resource utilization.

____________________
UltraSPARC-T1 | Cool Threads | Oracle Applications | E-Business Suite

Read More

Backing Up Sun Secure Application Switch Configuration

Multi-node deployments of enterprise applications like E-Business Suite (aka Oracle Applications) have a dependency on hardware load balancers like Sun Secure Application Switch^*. Just like any piece of hardware, Sun Secure Application Switches are also vulnerable to hardware failures. Naturally we want to make a backup copy of the switch configuration once everything is configured and tested properly. So, here are the steps to save the existing configuration:

1. Connect to Application Switch on the CLI. On the CLI, type, for example:
   

   
    % telnet 192.168.109.70
    Trying 192.168.109.70...
    Connected to 192.168.109.70.
    Escape character is '^]'.
    username: admin
    password:
   
    MDE> 
   
   

   
   
2. Save the configuration to any file with show runningConfig saveToFile command as shown below:
   

   
    MDE> show runningConfig saveToFile sunfish_3appnodes_010707.cfg.txt password admin nameValuePairs true
   
    Please confirm the password:
   
    Note: Opening file "/ftl0/usr/home/sunfish_3appnodes_010707.cfg.txt" to write running-config.
   
    MDE> quit
    Goodbye
    Connection to 192.168.109.70 closed by foreign host.
   
   

   
   
3. Finally download the configuration file from the switch {to the local host} with the help of trivial file transfer program (tftp) on port 69.
   

   
    % tftp 192.168.109.70 69
   
    tftp> get /ftl0/usr/home/sunfish_3appnodes_010707.cfg.txt
    Received 19672 bytes in 0.2 seconds
   
    tftp> quit
   
   

The configuration file may look like:

################################################################################
#  System Name     : Sun Secure Application Switch
#  Date            : Sun Jan  7 18:49:33 2007
#  Serial No       : 00608000124
#  Sofware Version : V3_1R0
################################################################################
commandModeEntry on
enable
configure

#
# Create all vSwitches upfront
#
vSwitch vSwitchName EBS 
    vRouter name default description {Default vRouter} 
    vRouter name default 
    exit;
exit;
vSwitch vSwitchName system description {System vSwitch} 
vSwitch vSwitchName system 
    vRouter name management description {System Management vRouter} 
    vRouter name management 
    exit;
    vRouter name shared description {Shared vRouter} 
    vRouter name shared 
    exit;
exit;

#
# Event configuration and statistics
#
event fileLogSize 4194304 
...
...

Sample Sun Secure Application Switch configuration:

Click here for the complete copy of the configuration file from an E-Business Suite 11.5.10 deployment with three application server nodes in the middle-tier.

Related documentation:

• Clustering and Securing Web Applications: A Tutorial
  
• Sun Secure Application Switch - N1000 Series documentation

Note:
Sun Secure Application Switch is more than a hardware load balancer. Check the home page of Sun Secure Application Switch for key features. By the way, Sun Secure Application Switch may also be referred as 'Nauticus Switch' or 'Sunfish' by some Sun folks.
____________________
Sun | Sun Secure Application Switch | Oracle Applications | E-Business Suite

Read More

Energy Policy Act of 2005 - Daylight Saving Time changes 2007

As you know, due to the Energy Policy Act of 2005, there will be some changes to the daylight saving time starting in 2007. As a hardware/software vendor, Sun Microsystems is ready with a solution (set of patches) to accommodate those changes; and the required information is well documented in the form of alerts.

Documentation

1. Required patches for Solaris, Java and Sun Fire firmware:
   
   BigAdmin tech note: 2007 Daylight Saving Time Changes in the U.S
   
   
   
2. The above tech note is referring to a Sun alert, 102178, which you can find on sunsolve web site at:
   
   Daylight Saving Time (DST) Changes for Australia (2006), United States (2007) and Others
   
   
   
3. Required firmware for timezone changes due to energy policy act of 2005 is at:
   
   Required Server Firmware for Timezone Changes Due to U.S. Energy Policy Act of 2005

In case if Sun didn't address some areas which may get impacted with the changes mentioned in energy policy act of 2005, feel free to file a bug at bugs.sun.com (Java bugs) or at bugs.opensolaris.org (Solaris bugs).
______________
Technorati tags:
Solaris | OpenSolaris | Java | Sun Fire | Sun Microsystems

Read More

Oracle: Lost Redo Log

Scenario:

The disk that is holding the most up-to-date redo log files experienced a fatal failure, and the database wouldn't come up any more. It complains about missing redo logs. So, you copy the redo log files from a recent backup; but still the database won't come up with error messages like ORA-00314: log 1 of thread 1, expected sequence# doesn't match.

SQL> startup
ORACLE instance started.

Total System Global Area 4294967296 bytes
Fixed Size                  1306568 bytes
Variable Size            2534150200 bytes
Database Buffers         1728053248 bytes
Redo Buffers               31457280 bytes
Database mounted.
ORA-00314: log 1 of thread 1, expected sequence#  doesn't match
ORA-00312: online log 1 thread 1: '/opt/oracle/oradata_4/RedoLogFiles/log1.dbf'

How to bring up the database with no redo logs?

In production environments, usually archive logging will be turned on. If that's the case, consult the Oracle system administration documentation to restore the lost transactions.

In all other cases it is very unlikely to recover the lost transactions. So the easiest way to bring up the database is to rebuild the control file. The steps to rebuild the control file with new redo log files are as follows:

1. Start the database up.
   
   SQL> startup <- let it complain about online redo log
   
   
   
2. Run the following command to dump the control file into a trace file.
   

   SQL> alter database backup controlfile to trace;
   Database altered.

   
   Go to udump location and locate the file which contains the necessary SQL text to build the control file. The easiest way is to grep for text like 'CREATE CONTROLFILE REUSE DATABASE'. Once you locate the trace file, change the extension from trc to sql and change NORESETLOGS to RESETLOGS. That is, a SQL like:
   
   CREATE CONTROLFILE REUSE DATABASE "XYZ" NORESETLOGS NOARCHIVELOG
   
   should be changed to
   
   CREATE CONTROLFILE REUSE DATABASE "XYZ" RESETLOGS NOARCHIVELOG
   
   Remove the lines starting from the line "RECOVER DATABASE USING BACKUP CONTROLFILE" to the end.
   
   
   
3. Shut down the database
   
   
   
4. Add the parameter _allow_resetlogs_corruption=TRUE to the init.ora file.
   
   
   
5. Run the script that you just modified -- it creates a new control file.
   
   
   
6. Shut down the database one more time
   
   
   
7. Start the database instance by running startup command as sysdba. It may fail with the error ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
   
   
   
8. Open the database with by running alter database open resetlogs;. Redo log files will be automatically created in this step.
   
   
   
9. Shutdown the database, remove the parameter _allow_resetlogs_corruption=TRUE from the init.ora file and restart the database instance. If the instance comes up fine, shut it down and take a backup. Otherwise, check the alert messages carefully and act accordingly to fix the things up.
   
   
   

Acknowledgements:
Akiko Marti

________________
Technorati tags:
Oracle

Read More