OBIEE 11g: Resolving Presentation Services Startup Failure

ISSUE:

Starting Presentation Services fail with the error:

[OBIPS] [ERROR:1] [] [saw.security.odbcuserpopulationimpl.getbisystemconnection] [ecid: ] [tid: ] Authentication Failure.
Odbc driver returned an error (SQLDriverConnectW).
State: 08004.  Code: 10018.  [NQODBC] [SQL_STATE: 08004] [nQSError: 10018] Access for the requested connection is refused.
[nQSError: 43113] Message returned from OBIS.
[nQSError: 43126] Authentication failed: invalid user/password. (08004)[[

Also connecting to the metadata repository (RPD) in online mode fails with similar error.

Looking through the BI server log, nqserver.log, you may find an error message similar to the following:

[OracleBIServerComponent] [ERROR:1] [] [] [ecid: 0001J1LfUetFCC3LVml3ic0000pp000000] [tid: 1] 
[13026] Error in getting roles from BI Security Service:  
'Error Message From BI Security Service: [nQSError: 46164] HTTP Server returned 404 (Not Found) for URL .' ^M

RESOLUTION:

• Connect to WebLogic Server (WLS) Console -> Deployments. Ensure that all deployed components are in 'Active' state.

• If any of the components is in 'Prepared' state, select that application and then click on "start servicing all requests"

• Restart BI Server and Presentation Services

In some cases, the following additional step might be needed to resolve the issue.

• Access the Enterprise Manager Fusion Middleware control: http://<host.domain>:port/em

• Navigate to Business Intelligence -> coreapplication

• 'Capacity Management' tab -> 'Scalability' sub-tab

• Click on 'Lock and Edit Configuration' button

• Enter the IP address in the 'Listen Address' field

• Click on 'Activate Changes' followed by 'Release Configuration' buttons
  

• Restart BI Server and Presentation Services

Also check these My Oracle Support (MOS) documents for more clues and information.

1387283.1 Authentication failed: invalid user/password
1251364.1 Error: "[nQSError: 10018] Access .. Refused. [nQSError: 43126] Authentication Failed .." when Installing OBIEE 11g
1410233.1.1 How To Bind Components / Ports To A Specific IP Address On Multiple Network Interface (NIC) Machines

Read More

Solaris Volume Manager (SVM) on Solaris 11

SVM is not installed on Solaris 11 by default.

# metadb
-bash: metadb: command not found

# /usr/sbin/metadb
-bash: /usr/sbin/metadb: No such file or directory

Install it using pkg utility.

# pkg info svm
pkg: info: no packages matching the following patterns you specified are
installed on the system.  Try specifying -r to query remotely:

        svm

# pkg info -r svm
          Name: storage/svm
       Summary: Solaris Volume Manager
   Description: Solaris Volume Manager commands
      Category: System/Core
         State: Not installed
     Publisher: solaris
       Version: 0.5.11
 Build Release: 5.11
        Branch: 0.175.0.0.0.2.1
Packaging Date: October 19, 2011 06:42:14 AM 
          Size: 3.48 MB
          FMRI: pkg://solaris/storage/svm@0.5.11,5.11-0.175.0.0.0.2.1:20111019T064214Z

# pkg install storage/svm
           Packages to install:   1
       Create boot environment:  No
Create backup boot environment: Yes
            Services to change:   1

DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1     104/104      1.6/1.6

PHASE                                        ACTIONS
Install Phase                                168/168 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 

# which metadb
/usr/sbin/metadb

This time metadb may fail with a different error.

# metadb
metadb: <HOST>: /dev/md/admin: No such file or directory

Check if md.conf exists.

# ls -l  /kernel/drv/md.conf 
-rw-r--r--   1 root     sys          295 Apr 26 15:07 /kernel/drv/md.conf

Dynamically re-scan md.conf so the device tree gets updated.

# update_drv -f md

# ls -l  /dev/md/admin
lrwxrwxrwx   1 root root 31 Apr 20 10:12 /dev/md/admin -> ../../devices/pseudo/md@0:admin

# metadb
metadb: <HOST>: there are no existing databases

Now Solaris Volume Manager is ready to use.

eg.,
#  metadb -f -a c0t5000CCA00A5A7878d0s0

# metadb
        flags           first blk       block count
     a        u         16              8192          /dev/dsk/c0t5000CCA00A5A7878d0s0

Read More

Resolving "PLS-00201: identifier 'DBMS_SYSTEM.XXXX' must be declared" Error

Here is a failure sample.

SQL> set serveroutput on
SQL> alter package APPS.FND_TRACE compile body;

Warning: Package Body altered with compilation errors.

SQL> show errors
Errors for PACKAGE BODY APPS.FND_TRACE:

LINE/COL ERROR
-------- -----------------------------------------------------------------
235/6    PL/SQL: Statement ignored
235/6    PLS-00201: identifier 'DBMS_SYSTEM.SET_EV' must be declared
..

By default, DBMS_SYSTEM package is accessible only from SYS schema. Also there is no public synonym created for this package. So, the solution is to create the public synonym and grant "execute" privilege on DBMS_SYSTEM package to all database users or a specific user.

eg.,

SQL> CREATE PUBLIC SYNONYM dbms_system FOR dbms_system;

Synonym created.

SQL> GRANT EXECUTE ON dbms_system TO APPS;

Grant succeeded.

- OR -

SQL> GRANT EXECUTE ON dbms_system TO PUBLIC;

Grant succeeded.

SQL>  alter package APPS.FND_TRACE compile body;

Package body altered.

Note that merely granting execute privilege is not enough -- creating the public synonym is as important to resolve this issue.

Read More

Oracle RDBMS & Solaris : Few Random Tips (Feb 2012)

These tips are just some quick solutions or workarounds. Use these quickies at your own risk.

[#1] Oracle Data Pump

Q: How to exclude the table definition while importing a table using Oracle Data Pump import utility?

A: Use EXCLUDE=TABLE/TABLE option.

eg.,

impdp login/password DUMPFILE=<DUMP_FILENAME> LOGFILE=<LOGFILE_NAME> \
 DIRECTORY=<DB_DIR_NAME> TABLES=<TABLE_NAME> EXCLUDE=TABLE/TABLE

---

[#2] Workaround to ORA-01089: immediate shutdown in progress - no operations are permitted

When the database is in the middle of an instance shutdown, if another shutdown or startup was attempted, Oracle RDBMS may throw the above ORA-01089 error. The workaround is to force Oracle to start the database instance using startup force option. This option will shutdown the database instance (if running) using the abort command and then starts it up.

eg.,

SQL> STARTUP FORCE

---

[#3] Quick steps to upgrade the Oracle database from version 11.2.0.[1 or 2] to 11.2.0.3

Execute the following in the same sequence as sysdba.

startup upgrade
!cd $ORACLE_HOME/rdbms/admin
@utlu112i.sql  /* pre-upgrade information tool */
exec dbms_stats.gather_dictionary_stats (DEGREE => 64);
@catupgrd.sql  /* create/modify data dictionary tables */
@utlu112s   /* all components should be in VALID state */
shutdown immediate
startup
@catuppst.sql  /* upgrade actions that do not require DB in UPGRADE mode */
@utlrp.sql  /* recompile stored PL/SQL and Java code */
SELECT count(*) FROM dba_invalid_objects;  
                        /* verify that all packages and classes are valid */
exit

---

[#4] Q: Solaris: how to get rid of zombie processes?

A: Run the following with appropriate user privileges.

ps -eaf | grep defunct | grep -v grep | preap `awk '{ print $2 }'`

Alternative way: (not as good as the previous one - still may work as expected)

prstat -n 500 1 1 | grep zombie | preap `awk '{ print $1 }'`

---

[Added on 03/01/2012]

[#5] Solaris: Many TCP listen drops

eg.,

# netstat -sP tcp | grep tcpListenDrop
        tcpListenDrop       =2442553     tcpListenDropQ0     =     0

To alleviate numerous TCP listen drops, bump up the value for the tunable tcp_conn_req_max_q

# ndd -set /dev/tcp tcp_conn_req_max_q <value>

---

[Added on 03/02/2012]

[#6] Solaris ZFS: listing all properties and values for a zpool

Run: zfs get all <zpool_name> as any OS user

eg.,

% zpool list
NAME    SIZE  ALLOC   FREE    CAP  HEALTH  ALTROOT
rpool   276G   167G   109G    60%  ONLINE  -
spec    556G   168G   388G    30%  ONLINE  -

% zfs get all rpool
NAME   PROPERTY              VALUE                  SOURCE
rpool  type                  filesystem             -
rpool  creation              Fri May 27 17:06 2011  -
...
rpool  compressratio         1.00x                  -
rpool  mounted               yes                    -
rpool  quota                 none                   default
rpool  reservation           none                   default
rpool  recordsize            128K                   default
...
rpool  checksum              on                     default
rpool  compression           off                    default
...
rpool  logbias               latency                default
rpool  sync                  standard               default
rpool  rstchown              on                     default

---

[#7] Solaris: listing all ZFS tunables

Run: echo "::zfs_params" | mdb -k with root/super-user privileges

eg.,

# echo "::zfs_params" | mdb -k
arc_reduce_dnlc_percent = 0x3
zfs_arc_max = 0x10000000
zfs_arc_min = 0x10000000
arc_shrink_shift = 0x5
zfs_mdcomp_disable = 0x0
zfs_prefetch_disable = 0x0
..
..
zio_injection_enabled = 0x0
zvol_immediate_write_sz = 0x8000

Read More

Solaris Tip: How-To Identify Memory Mapped Files

A memory mapped (mmap'd) file is a shared memory object, or a file with some portion or the whole file was mapped to virtual memory segments in the address space of an OS process. Here is one way to figure out if a given object (file or shared memory object) was memory mapped in a process or not.

1. find the file system inode number of the object
2. look for that inode number in the address space of a given process

And here is an example. We are about to check a log file and a shared memory segment in a Siebel object manager's process address space.

# pfiles 8251
8251:   siebmtshmw /siebel/siebsrvr/admin/Siebel81.isve02.s
..
   1: S_IFREG mode:0744 dev:256,65539 ino:246660 uid:1234 gid:30 size:0
      O_WRONLY|O_APPEND|O_CREAT
      /siebel/siebsrvr/enterprises/Siebel81/isve02/log/StdErrOut/stderrout_8251_23311913.log
...
   9: S_IFREG mode:0700 dev:256,65539 ino:246640 uid:1234 gid:30 size:6889472
      O_RDWR|O_CREAT|O_EXCL
      /siebel/siebsrvr/admin/Siebel81.isve02.shm
..

# pmap -sx 8251 | grep 246660    
#               <== stderrout_8251_23311913.log file was not a memory mapped file

# pmap -sx 8251 | grep 246640
F6400000      64      64       -       -   8K r--s-  dev:256,65539 ino:246640   
F6410000     136     136       -       -    - r--s-  dev:256,65539 ino:246640
F6432000     128     128       -       -   8K r--s-  dev:256,65539 ino:246640
...
                <== Siebel81.isve02.shm was a memory mapped object

Read More

Unwanted Software Installers

After all these years of software evolution, it is odd to see not much improvement in the area of software installation. Customers do not seem to mind dealing with different, complex installers. Nevertheless this whole process can be simplified to save time, effort and energy.

In an ideal world, a software installer is supposed to have just one function - copying the software bits to a designated location and nothing else. However today we interact with different installers that does variety of things -- some install the pre-compiled binaries, few come in ready-to-extract zip archives, few others compile the binary on-the-fly and install the binary. Most of the enterprise software installers configure the software as part of the installation process where as few installers install the software and simply quit leaving the configuration step for the experts. Some of the installers hard-code the hostname, IP address, absolute paths of certain files etc., into some of the files on target system, which makes it hard to re-use the software home directory on a different server. Few installers do sensible job by not tying anything to the host system where the software is being installed.

Here is my partial wish list of features for a software installer. I think it is enough to make a point.

— Idempotent installations : install the software once, run anywhere. Customers should be able to move the resulting home directories from one host to any location on another host without worrying about the underlying changes to the location of the home directory, hostname, IP address etc., One example is the Oracle RDBMS installation. Once installed, the ORACLE_HOME can be zipped up, moved to another host, extracted and used right away. ORACLE_HOME usually contains the binaries. Installation specific configuration is stored outside of ORACLE_HOME. Optional Oracle Grid Control configuration appears to be saved under ORACLE_HOME, which is an aberration though it can be easily reconfigured once moved to another host.

— Simplicity : providing the entire directory structure in an extractable compressed archive file will remove one or more layers of dependency that the software installer has. For example, some of the installers require Java run-time to show the graphical interface for the installer. I recently encountered an installer executable that has private/unsupported symbols statically linked to it. When those private interfaces were removed in a later version of the operating environment, installer crashed and failed to make any progress. Had the software been provided in an extractable archive, software would have been readily available in the latter case. It appears that Oracle Corporation is moving in the right direction by releasing WebLogic 12c software as a zip file.

— De-couple software installation from configuration : there should be clear separation between the installation and configuration. Once the software is in place, relevant folks can always configure the software as directed and needed. The customer just needs a simple tool or script to configure the software.

        => Off-topic: providing a web interface is even better. It gives the flexibility to configure the software from anywhere in the same network.

— Contain everything in a single top-level directory : it makes patching easier even if the top-level directory was moved to a different location or host. No point in spreading the pieces of software into multiple locations anyway. Going back to the example of ORACLE_HOME, one shortcoming in Oracle RDBMS installation is that few directories/files such as oraInventory reside outside of ORACLE_HOME - so, when moving ORACLE_HOME to another host, it is necessary to move all relevant files that are outside of ORACLE_HOME as well for successful database software patching.

With careful planning/design, Release Engineering can be as creative and innovative as the rest of the teams in delivering a software product out of the door --- but I guess it is up to the customers to demand that attitude.

PS:
This blog post can be improved a lot. However since it is mostly about an opinion and a wish list, there is not much motivation to put more effort into it. And of course it is a generic discussion - nothing specific to a particular software or corporation.

Read More

Oracle Application Testing Suite (OATS): Few Tips & Tricks

OATS is a suite of applications that can be used for performance and scalability testing, functional and regression testing. It is a thin client application that runs within a web browser - so, it is easy to use the tool from anywhere as long as the web server running on the host node is accessible. Hopefully the following tips and tricks will benefit some of the users of the Oracle Application Testing Suite.
Few technical details first - OATS is a 32-bit Java application that runs in a WebLogic container (WLS) with Oracle XE database being the backend store for test session data.

---

[Trick] Issue : OATS software fails to install on 64-bit Windows systems
Resolution:
Download and install 64-bit .NET framework manually before installing the OATS software. Look for .NET framework on Microsoft's downloads website.

---

[Trick] Issue : OATS software fails to install on systems with large number of [virtual] CPUs
Resolution:
On systems with many cores/vCPUs, Oracle database in general requires large amounts of memory to be configured for SGA - so, one solution would be to allocate as much memory as required. However Oracle XE limits the memory utilization within the database to 1 GB. Besides, XE uses only one CPU even if there are multiple CPUs available on a system. Hence one workaround is to limit the number of vCPUs that the system exposes during the installation of OATS software. The steps are shown below.

• Start button -> Run -> type "msconfig"
• Click on Boot tab -> Advanced Options
• Check "Number of processors" and set appropriate value (I believe we can go up to 16)
• Reboot Windows
• Uninstall failed OATS installation and try installing again
• Undo the above made changes after the successful installation of OATS
• Reboot Windows one final time

Thanks to my colleague Bao Doan for providing this workaround.

---

[Trick] Issue : During runtime, OATS drive the load and executes the test as expected but fails to collect runtime statistics
Resolution:
This is another limitation of Oracle XE database. Until 10g, XE limits the maximum amount of user data in the database to 4 GB. This limit was raised to 11 GB in release Oracle 11g XE. OATS 9.x releases bundle Oracle 10g XE. To take advantage of the larger limit for data, install Oracle 11g XE manually before installing OATS software. OATS installer gives the option to use an existing installation of Oracle XE. Besides, it is not possible to have multiple Oracle XE installations on a single box anyway (that's another XE limitation).
For existing installations, one workaround is to remove old and unwanted sessions to make room for new sessions in the database. Listed below are the steps.

• Connect to the Oracle Load Testing (OLT) tool
• Click on "Manage" top-level menu (upper right corner) -> Sessions
• Click on any unwanted session and press "Delete" button (I recommend deleting one session at a time)

---

[Trick] Issue : Under load, there are many network timeouts with ton of sockets in TIME_WAIT state on OATS agent systems including the OATS Controller node
Resolution:
Tune TCP/IP parameters on Windows as shown below.

• Launch Windows registry
• Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TcpIP\parameters
• Configure the following two parameters. If not found, create those parameters by selecting Edit -> New -> DWORD Value from the menu bar. Select "Decimal" under Base.
  TcpTimedWaitDelay : 30 [seconds] MaxUserPort : 65534
• Reboot Windows

Thanks to my colleagues Dino and Vishnu for sharing this workaround.

---

[Trick] Issue : OATS Controller does not show any graphs or analysis reports
Resolution:
Install Adobe Flash Plugin and try again.

---

[Trick] Issue : Under load, OATS Controller stops collecting runtime statistics at some random point
Resolution:
Check Oracle database alert log for some clue(s). If there is an error message such as "ORA-12516: TNS:listener could not find available handler with matching protocol stack", connect to the database, query v$resource_limit view and compare the values reported under CURRENT_UTILIZATION and MAX_UTILIZATION for the resource "processes". If the current utilization is pretty close to the configured maximum value, raise the value for processes parameter in [S]PFILE.

---

[Tip] Balancing the load among multiple OATS agent systems
One simple way is to create a VU Agent System Group based on the available agent systems. Steps listed below.

• Connect to the Oracle Load Testing (OLT) tool
• Click on "Manage" top-level menu (upper right corner) -> Systems
• Click on "VU Agent System Group" in the left hand side
• On the right hand side, click on "New" option
• Select all the agent systems that you want to be part of the "VU Agent System Group"
• Finally name the newly created system group and save

Note that it is not possible to attach weights to the agent systems - so, it is suggested to have agent systems with similar hardware configurations in the VU Agent System Group.

---

[Tip] Balancing the load among multiple web servers using OATS Controller
If there are multiple web server instances running in a enterprise application deployment; and OATS software is being used to test the performance and scalability of the application, parameterizing the web server hostname and port number in OATS test script will take care of the web server load balancing problem. Of course there are many alternatives to this approach such as using a hardware load balancer, using web server Reverse Proxy etc.,

---

[Added on 01/19/2012]

[Tip] How-To check the available space in USERS tablespace?
Run the following on OATS Controller node:
Start -> All Programs -> Oracle Database XX Express Edition -> Run SQL Command Line

SQL> connect / as sysdba

SQL> SELECT /* + RULE */  df.tablespace_name "Tablespace",
       df.bytes / (1024 * 1024) "Size (MB)",
       SUM(fs.bytes) / (1024 * 1024) "Free (MB)",
       Nvl(Round(SUM(fs.bytes) * 100 / df.bytes),1) "% Free",
       Round((df.bytes - SUM(fs.bytes)) * 100 / df.bytes) "% Used"
  FROM dba_free_space fs,
       (SELECT tablespace_name,SUM(bytes) bytes
          FROM dba_data_files
         GROUP BY tablespace_name) df
 WHERE fs.tablespace_name (+)  = df.tablespace_name
 GROUP BY df.tablespace_name,df.bytes
UNION ALL
SELECT /* + RULE */ df.tablespace_name tspace,
       fs.bytes / (1024 * 1024),
       SUM(df.bytes_free) / (1024 * 1024),
       Nvl(Round((SUM(fs.bytes) - df.bytes_used) * 100 / fs.bytes), 1),
       Round((SUM(fs.bytes) - df.bytes_free) * 100 / fs.bytes)
  FROM dba_temp_files fs,
       (SELECT tablespace_name,bytes_free,bytes_used
          FROM v$temp_space_header
         GROUP BY tablespace_name,bytes_free,bytes_used) df
 WHERE fs.tablespace_name (+)  = df.tablespace_name
 GROUP BY df.tablespace_name,fs.bytes,df.bytes_free,df.bytes_used
 ORDER BY 4 DESC;

Copy/paste the above SQL code in a text file with sql extension and execute that SQL statement by calling the SQL script from SQL> command prompt. eg., assuming the above code was saved in a plain text file called chktblspcusg.sql under C:\ drive, execute the SQL script as shown below:
SQL> @C:\chktblspcusg.sql

---

[Added on 06/27/2012]

[Trick] Issue : An attempt to open a test script in OpenScript fails with error

'Failed to open script' has encountered a problem.
Failed to open . See error log for details.

Clicking on "Details" button provides the following clue.

The project description file (.project) for '' is missing"

In addition the title bar shows "Relocating Eclipse Projects: The project description file (.project) for XXX is missing".

Resolution:
Navigate to C:\Documents and Settings\Administrator\osworkspace\.metadata\.plugins\org.eclipse.core.resources\.projects\

Look for the directory by name "<failing_script_name>" and remove it

---

[Added on 08/03/2012]

[Trick] Issue: Unexpected Agent exit. Code = 51 in the middle of an OLT load test

When running a load scenario in Oracle Load Testing (OLT) that uses a databank, the scenario runs fine for some time and then all of a sudden fails with the following error: Unexpected Agent exit. Code = 51.

Workaround:

The following settings may alleviate the issue.

• - toggle/experiment with the settings for "Clear cache between iterations" and "Clear cache before playing back"
  
  • those settings can be found under the test script preferences -> Playback -> Web Functional -> Miscellaneous
• - experiment with different values for "Maximum users per process" setting
  
  • this setting is under OLT -> Configure all parameters -> Advanced
• - increase the Java heap size (both min & max) in file <OATS_HOME>\agentmanager\bin\AgentManagerService.conf
  
  • default values: min heap size: 16 MB; max heap size: 64 MB

Contributors: John Snyder, Richard Barry

[Added 02/25/13]

Another colleague Dave Suri has an alternate tip to resolve the Agent 51 issue.

Edit <OATS_HOME>\agentmanager\processDescriptors\JavaAgent.properties

Change the following lines:

#process.debug=y
#process.debug.suspend=y
#process.debug.port=8123
#process.debug.custom=

To:

process.debug=y
#process.debug.suspend=y
#process.debug.port=8123
process.debug.custom=-verbose:gc -XX:+HeapDumpOnOutOfMemoryError -Xms512M 
-Xmx1536M -jrockit -Xrs -XgcPrio:deterministic -XpauseTarget=50ms 
-XX:+UseCallProfiling -XX:+UseAdaptiveFatSpin -XX:+ExitOnOutOfMemoryError 
-XXnoSystemGC -XX:+UseFastTime

---

See Also:

• Oracle Application Testing Suite
• Oracle Database XE Licensing Restrictions
• Check the documentation that is available as part of OATS installation

Read More