Monday, August 26, 2019

Migrating Single Node Apps/EBS R12 DB to RAC

Being running on single node for a while for EBS, it was the time to test integration / migration of EBS to HA on DB side i.e. to 11gR2 RAC. As you are aware that R12 is packed with 11.1.0.7 aka 11gR1 binaries. So you have to first migrate them to 11gR2 Binaries.

1. The migration of single node DB to 11gR2 binaries

2. Migrate the 11gR2 EBS DB to RAC Grid database aka multinode env.

This post assumes that you have installed and configured the 11gR2(11.2.0.1) Grid Home and 11gR2 RAC RDBMS Home.


Following are the high level steps for the entire process.


1) Install and configure 11.2.0.1 cluster on 2 nodes

2) Install 11gR2 RDBMS Oracle Home on 2 nodes
3) Install Application patches
4) convert non-rac database to RAC database
5) Enable autoconfig on database and application nodes
6) Startup services and verify services

For the sake of readability I have not covered the step 1 and step 2, as they are the SOP for any RAC setup. Also you will find numerous blogs and Oracle Documentation on the same, so no point in keep repeating :)
I will cover steps 3 to 6 in this write up 
On Node 1 - 
[oracle@apps_rac01 old]$ pwd
/u01/app/oracle/product/11.0/db_1/nls/data/old
[oracle@apps_rac01 old]$ perl cr9idata.pl 
Directory /u01/app/oracle/product/11.0/db_1/nls/data/9idata already exist. Overwriting...
Copying files to /u01/app/oracle/product/11.0/db_1/nls/data/9idata...
Copy finished. 
Please reset environment variable ORA_NLS10 to /u01/app/oracle/product/11.0/db_1/nls/data/9idata!
[oracle@apps_rac01 old]$ 

On Node 2- 
[oracle@apps_rac02 old]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/nls/data/old
[oracle@apps_rac02 old]$ perl ./cr9idata.pl 
Creating directory /u02/app/11.2.0/grid/nls/data/9idata ...
Copying files to /u02/app/11.2.0/grid/nls/data/9idata...
Copy finished. 
Please reset environment variable ORA_NLS10 to /u02/app/11.2.0/grid/nls/data/9idata!

Listener config - 
-- Create new listener (make sure you check on Node2 also)
[oracle@apps_rac01 bin]$ ./srvctl  add  listener -l listener_visr12 -o /u02/app/oracle/11.2.0/db_1 -p 1521
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: /u02/app/oracle/product/11.2.0/db_1
End points: TCP:1521

SCAN Listener - 

[oracle@apps_rac01 db_1]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  start scan_listener
[oracle@apps_rac01 bin]$ ./srvctl  status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node apps_rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node apps_rac02

-- Check env on both nodes 

On Node 1- 

[oracle@apps_rac01 old]$ env | grep ORACLE_HOME

ORACLE_HOME=/u01/app/oracle/product/11.0/db_1
[oracle@apps_rac01 old]$ env | grep ORA_NLS10
ORA_NLS10=/u01/app/oracle/product/11.0/db_1/nls/data/9idata
[oracle@apps_rac01 old]$ env | grep ORACLE_BASE
ORACLE_BASE=/u01/app/oracle
[oracle@apps_rac01 old]$ env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/u01/app/oracle/product/11.0/db_1/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/oracle/product/11.0/db_1/lib:/usr/dt/lib:/u01/app/oracle/product/11.0/db_1/ctx/lib
On Node 2- 
[oracle@apps_rac02 old]$ env | grep ORACLE_HOME
ORACLE_HOME=/u02/app/11.2.0/grid
[oracle@apps_rac02 old]$ env | grep PATH
LD_LIBRARY_PATH=/u02/app/11.2.0/grid/lib:/lib:/usr/lib
PATH=/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/oracle/bin:/u02/app/11.2.0/grid/bin
[oracle@apps_rac02 old]$ env | grep ORA_NLS10
ORA_NLS10=/u01/app/oracle/product/11.0/db_1/nls/data/9idata
[oracle@apps_rac02 old]$ env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/u01/app/oracle/product/11.0/db_1/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/oracle/product/11.0/db_1/lib:/usr/dt/lib:/u01/app/oracle/product/11.0/db_1/ctx/lib

Migration - 
-- shutdown the db and create spfile on ASM
SQL> create spfile='+DATA/spfile/spfilevisr12.ora' from pfile='/u01/app/oracle/product/11.0/db_1/dbs/initvisr12.ora';
File created.
-- Take backup of existing pfile and create new pfile as follows

[oracle@apps_rac01 dbs]$ pwd
/u01/app/oracle/product/11.0/db_1/dbs
[oracle@apps_rac01 dbs]$ scp initvisr12.ora initvisr12.ora.bkp
-- - edit the local pfile in old ORACLE_HOME and provide the location of spfile.

[oracle@apps_rac01 dbs]$ cat initvisr12.ora
spfile='+DATA/spfile/spfilevisr12.ora'
-- Startup the DB, it will now use the SPFILE from ASM Disk to start.
[oracle@apps_rac01 dbs]$ !sqlp
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 31 08:44:59 2012
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup 
ORACLE instance started.
Total System Global Area 1071333376 bytes
Fixed Size                  1341312 bytes
Variable Size             536873088 bytes
Database Buffers          520093696 bytes
Redo Buffers               13025280 bytes
Database mounted.
Database opened.
SQL > Show parameter spfile
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /u01/app/oracle/product/11.0/db_1/dbs/spfilevisr12.ora                                                      

Here it should have displayed the spfile from +DATA disk group but it was showing as from cooked file system.
The reason being is that, I already had an spfile under dbs on old Oracle Home. So I just renamed it and start the DB with pfile option and forcing DB to use spfile from +DATA Disk group as follows
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@apps_rac01 dbs]$ mv  spfilevisr12.ora  spfilevisr12.ora.bkp
[oracle@apps_rac01 dbs]$ !sqlp
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 31 08:51:06 2012
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup pfile='initvisr12.ora'
ORACLE instance started.
Total System Global Area 1071333376 bytes
Fixed Size                  1341312 bytes
Variable Size             490735744 bytes
Database Buffers          566231040 bytes
Redo Buffers               13025280 bytes
Database mounted.
Database opened.
SQL> sho parameter spfile
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/spfile/spfilevisr12.ora

So far so good. Now comes the most interesting part of the process i.e. Converting the Non RAC database to RAC database. There are several ways we can achieve this 
1. DBCA
2. rconfig
3. Oracle Enterprise Manager
4. Manually (old school way )

For this post, I am going to use rconfig, as its the more sophisticated way and less error prone to OEM. 
So first of all you have copy the convert template from default directory to your custom directory. Since I want to convert this DB as an Admin manged RAC DB, i will use ConvertToRAC_AdminManaged.xml template.
[oracle@apps_rac01 sampleXMLs]$ pwd
/u01/app/oracle/product/11.0/db_1/assistants/rconfig/sampleXMLs
[oracle@apps_rac01 sampleXMLs]$ ll
total 8

-rw-r--r-- 1 oracle dba 2591 Mar  4  2009 ConvertToRAC_AdminManaged.xml

-rw-r--r-- 1 oracle dba 2653 Mar 12  2009 ConvertToRAC_PolicyManaged.xml

[oracle@apps_rac01 sampleXMLs]$ scp ConvertToRAC_AdminManaged.xml /home/oracle/



Open the ConvertToRAC_AdminManaged.xml and make following changes based on your environment.

Before starting actual conversion, it is advisable to check the conversion process beforehand. So I will run the rconfig utility with ONLY option to check whether my conversion will work flawlessly or not. 



[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>

    <Convert>

      <Response>

        <Result code="1" >

          Got Exception

        </Result>
       <ErrorDetails>
             oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: SCAN Listener is not running.
Operation Failed. Refer logs at /u01/app/oracle/cfgtoollogs/rconfig/rconfig_07_31_12_09_23_49.log for more details.
       </ErrorDetails>
      </Response>
    </Convert>
  </ConvertToRAC></RConfig>

Since my SCAN Listeners were down, error was thrown as above. So its time to start SCAN Listener and run the utility again.

[oracle@apps_rac01 db_1]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  start scan_listener
[oracle@apps_rac01 bin]$ ./srvctl  status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node apps_rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node apps_rac02
<ErrorDetails>
             oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: Default Listener is not configured in Grid Infrastructure Home.
Operation Failed. Refer logs at /u01/app/oracle/cfgtoollogs/rconfig/rconfig_07_31_12_09_27_28.log for more details.
   </ErrorDetails>

The issue was that my listener was running from RAC ORACLE_HOME. The default listener has to running from GRID Home. 
-- First Removed the existing listener
[oracle@apps_rac01 bin]$ cd /u02/app/oracle/prodcut/11.2.0/db_1/bin/
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: /u02/app/oracle/prodcut/11.2.0/db_1
End points: TCP:1521
[oracle@apps_rac01 bin]$ srvctl  stop listener
[oracle@apps_rac01 bin]$ srvctl remove listener -l listener_visr12
[oracle@apps_rac01 bin]$ cd -
/u02/app/11.2.0/grid/bin

-- Create new listener from Grid Home
[oracle@apps_rac01 bin]$ ./srvctl  add  listener -l listener_visr12 -o /u02/app/11.2.0/grid -p 1521
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: <CRS Home>
End points: TCP:1521
[oracle@apps_rac01 bin]$ ssh apps_rac02
Last login: Tue Jul 31 02:54:28 2012 from apps_rac01.localdomain
[oracle@apps_rac02 ~]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac02 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: <CRS Home>
End points: TCP:1521

-- However after creating listener it still failed. After googling a bit, I found workaround as follows.
[oracle@apps_rac01 bin]$ pwd
/u02/app/11.2.0/grid/bin
[oracle@apps_rac01 bin]$ ./srvctl remove listener -l listener_visr12
Connection to apps_rac02 closed.
-- Remove the listener with custom name 
[oracle@apps_rac01 bin]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base for ORACLE_HOME=/u02/app/11.2.0/grid is /u01/app/oracle
-- Start the NETCA from grid home create the listener with default name "LISTENER"
[oracle@apps_rac01 bin]$ netca
Oracle Net Services Configuration:
Oracle Net Configuration Assistant is launched from Grid Infrastructure home. Ne
twork configuration will be clusterwide.
Configuring Listener:LISTENER
apps_rac01...
apps_rac02...
Listener configuration complete.
Oracle Net Listener Startup:
    Listener started successfully.
Oracle Net Services configuration successful. The exit code is 0
[oracle@apps_rac01 bin]$ ./srvctl  status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): apps_rac01,apps_rac02

With this RCONFIG went ahead, just to give another error. 
 <ErrorDetails>
             /u01/app/oracle/product/11.0/db_1/dbs/arch LOG ARCHIVE DEST does not exist on all nodes in the cluster
       </ErrorDetails>

Workaround was pretty simple this time and just created the arch directory on both nodes and there I was done.
RCONFIG didn't complain this time.
[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
    <Convert>
      <Response>
        <Result code="0" >
          Operation Succeeded
        </Result>
      </Response>
      <ReturnValue type="object">
There is no return value for this step     </ReturnValue>
    </Convert>
  </ConvertToRAC></RConfig>

Being said that, it's time to run RCONFIG with YES option to do the actual conversion and here I go 
[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
Converting Database "visr12" to Cluster Database. Target Oracle Home: /u02/app/oracle/prodcut/11.2.0/db_1. Database Role: PRIMARY.
Setting Data Files and Control Files
Adding Database Instances
Adding Redo Logs
Enabling threads for all Database Instances
Setting TEMP tablespace

Adding UNDO tablespaces

Adding Trace files

Setting Flash Recovery Area

Updating Oratab

Creating Password file(s)
Configuring Listeners
Configuring related CRS resources
Starting Cluster Database
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
    <Convert>
      <Response>
        <Result code="0" >
          Operation Succeeded
        </Result>
      </Response>
      <ReturnValue type="object">
<Oracle_Home>
         /u02/app/oracle/prodcut/11.2.0/db_1
       </Oracle_Home>
       <Database type="ADMIN_MANAGED"  >
 <InstanceList>
           <Instance SID="visr121" Node="apps_rac01"  >
           </Instance>
           <Instance SID="visr122" Node="apps_rac02"  >
           </Instance>
         </InstanceList>
       </Database>     </ReturnValue>
    </Convert>
  </ConvertToRAC></RConfig>

[oracle@apps_rac01 ~]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  config database -d visr12
Database unique name: visr12
Database name: visr12

Oracle home: /u02/app/oracle/prodcut/11.2.0/db_1

Oracle user: oracle

Spfile: +DATA/spfilevisr12.ora

Domain: 

Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: visr12
Database instances: visr121,visr122
Disk Groups: DATA
Services: 
Database is administrator managed

-- Copy appsutil directory from source ORACLE HOME to destination ORACLE HOME (on both nodes)
[oracle@apps_rac01 bin]$ scp -r appsutil /u02/app/oracle/prodcut/11.2.0/db_1/
[oracle@apps_rac01 bin]$ scp -r appsutil apps_rac02:/u02/app/oracle/prodcut/11.2.0/db_1/

-- Copy network/admin folder from source ORACLE HOME to destination ORACLE HOME  (on both nodes)
-- Change the path of ORACLE HOME from old to new (RAC) and modify the INSTANCE_NAME parameter on the respective nodes
[oracle@apps_rac01 bin]$ scp -r visr12_apps_rac01/ /u02/app/oracle/prodcut/11.2.0/db_1/network/admin/
[oracle@apps_rac01 bin]$ scp -r visr12_apps_rac01/ apps_rac02:/u02/app/oracle/prodcut/11.2.0/db_1/network/admin/visr12_apps_rac02/

-- You may also have to add Alias using NETCA (Run from GI Home)
Node 1 - 
[oracle@apps_rac01 bin]$ lsnrctl status listener
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 31-JUL-2012 13:47:16
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                31-JUL-2012 13:45:50
Uptime                    0 days 0 hr. 1 min. 25 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/apps_rac01/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.71)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.72)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "visr12" has 1 instance(s).

  Instance "visr121", status READY, has 1 handler(s) for this service...

The command completed successfully

Node 2 - 
[oracle@apps_rac02 visr12_apps_rac02]$ lsnrctl status listener
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 31-JUL-2012 08:46:09
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                31-JUL-2012 04:32:38
Uptime                    0 days 4 hr. 13 min. 33 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/apps_rac02/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.73)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.74)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "visr12" has 1 instance(s).
  Instance "visr122", status READY, has 1 handler(s) for this service...
The command completed successfully

-- Create context file with adbldxml.pl utility on both DB Nodes


[oracle@apps_rac01 bin]$ hostname
apps_rac01.localdomain
[oracle@apps_rac01 bin]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/bin

[oracle@apps_rac01 bin]$ perl adbldxml.pl appsuser=apps appspass=apps

The log file for this adbldxml session is located at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/adbldxml_07311354.log
AC-20010: Error: File - listener.ora could not be found at the location: 
        /listener.ora 
indicated by TNS_ADMIN. Context file can not be generated.
Could not Connect to the Database with the above parameters, Please answer the Questions below
Enter Hostname of Database server: 
Enter Hostname of Database server: apps_rac01            
Enter Port of Database server: 1521

Enter SID of Database server[visr12]:visr121
The context file has been created at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr121_apps_rac01.xml



[oracle@apps_rac02 bin]$ hostname 
apps_rac02.localdomain
[oracle@apps_rac02 bin]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/bin

[oracle@apps_rac02 bin]$ perl ./adbldxml.pl appsuser=apps appspass=apps
Starting context file generation for db tier..
Using JVM from /u02/app/oracle/prodcut/11.2.0/db_1/jdk/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/adbldxml_07310853.log

AC-20010: Error: File - listener.ora could not be found at the location: 

        /listener.ora 

indicated by TNS_ADMIN. Context file can not be generated.

Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: apps_rac02
Enter Port of Database server: 1521
Enter SID of Database server: visr122
Enter the value for Display Variable: 0
The context file has been created at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr122_apps_rac02.xml

-- Make sure your LOCAL_LISTENER are registered with DB 
Node 1 - 


NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      visr121
SQL> sho parameter listener
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------

listener_networks                    string

local_listener                       string      (DESCRIPTION=(ADDRESS_LIST=(AD

                                                 DRESS=(PROTOCOL=TCP)(HOST=apps

                                                 _rac01-vip)(PORT=1521))))

remote_listener                      string      apps-scan.localdomain:1521

Node 2 - 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      visr122
SQL> sho parameter listener
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
listener_networks                    string
local_listener                       string      (DESCRIPTION=(ADDRESS_LIST=(AD
                                                 DRESS=(PROTOCOL=TCP)(HOST=apps
                                                 _rac02-vip)(PORT=1521))))
remote_listener                      string      apps-scan.localdomain:1521


-- Run AutoConfig on both nodes 
Node 1 - 
[oracle@apps_rac01 bin]$ ./adconfig.sh contextfile=../visr121_apps_rac01.xml 
Enter the APPS user password: 
The log file for this session is located at: /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/visr121_apps_rac01/07311404/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u02/app/oracle/prodcut/11.2.0/db_1
        Classpath                   : :/u02/app/oracle/prodcut/11.2.0/db_1/jdbc/lib/ojdbc5.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java/xmlparserv2.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/netcfg.jar:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/ldapjclnt11.jar
        Using Context file          : /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr121_apps_rac01.xml
Context Value Management will now update the Context file
        Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 32 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.

Node 2 - 
[oracle@apps_rac02 bin]$ ./adconfig.sh contextfile=../visr122_apps_rac02.xml 
Enter the APPS user password: 
The log file for this session is located at: /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/visr122_apps_rac02/07310903/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u02/app/oracle/prodcut/11.2.0/db_1
        Classpath                   : :/u02/app/oracle/prodcut/11.2.0/db_1/jdbc/lib/ojdbc5.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java/xmlparserv2.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/netcfg.jar:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/ldapjclnt11.jar
        Using Context file          : /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr122_apps_rac02.xml
Context Value Management will now update the Context file
  Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 32 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.

-- In DB you can verify both the nodes 
SQL> select node_name from apps.fnd_nodes;
NODE_NAME
------------------------------
APPS_RAC01
APPS_RAC02

-- Check and set the DB Env for TNS_ADMIN
Node 1 - 
[oracle@apps_rac01 bin]$ ./srvctl  getenv database -d visr12
visr12:
[oracle@apps_rac01 bin]$ ./srvctl setenv database -d visr12 -T TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin
[oracle@apps_rac01 bin]$ ./srvctl  getenv database -d visr12
visr12:
TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin
Node 2 - 
[oracle@apps_rac02 bin]$ cd ../../bin/
[oracle@apps_rac02 bin]$ ./srvctl  getenv database -d visr12
visr12:
TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin

-- Modify the CONTEXT FILE on apps tier & add additional node info 
[oracle@appsnode_new appl]$ vi  $CONTEXT_FILE
 <jdbc_url oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=apps_rac01.localdomain)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=apps_rac02.localdomain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=visr12))))</jdbc_url>

-- Run AutoConfig on Apps tier
[oracle@appsnode_new appl]$ adautocfg.sh 
Enter the APPS user password:
The log file for this session is located at: /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/admin/log/07312029/adconfig.log
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
        Using CONFIG_HOME location     : /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new
Classpath                   : /u01/app/oracle/visr12/apps/apps_st/comn/java/lib/appsborg2.zip:/u01/app/oracle/visr12/apps/apps_st/comn/java/classes
     Configuring IES_TOP.......COMPLETED
        Configuring CSD_TOP.......COMPLETED
        Configuring IGC_TOP.......COMPLETED
AutoConfig completed successfully.

-- Start the Apps Tier Services 
[oracle@appsnode_new appl]$ adstrtal.sh apps/apps
You are running adstrtal.sh version 120.15.12010000.3
The logfile for this session is located at /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/appl/admin/log/adstrtal.log
Executing service control script:

Now log-in to apps console and navigate to system administrator responsibility -> OAM -> Database status. You can see now both the nodes available and active as follows.

Set Up Load Balancing On Apps Tier

Now its time to implement load balancing for the Oracle Applications database connections...

Run the Context Editor (through the Oracle Applications Manager interface) and set the value of "Tools OH TWO_TASK" (s_tools_two_task), "iAS OH TWO_TASK"(s_weboh_twotask) and "Apps JDBC Connect Alias" (s_apps_jdbc_connect_alias). 

To load balance the forms based applications database connections, set the value of "Tools OH TWO_TASK" to point to the <database_name>_balance alias generated in the tnsnames.ora file.

To load balance the self-service applications database connections, set the value of "iAS OH TWO_TASK" and "Apps JDBC Connect Alias" to point to the <database_name>_balance alias generated in the tnsnames.ora file.


Execute AutoConfig by running the command
[oracle@appsnode_new appl]$ adautocfg.sh

Restart the Applications processes, using the new scripts generated by AutoConfig.
Ensure that value of the profile option "Application Database ID" is set to dbc file name generated in $FND_SECURE. 

Hope that this post will help you in your Apps to RAC migration successfully. Comments are welcome.

JDBC Connections from E-Business Suite Application Tier Fail with "java.sql.SQLException: Io exception: There is no process to read data written to a pipe." (Doc ID 734293.1)

APPLIES TO:

Oracle Applications Utilities - Version 11.5.10.2 to 12.1.3 [Release 11.5.10 to 12.1]
IBM AIX on POWER Systems (64-bit)
IBM AIX Based Systems (64-bit)
AIX5L Based Systems (64-bit)
Checked for relevance on June 4th, 2018

SYMPTOMS

All the JDBC Thin Client connections from E-Business Suite are waiting long and then failing with error :
...
java.sql.SQLException: Io exception: There is no process to read data written to a pipe.
...
SQL*Plus connections are working fine.

CAUSE

This issue is usually caused by "TimeOut" issue under the following levels :
1. Firewall level timeout
2. SQL*Net level timeout
3. Listener level timeout
Please test the JDBC connection using the following code :

//=======================================================
// Test Code: cut from here
//=======================================================
import java.sql.*;
class oraConn
{
public static void main (String args []) throws SQLException
{
try {
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
java.util.Properties ora_property = new java.util.Properties();
Connection conn =
DriverManager.getConnection("jdbc:oracle:thin:apps/<passwd>@<host>.<domain>:<port>:<SID>");
Statement stmt = conn.createStatement();
ResultSet rset = stmt.executeQuery("select user from dual");
while (rset.next())
System.out.println(rset.getString(1));
} catch (SQLException ex) {

// A SQLException was generated. Catch it and
// display the error information. Note that there
// could be multiple error objects chained
// together.

System.out.println ("\n*** SQLException caught ***\n");

// Walk through the list of SQL exceptions and print
// each SQL exception's information
while (ex != null) {
System.out.println ("SQLState: " + ex.getSQLState ());
System.out.println ("Message: " + ex.getMessage ());
System.out.println ("Vendor: " + ex.getErrorCode ());
ex.printStackTrace();

ex = ex.getNextException ();
System.out.println ("");
}
}
}
}
//=======================================================
// End-of-Code
//======================================================= 

Check if it is able to create a JDBC connection on the server. Monitor it with :
SQL> SELECT sid, serial#, logon_time, program
       FROM v$session
      WHERE program LIKE 'JDBC%';

Check if the test code returns following error after the above session gets killed :
...

*** SQLException caught ***

SQLState: null
Message: Io exception: There is no process to read data written to a pipe.
Vendor: 17002
java.sql.SQLException: Io exception: There is no process to read data written to a pipe.
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:179)
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:334)
at oracle.jdbc.ttc7.TTC7Proto .handleIOException(TTC7Protocol.java:3664)
at oracle.jdbc.ttc7.TTC7Protocol.logon(TTC7Protocol.java:353)
at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:371)
at oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:551)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:351)
at java.sql.DriverManager.getConnection(DriverManager.java:559)
at java.sql.DriverManager.getConnection(DriverManager.java:211)
at oraConn.main(oraConn.java:9) 
...

SOLUTION

To implement the solution, please execute the following steps :

1. Unset any timeout value at network level or firewall level, if any.

2. Modify timeout settings :
listener.ora :
INBOUND_CONNECT_TIMEOUT_LISTENER = 0
sqlnet.ora :
SQLNET.INBOUND_CONNECT_TIMEOUT = 0

3. Re-start the Database and the DB Listener

4. Retest the issue

5. Migrate the solution to appropriate Environment
 

Was this document helpful?

 
   
 

Document Details

 
Email link to this documentOpen document in new windowPrintable Page
PROBLEM
PUBLISHED
Jul 29, 2019
Jul 29, 2019
   
 

Related Products

 
Oracle Applications Utilities
   
 

Document References

 
No References available for this document.
   
Didn't find what you are looking for?

Login Page Hangs With Error In oacore Log: There is no process to read data written to a pipe (Doc ID 1459761.1)

To BottomTo Bottom

In this Document
Symptoms
Changes
Cause
Solution
References


APPLIES TO:

Oracle E-Business Suite Technology Stack - Version 12.1.3 to 12.1.3 [Release 12.1]
Information in this document applies to any platform.

SYMPTOMS

On Oracle Applications release 12.0.6, IAS for Applications Technology

Actual Behavior
The OA_HTML/Applications Login is hanging for end users.

Expected Behavior
New logins can successfully open the login page.

ERROR in the oacore application.log file:
html: There is no process to read data written to a pipe.
  html: Servlet error
java.io.IOException: There is no process to read data written to a pipe.
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
at sun.nio.ch.IOUtil.write(IOUtil.java:103)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:329)
at java.nio.channels.Channels.write(Channels.java:74)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:148)
at com.evermind.server.http.AJPOutputStream.endRequest(AJPOutputStream.java:117)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:306)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:187)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:810)

Error in new connection to database via sqlplus

SQL> conn <apps_user>/<apps_password>
ERROR:
ORA-00020: maximum number of processes (%s) exceeded

CHANGES


CAUSE

This is justified with the following :

1. Result of v$resource_limit:
RESOURCE_NAME CURRENT_UTILIZATION MAX_UTILIZATION INITIAL_ALLOCATION LIMIT_VALUE
processes 1199 1200 1200 1200
sessions 1236 1305 2500 2500

The value of processes setup in database initialization parameter file is 1200, but at the peak time (about 374 users) the max utilization processes is 1200, so no available process to serve new user connection request to database.

There is not enough processes in database to serve the new user connection.

SOLUTION

To implement the solution, please execute the following steps:

1. Increase the value of PROCESS in database initialization parameter file:
(Note : For Oracle recommended value, refer to Section 7: Database Initialization Parameter Sizing in Note 396009.1 )
a. Identify the current value of the processes:
SQL > show parameter processes;

b. The parameter could be changed without shutting down the instance by use of the following command:
SQL> alter system set processes=<the number one wants to set> scope=spfile;

c. Following the command in b above, use the following to confirm the change:
SQL > show parameter processes;

2. Retest the issue.

3. Migrate the solution as appropriate to other environments.

REFERENCES

NOTE:329456.1 - Troubleshooting MOD_OC4J Errors and "oc4j_socket_recvfull timed out" Warning in the HTTP Server error_log File
NOTE:859747.1 - JTF-1400: Region JTFSCHNF Causes JTF Regionexception
NOTE:276557.1 - Firewall BLACKOUT and JDBC connections with Oracle Applications 11i and 12
NOTE:396009.1 - Database Initialization Parameters for Oracle E-Business Suite Release 12
NOTE:734293.1 - JDBC Connections from E-Business Suite Application Tier Fail with "java.sql.SQLException: Io exception: There is no process to read data written to a 

Friday, August 23, 2019

access.log analysis

access log

LOCATION :


$IAS_ORACLE_HOME/Apache/Apache/logs



SIGNIFICANCE :
An access log is a list of all the requests for individual files that people have requested from a Web site. These files will include the HTML files and their imbedded graphic images and any other associated files that get transmitted.



IMPORTANT POINTS :


>> We can modify setting in httpd.conf to rename the access log as below
---------------------------------------------------------------------------------------------------------------------
1- Stop the HTTP Server
2- Make a backup of $ORACLE_HOME/Apache/Apache/conf/httpd.conf:

3- Open the file httpd.conf and search for something like the following line:
...
CustomLog "|D:\oracle\FRM_REP_904\Apache\Apache\bin\rotatelogs.exe logs/access_log 43200" common
...
where ORACLE_HOME=D:\oracle\FRM_REP_904

4 - Modify the httpd.conf file like this example; this will create a new access_log every 24 hours
- 86400 seconds:
...
#
#CustomLog "|D:\oracle\FRM_REP_904\Apache\Apache\bin\rotatelogs.exe logs/access_log 43200" common
#
CustomLog "|D:\oracle\FRM_REP_904\Apache\Apache\bin\rotatelogs.exe logs/access_log.%d%m%Y 86400"
common
#
...

5 - Save the httpd.conf file

----------------------------------------------------------------------------------------------------------------------









FILE :

#########################################################################################################


132.226.187.252 - - [06/Jul/2010:20:00:17 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0

132.226.187.252 
This is the IP address of the client (remote host) which made the request to the server.The IP address reported here is not necessarily the address of the machine at which the user is sitting. If a proxy server exists between the user and the server, this address will be the address of the proxy, rather than the originating machine.

first -

The "hyphen" in the output indicates that the requested piece of information is not available. In this case, the information that is not available is the RFC 1413 identity of the client determined by identd on the clients machine.

second -

This is the userid of the person requesting the document as determined by HTTP authentication. If the document is not password protected, this part will be "-" just like the previous one.

[06/Jul/2010:20:00:17 -0400]

The time that the request was received. The format is:
[day/month/year:hour:minute:second zone]

"GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 

The request line from the client is given in double quotes. The request line contains a great deal of useful information. First, the method used by the client is GET. Second, the client requested the resource ICXINDEX.htm and third, the client used the protocol HTTP/1.0.


200

This is the status code that the server sends back to the client. This information is very valuable, because it reveals whether the request resulted in a successful response (codes beginning in 2), a redirection (codes beginning in 3), an error caused by the client (codes beginning in 4), or an error in the server (codes beginning in 5). 

see the status code details below :
-------------------------------------------------------------------------
Informational 1xx

100 Continue
101 Switching Protocols

Successful 2xx
200 OK
201 Created
202 Accepted
203 Non-Authoritative Information
204 No Content
205 Reset Content
206 Partial Content

Redirection 3xx

300 Multiple Choices
301 Moved Permanently
302 Found
303 See Other
304 Not Modified
305 Use Proxy
306 (Unused)
307 Temporary Redirect
Client Error 4xx 

400 Bad Request
401 Unauthorized
402 Payment Required
403 Forbidden
404 Not Found
405 Method Not Allowed
407 Proxy Authentication Required
408 Request Timeout
409 Conflict
410 Gone
411 Length Required
412 Precondition Failed
413 Request Entity Too Large
414 Request-URI Too Long
415 Unsupported Media Type
416 Requested Range Not Satisfiable
417 Expectation Failed

Server Error 5xx 

500 Internal Server Error
501 Not Implemented
502 Bad Gateway
503 Service Unavailable
504 Gateway Timeout
505 HTTP Version Not Supported
-------------------------------------------------------------------------------

418

The last part indicates the size of the object returned to the client, not including the response headers. If no content was returned to the client, this value will be "-". 






132.226.187.253 - - [06/Jul/2010:20:00:29 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.252 - - [06/Jul/2010:20:00:51 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.253 - - [06/Jul/2010:20:01:02 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.252 - - [06/Jul/2010:20:01:25 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.253 - - [06/Jul/2010:20:01:36 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.252 - - [06/Jul/2010:20:01:59 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.253 - - [06/Jul/2010:20:02:10 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.252 - - [06/Jul/2010:20:02:33 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.253 - - [06/Jul/2010:20:02:44 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.252 - - [06/Jul/2010:20:03:07 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.253 - - [06/Jul/2010:20:03:19 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.252 - - [06/Jul/2010:20:03:41 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.253 - - [06/Jul/2010:20:03:53 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.252 - - [06/Jul/2010:20:04:15 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.253 - - [06/Jul/2010:20:04:26 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0
132.226.187.252 - - [06/Jul/2010:20:04:49 -0400] "GET /OA_HTML/US/ICXINDEX.htm HTTP/1.0" 200 418 0


147.154.160.171 - - [06/Jul/2010:23:24:12 -0400] "GET /OA_HTML/AppsLocalLogin.jsp HTTP/1.1" 200 6659 0

147.154.160.171 - - [06/Jul/2010:23:24:12 -0400] "GET /OA_HTML/cabo/styles/cache/oracle-desktop-2_2_24_3-en-ie-6-windows.css HTTP/1.1" 200 21976 0

147.154.160.171 - - [06/Jul/2010:23:24:13 -0400] "GET /OA_HTML/cabo/jsLibs/Common2_2_24_3.js HTTP/1.1" 200 92167 0

147.154.160.171 - - [06/Jul/2010:23:24:14 -0400] "GET /OA_HTML/cabo/images/t.htm HTTP/1.1" 200 28 0

147.154.160.171 - - [06/Jul/2010:23:24:14 -0400] "GET /OA_HTML/cabo/images/t.gif HTTP/1.1" 200 85 0
147.154.160.171 - - [06/Jul/2010:23:24:14 -0400] "GET /OA_MEDIA/FNDSSCORP.gif HTTP/1.1" 200 1435 0
147.154.160.171 - - [06/Jul/2010:23:24:14 -0400] "GET /OA_MEDIA/fndpbs.gif HTTP/1.1" 200 211 0
147.154.160.171 - - [06/Jul/2010:23:24:15 -0400] "GET /OA_HTML/cabo/images/cache/cghes-2.gif HTTP/1.1" 200 109 0

Using File Storage Parallel Tools | Parallel Unix Commands

The Parallel File Tools suite provides parallel versions of  tar ,  rm , and  cp . These tools can run requests on large file systems in par...