Friday, December 14, 2018

[Troubleshooting] Oracle Apps R12.2 Online Patching ADOP : Prepare Phase Issue

This post covers, Online Patching (ADOP)  issue reported by one of the trainees in Oracle AppsDBA Training  with prepare phase.
Note: There are five phases in Online Patching (ADOP) in R12.2  PREPARE -> APPLY -> FINALIZE -> CUTOVER  -> CLEANUP .
  • To know more about Online Patching (ADOP) click here or for FREE video on How to find patches in EBS R12, check here 

Troubleshooting ADOP Prepare Phase Issue:


1. Run Prepare Phase:

Any Patching Cycle starts with Prepare Phase and to start it run below command
adop phase=prepare
Prepare phase is failing with error message
    [UNEXPECTED]fs_clone has failed.
        [UNEXPECTED]Error calling runPendingConfigClone subroutine.

2. Look at ADOP logs:

First thing in troubleshooting is to look at log file (in this case ADOP log file)
Note: Kindly refer this post to know more on log file location in R12.2 Oracle Apps R12.2 Log Files Location every Apps DBA must know

A) Error in adop logs: 

/u01/oracle/PRD122/fs_ne/EBSapps/log/adop/6/prepare_20171109_032332/PRD122_1705ecloud05/log/adop.log
Note: Here 
  • /u01/oracle/PROD is Oracle EBS install base directory
  • fs_ne Non-Editioned File System
  • 6 – adop session id
  • prepare_20171109_032332 – <phase>_<date>_<time>
  • PRD122_1705ecloud05 – CONTEXT NAME
Oracle Apps R12.2 has Dual File System (fs1 & fs2), which helps organizations to have less downtime during patching.  By introducing duplicate file system, Oracle has reduced patching downtime and improved availability of the system.
We can focus on three major parts of the E-Business Suite 12.2 file system:
  • File System 1 (fs1):  /u01/oracle/PROD/fs1
  • File System 2 (fs2):  /u01/oracle/PROD/fs2
  • Non-Editioned File System (fs_ne):  /u01/oracle/PROD/fs_ne
Thu Nov  9 03:34:15 2017
*******FATAL ERROR*******
PROGRAM : (/u01/oracle/PRD122/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Thu Nov  9 03:34:15 2017
FUNCTION: main::migrateCloneComponentApply [ Level 1 ] 
ERRORMSG: /u01/oracle/PRD122/fs1/EBSapps/comn/adopclone_1705ecloud05/bin/adclone.pl did not go through successfully.
[UNEXPECTED]Error occurred running “perl /u01/oracle/PRD122/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -patchcontextfile=/u01/oracle/PRD122/fs2/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -promptmsg=hide -console=off -mode=migrate -sync_mode=copy -sessionid=6 -timestamp=20171109_032332 -outdir=/u01/oracle/PRD122/fs_ne/EBSapps/log/adop/6/20171109_032332/prepare/1705ecloud05”
        [UNEXPECTED]occurred during CONFIG_CLONE Patch File System from Run File System, running command: “perl /u01/oracle/PRD122/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -patchcontextfile=/u01/oracle/PRD122/fs2/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -promptmsg=hide -console=off -mode=migrate -sync_mode=copy -sessionid=6 -timestamp=20171109_032332 -outdir=/u01/oracle/PRD122/fs_ne/EBSapps/log/adop/6/20171109_032332/prepare/1705ecloud05”.
        [PROCEDURE] [START 2017/11/09 03:34:15] Updating status of patches
        [STATEMENT] SQL statement : ”  update ad_adop_session_patches
  set status=’F’
  where applied_file_system_base = ‘/u01/oracle/PRD122/fs1’ and
  patch_file_system_base = ‘/u01/oracle/PRD122/fs2′ and bug_number=’CONFIG_CLONE’ and status <> ‘Y’
  and appltop_id=195 and node_name=’1705ecloud05′”
        [PROCEDURE] [END   2017/11/09 03:34:16] Updating status of patches
 [STATEMENT] SQL statement : ”  update ad_adop_session_patches
  set status=’F’
  where applied_file_system_base = ‘/u01/oracle/PRD122/fs1’ and
  patch_file_system_base = ‘/u01/oracle/PRD122/fs2′ and bug_number=’CONFIG_CLONE’ and status <> ‘Y’
  and appltop_id=195 and node_name=’1705ecloud05′”
        [PROCEDURE] [END   2017/11/09 03:34:16] Updating status of patches
        [UNEXPECTED]fs_clone has failed.
        [UNEXPECTED]Error calling runPendingConfigClone subroutine.

B) Error in fs_clone logs: 

Note: Prepare Phase, creates clone of Patch File System from Run File System and as per previous log, error was during this Cloning so we need to look at clone logs.

$INST_TOP/admin/log/clone/FSCloneApplyAppsTier_<timestamp>.log

 START: Instantiating the OHS Config.
Running /u01/oracle/PRD122/fs2/FMW_Home/webtier/perl/bin/perl  -I/u01/oracle/PRD122/fs2/FMW_Home/webtier/perl/lib/5.10.0 -I/u01/oracle/PRD122/fs2/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I/u01/oracle/PRD122/fs2/EBSapps/appl/au/12.0.0/perl -I/u01/oracle/PRD122/fs2/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u01/oracle/PRD122/fs2/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetOHSConfig.pl cfgCloneInstance -ctxfile=/u01/oracle/PRD122/fs2/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml -archvloc=/u01/oracle/PRD122/fs1/EBSapps/comn/adopclone_1705ecloud05/FMW/OHS/ohsarchive.jar -mvpln=/u01/oracle/PRD122/fs1/EBSapps/comn/adopclone_1705ecloud05/FMW/OHS/moveplan.xml -t2plogloc=/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/admin/log/clone/ohsT2PApply -logdir=/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/admin/log/clone -console=off -promptmsg=hide
Script Executed in 37703 milliseconds, returning status 255
ERROR: Script failed, exit code 255
Error Message  :1
  [PLUGIN][OHS] – ERROR – Nov 9, 2017 03:34:15 – CLONE-26009   OHS T2P failed.
  [PLUGIN][OHS] – CAUSE – Nov 9, 2017 03:34:15 – CLONE-26009   Unable to start OS component.
  [PLUGIN][OHS] – ACTION – Nov 9, 2017 03:34:15 – CLONE-26009   Check clone log and error file and ohs log file /u01/oracle/PRD122/fs2/FMW_Home/webtier/instances/EBS_web_PRD12111_OHS1/diagnostics/logs/OHS/EBS_web_PRD12111/console~OHS~1.log for root cause.
 

C) Error in OHS logs: 

Note: Clone process clones, Fusion Midddleware (WebLogic & OHS) as per the previous log, the error was during this startup of OHS server so we need to look at OHS logs.

/u01/oracle/PRD122/fs2/FMW_Home/webtier/instances/EBS_web_PRD12111_OHS1/diagnostics/logs/OHS/EBS_web_PRD12111/console~OHS~1.log

 /u01/oracle/PRD122/fs2/FMW_Home/webtier/ohs/bin/apachectl startssl: execing httpd
httpd.worker: bad user name applmgr11

3. Root Cause:

Root cause as identified in previous OHS log was, wrong owner permission for OHS files
Checked httpd.conf file and wherever it is having applmgr11,  changed to applmgr user permission
 cd /u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/cutover/OHS/EBS_web_PRD12111
[root@1705ecloud05 EBS_web_PRD12111]# grep applmgr11 *
httpd.conf:        User applmgr11
[root@1705ecloud05 EBS_web_PRD12111]# chown applmgr:dba httpd.conf
[root@1705ecloud05 EBS_web_PRD12111]#
   User applmgr
        #Group GROUP_TEMPLATE
</IfModule>

4. Change in Context File

Note: After updating user to applmgr in httpd.conf, OHS start-up was still failing
So checked $CONTEXT_FILE and below two entries having applmgr11 user on both run (fs1) and patch (fs2) file system and causing the issue
<appsuser oa_var=”s_appsuser” osd=”unix”>applmgr11</appsuser>
<sysadminmail oa_var=”s_sysadmin_mail” osd=”unix”>applmgr11@1705ecloud05.k21academy.com</sysadminmail>
/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml
Note: $CONTEXT_FILE, captured all configuration parameters. In Oracle R12, we are having two context file:
Application context file called the <CONTEXT_NAME>.xml contains the configuration information for the application tier & is located in $APPl_TOP/admin for each node of application tier.
Database context file called the <CONTEXT_NAME>.xml contains the configuration information for the database tier & is located in $ORACLE_HOME/appsutil/

Fix:

1. Changed user to applmgr on both run and patch file system under $CONTEXT_FILE and run autoconfig, then restart prepare phase again
<appsuser oa_var=”s_appsuser” osd=”unix”>applmgr</appsuser>
<sysadminmail oa_var=”s_sysadmin_mail” osd=”unix”>applmgr@1705ecloud05.k21academy.com</sysadminmail>
/u01/oracle/PRD122/fs1/inst/apps/PRD12111_1705ecloud05/appl/admin/PRD12111_1705ecloud05.xml

Run Prepare Phase Again:

adop phase=prepare restart=yes
The Node Manager is already shutdown
NodeManager log is located at /u01/oracle/PRD122/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1
adnodemgrctl.sh: exiting with status 2
adnodemgrctl.sh: check the logfile /u01/oracle/PRD122/fs2/inst/apps/PRD12111_1705ecloud05/logs/appl/admin/log/adnodemgrctl.txt for more information …
The prepare phase completed successfully.
adop exiting with status = 0 (Success)

Checked ADOP Status 

adop -status
ADOP (C.Delta.8)
Session Id: 6
Command: status
Output: /u01/oracle/PRD122/fs_ne/EBSapps/log/adop/6/20171109_085910/adzdshowstatus.out
===============================================================
Node Name       Node Type  Phase           Status          Started             Finished      Elapsed
————— ———- ————— ————— ——————– ——————– ————
1705ecloud05    master     PREPARE         COMPLETED       2017/11/06 22:41:13 2017/11/09 08:47:02  58:05:49
APPLY           NOT STARTED
FINALIZE        NOT STARTED
CUTOVER         NOT STARTED
CLEANUP         NOT STARTED
File System Synchronization Type: Light
adop exiting with status = 0 (Success)

In this scenario all you need to do is apply the pre-requisite patch first.
1)
Adop phase=apply patch=abcdef
Then once this is done continue with the patch you were applying earlier.
Adop phase=apply patches=11111,22222,33333,44444,55555
Note: There is another way to do it if there were no patches appled prior to to the above patches in this patching cycle you could also abort the patching cycle and start fresh patching cycle
2)
Adop phase=abort,cleanup
adop phase=prepare
Then continue with above mentioned prerequisite patch ( abcdef) and the respective patch later (11111,22222,33333,44444,55555)
But in this scenario the second mentioned is not required.
at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Database Options/Management Packs Usage Reporting for Oracle Databases 11.2 and later (Doc ID 1317265.1)

  Database Options/Management Packs Usage Report You can determine whether an option is currently in use in a database by running options_pa...