Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching (adop) Cutover (Doc ID 1584097.1)

This document describes how you can, if required, restore an Oracle E-Business Suite Release 12.2 system to the state it was in before an Online Patching cutover phase was run. It is only intended for use in specific scenarios where cutover has failed for some reason.

Note: This document supplements the main Online Patching documentation, which is primarily provided in Oracle E-Business Suite Concepts and Oracle E-Business Suite Maintenance Guide. Both these books are available as part of the Release 12.2 Documentation Library.

In This Document

1. Prerequisites and Restrictions
2. Setting Up Flashback
3. Problem Scenario
4. Flashing Back the Database
5. Restoring the File Systems
6. References
7. Change Record
8. Documentation Notices

1. Prerequisites and Restrictions

Oracle E-Business Suite Release 12.2 supports Online Patching, which allows patches to be applied to a copy of the system while users are working on the existing system. The Online Patching cycle consists of several phases, with the critical phase where the changes are committed being called cutover. Up to this phase, you can run a special phase called abort, which will undo the changes made so far in the patching cycle. After cutover is complete, however, you cannot do this.

Note: The procedure described in this document is only intended for use in the event of a failed cutover, and as a last resort. It is not supported for use as a rollback (restoration) option after a successful cutover.
If a cutover error occurs, you should first check the error message and try to determine if the problem can be fixed easily, or (as is true in many cases) cutover can be made to succeed simply by running the command again. Restoring to a point before cutover via Flashback recovery should only be done when the error cannot easily be fixed, and continues to fail on subsequent cutover attempts.
So, before proceeding further with the instructions in this document:

1. Review failure messages and cutover logs, identify problems, and make corrections as applicable. Issues such as running out of disk space can be corrected easily, whilst issues such as timeouts, deadlocks, and network issues may prove to be transient.
2. Retry the cutover command.
3. If cutover still fails, follow the instructions in the rest of this document to restore system availability while you take further diagnostic and corrective actions.

If after cutover you want to revert to the state of the system before the patching cycle was started, you can use the Oracle Database Flashback feature to go back to a designated point in time (a restore point). You should create the restore point just before running the cutover phase.

Note: Before creating the restore point, it is advisable to issue a suitable downtime notification and shut down the web services. This will ensure you do not lose any transactional data, and in effect simply extends slightly the cutover downtime.

Depending on exactly when the failure occurred, you may also need to restore the application tier file systems.

Note: The following prequisites must be met before you proceed with the instructions in this document:

1. You are ready to perform cutover.
2. All concurrent managers have been shut down cleanly.
3. There are no current database transactions being performed by any third-party applications.

Top

---

2. Setting Up Flashback

All the steps in this section are performed on the database tier, as sysdba.

2.1 Set ARCHIVELOG mode

Ensure the database is in ARCHIVELOG mode. Refer to the documentation on Changing the Database Archiving Mode.

2.2 Enable Fast Recovery Area

You enable the Fast Recovery Area (FRA) by setting two database initialization parameters:

• DB_RECOVERY_FILE_DEST_SIZE - Specifies the size of the Fast Recovery Area.
• DB_RECOVERY_FILE_DEST - Specifies the physical location of the Flashback recovery files.

Note: If you are using the Fast Recovery Area to store RMAN backups to disk, the FRA needs to be sized to hold all your datafiles, any incremental backups, and the flashback logs generated during cutover.
If you are not using the Fast Recovery Area to store RMAN backups, the FRA only needs to be large enough to contain the flashback logs for the duration of cutover. The size will depend on your selected retention policy (based on the time needed to perform cutover), and should be sufficient to accommodate your online redo log files. If you set the size too small, you may experience space availability issues, resulting in restoration failure. It is therefore advisable to monitor space utilization: any shortages will be recorded in the database alert log.
To minimize FRA space requirements, the Online Patching cutover phase should be scheduled for a time when there are few online transactions, and batch processing is minimal.

SQL>alter system set db_recovery_file_dest_size = 10G scope=BOTH SID='*';System altered.
SQL>alter system set db_recovery_file_dest = '/d1/oracle/test/ARCH' scope=BOTH SID='*';
System altered.

Note: It is generally advisable to use a server parameter file ("spfile") for managing the database initialization parameters. Refer to the documentation on Managing Initialization Parameters Using a Server Parameter File.

2.3 Specify maximum flashback time

The next command specifies the maximum number of minutes the database may be flashed back. This parameter determines how much flashback log data is kept in the recovery area.

SQL>alter system set db_flashback_retention_target=120;
System altered.

Note: The amount of retention time and space needed will be governed by the amount of time required for cutover. Setting the flashback retention target too high may result in issues if DB_RECOVERY_FILE_DEST_SIZE is set to a large value.

2.4 Activate Flashback

Flashback is activated using the following command:

SQL>alter database flashback on;
Database altered.

2.5 Create restore point

This step will create a restore point called BEFORE_CUTOVER. As shown in the example below, it is also recommended to force a logfile switch both before and after the restore point is created.

SQL>alter system switch logfile;
System altered.
SQL>create restore point BEFORE_CUTOVER guarantee flashback database;
Restore point created.
SQL>alter system switch logfile;
System altered.

You are now ready to flashback the database if needed.

Note: As noted under the FRA description, the Online Patching cutover phase should be scheduled for a time when there are few online transactions and batch processing is minimal. You should confirm that critical concurrent requests are not executing during cutover.  You should also consider putting scheduled concurrent requests on hold prior to creating the BEFORE_CUTOVER flashback restore point.

Top

---

3. Problem Scenario

You are running an Online Patching cycle:

$ adop phase=prepare...
$ adop phase=apply patches=11111111,22222222
...
$ adop phase=finalize
...
$ adop phase=cutover

Cutover fails, and you need to go back to the state of the system before you ran the cutover phase.

Note: If you had not run the cutover phase, you would have been able to roll back the patch application process by running the adop abort phase. However, this is not possible once cutover has been run.

There are two main parts to the restore procedure:

• You will at least need to restore the database using the Flashback feature (described in Section 4)
• Depending on when cutover failed, you may also need to restore the application tier file systems (described in Section 5).

These activities will be considered in the next two sections.

Top

---

4. Flashing Back the Database

1. First, shut down the database, then start it up in mount state:
   SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.
2. Restore the flashback to the specified restore point:
   SQL>flashback database to restore point BEFORE_CUTOVER;
Flashback complete.
3. Start the database in read-only mode:
   SQL>alter database open read only;
Database altered.
   Check all looks as expected.


4. Shut down the database, start it up in mount state, then open it with the resetlogs option:
   SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.
Total System Global Area 2142679040 bytes
Fixed Size 1346140 bytes
Variable Size 520095140 bytes
Database Buffers 1593835520 bytes
Redo Buffers 27402240 bytes
Database mounted.
SQL>alter database open resetlogs;
Database altered.
5. Disable flashback:
   SQL>alter database flashback off;
Database altered.
6. Drop the restore point:
   SQL>drop restore point BEFORE_CUTOVER;
Restore point dropped.
7. Set recovery file destination:
   SQL>alter system set db_recovery_file_dest='';
System altered.
8. Confirm that Flashback has been deactivated:
   SQL>select FLASHBACK_ON from v$database;
FLASHBACK_ON
------------
NO

This concludes the Flashback activities.

Note: If you are not using FRA, you may want to clean up the flashback space (unless you intend to keep it for the next adop cycle).

Top

---

5. Restoring the File Systems

Whether you need to perform this step is conditional, depending on whether cutover failed before the file systems were switched. You can identify which of these cases applies by referring to the cutover logs in $NE_BASE/EBSapps/log/adop/<current_session_id>/cutover_<timestamp>/ for your current session id.

Case 1 - If the log messages indicate that cutover failed before the file systems were switched, do a clean shutdown of any services that are running. Then restart all the services using the normal startup script, and go to Section 6.

Case 2 - If the log messages indicate that cutover failed after the file systems were switched, follow Step 5.1 to shut down any services that have started from the new run file system, then follow Step 5.2 to switch the file systems back. After that, go to Section 6.

5.1 Shut down services started from new run file system

1. Source the environment on the new run file system.
2. From $ADMIN_SCRIPTS_HOME, shut down all the services (using adstpall.sh on UNIX).
3. In a multi-node environment, repeat the preceding two steps on all nodes, leaving the admin node until after all the slave nodes.

5.2 Switch file systems back

1. On all nodes where file systems have been switched, run the following command to switch the file systems back:
   $ perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl \
-action=ctxupdate \
-contextfile=<full path to new run context file> \
-patchcontextfile=<full path to new patch file system context file> \
-outdir=<full path to out directory>
2. Start up all services from the old run file system (using adstrtal.sh on UNIX).
3. In a multi-node environment, repeat the preceding two steps on all nodes, starting with the admin node and then proceeding to the slave nodes.

Top

---

6. Options and Next Steps

After the restore is complete, you have two basic options for proceeding:

• Abort the current patching cycle, if the issue that required you to restore was caused by the patches you were attempting to apply.
• Identify and fix any other issues in the current patching cycle, and proceed with patching.

Top

---

7. References

For more information on the database features involved, plus guidance on setting the parameters to meet your specific requirements, refer to:

• Chapter 5, Configuring the RMAN Environment, in Oracle Database Backup and Recovery User's Guide 11g Release 2 (11.2), Part No. E10642.

For more information on the Oracle E-Business Suite Online Patching cycle, refer to:

• Chapter 3, Patching Procedures, in Oracle E-Business Suite Maintenance Guide, Part No.E22954.

Top

---

7. Change Record

Date	Description
02-Jun-2014	Added adop to title.
21-Oct-2013	Made minor additions.
05-Sep-2013	Added prerequisites.
30-Aug-2013	Updated FRA sizing guidance.
28-Aug-2013	Updated Section 5.
09-Aug-2013	Initial creation.

Top

My Oracle Support Knowledge Document 1584097.1 by Oracle E-Business Suite Development

8. Documentation Notices

Copyright © 2013, 2014, Oracle and/or its affiliates. All rights reserved.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.

For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.

Access to Oracle Support

Oracle customers have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.

12.2 E-Business Suite Applications DBA Steps To Check if a Patch is Applied in 12.2.x using SQL*PLUS (Doc ID 1963046.1)

APPLIES TO:

Oracle E-Business Suite Technology Stack - Version 12.2 to 12.2.7 [Release 12.2]
Oracle Applications Manager - Version 12.2 to 12.2.7 [Release 12.2Cloud to 12.2]
Oracle Trade Management - Version 12.2 to 12.2.7 [Release 12.2]
Oracle Application Object Library - Version 12.2 to 12.2.7 [Release 12.2]
Oracle Applications DBA - Version 12.2 to 12.2.7 [Release 12.2Cloud to 12.2]
Information in this document applies to any platform.

GOAL

The purpose of this document is to explain how to check if a Patch is applied on an environment in Release 12.2.x.

SOLUTION

In eBusiness Suite (EBS) 12.2.x you cannot query the AD_BUGS table to check if patches have been applied.

The AD_BUGS table may have entries for patches that were applied but later the patching cycle was aborted (not really applied).

To check whether a patch is really applied use the AD_PATCH.IS_PATCH_APPLIED pl/sql function.

Using this API is an alternative method for users without access to Oracle Applications Manager's "Patching and Utilities" feature to determine if a certain patch is applied.

Usage:

select AD_PATCH.IS_PATCH_APPLIED(\'$release\',\'$appl_top_id\',\'$patch_no\',\'$language\')
from dual;

To obtain the appl_top_id use the following select to list the appl_tops:

SELECT APPL_TOP_ID , NAME , DESCRIPTION , APPLICATIONS_SYSTEM_NAME , ACTIVE_FLAG FROM APPLSYS.AD_APPL_TOPS;

Example sql using the APPL_TOP ID for a specific node in a multinode environment (1045 is the APPL_TOP ID):

SELECT adb.bug_number,ad_patch.is_patch_applied('R12', 1045, adb.bug_number)
FROM ad_bugs adb
WHERE adb.bug_number in (20034256);

or for single app tier installations:

select ad_patch.is_patch_applied('R12',-1,20034256) from dual;

expected results:
EXPLICIT = applied
NOT APPLIED = not applied / aborted

Note: If you are sure patch is applied but it is not showing as applied via the above steps, then update the snapshot manually with the steps below:

1. Start adadmin after source the RUN FS env.
2. Select "2. Maintain Applications Files menu" in "AD Administration Main Menu".
3. In "Maintain Applications Files", select "4. Maintain snapshot information".
4. Select "2. Update current view snapshot" in the "Maintain Snapshot Information".
5. Select "1. Update Complete APPL_TOP" in the "Maintain Current View Snapshot Information".

E-Business Suite Applications Technology Stack Utility / Script To Check The Techstack Component Versions (Forms, Http Server, JDK, Framework, Database, etc) (Doc ID 601736.1)

APPLIES TO:

Oracle E-Business Suite Technology Stack - Version 11.5.10.2 to 12.2 [Release 11.5.10 to 12.2]
Information in this document applies to any platform.

GOAL

How to find the versions of Technology stack components like Forms, iAS, Framework, JDK, OJSP, Database, etc.?

SOLUTION

On Application Tier

1. Establish the needed environment parameters by sourcing the Applications environment file as the owner of the application tier file system.
2. Ensure "APPLRGF" variable is set in environment. If not, set it to the same value as "APPLTMP" (Note: If you set the output file directory using -outfile parameter, skip this step and the file will be generated under $APPLRGF/TXK directory).
3. Navigate to <FND_TOP>/patch/115/bin. Run the utility/command  as follows :
   
   

   Operation System	Command Line
   Unix or Linux	$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \ -script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \ -contextfile=$CONTEXT_FILE \ -appspass=apps \ -outfile=$APPLTMP/Report_App_Inventory.html Note: - Run the above command in single line - Remove the '\' after each parameter when running the command on a single line - Supply the apps user password as is applicable for parameter "-appspass"
   Windows	­PERLPRG% %FND_TOP%\patch\115\bin\TXKScript.pl -script=%FND_TOP%\patch\115\bin\txkInventory.pl -txktop=%APPLTMP% -contextfile=%CONTEXT_FILE% -appspass=apps -outfile=%APPLTMP%\Report_App_Inventory.html Note: - Run the above command in single line - Remove the '\' after each parameter when running the command on a single line - Supply the apps user password as is applicable for parameter "-appspass"

   
   Where:

   txktop	Temporary working directory use by Perl Modules. Should not be an empty string.
   contextfile	Location of the context file. If not passed, default is picked from the environment.
   appspass	APPS schema password. If not passed, default password is used.
   outfile	Location for the report being generated. If not passed, the default location is <APPLTMP>/TXK

   
   To generate the report in text format, the parameter "-reporttype=text" needs to be passed to the above commands. For example:
   

   outfile=$APPLTMP/Report_App_Inventory.txt -reporttype=text

4. Once the command executes successfully, it should generate the report file in the location specified for "outfile" parameter in above script
5. In case you have more than one application tier's (multi-node architecture), follow the steps 1-4 on each of the Application tiers
6. Upload the report output file (default: $APPLTMP/Report_App_Inventory.html )to Oracle Support for review

On Database Tier

1. Establish the needed environment parameters by sourcing the Database environment file as the owner of the Database tier file system. |
2. Ensure "ORACLE_HOME" variable is set in environment and correctly pointing to your database ORACLE_HOME.
3. Navigate to <ORACLE_HOME>/appsutil/bin. Run the utility/command as below (all on a single line):
   
   

   Operation System	Command Line
   Unix or Linux	$ADPERLPRG $ORACLE_HOME/appsutil/bin/TXKScript.pl -script=$ORACLE_HOME/appsutil/bin/txkInventory.pl -txktop=$ORACLE_HOME/appsutil/temp -contextfile=$CONTEXT_FILE -appspass=apps -outfile=$ORACLE_HOME/appsutil/temp/Report_DB_Inventory.html Note:- Run the above command in single line - Supply the apps user password as is applicable for parameter "-appspass"
   Windows	­PERLPRG% %ORACLE_HOME%/appsutil/bin/TXKScript.pl -script=%ORACLE_HOME%/appsutil/bin/txkInventory.pl -txktop=%ORACLE_HOME%/appsutil/temp -contextfile=%CONTEXT_FILE% -appspass=apps -outfile=%ORACLE_HOME%/appsutil/temp/Report_DB_Inventory.html Note: - Run the above command in single line - Supply the apps user password as is applicable for parameter "-appspass"

   
   Where
    

   txktop	Temporary working directory use by Perl Modules. Should not be an empty string.
   contextfile	Location of the context file.
   appspass	APPS schema password.
   outfile	Location for the report being generated.

   
   To generate the report in text format, the parameter "-reporttype=text" needs to be passed to the above commands. For example:
   
   

   outfile=$ORACLE_HOME/appsutil/temp/Report_DB_Inventory.txt -reporttype=text

4. Once the command executes successfully, it should generate the report file in the location specified for "outfile" parameter in above script
5. Upload the report output file (default: $ORACLE_HOME/appsutil/temp/Report_DB_Inventory.html ) to Oracle Support for review

If the Admin Password of an EBS WebLogic Domain is lost or forgotten

As noted earlier, the EBS WebLogic domain uses Node Manager to control startup of the AdminServer and Managed Servers. For the EBS WebLogic domain, the Node Manager and WebLogic AdminServer passwords must be same. If the passwords are different, the AD control scripts will not work properly.

If the AdminServer password has been lost or forgotten, it can be reset by carrying out the following steps on the run file system. As described in the final step, an fs_clone operation should then be performed to synchronize the run and patch file systems.

1. Shut down all running services. Since the AdminServer password is not known, the servers cannot be stopped from the console and so must be killed as follows.
   1. Connect to the Oracle E-Business Suite instance and source the application tier environment file.
   2. Identify the PIDs of Node Manager, AdminServer, and all running Managed Servers:

      $ ps -ef | grep "NodeManager"
      $ ps -ef | grep "weblogic.Name=AdminServer"
      $ ps -ef | grep "weblogic.Name=forms-c4ws_server"
      $ ps -ef | grep "weblogic.Name=forms_server"
      $ ps -ef | grep "weblogic.Name=oafm_server"
      $ ps -ef | grep "weblogic.Name=oacore_server"

   3. Kill all these processes, starting with Node Manager and followed by the Managed Servers.
2. Back up these folders, and then delete them:

   <EBS_DOMAIN_HOME>/security/ DefaultAuthenticatorInit.ldift
   <EBS_DOMAIN_HOME>/servers/<server_name>/data/ldap
   <EBS_DOMAIN_HOME>/servers/<server_name>/security/boot.properties
   <EBS_DOMAIN_HOME>/servers/<server_name>/data/nodemanager/boot.properties
   
   

   Where:
   • <EBS_DOMAIN_HOME> is the absolute path of the EBS WebLogic domain
   • <server_name> is the name of the server directory under <EBS_DOMAIN_HOME>.
   If the password is not reset correctly, the backed up files and folders can be restored.
   Note: For certain servers, the boot.properties file may be present in only one location of the two specified above. In such a case, back it up and then delete it.
3. Set up a new environment to change the WLS AdminServer password.
   1. Start a new session and connect to the Oracle E-Business Suite instance.
   2. Do not source the application tier environment file.
   3. Run the following command to source the WebLogic Server domain environment:

      $ cd <EBS_DOMAIN_HOME>/bin
      $ source setDomainEnv.sh

   4. Run the following commands:

      $ cd <EBS_DOMAIN_HOME>/security
      $ java weblogic.security.utils.AdminAccount <wls_adminuser> <wls_admin_new_password> .

      Where:
      • <wls_adminuser> is the same as the value of context variable s_wls_admin_user
      • <wls_admin_new_password> is the new WLS AdminServer password you wish to set.
      Note: Do not omit the trailing period ('.') in the above command: it is needed to specify the current domain directory.
4. Start AdminServer from the command line. You will be prompted for the WebLogic Server username and password, so that the AdminServer boot.properties file can be generated.
   1. Go to the EBS Domain Home:

      $ cd <EBS_DOMAIN_HOME>

   2. Start AdminServer:

      $ java <s_nm_jvm_startup_properties> -Dweblogic.system.StoreBootIdentity=true -Dweblogic.Name=AdminServer weblogic.Server

      Where:
      • <s_nm_jvm_startup_properties> is the same as the value of context variable ss_nm_jvm_startup_properties
      The above command prompts for the WebLogic Server username and password:

      Enter username to boot WebLogic server:
      Enter password to boot WebLogic server: 

      Provide the same credentials as you provided in Step 3.
5. Change the Node Manager password.
   1. Log in to the WebLogic Administration console.
   2. Click the 'Lock & Edit' button.
   3. In the left panel, click on the EBS Domain link.
   4. Select the 'Security' tab.
   5. Click on the 'Advanced' link.
   6. Edit the 'Node Manager password' field and set it to the new WebLogic Server password. The password should be same as set in Step 3.
   7. Edit the 'Confirm Node Manager Password' field and set it to the new WebLogic Server password. The password should be same as set in Step 3.
   8. Save and activate the changes.
6. The first time, AdminServer has to be stopped from the Admin console. Follow these steps:
   1. Log in to the WebLogic Administration console.
   2. Shut down AdminServer.
7. Set up your environment to start AdminServer again. AdminServer should now be started using the normal AD script, which will also start Node Manager using the new password.
   1. Launch a new session and connect to the Oracle E-Business Suite instance.
   2. Source the application tier environment file.
   3. Start AdminServer with the following command:

      $ $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

8. Start the Managed Servers. For the first time, all Managed Servers should be started from the WebLogic Server Admin console. This step will create boot.properties files for the respective Managed Servers. Follow these steps:
   1. Log in to the WebLogic Server Administration Console.
   2. Start all Managed Servers, one at a time.
9. Shut down all the Managed Servers. This is so the new credentials will be picked up at the next startup. Follow these steps:
   1. Log in to the WebLogic AdminServer console.
   2. Shut down all Managed Servers.
   3. Shut down AdminServer.
10. Shut down Node Manager using the normal AD script.

    $ $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

11. Copy the boot.properties file for each Managed Server.
    WebLogic Server native scripts use the boot.properties file. The above steps have created the boot.properties file under <EBS_DOMAIN_HOME>/servers/<Managed Server name>/data/nodemanager, which is used by Node Manager. For each Managed Server, copy the newly-generated boot.properties file from<EBS_DOMAIN_HOME>/servers/<Managed Server name>/data/nodemanager to <EBS_DOMAIN_HOME>/servers/<Managed Server name>/security.
    The EBS WebLogic Server domain password has now been changed, and all servers can now be started using the normal AD scripts.
    To start AdminServer:

    $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

    To start the Managed Servers:

    $ $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start <managed_server_name>

12. The above steps have changed the Oracle WebLogic AdminServer password on the run file system. You now need to perform an fs_clone operation, to change the WebLogic EBS Domain password on the patch file system:
    1. Launch a new session and connect to the Oracle E-Business Suite instance.
    2. Source the application tier environment file.
    3. Run the following command:

       $ adop phase=fs_clone

Adding connection filter in EBS

cd $ADMIN_SCRIPTS_HOME

sh adadminsrvctl.sh start

-- add all nodes from run fs

--------------------------

. ~/EBSapps.env run

java -cp $CLASSPATH:$FMW_HOME/wlserver_10.3/server/lib/weblogic.jar oracle.apps.ad.tools.configuration.RegisterNodePreReq add-filter-rule \

-contextfile $CONTEXT_FILE -hostname <isupplier node hostname>

--------------------------

perl adpreclone.pl appsTier -- ## 25 minutes -- started 3.31pm