Showing posts with label adop. Show all posts
Showing posts with label adop. Show all posts

Monday, March 18, 2024

New EBS Technology Patch Automation Tool for Application Tier (ETPAT-AT)

 This is the first of two articles about a pair of brand new tools designed to increase the productivity of Oracle E-Business Suite (EBS) DBAs and related staff when upgrading and patching EBS environments.

The tools are called:

  • EBS Technology Patch Automation Tool - Application Tier (ETPAT-AT)
  • EBS 12.2 Upgrade Readiness Checker Tool - Database Tier (EURC-DT)

Introducing ETPAT-AT

We are pleased to announce availability of the EBS Tech Patch Automation Tool - Application Tier (ETPAT-AT) tool, which automates patching of key EBS Release 12.2.0 application tier technology stack components such as Fusion Middleware and WebLogic Server, saving time and reducing the likelihood of error.

Capabilities

Available for all platforms on which Oracle E-Business Suite is supported, the new ETPAT-AT tool complements the capabilities of the EBS Technology Codelevel Checker (ETCC).

Specifically, ETPAT-AT:

  • Supports EBS environments either on-premises or on Oracle Cloud Infrastructure (Compute).
  • Can be run on an EBS Release 12.2.0 application tier for new installations or upgrades.
  • Runs ETCC-MT both before and after each patching session.
  • Provides comprehensive reports detailing:
    • All components that were patched in the patching session.
    • The resulting updated application tier technology stack inventory.
    • Post-patching ETCC-MT actions and results.

For detailed instructions on running the new tool, refer to EBS Technology Patch Automation Tool for Application Tier (ETPAT-AT) (MOS Note 2749774.1).

Obtaining ETPAT-AT

You can download the new ETPAT-AT tool from My Oracle Support as Patch 32208510.

The next article in this two-part series will cover the related tool, EBS 12.2 Upgrade Readiness Checker for Database Tier (EURC-DT).

References

Related Articles

https://blogs.oracle.com/ebstech/post/new-ebs-technology-patch-automation-tool-for-application-tier-etpat-at

Friday, August 11, 2023

Oracle E-Business Suite Release 12.2: Recovery Guidelines For Failed Online Patching (adop) Cutover

 This document describes how you can, if required, restore an Oracle E-Business Suite Release 12.2 system to the state it was in before an Online Patching cutover phase was run. It is only intended for use in specific scenarios where cutover has failed for some reason.

Note: This document supplements the main Online Patching documentation, which is primarily provided in Oracle E-Business Suite Concepts and Oracle E-Business Suite Maintenance Guide. Both these books are available as part of the Release 12.2 Documentation Library.

In This Document

  1. Prerequisites and Restrictions
  2. Setting Up Flashback
  3. Problem Scenario
  4. Flashing Back the Database
  5. Restoring the File Systems
  6. References
  7. Change Record
  8. Documentation Notices

1. Prerequisites and Restrictions

Oracle E-Business Suite Release 12.2 supports Online Patching, which allows patches to be applied to a copy of the system while users are working on the existing system. The Online Patching cycle consists of several phases, with the critical phase where the changes are committed being called cutover. Up to this phase, you can run a special phase called abort, which will undo the changes made so far in the patching cycle. After cutover is complete, however, you cannot do this.

Note: The procedure described in this document is only intended for use in the event of a failed cutover, and as a last resort. It is not supported for use as a rollback (restoration) option after a successful cutover.

If a cutover error occurs, you should first check the error message and try to determine if the problem can be fixed easily, or (as is true in many cases) cutover can be made to succeed simply by running the command again. Restoring to a point before cutover via Flashback recovery should only be done when the error cannot easily be fixed, and continues to fail on subsequent cutover attempts.

So, before proceeding further with the instructions in this document:

  1. Review failure messages and cutover logs, identify problems, and make corrections as applicable. Issues such as running out of disk space can be corrected easily, whilst issues such as timeouts, deadlocks, and network issues may prove to be transient.
  2. Retry the cutover command.
  3. If cutover still fails, follow the instructions in the rest of this document to restore system availability while you take further diagnostic and corrective actions.

If after cutover you want to revert to the state of the system before the patching cycle was started, you can use the Oracle Database Flashback feature to go back to a designated point in time (a restore point). You should create the restore point just before running the cutover phase.

Note: Before creating the restore point, it is advisable to issue a suitable downtime notification and shut down the web services. This will ensure you do not lose any transactional data, and in effect simply extends slightly the cutover downtime.

Depending on exactly when the failure occurred, you may also need to restore the application tier file systems.

Note: The following prequisites must be met before you proceed with the instructions in this document:

  1. You are ready to perform cutover.
  2. All concurrent managers have been shut down cleanly.
  3. There are no current database transactions being performed by any third-party applications.

Top


2. Setting Up Flashback

All the steps in this section are performed on the database tier, as sysdba.

2.1 Set ARCHIVELOG mode

Ensure the database is in ARCHIVELOG mode. Refer to the documentation on Changing the Database Archiving Mode.

2.2 Enable Fast Recovery Area

You enable the Fast Recovery Area (FRA) by setting two database initialization parameters:

  • DB_RECOVERY_FILE_DEST_SIZE - Specifies the size of the Fast Recovery Area.
  • DB_RECOVERY_FILE_DEST - Specifies the physical location of the Flashback recovery files.
Note: If you are using the Fast Recovery Area to store RMAN backups to disk, the FRA needs to be sized to hold all your datafiles, any incremental backups, and the flashback logs generated during cutover.

If you are not using the Fast Recovery Area to store RMAN backups, the FRA only needs to be large enough to contain the flashback logs for the duration of cutover. The size will depend on your selected retention policy (based on the time needed to perform cutover), and should be sufficient to accommodate your online redo log files. If you set the size too small, you may experience space availability issues, resulting in restoration failure. It is therefore advisable to monitor space utilization: any shortages will be recorded in the database alert log.

To minimize FRA space requirements, the Online Patching cutover phase should be scheduled for a time when there are few online transactions, and batch processing is minimal.

SQL>alter system set db_recovery_file_dest_size = 10G scope=BOTH SID='*';
System altered.
SQL>alter system set db_recovery_file_dest = '/d1/oracle/test/ARCH' scope=BOTH SID='*';
System altered.

Note: It is generally advisable to use a server parameter file ("spfile") for managing the database initialization parameters. Refer to the documentation on Managing Initialization Parameters Using a Server Parameter File.

2.3 Specify maximum flashback time

The next command specifies the maximum number of minutes the database may be flashed back. This parameter determines how much flashback log data is kept in the recovery area.

SQL>alter system set db_flashback_retention_target=120;
System altered.

Note: The amount of retention time and space needed will be governed by the amount of time required for cutover. Setting the flashback retention target too high may result in issues if DB_RECOVERY_FILE_DEST_SIZE is set to a large value.

2.4 Activate Flashback

Flashback is activated using the following command:

SQL>alter database flashback on;
Database altered.

2.5 Create restore point

This step will create a restore point called BEFORE_CUTOVER. As shown in the example below, it is also recommended to force a logfile switch both before and after the restore point is created.

SQL>alter system switch logfile;
System altered.
SQL>create restore point BEFORE_CUTOVER guarantee flashback database;
Restore point created.

SQL>alter system switch logfile;
System altered.

You are now ready to flashback the database if needed.

Note: As noted under the FRA description, the Online Patching cutover phase should be scheduled for a time when there are few online transactions and batch processing is minimal. You should confirm that critical concurrent requests are not executing during cutover.  You should also consider putting scheduled concurrent requests on hold prior to creating the BEFORE_CUTOVER flashback restore point.

Top


3. Problem Scenario

You are running an Online Patching cycle:

$ adop phase=prepare
...
$ adop phase=apply patches=11111111,22222222
...
$ adop phase=finalize
...
$ adop phase=cutover

Cutover fails, and you need to go back to the state of the system before you ran the cutover phase.

Note: If you had not run the cutover phase, you would have been able to roll back the patch application process by running the adop abort phase. However, this is not possible once cutover has been run.

There are two main parts to the restore procedure:

  • You will at least need to restore the database using the Flashback feature (described in Section 4)
  • Depending on when cutover failed, you may also need to restore the application tier file systems (described in Section 5).

These activities will be considered in the next two sections.

Top


4. Flashing Back the Database

  1. First, shut down the database, then start it up in mount state:
    SQL>shutdown immediate
    Database closed.
    Database dismounted.
    ORACLE instance shut down.
    SQL>startup mount
    ORACLE instance started.
  2. Restore the flashback to the specified restore point:
    SQL>flashback database to restore point BEFORE_CUTOVER;
    Flashback complete.
  3. Start the database in read-only mode:
    SQL>alter database open read only;
    Database altered.
    Check all looks as expected.

  4. Shut down the database, start it up in mount state, then open it with the resetlogs option:
    SQL>shutdown immediate
    Database closed.
    Database dismounted.
    ORACLE instance shut down.
    SQL>startup mount
    ORACLE instance started.
    Total System Global Area 2142679040 bytes
    Fixed Size 1346140 bytes
    Variable Size 520095140 bytes
    Database Buffers 1593835520 bytes
    Redo Buffers 27402240 bytes
    Database mounted.
    SQL>alter database open resetlogs;
    Database altered.
  5. Disable flashback:
    SQL>alter database flashback off;
    Database altered.
  6. Drop the restore point:
    SQL>drop restore point BEFORE_CUTOVER;
    Restore point dropped.
  7. Set recovery file destination:
    SQL>alter system set db_recovery_file_dest='';
    System altered.
  8. Confirm that Flashback has been deactivated:
    SQL>select FLASHBACK_ON from v$database;
    FLASHBACK_ON
    ------------
    NO

This concludes the Flashback activities.

Note: If you are not using FRA, you may want to clean up the flashback space (unless you intend to keep it for the next adop cycle).

Top


5. Restoring the File Systems

Whether you need to perform this step is conditional, depending on whether cutover failed before the file systems were switched. You can identify which of these cases applies by referring to the cutover logs in $NE_BASE/EBSapps/log/adop/<current_session_id>/cutover_<timestamp>/ for your current session id.

Case 1 - If the log messages indicate that cutover failed before the file systems were switched, do a clean shutdown of any services that are running. Then restart all the services using the normal startup script, and go to Section 6.

Case 2 - If the log messages indicate that cutover failed after the file systems were switched, follow Step 5.1 to shut down any services that have started from the new run file system, then follow Step 5.2 to switch the file systems back. After that, go to Section 6.

5.1 Shut down services started from new run file system

  1. Source the environment on the new run file system.
  2. From $ADMIN_SCRIPTS_HOME, shut down all the services (using adstpall.sh on UNIX).
  3. In a multi-node environment, repeat the preceding two steps on all nodes, leaving the admin node until after all the secondary nodes.

5.2 Switch file systems back

  1. On all nodes where file systems have been switched, run the following command to switch the file systems back:
    $ perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl \
    -action=ctxupdate \
    -contextfile=<full path to new run context file> \
    -patchcontextfile=<full path to new patch file system context file> \
    -outdir=<full path to out directory>
  2. Start up all services from the old run file system (using adstrtal.sh on UNIX).
  3. In a multi-node environment, repeat the preceding two steps on all nodes, starting with the admin node and then proceeding to the secondary nodes.

Top


6. Options and Next Steps

After the restore is complete, you have two basic options for proceeding:

  • Abort the current patching cycle, if the issue that required you to restore was caused by the patches you were attempting to apply.
  • Identify and fix any other issues in the current patching cycle, and proceed with patching.

Top


7. References

For more information on the database features involved, plus guidance on setting the parameters to meet your specific requirements, refer to:

  • Chapter 5, Configuring the RMAN Environment, in Oracle Database Backup and Recovery User's Guide 11g Release 2 (11.2), Part No. E10642.

For more information on the Oracle E-Business Suite Online Patching cycle, refer to:

  • Chapter 3, Patching Procedures, in Oracle E-Business Suite Maintenance Guide, Part No.E22954.

Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching (adop) Cutover (Doc ID 1584097.1)

Saturday, October 8, 2022

adop fs_clone failure

adop fs_clone failure

Validating credentials.


Initializing.

    Run Edition context  : /ascprd/app/applmgr/IRASCPRD_R122/fs1/inst/apps/IRASCPRD_Admin_host/appl/admin/IRASCPRD_Admin_host.xml

    Patch edition context: /ascprd/app/applmgr/IRASCPRD_R122/fs2/inst/apps/IRASCPRD_Admin_host/appl/admin/IRASCPRD_Admin_host.xml

    Patch file system free space: 696.66 GB


Validating system setup.

    [UNEXPECTED]Invalid worker Count: 0

    [UNEXPECTED]Error validating worker count



[STATEMENT] Please run adopscanlog utility, using the command


"adopscanlog -latest=yes"


to get the list of the log files along with snippet of the error message corresponding to each log file.



adop exiting with status = 2 (Fail)



Fix : Run with one worker

$ adop phase=fs_clone worker=1




Solution worked for others


Solution:
The default worker count information(recomm & max)  is stored in a file adpawc.xml that can be found under $APPL_TOP/admin/$TWO_TASK/log

Somehow this file has been modified with recommended value as 0 and max value as 1 during node addition.

FileContent:
<?xml version="1.0"?>
<WORKER_COUNT>
        <RECOMMENDED>0</RECOMMENDED>
        <MAX>1</MAX>
</WORKER_COUNT>

Update recommended &  max value based on cpu count.

Here I updated recommended value as 8 and max value as 64 and saved the file.

Once the changes are made, fs_clone went smoothly.

How to choose the number of workers for adop or adpatch

 How to choose the number of workers:


For less than 32 cores set:

• parallel_max_servers = 2 x number of CPU cores
• AD Parallel workers – start with 1.5 x number of CPU cores. Possibly increase to 2 x number of CPU cores
• job_queue_processes = 2 x number of CPU cores

For 32 cores and above, start with:

• parallel_max_servers = 1.5 x number of CPU cores
• AD Parallel workers = between 1.0 and and 1.5 x number of CPU cores
• job_queue_processes = 1.5 x number of CPU cores


EBS 12.2.9 RUP Patch 28840850 Hung with Error "aiosp2() Error: failure in usdspn() Contents of error buffer are: usdsop cannot create a new process" (Doc ID 2878394.1)

Oracle E-Business Suite Release 12.2 System Schema Migration

In This Document Section 1: Overview of the EBS System Schema Section 2: Requirements for Using the EBS System Schema Section 3: Migrating t...