Tuesday, August 23, 2022

Common Tracing Techniques in Oracle E-Business Applications 11i and R12 (Doc ID 296559.1)

 In this Document

Purpose
Questions and Answers
 1a. What is a trace file?
 Example #1: ORA-Error
 Example #2: Rollback
 1b. What are bind variables?
 Example #1: Binds
 Example #2: Binds
 2. How to Enable Trace In...
 2a. Forms
 2b. Concurrent Program
 Checkbox
 Debug Options Button! (Binds and Waits)
 SQL Program (Binds and Waits)
 2c. Trace ALL user actions
 2d. Web-based screen
 3. Finding Trace Files?
 
3a. In 11G:
 3b. Before 11G:
 4. Other Tools
 4a. Using TKPROF
 4b. Using TraceAnalyzer
 4c. SQLT
 4d. RDA
 4. Before 11i?
 5. Tracing API Calls
References

APPLIES TO:

Oracle E-Business Suite Performance - Version 11.5.10.2 to 12.2 [Release 11.5.10 to 12.2]
Oracle Inventory Management - Version 12.1.3 to 12.1.3 [Release 12.1]
Information in this document applies to any platform.

PURPOSE

Here are some common questions or tips about Tracing, SQL Trace, TKProf, and Trace Analyzer related to the Oracle E-Business Applications 11i and R12.

QUESTIONS AND ANSWERS

1. Overview

1a. What is a trace file?

A trace file is a log of SQL run in a particular session or sessions focused on selects, inserts, updates, and deletes. A trace file can be used in many circumstances including reviewing performance, finding tables and views referenced, or finding the root of an error. More details about trace files can be read in the database user's guide related to your version.

Example #1: ORA-Error

The following is an snippet from a trace file that had an Oracle standard error. In this case it is a unique constraint violation, ORA 1 unique constraint. See the line: ERROR #117:err=1 tim=1443801753.  The ERROR flags the line and the 'err=1' lists the Oracle error as ORA-1:

=====================
PARSING IN CURSOR #117 len=1828 dep=1 uid=173 oct=2 lid=173 tim=1203661932011076 hv=1015075934 ad='cc60a820'
INSERT INTO BOM_SUB_OPERATION_RESOURCES (OPERATION_SEQUENCE_ID, SUBSTITUTE_GROUP_NUM, RESOURCE_ID, SCHEDULE_SEQ_NUM,
REPLACEMENT_GROUP_NUM, ACTIVITY_ID, STANDARD_RATE_FLAG, ASSIGNED_UNITS, USAGE_RATE_OR_AMOUNT, USAGE_RATE_OR_AMOUNT_INVERSE, BASIS_TYPE,
SCHEDULE_FLAG, LAST_UPDATE_DATE, LAST_UPDATED_BY, CREATION_DATE, CREATED_BY, LAST_UPDATE_LOGIN, RESOURCE_OFFSET_PERCENT, AUTOCHARGE_TYPE,
ATTRIBUTE_CATEGORY, REQUEST_ID, PROGRAM_APPLICATION_ID, PROGRAM_ID, PROGRAM_UPDATE_DATE, ATTRIBUTE1, ATTRIBUTE2, ATTRIBUTE3, ATTRIBUTE4,
ATTRIBUTE5, ATTRIBUTE6, ATTRIBUTE7, ATTRIBUTE8, ATTRIBUTE9, ATTRIBUTE10, ATTRIBUTE11, ATTRIBUTE12, ATTRIBUTE13, ATTRIBUTE14, ATTRIBUTE15,
PRINCIPLE_FLAG, SETUP_ID, CHANGE_NOTICE, ACD_TYPE, ORIGINAL_SYSTEM_REFERENCE ) SELECT DISTINCT A.OPERATION_SEQUENCE_ID,
B.SUBSTITUTE_GROUP_NUM, D.RESOURCE_ID, B.SCHEDULE_SEQ_NUM, B.REPLACEMENT_GROUP_NUM, B.ACTIVITY_ID, B.STANDARD_RATE_FLAG,
B.ASSIGNED_UNITS, B.USAGE_RATE_OR_AMOUNT, B.USAGE_RATE_OR_AMOUNT_INVERSE, B.BASIS_TYPE, B.SCHEDULE_FLAG, SYSDATE, :B4 ,
SYSDATE, :B4 , NULL, B.RESOURCE_OFFSET_PERCENT, B.AUTOCHARGE_TYPE, B.ATTRIBUTE_CATEGORY, FND_GLOBAL.CONC_REQUEST_ID, NULL,
FND_GLOBAL.CONC_PROGRAM_ID, SYSDATE, B.ATTRIBUTE1, B.ATTRIBUTE2, B.ATTRIBUTE3, B.ATTRIBUTE4, B.ATTRIBUTE5, B.ATTRIBUTE6, B.ATTRIBUTE7,
B.ATTRIBUTE8, B.ATTRIBUTE9, B.ATTRIBUTE10, B.ATTRIBUTE11, B.ATTRIBUTE12, B.ATTRIBUTE13, B.ATTRIBUTE14, B.ATTRIBUTE15, B.PRINCIPLE_FLAG,
B.SETUP_ID, DECODE (:B2 , 1, B.CHANGE_NOTICE, :B3 ), DECODE (:B2 , 1, B.ACD_TYPE, 1), B.ORIGINAL_SYSTEM_REFERENCE
FROM BOM_OPERATION_RESOURCES A, BOM_SUB_OPERATION_RESOURCES B, BOM_RESOURCES C, BOM_RESOURCES D
WHERE A.LAST_UPDATED_BY = B.OPERATION_SEQUENCE_ID AND B.RESOURCE_ID = C.RESOURCE_ID AND C.RESOURCE_CODE = D.RESOURCE_CODE
AND D.ORGANIZATION_ID = :B1 AND NVL (D.DISABLE_DATE, SYSDATE + 1) > SYSDATE
END OF STMT
EXEC #117:c=2099681,e=2436776,p=315,cr=5416,cu=19,mis=0,r=0,dep=1,og=1,tim=1203661932011073
ERROR #117:err=1 tim=1443801753
=====================

 

Example #2: Rollback

Another common thing to look for in trace files are rollbacks. There are several formats that you might see for a rollback including the word 'rollback' or text like 'XCTEND rlbk=1, rd_only=1'.

=====================
PARSING IN CURSOR #118 len=30 dep=1 uid=173 oct=45 lid=173 tim=1203661932011639 hv=4099445834 ad='0'
ROLLBACK TO BEGIN_ROUTING_COPY
END OF STMT
EXEC #118:c=0,e=124,p=0,cr=0,cu=21,mis=0,r=0,dep=1,og=0,tim=1203661932011637
RPC EXEC:c=2105681,e=2524801
BINDS #116:
=====================

 

1b. What are bind variables?

A bind variable is a substitution value used within a trace file. To improve performance, the Oracle database will reuse SQL substituting values for variables as necessary. For example, the same SQL statement could be run for each
of your employees. The employee id might be a variable that is changed before you run each SQL. This could be translated to the database as a bind variable so that the same SQL can be used each time but the correct variable can be substituted just before the SQL is run.

If someone requests that you enable bind variables for your trace file, this causes the database to document the substitution variables within the trace file as well as the SQL run. For example, we may see that a select statement ran against the employee table ten times, but without the binds, we would not know what employee was selected each time.

Example #1: Binds

Here is an example of a trace file with binds. In this example, we see a query against a table called MTL_SYSTEM_ITEMS_VL. Bind variables are used in the where-clause to substitute for the following ":num", ":SEGMENT1", and ":FND_FLEX_WHERE_BIND_0".

=====================
PARSING IN CURSOR #78 len=785 dep=0 uid=173 oct=3 lid=173 tim=1203661928858323 hv=3885565797 ad='d16ab6f8'
SELECT ENABLED_FLAG, SUMMARY_FLAG, NVL(TO_CHAR(START_DATE_ACTIVE, 'J'), 0),NVL(TO_CHAR(END_DATE_ACTIVE, 'J'), 0),SEGMENT1,
DESCRIPTION, PRIMARY_UOM_CODE, LOCATION_CONTROL_CODE, RESTRICT_LOCATORS_CODE, RESTRICT_SUBINVENTORIES_CODE,
INVENTORY_ASSET_FLAG, product_family_item_id, BOM_ITEM_TYPE, SERIAL_NUMBER_CONTROL_CODE, INVENTORY_ITEM_ID
FROM MTL_SYSTEM_ITEMS_VL WHERE ORGANIZATION_ID = :num AND SEGMENT1 = :SEGMENT1
AND  ( bom_enabled_flag = 'Y' And bom_item_type <> 3 And pick_components_flag = 'N'  And eng_item_flag = 'N'
And not exists (Select null From bom_operational_routings bor Where bor.assembly_item_id = inventory_item_id
And bor.organization_id = :FND_FLEX_WHERE_BIND_0 And bor.alternate_routing_designator is null And bor.routing_type <> 1)
And eam_item_type IS NULL  ) 
END OF STMT
EXEC #78:c=0,e=108,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=1203661928858320
FETCH #78:c=0,e=76,p=0,cr=10,cu=0,mis=0,r=1,dep=0,og=1,tim=1203661928858425
BINDS #78:
kkscoacd
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=01 fl2=1000000 frm=00 csi=00 siz=2056 off=0
  kxsbbbfp=2a9cff79f0  bln=22  avl=03  flg=05
  value=101
 Bind#1
  oacdty=01 mxl=32(15) mxlc=00 mal=00 scl=00 pre=00
  oacflg=01 fl2=1000000 frm=01 csi=873 siz=0 off=24
  kxsbbbfp=2a9cff7a08  bln=32  avl=09  flg=01
  value="MRS6814MC"
 Bind#2
  oacdty=01 mxl=2000(255) mxlc=00 mal=00 scl=00 pre=00
  oacflg=01 fl2=1000000 frm=01 csi=873 siz=0 off=56
  kxsbbbfp=2a9cff7a28  bln=2000  avl=03  flg=01
  value="101"
EXEC #78:c=0,e=81,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=1203661928859390
FETCH #78:c=0,e=36,p=0,cr=10,cu=0,mis=0,r=1,dep=0,og=1,tim=1203661928859438
=====================

 

Example #2: Binds

Here is another example where you see the bind numbers. This is more common. Note that the binds are off by one -- bind#0 goes with the variable :b1, bind#1 goes with variable :b2, etc.

=====================
PARSING IN CURSOR #128 len=863 dep=0 uid=173 oct=6 lid=173 tim=1203661928862922 hv=1516422370 ad='d1d91940'
UPDATE BOM_OPERATIONAL_ROUTINGS SET ROUTING_SEQUENCE_ID = :b1,ASSEMBLY_ITEM_ID = :b2,ORGANIZATION_ID = :b3,ALTERNATE_ROUTING_DESIGNATOR = :b4,
LAST_UPDATE_DATE = :b5,LAST_UPDATED_BY = :b6,LAST_UPDATE_LOGIN = :b7 WHERE ROWID  =  :b8  
END OF STMT
PARSE #128:c=0,e=442,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,tim=1203661928862919
BINDS #128:
kkscoacd
 Bind#0
  oacdty=02 mxl=23(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=2200 off=0
  kxsbbbfp=2a9cff7cd8  bln=23  avl=04  flg=05
  value=30020
 Bind#1
  oacdty=02 mxl=23(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=2a9cff7cf0  bln=23  avl=04  flg=01
  value=13003
 Bind#2
  oacdty=02 mxl=23(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=48
  kxsbbbfp=2a9cff7d08  bln=23  avl=03  flg=01
  value=101
 Bind#3
  oacdty=01 mxl=2000(255) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=01 csi=873 siz=0 off=72
  kxsbbbfp=2a9cff7d20  bln=2000  avl=00  flg=01
 Bind#4
  oacdty=12 mxl=07(07) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=2072
  kxsbbbfp=2a9cff84f0  bln=07  avl=07  flg=01
  value="1/21/2009 8:56:55"
 Bind#5
  oacdty=02 mxl=23(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=2080
  kxsbbbfp=2a9cff84f8  bln=23  avl=03  flg=01
  value=1555
 Bind#6
  oacdty=02 mxl=23(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=0 off=2104
  kxsbbbfp=2a9cff8510  bln=23  avl=04  flg=01
  value=370867
  Bind#7
  oacdty=01 mxl=32(18) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=01 csi=873 siz=0 off=2168
  kxsbbbfp=2a9cfef550  bln=32  avl=18  flg=01
  value="AABDb6AAiAAAGBqAA3"
=====================

 

2. How to Enable Trace In...

2a. Forms

How does one enable trace in the Oracle Application screens / forms?

One can enable trace through the forms by using the Help menu, choosing the diagnostics menu, trace and then selecting the appropriate trace for your needs. Most commonly if debugging an error, you should at least provide trace with binds.  When debugging a performance issue, you may consider using trace with binds and waits.

For example, the following is the navigation to enable trace in a form:
1. Goto the Oracle Applications
2. Login
3. Open the form where the error occurs but do not yet cause the error.
4. Enable SQL*Trace by choosing Help > Diagnostics > Trace > Trace with binds
5. A message appears indicating that the trace will be recorded
6. Note the file name and location of the file
7. Now reproduce the error.
8. Once the error occurs, disable trace as soon as possible.
9. Disable SQL*Trace by choosing Help > Diagnostics > Trace > Trace off
10. The same file name and location will be noted in case you need it again.
11. Retrieve the trace file.

Enable trace on form.

2b. Concurrent Program

How does one enable trace for a concurrent program?

Note: For R12.1 and higher, the concurrent program submission screen now has a "Debug Options" button where you can enable trace. Make sure you have the Patch 8743459 described in Note 1273539.1 for this to work. Without the patch it will not work.

Checkbox

With system administration privileges, a simple way to enable trace in a concurrent program is to review the concurrent program definition and select trace enabled. This will enable trace without binds for all users that run the program.

For example, the following steps could be used.
1. Goto Sysadmin > Concurrent > Program > Define
2. Query the concurrent program
3. Check the trace box to enable trace
Enable trace with checkbox
If you require bind variables for your trace file from a concurrent program, a more sophisticated method is required as noted below.

Debug Options Button! (Binds and Waits)

When submitting a concurrent program in R12.1 and higher, the concurrent program submission screen has a "Debug Options" button where you can enable trace. The button opens a window with various SQL Tracing options including Binds and Waits. The selection applies just to the current run of the program.

Enable trace with debug options

Note: Make sure you have the patch described in Note 1273539.1 for this to work.

 

SQL Program (Binds and Waits)

How does one enable trace for a concurrent program INCLUDING bind variables and waits?

NOTE: See Note 301372.1 for another method using the Profile "Concurrent: Allow Debugging" for 11.5.10 and R12.x.

Trace can be enabled for a concurrent program by enabling trace on the program definition for all users as noted above, but this does not include bind variables or wait times. To enable trace with bind variables for a concurrent program, you may consider the following approach that uses the bde script for concurrent program tracing with binds and waits using event 10046 at level 12.  

*** This method will enable trace with binds for all new sessions and connections
1. Download and review the script, bde_system_event_10046.sql, from Note.179848.1.
2. Run the script and press enter when prompted:

SQL> bde_system_event_10046.sql

3. Return to the application
4. Execute the concurrent program
5. Monitor the program until it begins to run
6. Return to SQL*Plus and press enter to cancel tracing new programs

2c. Trace ALL user actions

How does one enable trace for all actions that occur for a user?

One can enable trace for all actions that a user takes by setting a profile option for the user. See Note 170223.1 for detailed screen shots. This will function when the user logs into the Oracle Applications forms or when logging into the Self Service Web Applications. This method uses an Event Trace.

The instructions for enabling the event trace follow:
a. First enable trace for a specific user:
- Bring up the Core application - System Administrator responsibility
- Move to Profile/System
- Check off the USER box - and enter your username that you are using when getting the error
- Search on the following profile option - 'Initialization SQL Statement - Custom'

Please set this at the user level with the following string:

BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'4269824.999' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;

(Cut and paste the above sql statement into the User Level profile option: "Initialization SQL - Custom")
This must be one complete line of text - so may be best to copy the string into notepad prior to putting it into the profile.

b. Once the profile is set, please change responsibilities and directly go and reproduce the problem.
After 11G and in 12C, etc. check the "Diag Trace". To locate the trace file, use the following sql in sqlplus:

SQL> SELECT value FROM V$DIAG_INFO WHERE NAME = 'Diag Trace';

Before 11G, to locate the trace file, use the following sql in sqlplus:

SQL> select name, value from v$parameter where name like 'user_dump_dest';

- The value is the location of the trace file on the database server

- The value is the location of the trace file on the database server
- Please move to this directory and search for the file having 4269824.999 in the filename
- This is the trace file created - please tkprof and upload BOTH the raw and tkprof trace file to My Oracle Support.

c. Ensure that the profile option is unset before exiting the Application.
This is a very important step. If this is not unset, unnecessary trace files will be generated.

d. Note that a similar technique can be used for enabling trace with binds for a user in SQL*Plus. The commands would look like the following:

SQL> ALTER SESSION SET TRACEFILE_IDENTIFIER='577' EVENTS ='10046 TRACE NAME CONTEXT FOREVER, LEVEL 12';
SQL> select 'x' from dual; -- or any SQL that you want to trace
SQL> alter session set sql_trace = false;

 

2d. Web-based screen

How does one enable trace for a web-based screen like found in the self service applications?

You have a few options with web-based screens. You can use Diagnostics > Enable trace.

Web-based trace

Or you can use the same method as noted above in "How does one enable trace for all actions that occur for a user?" Also review Note.245974.1, section "10. How To Generate and Retrieve Trace Files in SSA (Self Service Applications)".  Section 10 discusses techniques available within Self Service for debugging and tracing.

3. Finding Trace Files?

How does one find a trace file?

Some trace files like in forms are specified when you enable and disable trace. Others are a bit trickier to identify. In concurrent programs for example, the user is not given the trace file number and location.  It is common for people to look for the last updated trace files or to use UNIX functions like 'grep' to parse the trace files looking for the one related to their concurrent program. However, the following SQL can be used to assist in identifying the trace file associated to a concurrent program. The SQL prompts the user for the request id and outputs the trace file name and location.

Warning: Some users have pointed out that this SQL does not always point out the exact filename and location. On a multi-node database environment, the user might not know what machine to look at. Also, the construction of the actual filename may be different on your environment as this is configurable. If your filenames are configured differently, you may want to use number part of the filename (the process id), then search for files in your trace directory with the same id. A trick that may help you find the correct trace file directory is to enable trace via the Application user using Help > Diagnostics > Trace > Trace On. This will popup a window indicating the directory where trace files are written.

Finally, the oracle_process_id is selected from v$process so this ID is only available during runtime of the process.
It gets cleared shortly after the process terminated. Then you cannot find the trace file for a concurrent program anymore. Therefore, you should run this SQL shortly after running the concurrent program.


3a. In 11G:

This PL/SQL does not link to the Concurrent Request Id but attempts to point to the directory where trace files are located based on your database version. The trace directory parameter changed in 11G onward and 12C to "Diag Trace". It used to be "user_dump_dest".

Set Serveroutput ON

DECLARE
DB_Version VARCHAR2(2);
Trace_location VARCHAR2(240);
BEGIN
SELECT SUBSTR(vi.version, 1, INSTR(vi.version, '.')-1) INTO DB_Version FROM v$instance vi;

IF TO_NUMBER(DB_Version) >= 11 THEN
/* Following Line included as Execute Immediate as V$DIAG_INFO does not exist in DB Versions prior to 11g */
EXECUTE IMMEDIATE 'SELECT value FROM V$DIAG_INFO WHERE NAME = ''Diag Trace''' INTO Trace_location ;
ELSE
SELECT value INTO Trace_location FROM v$parameter WHERE name = 'user_dump_dest';
END IF ;
dbms_output.put_line('Trace File should be located in the directory :- '|| trace_location);
END;
/

 

3b. Before 11G:

This SQL attempts to lookup the trace filename and location based on the request id. This is no longer accurate  in 11G and higher.

SQL>

prompt
accept request prompt 'Please enter the concurrent request id for the appropriate concurrent program:'
prompt

column traceid format a8
column tracename format a80
column user_concurrent_program_name format a40
column execname format a15
column enable_trace format a12
set lines 80
set pages 22
set head off

SELECT 'Request id: '||request_id ,
'Trace id: '||oracle_Process_id,
'Trace Flag: '||req.enable_trace,
'Trace Name:
'||dest.value||'/'||lower(dbnm.value)||'_ora_'||oracle_process_id||'.trc',
'Prog. Name: '||prog.user_concurrent_program_name,
'File Name: '||execname.execution_file_name|| execname.subroutine_name ,
'Status : '||decode(phase_code,'R','Running')
||'-'||decode(status_code,'R','Normal'),
'SID Serial: '||ses.sid||','|| ses.serial#,
'Module : '||ses.module
from fnd_concurrent_requests req, v$session ses, v$process proc,
v$parameter dest, v$parameter dbnm, fnd_concurrent_programs_vl prog,
fnd_executables execname
where req.request_id = &request
and req.oracle_process_id=proc.spid(+)
and proc.addr = ses.paddr(+)
and dest.name='user_dump_dest'
and dbnm.name='db_name'
and req.concurrent_program_id = prog.concurrent_program_id
and req.program_application_id = prog.application_id
and prog.application_id = execname.application_id
and prog.executable_id=execname.executable_id;

 

4. Other Tools

Are there any other tools that can assist in reviewing trace files (what is TraceAnalyzer)?

4a. Using TKPROF

A trace file can be reviewed using TKPROF. TKPROF reformats the raw data so that it is easier to review. The TKPROF commands are normally run at the operating system command prompt.  This will be signified with a $ as this is a common prompt in UNIX.
1. Retrieve the trace file.
2. Issue a command like the following to create a TKPROF version of the trace file.  The explain option will look to see how each query is handled by the database in terms of what tables and indexes are referenced.

$ tkprof <filename.trc> <output_filename> sys=no explain=apps/<password>

However, TKPROF does not take into account bind variables. Therefore, these are ignored in the output. When you need to analyze bind variables, consider using TraceAnalyzer.

A handy technique when trouble shooting performance issues is to use TKPROF to look at the longest running queries.  Since trace files related to performance can be huge, one might spend hours looking through the results to find the queries causing the issue.  If you sort the file by the longest running queries first, it makes it much easier to investigate.  The following example sorts by longest running queries first with the sort options selected as "sort='(prsela,exeela,fchela)'" and limits the results to the "Top 10" long running queries with the "print=10" command:

$ tkprof <filename.trc> <output_filename> sys=no explain=apps/<password>
sort='(prsela,exeela,fchela)' print=10

4b. Using TraceAnalyzer

Please refer to Note 224270.1: Trace Analyzer

TraceAnalyzer reformats the raw data so that it is easier to review. It is a more powerful tool than TKPROF. It translates things like bind variables and make them easy to find.

1. Download TraceAnalyzer from Note 224270.1
2. Install trace analyzer in SQL:

SQL> start TRCACREA.sql

3. Retrieve the trace file.
4. In SQL*Plus, issue a command like the following to create a TraceAnalyzer version of the trace file:

SQL> start TRCANLZR.sql UDUMP vis015_ora_22854.trc

4c. SQLT

Please refer to Note 215187.1 for details on SQLTXPLAIN (SQLT). 

4d. RDA

Is RDA Another technique for finding / supplying trace files?

Check out the RDA output that can grab the last trace file and you could then upload the RDA to the Service Request.

  • Note 559800.1 How To Run Application Collection Tool (ACT/RDA) Diagnostics Test?

... 4e. AWR

See Note 748642.1 - What is AWR( Automatic workload repository ) and How to generate the AWR report?

 

4. Before 11i?

Will these techniques work in previous versions before 11i (For example, 10.7 and 11.0.3)?

Many of the same techniques will work in previous versions of the Oracle Applications. However, there are some limitations for example, in 10.7 and 11.0.3, the application did not indicate the trace file name when a user enabled trace using Help > Diagnostics > Trace. Also, the previous versions of the application did not allow for the user to enable trace with binds using the Help > Diagnostics navigation path.

See also : Note 171647.1 - Tracing Oracle Applications using Event 10046 in 10.7, 11.0, and 11i

5. Tracing API Calls

See Note 869386.1.

REFERENCES

NOTE:130182.1:142898.1
NOTE:301372.1:38018.1
NOTE:115698.1:117129.1
NOTE:179848.1:224270.1
NOTE:1007808.6:100964.1
NOTE:76338.1:869386.1
NOTE:245974.1:279132.1

Concurrent Processing - Command-Line Utility OR Recovery Wizard | cpadmin.sh

TROUBLESHOOTING STEPS

Concurrent Processing Command-Line Utility

The command-line utility cpadmin consolidates various existing utilities for concurrent processing into a single menu-based utility. This adadmin-style utility can be used for multiple tasks, including:
• Manager Status: Use this option to display the statuses of all managers.
• Clean CP Tables: Use this option to clean up the concurrent processing tables. This utility replaces cmclean.sql.
• Set Manager Diagnostics: Turn diagnostics on or off for individual managers with this option.
• Manager Control: Use this option to send a request such as start, stop, or verify to an individual manager.
• Rebuild Manager Views: Use this option to rebuild the FND_CONCURRENT_WORKER_REQUEST and FND_CONCURRENT_CRM_REQUESTS views.
• Move Request Files: Change request log and output file locations with this option.
• Analyze Request: Use this option to analyze a concurrent request.

Complete the steps below to run the command-line utility and its maintenance tasks.
1. Set the environment.
You must set the environment in order to apply the environment variables that define your system. This task is common to many utilities. See Setting the Environment, Oracle E-Business Suite Maintenance Guide for the preparatory steps.
2. From any directory, start cpadmin with this command:
$ cpadmin.sh
The utility starts and prompts you for the APPS password (required).
3. Respond to prompts.
Supply the information requested by cpadmin prompts. Prompts unique to an option are described with the option.
When you complete the prompts, the Main Menu appears.
4. Choose one of the tasks listed above.
5. Exit the cpadmin utility.

This is documented here: http://docs.oracle.com/cd/E26401_01/doc.122/e22953/T174296T575591.htm#8159578

Concurrent Manager Recovery Wizard

NOTE: The Concurrent Managers have to be stopped before the Wizard is run. Also insure that listeners are not stopped before the managers are stopped.

Please run the Concurrent Manager Recovery wizard to address Concurrent Manager failures within the Oracle Applications Manager.

Using the Concurrent Manager Recovery wizard is the method to clear the errors upon bringing the Internal Manager back up.

For Concurrent Internal Manager failures, it is recommended to run the Concurrent Manager Recovery feature using the Oracle Applications Manager. This feature should be used for recovering from when the Internal Manager won't start. This is accessed from the Troubleshooting wizards available within Oracle Applications Manager logged in as the sysadmin userid.

Navigate:

Oracle Applications Manager > Concurrent Managers OR Concurrent Requests > Site Map > Diagnostics and Repair > Concurrent Manager Recovery
Shows Navigation To Run The Wizard
For information on the Concurrent Manager Recovery feature, please reference the Oracle Applications System Administrator's Guide - Maintenance provides information for frequent tasks such as monitoring your system with Oracle Applications Manager, administering Oracle E-Business Suite Secure Enterprise.  Search, managing concurrent managers and reports, using diagnostic utilities including logging, managing profile options, and using alerts.

 

Concurrent Processing - Command-Line Utility OR Recovery Wizard (Doc ID 134007.1)

Thursday, August 18, 2022

Viewing Concurrent Request Output and Log Files in HTML/Browser

If the ouput and log files of the concurrent needs to be viewed in Browser, then we can perform the below action item. 1. Goto System Administrator responsibility and select Profile>System 2. Search for the 'Viewer:%' profile. 3. Set the profile Viewer: Application for Text to Browser. If this has to be set for all users then it has to be set at Site Level or for a particular user we can set it at user level. https://www.funoracleapps.com/2013/05/viewing-request-output-and-log-files-in.html

ORA-31626: job does not exist

dsvmofs> expdp system/XXXXX parfile=auexpfulltts.dat Export: Release 11.2.0.4.0 - Production on Wed Aug 17 05:29:16 2022 Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options ORA-31626: job does not exist ORA-31633: unable to create master table "SYSTEM.SYS_EXPORT_FULL_05" ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95 ORA-06512: at "SYS.KUPV$FT", line 1038 ORA-00372: file 7 cannot be modified at this time ORA-01110: data file 7: '/OFSSITDATA2/oradata/owad01.dbf' dsvmofs> cat auexpfulltts.dat directory=dmpdir metrics=y dumpfile=xttsfulltts%U.dmp filesize=1048576000 full=y exclude=STATISTICS logfile=expfulltts.log parallel=8 transportable=always version=12 dsvmofs> Root cause : Customer has SYS related tables in OWAPUB Tablespace, We made OWAPUB readonly as pert of TTS Activity. Solution : Moved the SYS related tables from OWAPUB tablespace to SYS. Other alternative solutions 1. Before restarting the expdp check if there are orphan Data Pump jobs left in database. Use << Note 336014.1 >> 2. After clearing the orphaned jobs if you still see the errors, Upload output of below queries. connect as sysdba spool boxa.html set pagesize 9999 set markup html on select name from v$database; SELECT * from v$version ; SELECT distinct(length(addr)*4) "Word Size" from v$process; SELECT owner,object_name,object_type,status from dba_objects where status = 'INVALID'; select owner,object_type,count(*) from dba_objects where status='INVALID' group by owner,object_type; SELECT SUBSTR(comp_id,1,15) comp_id, status, SUBSTR(version,1,10) version, modified, SUBSTR(comp_name,1,30) comp_name, substr(schema,1,15) as schema FROM dba_registry order by comp_id; select * from registry$history;

How To Drop A NOT RUNNING DataPump Job? (Doc ID 1456784.1)

A DataPump import job created through DBMS_DATAPUMP API is listed as NOT RUNNING when querying DBA_DATAPUMP_JOBS: SQL> connect / as sysdba SQL> SELECT o.status, o.object_id, o.object_type, o.owner||'.'||object_name "OWNER.OBJECT" FROM dba_objects o, dba_datapump_jobs j WHERE o.owner=j.owner_name AND o.object_name=j.job_name AND j.job_name NOT LIKE 'BIN$%' ORDER BY 4,2; STATUS OBJECT_ID OBJECT_TYPE OWNER.OBJECT ------- ---------- ------------------- -------------------------------------------------------- VALID 1015301 TABLE SYS.daily_import When trying to drop the master table it fails with: SQL> DROP TABLE SYS.daily_import; DROP TABLE SYS.daily_import * ERROR at line 1: ORA-00942: table or view does not exist You need to specify double quotes around the schema owner and the table name when dropping it: SQL> DROP TABLE "SYS"."daily_import"; SQL> purge recyclebin;

Wednesday, August 17, 2022

How To Cleanup Orphaned DataPump Jobs In DBA_DATAPUMP_JOBS ? (Doc ID 336014.1)

SOLUTION The jobs used in this example: - Export job .EXPDP_20051121 is a schema level export that is running - Export job .SYS_EXPORT_TABLE_01 is an orphaned table level export job - Export job .SYS_EXPORT_TABLE_02 is a table level export job that was stopped - Export job SYSTEM.SYS_EXPORT_FULL_01 is a full database export job that is temporary stopped Step 1. Determine in SQL*Plus which Data Pump jobs exist in the database: %sqlplus /nolog CONNECT / as sysdba SET lines 200 COL owner_name FORMAT a10; COL job_name FORMAT a20 COL state FORMAT a12 COL operation LIKE state COL job_mode LIKE state COL owner.object for a50 -- locate Data Pump jobs: SELECT owner_name, job_name, rtrim(operation) "OPERATION", rtrim(job_mode) "JOB_MODE", state, attached_sessions FROM dba_datapump_jobs WHERE job_name NOT LIKE 'BIN$%' ORDER BY 1,2; OWNER_NAME JOB_NAME OPERATION JOB_MODE STATE ATTACHED ---------- ------------------- --------- --------- ----------- -------- EXPDP_20051121 EXPORT SCHEMA EXECUTING 1 SYS_EXPORT_TABLE_01 EXPORT TABLE NOT RUNNING 0 SYS_EXPORT_TABLE_02 EXPORT TABLE NOT RUNNING 0 SYSTEM SYS_EXPORT_FULL_01 EXPORT FULL NOT RUNNING 0 Step 2. Ensure that the listed jobs in dba_datapump_jobs are not export/import Data Pump jobs that are active: status should be 'NOT RUNNING'. Step 3. Check with the job owner that the job with status 'NOT RUNNING' in dba_datapump_jobs is not an export/import Data Pump job that has been temporary stopped, but is actually a job that failed. (E.g. the full database export job by SYSTEM is not a job that failed, but was deliberately paused with STOP_JOB). Step 4. Identify orphan DataPump external tables. To do this, run the following as SYSDBA in SQL*Plus: set linesize 200 trimspool on set pagesize 2000 col owner form a30 col created form a25 col last_ddl_time form a25 col object_name form a30 col object_type form a25 select OWNER,OBJECT_NAME,OBJECT_TYPE, status, to_char(CREATED,'dd-mon-yyyy hh24:mi:ss') created ,to_char(LAST_DDL_TIME , 'dd-mon-yyyy hh24:mi:ss') last_ddl_time from dba_objects where object_name like 'ET$%' / select owner, TABLE_NAME, DEFAULT_DIRECTORY_NAME, ACCESS_TYPE from dba_external_tables order by 1,2 / Correlate the information from DBA_OBJECTS and DBA_EXTERNAL TABLES above to identify the temporary external tables that belong to the DataPump orphaned jobs. Drop the temporary external tables that belong to the DataPump orphaned job. eg: SQL> drop table system.&1 purge; Enter value for 1: ET$00654E1E0001 old 1: drop table system.&1 purge new 1: drop table system.ET$00654E1E0001 purge Step 5. Determine in SQL*Plus the related master tables: -- locate Data Pump master tables: COL owner.object FORMAT a50 SELECT o.status, o.object_id, o.object_type, o.owner||'.'||object_name "OWNER.OBJECT" FROM dba_objects o, dba_datapump_jobs j WHERE o.owner=j.owner_name AND o.object_name=j.job_name AND j.job_name NOT LIKE 'BIN$%' ORDER BY 4,2; STATUS OBJECT_ID OBJECT_TYPE OWNER.OBJECT ------- ---------- ------------ ------------------------- VALID 85283 TABLE .EXPDP_20051121 VALID 85215 TABLE .SYS_EXPORT_TABLE_02 VALID 85162 TABLE SYSTEM.SYS_EXPORT_FULL_01 select table_name, owner from dba_external_tables; Step 6. For jobs that were stopped in the past and won't be restarted anymore, delete the master table. E.g.: DROP TABLE .sys_export_table_02; -- For systems with recycle bin additionally run: purge dba_recyclebin; Note: ===== Following statement can be used to generate the drop table statement for the master table: SELECT 'DROP TABLE '||o.owner||'.'||object_name||' PURGE;' FROM dba_objects o, dba_datapump_jobs j WHERE o.owner=j.owner_name AND o.object_name=j.job_name AND j.job_name NOT LIKE 'BIN$%'; NOTE: In case the table name is mixed case, you can get errors on the drop, e.g.: SQL> drop table SYSTEM.impdp_schema_TEST_10202014_0; drop table SYSTEM.impdp_schema_TEST_10202014_0 * ERROR at line 1: ORA-00942: table or view does not exist Because the table has a mixed case, try using these statements with double quotes around the table name, for instance: drop table SYSTEM."impdp_SCHEMA_TEST_04102015_1"; drop table SYSTEM."impdp_schema_TEST_10202014_0"; Step 7. Re-run the query on dba_datapump_jobs and dba_objects (step 1 and 4). If there are still jobs listed in dba_datapump_jobs, and these jobs do not have a master table anymore, cleanup the job while connected as the job owner. E.g.: CONNECT / SET serveroutput on SET lines 100 DECLARE h1 NUMBER; BEGIN h1 := DBMS_DATAPUMP.ATTACH('SYS_EXPORT_TABLE_01',''); DBMS_DATAPUMP.STOP_JOB (h1); END; / Note that after the call to the STOP_JOB procedure, it may take some time for the job to be removed. Query the view user_datapump_jobs to check whether the job has been removed: CONNECT / SELECT * FROM user_datapump_jobs; Step 8. Confirm that the job has been removed: CONNECT / as sysdba SET lines 200 COL owner_name FORMAT a10; COL job_name FORMAT a20 COL state FORMAT a12 COL operation LIKE state COL job_mode LIKE state COL owner.object for a50 -- locate Data Pump jobs: SELECT owner_name, job_name, rtrim(operation) "OPERATION", rtrim(job_mode) "JOB_MODE", state, attached_sessions FROM dba_datapump_jobs WHERE job_name NOT LIKE 'BIN$%' ORDER BY 1,2; OWNER_NAME JOB_NAME OPERATION JOB_MODE STATE ATTACHED ---------- ------------------- --------- --------- ----------- -------- EXPDP_20051121 EXPORT SCHEMA EXECUTING 1 SYSTEM SYS_EXPORT_FULL_01 EXPORT FULL NOT RUNNING 0 -- locate Data Pump master tables: SELECT o.status, o.object_id, o.object_type, o.owner||'.'||object_name "OWNER.OBJECT" FROM dba_objects o, dba_datapump_jobs j WHERE o.owner=j.owner_name AND o.object_name=j.job_name AND j.job_name NOT LIKE 'BIN$%' ORDER BY 4,2; STATUS OBJECT_ID OBJECT_TYPE OWNER.OBJECT ------- ---------- ------------ ------------------------- VALID 85283 TABLE .EXPDP_20051121 VALID 85162 TABLE SYSTEM.SYS_EXPORT_FULL_01 Remarks: 1. Orphaned Data Pump jobs do not have an impact on new Data Pump jobs. The view dba_datapump_jobs is a view, based on gv$datapump_job, obj$, com$, and user$. The view shows the Data Pump jobs that are still running, or jobs for which the master table was kept in the database, or in case of an abnormal end of the Data Pump job (the orphaned job). If a new Data Pump job is started, a new entry will be created, which has no relation to the old Data Pump jobs. 2. When starting the new Data Pump job and using a system generated name, we check the names of existing Data Pump jobs in the dba_datapump_job in order to obtain a unique new system generated jobname. Naturally, there needs to be enough free space for the new master table to be created in the schema that started the new Data Pump job. 3. A Data Pump job is not the same as a job that is defined with DBMS_JOBS. Jobs created with DBMS_JOBS use there own processes. Data Pump jobs use a master process and worker process(es). In case a Data Pump still is temporary stopped (STOP_JOB while in interactive command mode), the Data Pump job still exists in the database (status: NOT RUNNING), while the master and worker process(es) are stopped and do not exist anymore. The client can attach to the job at a later time, and continue the job execution (START_JOB). 4. The possibility of corruption when the master table of an active Data Pump job is deleted, depends on the Data Pump job. 4.a. If the job is an export job, corruption is unlikely as the drop of the master table will only cause the Data Pump master and worker processes to abort. This situation is similar to aborting an export of the original export client. 4.b. If the job is an import job then the situation is different. When dropping the master table, the Data Pump worker and master processes will abort. This will probably lead to an incomplete import: e.g. not all table data was imported, and/or table was imported incomplete, and indexes, views, etc. are missing. This situation is similar to aborting an import of the original import client. The drop of the master table itself, does not lead to any data dictionary corruption. If you keep the master table after the job completes (using the undocumented parameter: KEEP_MASTER=Y), then a drop of the master table afterwards, will not cause any corruption. 5. Instead of the status 'NOT RUNNING' the status of a failed job could also be 'DEFINING'. When trying to attach to such a job, this would fail with: $ expdp system/ attach=system.sys_export_schema_01 Export: Release 11.2.0.4.0 - Production on Tue Jan 27 10:14:27 2015 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options ORA-31626: job does not exist ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79 ORA-06512: at "SYS.KUPV$FT", line 405 ORA-31638: cannot attach to job SYS_EXPORT_SCHEMA_01 for user SYSTEM ORA-31632: master table "SYSTEM.SYS_EXPORT_SCHEMA_01" not found, invalid, or inaccessible ORA-00942: table or view does not exist The steps to cleanup these failed/orphaned jobs are the same as mentioned above.

Friday, August 12, 2022

RMAN DUPLICATE STANDBY

run { allocate channel tgt1 device type disk ; allocate channel tgt2 device type disk ; allocate channel tgt3 device type disk ; allocate channel tgt4 device type disk ; allocate channel tgt5 device type disk ; allocate channel tgt6 device type disk ; allocate channel tgt7 device type disk ; allocate channel tgt8 device type disk ; allocate channel tgt9 device type disk ; allocate channel tgt10 device type disk ; allocate channel tgt11 device type disk ; allocate channel tgt12 device type disk ; allocate channel tgt13 device type disk ; allocate channel tgt14 device type disk ; allocate channel tgt15 device type disk ; allocate channel tgt16 device type disk ; allocate auxiliary channel aux1 device type disk ; allocate auxiliary channel aux2 device type disk ; allocate auxiliary channel aux3 device type disk ; allocate auxiliary channel aux4 device type disk ; allocate auxiliary channel aux5 device type disk ; allocate auxiliary channel aux6 device type disk ; allocate auxiliary channel aux7 device type disk ; allocate auxiliary channel aux8 device type disk ; allocate auxiliary channel aux9 device type disk ; allocate auxiliary channel aux10 device type disk ; allocate auxiliary channel aux11 device type disk ; allocate auxiliary channel aux12 device type disk ; allocate auxiliary channel aux13 device type disk ; allocate auxiliary channel aux14 device type disk ; allocate auxiliary channel aux15 device type disk ; allocate auxiliary channel aux16 device type disk ; DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE DORECOVER SPFILE SET db_unique_name=‘cvs19qas_stby’ COMMENT ‘Is standby’ set DB_CREATE_FILE_DEST = ‘+DATAC1’ set DB_CREATE_ONLINE_LOG_DEST_1 = ‘+DATAC1’ set DB_RECOVERY_FILE_DEST = ‘+RECOC1’ set diagnostic_dest=‘/u02/app/oracle’ set DB_RECOVERY_FILE_DEST_SIZE=‘1500G’ set audit_file_dest=‘/u02/app/oracle/product/19.0.0.0/dbhome_1/admin/cvs19qas/adump’ SET job_queue_processes=‘0’ SET LOCAL_LISTENER=‘’ SET REMOTE_LISTENER=‘’ set cluster_database=‘false’ NOFILENAMECHECK; }

Database Options/Management Packs Usage Reporting for Oracle Databases 11.2 and later (Doc ID 1317265.1)

  Database Options/Management Packs Usage Report You can determine whether an option is currently in use in a database by running options_pa...