12.2 E-Business Suite - Collecting Fusion Middleware Log Files (Doc ID 1362900.1)

In this Document

Goal

Solution

Section 1: Steps to collect log files

Alternative...

Section 2: Steps to configure higher Debug levels and collect log files

Section 4: Collecting configuration Information

Appendix A: Script for automatically clearing contents of log files

References

---

APPLIES TO:

Oracle E-Business Suite Technology Stack - Version 12.2 to 12.2.6 [Release 12.2]
Oracle E-Business Suite Performance - Version 12.2 to 12.2 [Release 12.2]
Information in this document applies to any platform.

GOAL

This note is intended to be used as a debugging tool for the Fusion Middleware 11g components of E-Business Suite R12.2 which include OPMN, OHS and WebLogic components.

These include:

• Oracle Process Manager and Notifications Server which manages the HTTP Server

• Oracle HTTP Server (Apache) which acts as a single Entry Point Proxy Server

• WebLogic Server which contains Oracle E-Business Suite Application Server Deployments

This note provides steps to enable extra debug and automate collection of log files required for debugging issues. Any changes made to increase log levels should be reverted once the information is collected. Oracle does not recommend detailed logging be turned on for extended periods, especially in production.

SOLUTION

Section 1: Steps to collect log files

This section should be followed by customers who are creating service requests for issues that have perhaps already occurred or which are not easily reproducible. This section does not enable extra debug - it simply collects log files at whatever debug levels are currently configured. In some cases the debug collected here may not suffice and in that case you may be directed by support to follow the steps in Section 2 to enable extra debug.

a) OPMN and OHS log files

Run this script as the owner of the applications file system to gather the log files for OPMN and OHS services:

zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_OPMN_OHS.zip $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/*/*log* $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/*

b) WebLogic Server

Run this script as the owner of the applications file system to gather the log files for Fusion Middleware components such as NodeManager, AdminServer, forms, oacore and oafm services:

zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_FMW.zip $IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager $EBS_DOMAIN_HOME/servers/oa*/logs/*out* $EBS_DOMAIN_HOME/servers/oa*/logs/*log* $EBS_DOMAIN_HOME/servers/forms*/logs/*out* $EBS_DOMAIN_HOME/servers/forms*/logs/*log* $EBS_DOMAIN_HOME/servers/AdminServer/logs/*out* $EBS_DOMAIN_HOME/servers/AdminServer/logs/*log* $EBS_DOMAIN_HOME/sysman/log/* $EBS_DOMAIN_HOME/servers/oac*/adr/diag/ofm/EBS_domain*/oac*/incident/* $EBS_DOMAIN_HOME/servers/forms_s*/adr/diag/ofm/EBS_domain*/forms_s*/incident/*

These 2 commands will create zip files in the /tmp directory with names including server name and date. e.g. server.customer.com_092811.2005_OPMN_OHS.zip.  Please collect these files from the /tmp directory and upload them to your service request.

Alternative...

Use the following script to gather the Apache, WLS and Apps log files which have been modified in the last 1 day

#####################################################################################
####     Start of script
#####################################################################################
(
# pick up files which have been modified in the last 1 day only
HowManyDaysOld=1
echo "Picking up files which have been modified in the last ${HowManyDaysOld} days"
set -x
find $LOG_HOME -type f -mtime -${HowManyDaysOld} > m.tmp
find $FMW_HOME/webtier/instances/*/diagnostics/logs -type f -mtime -${HowManyDaysOld} >> m.tmp
find $FMW_HOME/wlserver_10.3/common/nodemanager/nmHome*/*.log -type f -mtime -${HowManyDaysOld} >> m.tmp
##  Get log files for only the WLS servers needed.   Valid server names are one or more of:
##      AdminServer forms-c4ws_server  forms_server  oacore_server  oaea_server  oafm_server
for SERVERNAME in AdminServer oacore_server forms_server oafm_server
do
   find $EBS_DOMAIN_HOME/servers/${SERVERNAME}*/logs -type f -mtime -${HowManyDaysOld} >> m.tmp
   find $EBS_DOMAIN_HOME/servers/${SERVERNAME}*/adr/diag/ofm/EBS_domain*/${SERVERNAME}*/incident -type f -mtime -${HowManyDaysOld} >> m.tmp
done
zip -r mzAppsLogFiles_`hostname`_`date '+%m%d%y'`.zip -@ < m.tmp
rm m.tmp
) 2>&1 | tee mzLogZip.out
#####################################################################################
####     End of script
#####################################################################################

Section 2: Steps to configure higher Debug levels and collect log files

This section explains the steps to enable higher logging levels to capture debug information required for resolving problems. You may be directed to this section by Oracle Support. Follow the steps below to enable debug and collect the log files generated after reproducing the issue. Generally speaking, increasing the log levels as per below should have only a minimal impact on performance (max 5% degradation)

a) OPMN
To enable debug for opmn, edit $IAS_ORACLE_HOME/instances/<instance>/config/OPMN/opmn/opmn.xml

Change the following line

<debug comp="" rotation-size="1500000"/>
to
<debug comp="ons[all];pm[all]" rotation-size="1500000"/>

Alternatively you can increase OPMN logging for the current session by issuing command

$IAS_ORACLE_HOME/instances/<yourinstance>/bin/opmnctl set target=debug comp="ons[all];pm[all]"

This would enable debug for all components running on the OPMN process. However, this may not be practical in most situations.


b) Increasing OHS Logging
To adjust HTTP logging levels access the Enterprise Manager console as follows
http://server.domain:wls_admin_port/em
login as weblogic/<weblogic admin password>

Expand the Web Tier menu
Right click EBS_web_component and choose administration - log configuration. Here you have the option to set the Log level. For debug purposes you can set this to "Trace:32"

c) Node Manager
To adjust log levels for Node Manager edit $IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager/nodemanager.properties

adjust LogLevel=INFO to LogLevel=FINEST

d) Start and Stop LoggingTo adjust start/stop log levels access the Enterprise Manager console as follows
http://server.domain:wls_admin_port/em
login as weblogic/welcome1
Expand WebLogic Domain then expand EBS_domain_X
Right click AdminServer and select logs - log configuration
Change the logging level to Trace:32

e) Web Logic Logging

Step 1: Enable log recording levelsTo adjust WebLogic log levels access the WebLogic console as follows
http://server.domain:wls_admin_port/console
login as weblogic/<password>

Expand environment - servers

Select the servers to enable debug for. In most cases this will be the oacore servers. If you have more than one server defined in a cluster you will need to enable debug for each server individually. Only enable logging for the managed servers that are having issues.

Choose the logging tab, select the advanced hyperlink
Change severity level to Trace
Change Severity levels for standard out and domain log broadcaster to Debug

On this same configuration page you will see the setting "limit number of retained files". You may need to increase this value as extra logging may result in log files being overwritten due to excessive log file rotation.

Step 2: Enable debug areas
Continuing from step 1 above, select the debug tab
Expand the Weblogic section
Select the Servlet group which you wish to enable debug for (see below)
Choose enable to activate debug for these components.

Servlet	Enable this for all types of issues
JDBC	Enable logging for this component if your issue relates to JDBC connection problems
Classloader	Enable logging for this component for issues relating to classloading. Such as missing classes on startup of services

Note: If support or the customer is directed to enable any other component besides JDBC, servlet or classloader please inform the note author so these can be considered for addition to this note.


f) Restart Application Tier Services and clear log filesStop application tier services using the scripts located in $ADMIN_SCRIPTS_HOME. Clear the log files and then restart services
Note: Ideally at this stage it is a good idea to clear log files so that the problem can be pinpointed easily in the log files.This is especially true of production systems where logs can be large and almost un-manageable to debug. Log files should be cleared when services are down. You can do this manually or use the script provided in Appendix A

g) Reproduce issue and upload log files
Reproduce the issue and use the scripts in Section 1 to gather the log files. Upload the log files to your service request

Section 4: Collecting configuration Information

a) Run the technology stack inventory report to collect a list of patches applied to all middle tier homes (besides Weblogic). This script will also report the patch level for each oracle home:

$ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl -script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -contextfile=$CONTEXT_FILE -appspass=<appspassword> -outfile=$APPLTMP/Report_App_Inventory.html

b) Provide the following files which contain managed server configuration information:

$IAS_ORACLE_HOME/instances/EBS_web_<SID>_OHS1/config/OHS/EBS_web_<SID>  (ZIP all files)
$INST_TOP/appl/admin/*.properties
$EBS_DOMAIN_HOME/config/config.xml (contains managed server start / stop options and memory settings)
$EBS_APPS_DEPLOYMENT_DIR/[forms  forms-c4ws  oacore  oafm] /APP-INF/node_info.txt

---

Appendix A: Script for automatically clearing contents of log files

The following script can be run as the applmgr user with applications environment sourced. This script will automatically clear the contents of the log files so that fresh logs can be captured. All application tier services should be shutdown prior to running this script.

for files in $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn $EBS_DOMAIN_HOME/sysman/log
do
for file in `ls $files/*`
do
echo "Clearing file - " $file
cat /dev/null > $file
done
done
for files in $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/EBS_web_component $EBS_DOMAIN_HOME/servers/*/logs $IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager
do
for file in `ls $files/*log* $files/*out*| grep -v lck`
do
echo "Clearing file - " $file
cat /dev/null > $file
done
done

Note: one can comment the following line
#cat /dev/null > $file
so that the command identifies the files without making any changes. Once comfortable with the changes the comment can be removed and the script run again to clear the files


References
For more information on logging definition, rotation, location and control review Document 1366187.1 - Oracle Applications E-Business Suite 12.2 Fusion Middleware Log Files: Locate,View, and Control