Debug Startup services in an EBusiness Suite Release 12 environment (Doc ID 743518.1)

Starting up AS10g services in an EBusiness Suite Release 12 environment Introduction This article is an introduction into how AS10g is started in an eBiz specific environment What processes/port numbers to expect Which configuration and log files to look for Some hints towards troubleshooting startup issues Architecture AS10g in Rel 12 is conceptually similar to Rel 11i, in that there is an Apache listener which receives HTTP requests from clients. If the request is to process some Java code it will pass the request to a separate JVM process, which will provide the response back to the Client. In R12 this JVM is provided for by OC4J Additionally, the Apache and OC4J processes are not started independently but are started, stopped and managed by the OPMN process This means that the OPMN process must be started first, then you ask OPMN to startup Apache and any OC4J processes that are needed. R12 creates 3 OC4J instances: OACore - runs OA Framework-based applications Forms - runs Forms-base applications OAFM (Oracle Apps Fusion Middleware) - runs web services, mapviewer, ascontrol Rel 12 File System location - $INST_TOP Release 12 introduces the concept of an Instance Top directory location accessed by the environment variable $INST_TOP, below which all configuration files and log files for a particular instance can be found. Procedure to startup the AS10g services The startup/shutdown scripts for EBiz are located in directory defined by the environment variable $ADMIN_SCRIPTS_HOME Most of these scripts will setup the required environment variables by sourcing in the environment file located at     $INST_TOP/ora/10.1.3/$APPLFENV The scripts we will investigate further in this document are adstrtal.sh : Starts all enabled services adopmnctl.sh : Start just the OPMN process adapcctl.sh : Startup Apache listener adoacorectl.sh : Startup "oacore" OC4J adformsctl.sh : Startup "forms" OC4J adoafmctl.sh : Startup "oafm" OC4J adstrtal.sh : Starts all enabled services This script sends it's output to the log file $LOG_HOME/appl/admin/log/adstrtal.log It first determines what services need to be started by scanning the $CONTEXT_FILE for "oa_service_group" entries that have "oa_service_group_status" = "enabled" and starts the services where "oa_service_status" is also set to "enabled" The order of the entries in the $CONTEXT_FILE determines the order in which the enabled services are started The "ctrl_script" entry defines the script that is executed to start the service, i.e. this links to the individual ad startup script For example, given the following section in the $CONTEXT_FILE     <oa_service_group type="root" title="Root Service">         <oa_service_group_status oa_var="s_root_status">enabled</oa_service_group_status>         <oa_service type="opmn">             <oa_service_name oa_var="s_opmnname">Oracle Process Manager for VIS_myHost</oa_service_name>             <oa_service_status oa_var="s_opmnstatus">enabled</oa_service_status>             <oa_service_log oa_var="s_opmnlog">/oracle/VIS/inst/apps/VIS_myHost/logs/appl/admin/log/adopmnctl.txt</oa_service_log>             <oa_service_env>                 <oa_env_include>web_home</oa_env_include>             </oa_service_env>             <timeout oa_var="s_opmntimeout">100</timeout>             <ctrl_script oa_var="s_opmnctrl" osd="unix">/oracle/VIS/inst/apps/VIS_myHost/admin/scripts/adopmnctl.sh</ctrl_script>         </oa_service>     </oa_service_group>  adstrtal.sh will see the "oa_service_group_status" is set to "enabled" so looks inside this section. "oa_service_status" is also set to "enabled" so the script defined in the "ctrl_script" line (i.e. "adopmnctl.sh") will be executed with "start" passed as an argument to the script The "s_opmntimeout" variable specifies that the script will be given 100 seconds to complete the startup process, otherwise will generate an error. Each service has it's own timeout variable adopmnctl.sh : Start just the OPMN process It is the OPMN process that actually starts and manages all other AS10g services, so this MUST be started correctly first in order for any of the other services to also start Script arguments: start = starts the OPMN service status = show current status of all services stop = stop OPMN service This is an Ebiz specific wrapper script which sets the environment then eventually calls:     $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl Output from the script is captured in log file     $LOG_HOME/appl/admin/log/adopmnctl.txt Also check the OPMN log files $LOG_HOME/ora/10.1.3/opmn directory     opmn.dbg     opmn.out     opmn.log OPMN is using the configuration file located at $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml This file defines the "ias-component" that are being managed and control information about these modules. For example:- ...................... <ias-component id="IASPT" status="disabled"> ...................... ...................... <ias-component id="OC4J">     <process-type id="home" module-id="OC4J" status="disabled" ...................... ......................     <process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">         <module-data>             <category id="start-parameters">                 <data id="java-options" value="-server -verbose:gc -Xmx512M ............/>                 <data id="java-bin" value="/oracle/VIS/inst/apps/VIS_myHost/admin/scripts/java.sh"/>                 <data id="oc4j-options" value="-out ............./>             </category>             <category id="stop-parameters">                 <data id="java-options" value="-server -verbose:gc -Xmx512M ............/>             </category>         </module-data>         <start timeout="600" retry="2"/>         <stop timeout="120"/>         <restart timeout="720" retry="2"/>         <port id="default-web-site" range="21500-21504" protocol="ajp"/>         <port id="rmi" range="20000-20004"/>         <port id="jms" range="23000-23004"/>         <process-set id="default_group" numprocs="2"/>     </process-type>  NOTE - If the service is enabled in CONTEXT.xml file, but disabled in opmn.xml then the eBiz script will run and complete successfully, but the service will not be started by OPMN. The log files will show why the service is not started in this case You can check the OPMN processes are running with the following command:-     ps -ef | grep opmn | grep -v grep There should be two processes listed If only the OPMN service is started, then you would expect to see opmn listening on some ports. You can use the following script to check the ports running:- ##################################################################### ### START OF SCRIPT ### ##################################################################### ( echo "Run started " date # Send script output to screen and log file OPMN_PORTS=`cat $INST_TOP/ora/10.1.3/opmn/conf/opmn.xml|grep "port local"|awk ' BEGIN { FS="\"" } { print $2, $4, $6 }'` for ports in $OPMN_PORTS do echo "Checking for port $ports" netstat -an | grep $ports done ## echo "Run completed " date ) 2>&1 | tee mzOPMNports.txt ##################################################################### ### END OF SCRIPT ### #####################################################################  The output from this should be something like this:-     Checking for port 6100     tcp 0 0 localhost.localdomain:6100 *:* LISTEN      Checking for port 6200     tcp 0 0 myHost.uk.domain:6200 *:* LISTEN      Checking for port 6500     tcp 0 0 localhost.localdomain:6500 *:* LISTEN     tcp 0 0 myHost.uk.domain:6500 *:* LISTEN  Additional Debugging if required Remove the "states" files If having startup problems, it can sometimes help to shutdown the services, then remove the "states" files to ensure a fresh startup:- Shutdown eBiz AS10g services (adstpall.sh) Run the following command to clear out the states files     rm $ORA_CONFIG_HOME/10.1.3/opmn/logs/states/* Restart eBiz Services and confirm if problem persists Sometimes starting OPMN with the eBiz wrapper script fails, but there is no obvious reasons given in the log files. In this case the next step is to enable the OPMN debug output Refer Note 395730.1 "How to Enable OPMN Logging in OracleAS 10g Rel 3 (10.1.3)?" which describes how to enable additional debug messages. The basic steps are repeated below for your convenience:- Edit the $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml file Change the following line <debug path="/oracle/VIS/inst/apps/VIS_myHost/logs/ora/10.1.3/opmn/opmn.dbg" comp="" rotation-size="1500000" rotation-hour="0" /> to be <debug path="/oracle/VIS/inst/apps/VIS_myHost/logs/ora/10.1.3/opmn/opmn.dbg" comp="internal;ons;pm" rotation-size="1500000" rotation-hour="0" />  Restart OPMN for these changes to take effect Notice that there are some messages splashed to console, but most of the debug information is written to opmn.dbg file If the OPMN debug output still gives no additional help towards solving the reason for OPMN not starting, then you can manually emulate the same steps as the eBiz script using the following commands:- ##################################################################### ### START OF SCRIPT ### ##################################################################### # ( # source in 10.1.3 environment . $ORA_CONFIG_HOME/10.1.3/$APPLFENV # check OPMN status (this should fail as OPMN is not running) $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl status # echo "Attempting to startup OPMN" $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl verbose start exit_code=$?; echo "Exit status code of startup command is $exit_code" echo " " # check for status of processes $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl status # check processes are up echo "Checking the OPMN processes are running " ps -ef | grep "opmn -d" | grep -v grep # ) 2>&1 | tee mzManualStartOPMN.txt # ##################################################################### ### END OF SCRIPT ### #####################################################################  The two main purposes to do this are firstly to see if there are any additional messages being shown, but this also allows eBiz Support to identify if there is likely to be a more generic AS10g issue which may need to be addressed by the AS10g Support team NOTE - once the OPMN process is running, then you can attempt to startup the other services listed below. These services are independent from one another, so can be started in any order, but eBiz will start them in the order they are listed. Also note that these scripts will first check to see if OPMN is running, and if not will try to start OPMN first. adapcctl.sh : Startup Apache listener Script arguments: start = starts the Apache service status = show current status of all services stop = stop Apache service This is an Ebiz specific wrapper script that sets the environment then executes the script:-     $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl start     $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl startproc type=ohs Output from the script is captured in the log file located at     $LOG_HOME/appl/admin/log/adapcctl.txt The OPMN log files that are updated by starting Apache are located in $LOG_HOME/ora/10.1.3/opmn directory:     opmn.dbg     opmn.out     opmn.log     HTTP_Server~1  Apache log files are located in the $LOG_HOME/ora/10.1.3/Apache directory:     ssl_engine_log     access_log.999999999     error_log.999999999  The configuration files related to Apache are     $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml     $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf       (and all the "include" files)  You can check the status of the service via OPMN using the command:-     $ADMIN_SCRIPTS_HOME/adapcctl.sh status         and/or     $ADMIN_SCRIPTS_HOME/adopmnctl.sh status  If only the Apache service has been started, the output should look something like this:- You are running adopmnctl.sh version 120.4.12000000.3 Checking status of OPMN managed processes... Processes in Instance: VIS_myHost.myHost.domain -------------------+--------------------+---------+--------- ias-component | process-type | pid | status -------------------+--------------------+---------+--------- OC4J | oafm | N/A | Down OC4J | forms | N/A | Down OC4J | oacore | N/A | Down OC4J | oacore | N/A | Down HTTP_Server | HTTP_Server | 13538 | Alive adopmnctl.sh: exiting with status 0 adopmnctl.sh: check the logfile /oracle/VIS/inst/apps/VIS_myHost/logs/appl/admin/log/adopmnctl.txt for more information ...  You can also verify the Apache processes are running with the following command     ps -ef | grep `cat $INST_TOP/pids/10.1.3/Apache/httpd.pid`| grep -v grep Check TCP port is being LISTENed to with the following     netstat -an | grep `grep "^Listen" $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf | awk '{print $2}'` URLs that should now work (with just Apache listener running)     http://HOST.DOMAIN:PORT/OA_MEDIA/FNDLOGOL.gif     http://HOST.DOMAIN:PORT/  Any URL for Java code should fail at this stage. For example this URL will fail:     http://myHost.domain:8000/OA_HTML/ServletPing  Additional Debugging if required 1. Enable debug logging In the httpd.conf file set     LogLevel debug Review Note 419839.1 "How to enable Apache, OC4J and OPMN logging in Oracle Applications R12" for more details  2. You could also try starting Apache outside of OPMN entirely (just as a test of course!) Assuming you have sourced in the 10.1.3 environment you can run the following command:-     $ORACLE_CONFIG_HOME/Apache/Apache/bin/apachectl \         start \         -f $ORACLE_CONFIG_HOME/Apache/Apache/conf/httpd.conf  When using SSL, use "startssl" instead of "start" for the above command adoacorectl.sh : Startup "oacore" OC4J Script arguments: start = starts the OACORE service status = show current status of all services stop = stop OACORE service This is an Ebiz specific wrapper script that sets the environment then calls:     $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl start     $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl startproc type=oc4j instancename=oacore Output from the script is captured in log file     $LOG_HOME/appl/admin/log/adoacorectl.txt The OC4J log files are located in the $LOG_HOME/ora/10.1.3 directory:-     opmn/OC4J~oacore~default_group~1     opmn/oacore_default_group_1/oacorestd.err     opmn/oacore_default_group_1/oacorestd.out     j2ee/oacore/oacore_default_group_1/jms.log     j2ee/oacore/oacore_default_group_1/application.log     j2ee/oacore/oacore_default_group_1/rmi.log     j2ee/oacore/oacore_default_group_1/server.log     j2ee/oacore/oacore_default_group_1/global-application.log     j2ee/oacore/oacore_default_group_1/log.xml     j2ee/oacore/oacore_default_group_1/system-application.log  You can check the status of the services via OPMN using the command:-     $ADMIN_SCRIPTS_HOME/adopmnctl.sh status  You can also check OACore JVM process(es) are running using the following commands ps -ef | grep CLIENT | grep oacore     OR ##### Start ##### ( for corePID in `$ADMIN_SCRIPTS_HOME/adopmnctl.sh status | grep oacore | cut --delimiter=\| -f 3` do     ps -ef | grep $corePID | grep -v grep done ) 2>&1 | tee mzOACoreProcs.txt ##### End #####  You can check which TCP port is being LISTENed to by the OACORE JVM(s):- ############################################################## ### START OF SCRIPT ### ############################################################## ( echo "==========================================" echo "Run started " date echo "==========================================" for fn in `grep "^oacore" $ORA_CONFIG_HOME/10.1.3/opmn/logs/states/* | cut --delimiter=\: -f 1` do     echo $fn     echo "Check the following ports are listening..."     awk ' { if ( FNR>=15 && FNR<=17) print $1 } ' $fn     echo "------------------------------------------"     for corePort in `awk ' { if ( FNR>=15 && FNR<=17) print $1 } ' $fn`     do         netstat -an | grep $corePort | grep LISTEN     done     echo "------------------------------------------" done echo "Run completed " date echo "==========================================" ) 2>&1 | tee mzOACorePorts.txt ############################################################## ### END OF SCRIPT ### ##############################################################  Test URL that should work now OACORE OC4J is started:-     http://HOST.DOMAIN:PORT/OA_HTML/ServletPing     http://HOST.DOMAIN:PORT/OA_HTML/OAInfo.jsp     http://HOST.DOMAIN:PORT/OA_HTML/jsp/fnd/aoljtest.jsp     http://HOST.DOMAIN:PORT/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE     http://HOST.DOMAIN:PORT/  The configuration files are numerous, so only a couple of the main files are mentioned here:- $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml     Picks up the following entries     start-parameters     port id  $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/server.xml     Top level OC4J configuration file Additional Debugging if required Increase OACore J2EE logging level Edit the $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/j2ee-logging.xml file: Change the section that currently reads:     <!-- Default Logger, useParentHandlers should be set to false because     the root Logger (named the empty string "") will log to console -->     <logger name='oracle' level='NOTIFICATION:1' useParentHandlers='false'> to be     <!-- Default Logger, useParentHandlers should be set to false because     the root Logger (named the empty string "") will log to console -->     <logger name='oracle' level='TRACE:32' useParentHandlers='false'>  This provides fine detailed logging information in the oacorestd.out file and the log.xml. These log files could get very large, very quickly, so should not be left enabled at this level for very long You can review the "Oracle Containers for J2EE Configuration and Administration Guide 10g Release 3 (10.1.3)" (http://download.oracle.com/docs/cd/B25221_04/web.1013/b14432/logadmin.htm#sthref437) for more information about these logging levels and their effect Enabling debug mode for JSPs Edit the file $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml and change the section:     <init-param>     <param-name>debug_mode</param-name>     <param-value>false</param-value> to be     <init-param>     <param-name>debug_mode</param-name>     <param-value>true</param-value>  This change will output the stack trace when certain runtime exceptions occur This is documented in "Oracle Containers for J2EE Support for JavaServer Pages Developer's Guide 10g Release 3 (10.1.3) " http://download.oracle.com/docs/cd/B25221_04/web.1013/b14430/devconfig.htm#sthref203 See additional detail in the OPMN Status output The following command shows additional information     $ORA_CONFIG_HOME/10.1.3/opmn/bin/opmnctl status -l The output will look similar to shown below:- Processes in Instance: VIS_myHost.myHost.domain -------------------+--------------------+---------+----------+------------+----------+-----------+------ ias-component | process-type | pid | status | uid | memused | uptime | ports -------------------+--------------------+---------+----------+------------+----------+-----------+------ OC4J | oafm | 25964 | Alive | 1211722164 | 138596 | 0:04:39 | jms:24500,ajp:25000,rmi:25500 OC4J | forms | 25899 | Alive | 1211722163 | 132904 | 0:04:48 | jms:23500,ajp:22000,rmi:20500 OC4J | oacore | 25829 | Alive | 1211722162 | 177344 | 0:05:00 | jms:23000,ajp:21500,rmi:20000 HTTP_Server | HTTP_Server | 25766 | Alive | 1211722161 | 120368 | 0:04:37 | https1:4443,http1:8000 ASG | ASG | N/A | Down | N/A | N/A | N/A | N/A adformsctl.sh : Startup "forms" OC4J Similar process to OACore OC4J so not described in detail adoafmctl.sh : Startup "oafm" OC4J Similar process to OACore OC4J so not described in detail Gathering files for further investigation a. Collate the latest versions of all configuration files for Apache and OC4J configuration files, which are mainly located at $ORACLE_CONFIG_HOME Use the following command to package up these files:     tar cvf configFiles_`hostname`.tar $ORA_CONFIG_HOME/10.1.2/* \         $CONTEXT_FILE \         $ADMIN_SCRIPTS_HOME \         $APPL_TOP/*.env \         $ORA_CONFIG_HOME/10.1.3/Apache \         $ORA_CONFIG_HOME/10.1.3/config \         $ORA_CONFIG_HOME/10.1.3/j2ee/oacore \         $ORA_CONFIG_HOME/10.1.3/j2ee/forms \         $ORA_CONFIG_HOME/10.1.3/j2ee/oafm \         $ORA_CONFIG_HOME/10.1.3/network \         $ORA_CONFIG_HOME/10.1.3/opmn \         $INST_TOP/ora/10.1.3/*.env b. Log files are located in /oracle/<INSTANCE>/inst/apps/<INSTANCE>_<HOST>/logs, which is configured as $LOG_HOME environment variable i) For test systems with little activity, the easiest way is to simply gather ALL log files     tar cvf logFiles_`hostname`.tar $LOG_HOME ii) Production systems or busy test systems will likely have HUGE number of log files we don't want to see.... so would need to be more selective     tar cvf ApacheLogFiles_`hostname`.tar $LOG_HOME/ora/10.1.3     tar cvf AppsLogFiles_`hostname`.tar $LOG_HOME/appl/admin $LOG_HOME/appl/rgf iii) An alternative command for the ApacheLogFiles is listed below, which gathers only the log files changed in the last 24 hours Linux     find $LOG_HOME/ora/10.1.3 -type f -mtime -1>m.tmp; tar cvf ApacheLogFiles_`hostname`.tar -files-from=m.tmp Solaris     find $LOG_HOME/ora/10.1.3 -type f -mtime -1 > m.tmp; tar cvf ApacheLogFiles_`hostname`.tar -I m.tmp  Potential issues that may be encountered All services show as having started up OK but login page shows as blank page Issue 1 - multiple JVMs have just been implemented log.xml file shows the message:-     Internal error raised tyring to instantiate web-application: html defined in web site OC4J 10g (10.1.3) Default Web Site. Error compiling :/oracle/VIS/apps/apps_st/comn/webapps/oacore/html: Syntax error in source or compilation failed in: /oracle/VIS/inst/apps/VIS_myHost/ora/10.1.3/j2ee/oacore/application-deployments/oacore/html/oracle/webdb/wsrp/runtime/ReturnAny__LiteralSerializer.java This is likely to be the issue described in Note 731115.1 "Increasing Number of JVMs Allocated to oacore Process causes Login Problems" Issue 2 - run out of tablespace @This issue reported in internal bug 7376524 "Login page not coming up" Check the Database alert.log for errors like     ORA-1653: unable to extend table APPLSYS.FND_LOG_MESSAGES by 16 in tablespace APPS_TS_TX_DATA You may also see errors in the Apache error log     [error] [client 192.168.2.1] mod_oc4j: request to OC4J myhost:123456 failed: Connect failed (errno=111) In this case the error is obvious, you have run out of space in the APPS_TS_TX_DATA tablespace which must be fixed to proceed. In this case you may also wish to confirm the number of rows in the APPLSYS.FND_LOG_MESSAGES table as it may need to be purged Issue 3 - Java Object Cache not initializing On the eBiz Web Node, the oacorestd.err file shows the error message:-     Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initialization failed. The Distributed Caching System failed to initialize on port: 12345. The list of hosts in the distributed caching system is: myHost.uk.oracle.com . The port 12345 should be free on each host running the JVMs. application.log file shows the message:-     html: chain failed         javax.servlet.ServletException         at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:780)         at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.EvermindPageContext.handlePageException(EvermindPageContext.java:727)         at _AppsLocalLogin._jspService(_AppsLocalLogin.java:296)  You may also see following error in the browser :-     500 Internal Server Error     java.lang.NoClassDefFoundError         at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:126)         at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:170)  On the eBiz Web Node, when running the command "netstat -an | grep 12345" shows the following output:     tcp 0 0 IP_ADDR:12345 0.0.0.0:* LISTEN Running the "netstat -an | grep 12345" command on the Concurrent Manager node shows the following :-     tcp 0 0 ::ffff:IP_ADDR:12345 :::* LISTEN  With these symptoms, the issue is caused by the Java Object Cache (JOC) not initializing correctly. There are various reasons why this can occur, but in this case the Web Node is unable to connect to the Concurrent Manager node (which is already running the JOC coordinator process) When starting the Mid Tier Node although the OACore listener will start first the OAFM process will initiate the cache process and become the node listener. This is because the JOC member initialization code is not invoked in OACore until a request for login is made to the instance. Stopping the eBiz services on the Concurrent Manager node, then restarting the services on the Web Node should provide a temporary resolution to this specific situation Additional investigations for Bug 8882465 discovered this can be caused by a mismatch of file versions between the different nodes in the instance. This may be either of the files :     $IAS_ORACLE_HOME/javacache/lib/cache.jar     $COMMON_TOP/java/lib/appsborg.zip You can run the "Regenerate JAR Files" option on each Apps node to ensure the appsborg.zip file is updated, and also check the file size of cache.zip to confirm they are the same on all nodes :     cksum $IAS_ORACLE_HOME/javacache/lib/cache.jar Some or all the startup scripts report a timeout error Could be similar to the issue described in Note 564536.1 "Release 12 : ADSTRTAL.sh returns error : TIMED OUT( 100000 ) : INTERRUPTED EXCEPTION" Each service has 100 seconds to complete the startup process before being timed out. Increasing the services timeout value may provide a solution. For example "s_apctimeout" for Apache Listener or "s_oacoretimeout" for the OACore OC4J service Could also be caused by issue with server network settings This is particularly relevant if the issue has only started to occur after a hard reboot of the server. For example, there was a case where there was an old domain entry in "/etc/resolv.conf" where the IP address did not exist . After removing this entry the starting and stopping of the application started to work without any timeouts. You should therefore review the /etc/resolv.conf file and make sure that ALL the DNS server IP addresses listed are pingable and the DNS server is responding on each IP address listed. adopmnctl.sh script fails with error : bind (can't assign requested address) In addition to the above debugging steps, you can also force OPMN to bind to a specific IP address with the modification below: Modify $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml to specify the explicit IP address for this server      <notification-server interface="ipv4">          <ipaddr remote="IP_ADDR" request="IP_ADDR" /> <== ADD THIS ROW          <port ....>          .          .     </notification-server>  If this change makes no difference then when running "opmnctl start" generate a trace file with tusc, for example:     tusc -f -o opmn.trc adopmnctl start This issue was resolved when it was identified that the "localhost" hostname was being resolved to an IP address other than 127.0.0.1 Ensure "localhost" hostname resolved to the IP address 127.0.0.1     nslookup localhost     nslookup localhost.uk.domain     nslookup localhost.localdomain Should all return either "127.0.0.1" or a message saying it cannot be found in DNS However as OPMN uses a call to "getaddrinfo" you also need check the /etc/resolv.conf to ensure it has no old domain information left in there, for example in the "search" entry  adapcctl.sh: exiting with status 204 Various possibilities for this. Mostly to do with operating system level issues, for example:- Note 730982.1 "Starting R12 Apache errors with status 204 after Cloning SSO enabled E-Business Suite Instance" Where source instance is SSO enabled, but target instance is not Note 552186.1 "Adapcctl.Sh Exists With Status 204 after RUP4 (12.0.4) on HP-UX PA-RISC" After upgrading to 12.0.4 on HP-UX, missed pre-req patch 6616965 Note 412980.1 "oafm,forms and oacore not starting up after IP adress change" Need to remove the old OC4J status files which contain references to the old IP address Bug 7244575 "adapcctl.sh exists with status 204" $INST_TOP/ora/10.1.3/Apache/Apache/bin/.apachectl was not able access root privileges due to o/s security setup Note 416006.1 "HTTP Does Not Start After Cloning the Applications Tier Using OCSF as Storage Device" OCSF currently not supported for an applications filesystem Bug 6854040 "Apache not coming up in AIX 5.2 with AS 10.1.3.3" Some o/s libraries were not copied correctly when installing 10.1.3.3. Need to follow "Appendix A: IBM AIX specific 10.1.3.3 Patch Set Installation Instruction" from Note 454811.1 "Upgrading Oracle E-Business Suite Release 12 10.1.3 Oracle Home to Oracle 10gAS Patch Set 3 (10.1.3.3)" adoacorectl.sh: exiting with status 150 Various possibilities for this as well. For example:- This error ccould simply indicate that the service is taking too long to startup and the OACore timeout has been hit. This may be due to the load on the server, or some networking issues for example. The default time for all services to start is 100 seconds as already described, but can be increased if necessary Each service has it's own timeout value in AutoConfig, in this case you may wish to increase the time for OACore which uses the "s_oacoretimeout" AutoConfig variable Internal Bug 7456407 "UNABLE TO START OPMN SERVICE" describes a situation when using NFS fileystem whereby the NFS mount options were not correct and were causing this issue Oracle eBusiness Support are unable to give vendor specific tuning advice for NFS mounts, beyond that given in Note 384248.1 "Sharing The Application Tier File System in Oracle E-Business Suite Release 12" In the specific case for which the bug was raised, the "nolock" NFS mount option was omitted when using Redhat Linux Additional general thoughts More on debugging startup issues Check for core files which may indicate processes are crashing     find $INST_TOP -name "core*" -print  Check all pre-requisite operating system patches have been applied These requirements are listed in the eBusiness Suite installation documentation. For example:     Note 402306.1 : Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for AIX-Based Systems     Note.402308.1 : Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for HP-UX PA-RISC     Note.402310.1 : Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for Linux (32-bit)     Note.402311.1 : Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for Microsoft Windows     Note.402312.1 : Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for Solaris Operating System (SPARC)  Ensure the hostname resolution is returning the correct hostname and IP address for the server  Addiitonal arguments you can use with the ad<xx>.sh scripts Whilst the normal usage of these scripts is simply "start", "stop" or "status" the adopmnctl.sh, adapcctl.sh, adoacorectl.sh, adformsctl.sh and adoafmctl.sh scripts will pass through any other arguments to be processed by opmn itself. The output is not echoed to screen but gets written into the script log file and could be useful for debugging in some cases Any of the OPMN arguments will be passed through, so for example you can execute:- adopmnctl.sh ping adoafmctl.sh dmsdump etc.... Additional Reading Metalink Notes Note 454178.1 : Oracle Application Server Diagnostic Tools and Log Files in Applications Release 12 Note 395730.1 : How to Enable OPMN Logging in OracleAS 10g Rel 3 (10.1.3)? Note 419839.1 : How to enable Apache, OC4J and OPMN logging in Oracle Applications R12 Note 422419.1 : R12 - How To Enable and Collect Debug for HTTP, OC4J and OPMN Note 564536.1 : Release 12 : ADSTRTAL.sh returns error : TIMED OUT( 100000 ) : INTERRUPTED EXCEPTION Note 412980.1 : oafm,forms and oacore not starting up after IP adress change Note 552186.1 : Adapcctl.Sh Exists With Status 204 after RUP4 (12.0.4) on HP-UX PA-RISC Note 748191.1 : Oacore Processes Cannot Be Started When nprocs Is Configured > 1 on IBM AIX Note 952583.1 : OAFM OC4J is Not Starting. Error: "time out while waiting for a managed process to start" Interesting Bugs Bug 6702510 : server.xml gets corrupted with extra line when OPMN restarts the container REFERENCES NOTE:552186.1 - R12.0.4: Adapcctl.Sh Exists With Status 204 after RUP4 (12.0.4) on HP-UX PA-RISC NOTE:564536.1 - Release 12 : ADSTRTAL.sh returns error : TIMED OUT( 100000 ): INTERRUPTED EXCEPTION NOTE:748191.1 - R12: OACore Processes Cannot Be Started When nprocs Is Configured > 1 on IBM AIX JDK pap3260sr1 NOTE:454178.1 - Oracle Application Server Diagnostic Tools and Log Files in Applications Release 12 NOTE:454811.1 - Upgrading to the Latest OracleAS 10g 10.1.3.x Patch Set in Oracle E-Business Suite Release 12 NOTE:730982.1 - Starting R12 Apache Errors With Status 204 after Cloning SSO Enabled E-Business Suite Instance BUG:8882465 - CC DISTRIBUTED CACHE DOES NOT WORK AFTER PATCH 7484199 NOTE:731115.1 - Increasing Number of JVMs Allocated to oacore Process Causes Login Problems NOTE:384248.1 - Sharing The Application Tier File System in Oracle E-Business Suite Release 12.0 and 12.1 BUG:7244575 - R12UPG: ADAPCCTL.SH EXISTS WITH STATUS 204 NOTE:395730.1 - How to Enable OPMN Logging in OracleAS 10g Rel 3 (10.1.3)? NOTE:402306.1 - Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for AIX-Based Systems NOTE:402310.1 - Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for Linux (32-bit) NOTE:402311.1 - Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for Microsoft Windows NOTE:952583.1 - OAFM OC4J is Not Starting. Error: "time out while waiting for a managed process to start" NOTE:412980.1 - R12 E-Business Suite Applications Technology Stack Forms, OAFM, and Oacore Services Are Not Starting Up After The IP Address Is Changed NOTE:416006.1 - HTTP Does Not Start After Cloning the Applications Tier Using OCSF as Storage Device NOTE:419839.1 - How To Enable Apache, OC4J and OPMN Logging In Oracle E-Business Suite R12 NOTE:422419.1 - R12, 12.1 - How To Enable and Collect Debug for HTTP, OC4J and OPMN NOTE:402308.1 - Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for HP-UX PA-RISC NOTE:402312.1 - Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for Solaris Operating System (SPARC)