Wednesday, April 17, 2019

Garbage First Garbage Collector Tuning

Learn about how to adapt and tune the G1 GC for evaluation, analysis and performance.
Published August 2013
The Garbage First Garbage Collector (G1 GC) is the low-pause, server-style generational garbage collector for Java HotSpot VM. The G1 GC uses concurrent and parallel phases to achieve its target pause time and to maintain good throughput. When G1 GC determines that a garbage collection is necessary, it collects the regions with the least live data first (garbage first).
A garbage collector (GC) is a memory management tool. The G1 GC achieves automatic memory management through the following operations:
  • Allocating objects to a young generation and promoting aged objects into an old generation.
  • Finding live objects in the old generation through a concurrent (parallel) marking phase. The Java HotSpot VM triggers the marking phase when the total Java heap occupancy exceeds the default threshold.
  • Recovering free memory by compacting live objects through parallel copying.
Here, we look at how to adapt and tune the G1 GC for evaluation, analysis and performance—we assume a basic understanding of Java garbage collection.
The G1 GC is a regionalized and generational garbage collector, which means that the Java object heap (heap) is divided into a number of equally sized regions. Upon startup, the Java Virtual Machine (JVM) sets the region size. The region sizes can vary from 1 MB to 32 MB depending on the heap size. The goal is to have no more than 2048 regions. The eden, survivor, and old generations are logical sets of these regions and are not contiguous.
The G1 GC has a pause time-target that it tries to meet (soft real time). During young collections, the G1 GC adjusts its young generation (eden and survivor sizes) to meet the soft real-time target. During mixed collections, the G1 GC adjusts the number of old regions that are collected based on a target number of mixed garbage collections, the percentage of live objects in each region of the heap, and the overall acceptable heap waste percentage.
The G1 GC reduces heap fragmentation by incremental parallel copying of live objects from one or more sets of regions (called Collection Set (CSet)) into different new region(s) to achieve compaction. The goal is to reclaim as much heap space as possible, starting with those regions that contain the most reclaimable space, while attempting to not exceed the pause time goal (garbage first).
The G1 GC uses independent Remembered Sets (RSets) to track references into regions. Independent RSets enable parallel and independent collection of regions because only a region's RSet must be scanned for references into that region, instead of the whole heap. The G1 GC uses a post-write barrier to record changes to the heap and update the RSets.

Garbage Collection Phases

Apart from evacuation pauses (described below) that compose the stop-the-world (STW) young and mixed garbage collections, the G1 GC also has parallel, concurrent, and multiphase marking cycles. G1 GC uses the Snapshot-At-The-Beginning (SATB) algorithm, which takes a snapshot of the set of live objects in the heap at the start of a marking cycle. The set of live objects is composed of the live objects in the snapshot, and the objects allocated since the start of the marking cycle. The G1 GC marking algorithm uses a pre-write barrier to record and mark objects that are part of the logical snapshot.

Young Garbage Collections

The G1 GC satisfies most allocation requests from regions added to the eden set of regions. During a young garbage collection, the G1 GC collects both the eden regions and the survivor regions from the previous garbage collection. The live objects from the eden and survivor regions are copied, or evacuated, to a new set of regions. The destination region for a particular object depends upon the object's age; an object that has aged sufficiently evacuates to an old generation region (that is, promoted); otherwise, the object evacuates to a survivor region and will be included in the CSet of the next young or mixed garbage collection.

Mixed Garbage Collections

Upon successful completion of a concurrent marking cycle, the G1 GC switches from performing young garbage collections to performing mixed garbage collections. In a mixed garbage collection, the G1 GC optionally adds some old regions to the set of eden and survivor regions that will be collected. The exact number of old regions added is controlled by a number of flags that will be discussed later (see "Taming Mixed GCs"). After the G1 GC collects a sufficient number of old regions (over multiple mixed garbage collections), G1 reverts to performing young garbage collections until the next marking cycle completes.

Phases of the Marking Cycle

The marking cycle has the following phases:
  • Initial mark phase: The G1 GC marks the roots during this phase. This phase is piggybacked on a normal (STW) young garbage collection.
  • Root region scanning phase: The G1 GC scans survivor regions of the initial mark for references to the old generation and marks the referenced objects. This phase runs concurrently with the application (not STW) and must complete before the next STW young garbage collection can start.
  • Concurrent marking phase: The G1 GC finds reachable (live) objects across the entire heap. This phase happens concurrently with the application, and can be interrupted by STW young garbage collections.
  • Remark phase: This phase is STW collection and helps the completion of the marking cycle. G1 GC drains SATB buffers, traces unvisited live objects, and performs reference processing.
  • Cleanup phase: In this final phase, the G1 GC performs the STW operations of accounting and RSet scrubbing. During accounting, the G1 GC identifies completely free regions and mixed garbage collection candidates. The cleanup phase is partly concurrent when it resets and returns the empty regions to the free list.

Important Defaults

The G1 GC is an adaptive garbage collector with defaults that enable it to work efficiently without modification. Here is a list of important options and their default values. This list applies to the latest Java HotSpot VM, build 24. You can adapt and tune the G1 GC to your application performance needs by entering the following options with changed settings on the JVM command line.
  • -XX:G1HeapRegionSize=n

    Sets the size of a G1 region. The value will be a power of two and can range from 1MB to 32MB. The goal is to have around 2048 regions based on the minimum Java heap size.
  • -XX:MaxGCPauseMillis=200

    Sets a target value for desired maximum pause time. The default value is 200 milliseconds. The specified value does not adapt to your heap size.
  • -XX:G1NewSizePercent=5

    Sets the percentage of the heap to use as the minimum for the young generation size. The default value is 5 percent of your Java heap. This is an experimental flag. See "How to unlock experimental VM flags" for an example. This setting replaces the -XX:DefaultMinNewGenPercent setting. This setting is not available in Java HotSpot VM, build 23.
  • -XX:G1MaxNewSizePercent=60

    Sets the percentage of the heap size to use as the maximum for young generation size. The default value is 60 percent of your Java heap. This is an experimental flag. See "How to unlock experimental VM flags" for an example. This setting replaces the -XX:DefaultMaxNewGenPercent setting. This setting is not available in Java HotSpot VM, build 23.
  • -XX:ParallelGCThreads=n

    Sets the value of the STW worker threads. Sets the value of n to the number of logical processors. The value of n is the same as the number of logical processors up to a value of 8.
    If there are more than eight logical processors, sets the value of n to approximately 5/8 of the logical processors. This works in most cases except for larger SPARC systems where the value of n can be approximately 5/16 of the logical processors.
  • -XX:ConcGCThreads=n

    Sets the number of parallel marking threads. Sets n to approximately 1/4 of the number of parallel garbage collection threads (ParallelGCThreads).
  • -XX:InitiatingHeapOccupancyPercent=45

    Sets the Java heap occupancy threshold that triggers a marking cycle. The default occupancy is 45 percent of the entire Java heap.
  • -XX:G1MixedGCLiveThresholdPercent=65

    Sets the occupancy threshold for an old region to be included in a mixed garbage collection cycle. The default occupancy is 65 percent. This is an experimental flag. See "How to unlock experimental VM flags" for an example. This setting replaces the -XX:G1OldCSetRegionLiveThresholdPercent setting. This setting is not available in Java HotSpot VM, build 23.
  • -XX:G1HeapWastePercent=10

    Sets the percentage of heap that you are willing to waste. The Java HotSpot VM does not initiate the mixed garbage collection cycle when the reclaimable percentage is less than the heap waste percentage. The default is 10 percent. This setting is not available in Java HotSpot VM, build 23.
  • -XX:G1MixedGCCountTarget=8

    Sets the target number of mixed garbage collections after a marking cycle to collect old regions with at most G1MixedGCLIveThresholdPercent live data. The default is 8 mixed garbage collections. The goal for mixed collections is to be within this target number. This setting is not available in Java HotSpot VM, build 23.
  • -XX:G1OldCSetRegionThresholdPercent=10

    Sets an upper limit on the number of old regions to be collected during a mixed garbage collection cycle. The default is 10 percent of the Java heap. This setting is not available in Java HotSpot VM, build 23.
  • -XX:G1ReservePercent=10

    Sets the percentage of reserve memory to keep free so as to reduce the risk of to-space overflows. The default is 10 percent. When you increase or decrease the percentage, make sure to adjust the total Java heap by the same amount. This setting is not available in Java HotSpot VM, build 23.

How to Unlock Experimental VM Flags

To change the value of experimental flags, you must unlock them first. You can do this by setting -XX:+UnlockExperimentalVMOptions explicitly on the command line before any experimental flags. For example:
> java -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=10 -XX:G1MaxNewSizePercent=75  G1test.jar

Recommendations

When you evaluate and fine-tune G1 GC, keep the following recommendations in mind:
  • Young Generation Size: Avoid explicitly setting young generation size with the -Xmn option or any or other related option such as -XX:NewRatio. Fixing the size of the young generation overrides the target pause-time goal.
  • Pause Time Goals: When you evaluate or tune any garbage collection, there is always a latency versus throughput trade-off. The G1 GC is an incremental garbage collector with uniform pauses, but also more overhead on the application threads. The throughput goal for the G1 GC is 90 percent application time and 10 percent garbage collection time. When you compare this to Java HotSpot VM's throughput collector, the goal there is 99 percent application time and 1 percent garbage collection time. Therefore, when you evaluate the G1 GC for throughput, relax your pause-time target. Setting too aggressive a goal indicates that you are willing to bear an increase in garbage collection overhead, which has a direct impact on throughput. When you evaluate the G1 GC for latency, you set your desired (soft) real-time goal, and the G1 GC will try to meet it. As a side effect, throughput may suffer.
  • Taming Mixed Garbage Collections: Experiment with the following options when you tune mixed garbage collections. See "Important Defaults" for information about these options:

    • -XX:InitiatingHeapOccupancyPercent
      For changing the marking threshold.
    • -XX:G1MixedGCLiveThresholdPercent and -XX:G1HeapWastePercent
      When you want to change the mixed garbage collections decisions.
    • -XX:G1MixedGCCountTarget and -XX:G1OldCSetRegionThresholdPercent
      When you want to adjust the CSet for old regions.

Overflow and Exhausted Log Messages

When you see to-space overflow/exhausted messages in your logs, the G1 GC does not have enough memory for either survivor or promoted objects, or for both. The Java heap cannot expand since it is already at its max. Example messages:
924.897: [GC pause (G1 Evacuation Pause) (mixed) (to-space exhausted), 0.1957310 secs]

OR

924.897: [GC pause (G1 Evacuation Pause) (mixed) (to-space overflow), 0.1957310 secs]
To alleviate the problem, try the following adjustments:
Increase the value of the -XX:G1ReservePercent option (and the total heap accordingly) to increase the amount of reserve memory for "to-space".
Start the marking cycle earlier by reducing the -XX:InitiatingHeapOccupancyPercent.
You can also increase the value of the -XX:ConcGCThreads option to increase the number of parallel marking threads.
See "Important Defaults" for a description of these options.

Humongous Objects and Humongous Allocations

For G1 GC, any object that is more than half a region size is considered a "Humongous object". Such an object is allocated directly in the old generation into "Humongous regions". These Humongous regions are a contiguous set of regions. StartsHumongous marks the start of the contiguous set and ContinuesHumongous marks the continuation of the set.
Before allocating any Humongous region, the marking threshold is checked, initiating a concurrent cycle, if necessary.
Dead Humongous objects are freed at the end of the marking cycle during the cleanup phase also during a full garbage collection cycle.
In-order to reduce copying overhead, the Humongous objects are not included in any evacuation pause. A full garbage collection cycle compacts Humongous objects in place.
Since each individual set of StartsHumongous and ContinuesHumongous regions contains just one humongous object, the space between the end of the humongous object and the end of the last region spanned by the object is unused. For objects that are just slightly larger than a multiple of the heap region size, this unused space can cause the heap to become fragmented.
If you see back-to-back concurrent cycles initiated due to Humongous allocations and if such allocations are fragmenting your old generation, please increase your -XX:G1HeapRegionSize such that previous Humongous objects are no longer Humongous and will follow the regular allocation path.

Conclusion

G1 GC is a regionalized, parallel-concurrent, incremental garbage collector that provides more predictable pauses compared to other HotSpot GCs. The incremental nature lets G1 GC work with larger heaps and still provide reasonable worst-case response times. The adaptive nature of G1 GC just needs a maximum soft-real time pause-time goal along-with the desired maximum and minimum size for the Java heap on the JVM command line.

See Also

About the Author

Monica Beckwith, Principal Member of Technical Staff at Oracle, is the performance lead for the Java HotSpot VM's Garbage First Garbage Collector. She has worked in the performance and architecture industry for over 10 years. Prior to Oracle and Sun Microsystems, Monica lead the performance effort at Spansion Inc. Monica has worked with many industry standard Java based benchmarks with a constant goal of finding opportunities for improvement in the Java HotSpot VM.

Join the Conversation

Join the Java community conversation on Facebook, Twitter, and the Oracle Java Blog!

at No comments:

Tuesday, April 2, 2019

Get Proactive: Fusion Middleware Product Information Centers (Doc ID 1388276.1)

Information Centers: Fusion Middleware

 https://docs.oracle.com/middleware/12213/cross/upgrade.htm


Index
| Application Server | Oracle Secure Enterprise Search | Developer Tools | Portals | Content Management | Identity Management | Service Oriented Architecture | Data Integration | Collaborative Technologies |
Application Server

Oracle Secure Enterprise Search

Developer Tools

Portals

Content Management

Identity Management

Service — Oriented Architecture

Data Integration

Data Quality

Collaborative Technologies
at No comments:

Different ways to take thread dumps in WebLogic Server

Purpose

This article provides information about different ways on taking java thread dumps in a WebLogic Server environment.
Thread dumps are essential diagnosis information used to analyze and troubleshoot performance related issues such as server hangs, deadlocks, slow running, idle or stuck applications, slow database interactions etc...

Different ways to take thread dumps in WebLogic Server

WebLogic Server (WLS) and Java offer several ways to generate thread dumps. They are detailed below. It is always recommended to obtain the thread dumps by using operating system (OS) commands rather than by using Java classes or the Administration Console, because if the console is hanging, users won't be able to connect to it to issue thread dumps.

  1. Use operating system commands to get the thread dumps when WLS starts up from a command-line script:
    • On Windows OSes, thread dumps can be created by
      <ctrl>+<break> -- the thread dumps are generated in the server stdout
    • On POSIX-compliant platforms (e.g. Solaris and Linux), first identify the process ID (pid) using the command ps -ef | grep java, then run
      kill -3 <pid> 2>&1
      Signal 3 is equivalent to SIGQUIT. Note that in Solaris, the thread dump is generated in the current shell, but in Linux, the thread dump is generated in the shell which started the java process specified by the pid.
  2. Using beasvc (up to WLS 10.3.5 included):
    beasvc -dump -svcname:<service_name>
    • service_name is the Windows service that is running the server instance (e.g. mydomain_myserver)
  3. Using wlsvc (from 10.3.6/12.1.1):
    wlsvc -dump -svcname:<service_name>
  4. Using weblogic.WLST:
    setDomainEnv.cmd or setDomainEnv.sh depending on the OS
    java weblogic.WLST
    connect("<username>","<password>","t3://<url>:<port>")
    threadDump()
    The thread dump will be generated in Thread_Dump_AdminServer.txt. Note 1274713.1 addresses WLST thread dumps in more detail with examples on how to define sleep time between each dump and number of dumps to take.
  5. From a command line or shell, a thread dump can be generated via the following command (deprecated from WLS 9.0, and is only available till WLS release 12.1.3. The weblogic.Admin utility is no longer shipped from 12.2.1.x onwards):
    setDomainEnv.cmd or setDomainEnv.sh depending on the OS
    java weblogic.Admin -url <url>:<port> -username <username> -password <password> THREAD_DUMP
    Example: java weblogic.Admin -url t3://localhost:7001 -username weblogic -password welcome1 THREAD_DUMP (The thread dump will be generated in the defined server stdout).
  6. From the WLS Administration Console, a thread dump can be created by navigating to Servers -> <server_name> -> Monitoring -> Threads -> Dump threads stack. This method could lead to truncated or incomplete thread dumps.
  7. From the Services Administration Tools when WLS runs as a Windows Service, see Note 1348645.1
  8. Java VisualVM can also be used to take thread dumps while applications are running, see http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/applications_local.html for more details
  9. With jstack
    jstack <pid> or jstack -l <pid> to print additional information about locks
  10. From the JRockit command line:
    jrcmd <pid> print_threads
  11. From Java Mission Control with JDK 7:
    jcmd <pid> Thread.print 


at No comments:

Monday, March 18, 2019

ADRCI how to purge oracle trace files

ADRCI – Auto Purge Old Trace and Incident files
How to purge Oracle trace files through adrci
What is ADRCI?
ADRCI is a new feature introduced from 11G onwards. It is used to deal with diagnostic information.
It has multiple features such as:
  1. Analyze diagnostic data within the Automatic Diagnostic Repository (ADR).
  2. Create a package with incident/problem information for Oracle Support.
  3. Auto purge of old diagnostic files.
Auto purge of old Diagnostic files:
One of the feature of ADRCI is to purge the old diagnostic information/files. This is controlled by two parameters in ADRCI utility.
  1. SHORTP_POLICY:  Which is used for automatically purging short-lived files, i.e. core dump files and traces, expressed in hours and defaults to 30 days.
  1. LONGP_POLICY: Which is used for automatically purging long-lived files, i.e. incidents and health monitor warnings, expressed in hours and defaults to 1 year.
How the purge cycle is defined:
The auto-purge will ONLY purge every 7 days and this has always been the way it was designed. It then uses the SHORTP_POLICY and LONGP_POLICY values to determine how much information to purge. The ADR purge is run towards the end of the regular AWR purge.
There is no control on the purge frequency, it is fixed to be 48 hours after startup and then every 7 days. Manual purge is the only way to control purge actions in between.
Some commands for ADRCI:
Set environment variable
. oraenv
SID
After set environment Enter adrci command to connect adrci prompt


[oracle@prod ~]$ adrci
ADRCI: Release 11.2.0.1.0 – Production on Mon Feb 6 00:42:15 2017
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
ADR base = “/u01/app/oracle”

Adrci> show homes (This command will list all the available database trace directories.)
adrci> show homes
ADR Homes:
diag/tnslsnr/prod/lsnrctl_prod
diag/tnslsnr/prod/prod
diag/tnslsnr/prod/listener
diag/tnslsnr/prod/listener_prod
diag/rdbms/prod/prod
diag/rdbms/chicago/chicago
diag/rdbms/india/india
diag/rdbms/qdev1/qdev1
adrcihome
Adrci> set homepath <home path to set>(This command will use to set a specific home value (trace directory)
adrci> set homepath diag/rdbms/chicago/Chicago
sethome
Adrci> show control
This command will list the settings of “shortp_policy”, “longp_policy” and when was the last auto purge happened.
show control1
adrci > set control (SHORTP_POLICY = 168)
adrci> set control (LONGP_POLICY = 168)
This command is use to reset the short policy and long policy and this is set in hours.
After set control now sort policy and long policy is 168hr /7 days.
Adrci>Show control
showcontrol2
SHORTP POLICY (ADRCI SETTINGS) –
168 hours / 7 days (will keep last 7 days of files and delete all)
LONGP POLICY (ADRCI SETTINGS) –
168 hours / 7 days (will keep last 7 days of files and delete all)
ADRCI SETTINGS and PURGE:
Adrci> set homepath <home path to set>
adrci > set control (SHORTP_POLICY = 168)
adrci> set control (LONGP_POLICY = 168)
Purge – this will purge all the files by keeping last 7 days of files.
Adrci> purge
Manually purge the traces-
Adrci> help
Use to get help on commands.


====================================================
Automatically purging
Sometimes we are facing with space related issues due to the huge number of trace file generation. Automatic purging can help us in this situations if set short retention time for trace files.
  1. Type show homes to see current homes supported by ADRCI and set homepath.
    adrci> show homes
ADR Homes: 
diag/rdbms/db12c_stby/db12c_stby
diag/rdbms/db12c_stby/db12c
diag/rdbms/db12c_stby/DB12c
diag/tnslsnr/Standby/listener
diag/clients/user_oracle/host_2436394132_82
 
adrci> set homepath diag/rdbms/db12c_stby/db12c_stby

  2. The automatic purging runs on schedule defined in retention policy .To check current policy for home, we can use two methods :
    adrci> show control

ADR Home = /u01/app/oracle/diag/rdbms/db12c_stby/db12c_stby:
*************************************************************************
ADRID                SHORTP_POLICY        LONGP_POLICY         LAST_MOD_TIME                            LAST_AUTOPRG_TIME                        LAST_MANUPRG_TIME                        ADRDIR_VERSION       ADRSCHM_VERSION      ADRSCHMV_SUMMARY     ADRALERT_VERSION     CREATE_TIME                              
-------------------- -------------------- -------------------- ---------------------------------------- ---------------------------------------- ---------------------------------------- -------------------- -------------------- -------------------- -------------------- ---------------------------------------- 
3514698554           720                  8760                 2016-05-31 00:54:57.566484 +05:00        2016-07-07 23:27:37.789953 +05:00                                                 1                    2                    82                   1                    2016-05-31 00:54:57.566484 +05:00       
1 rows fetched
    or
    adrci> select SHORTP_POLICY,LONGP_POLICY from ADR_CONTROL;

ADR Home = /u01/app/oracle/diag/rdbms/db12c_stby/db12c_stby:
*************************************************************************
SHORTP_POLICY        LONGP_POLICY         
-------------------- -------------------- 
720                  8760                
1 rows fetched
    By default retention is 720 hours for short policy and 8760 hours for long policy.
    Short policy include the following files :(Trace files, Core dump files, Packaging information)
    Long policy include the following files:(Incident information, Incident dumps, Alert logs)
  3. To change the retention policy :
    adrci> set control (SHORTP_POLICY=120); -- 5 days

adrci> set control (LONGP_POLICY=720); -- 30 days


--check changes

adrci> select SHORTP_POLICY,LONGP_POLICY,LAST_AUTOPRG_TIME,LAST_MANUPRG_TIME from ADR_CONTROL; 

ADR Home = /u01/app/oracle/diag/rdbms/db12c_stby/db12c_stby:
*************************************************************************
SHORTP_POLICY LONGP_POLICY LAST_AUTOPRG_TIME LAST_MANUPRG_TIME 
-------------------- -------------------- ---------------------------------------- ---------------------------------------- 
120 720 2016-07-07 23:27:37.789953 +05:00 2016-07-16 22:37:52.700123 +05:00 
1 rows fetched

adrci>
    Initially automatic purging runs 2 days after instance startup, then by policy defined interval.


Manual purging
  1. Type show homes to see current homes supported by ADRCI and set homepath.
    [oracle@MAINDB ~]$ adrci

ADRCI: Release 12.1.0.2.0 - Production on Tue Feb 23 19:18:42 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.

ADR base = "/u01/app/oracle"

adrci> show homes
ADR Homes: 
diag/tnslsnr/MAINDB/listener
diag/rdbms/cdb1/cdb1

adrci> set homepath diag/rdbms/cdb1/cdb1

  2. To purge diagnostic data that is older than the amount of time (minutes) given in the purge command. For ex to purge diagnostic data that is over 1 day old (1440 minutes).
    adrci> PURGE -age 1440 -type ALERT

--You can also pruge TRACE files with same method:
adrci> PURGE -age 1440 -type TRACE
  3. To remove all data older than one minute use:
    adrci> PURGE -age 1 -type ALERT
adrci> SHOW ALERT

ADR Home = /u01/app/oracle/diag/rdbms/cdb1/cdb1:
*************************************************************************

No alert log in selected home
adrci> exit
at No comments:

Database Hung! what to do?

Development team reported that your oracle database is hung. Users can't log in and exiting users can't complete their transaction. The DBA's with SYSDBA privileges may also be unable to log into the database. Your challenge is to troubleshoot the problem what is causing this database to hang and then fix the problem ASAP.


Step 1: Check the alert log to see if database has reported any errors, which may indicate why the database is hanging.

Step 2: See if you can get AWR or ASH report or query some of the ASH views. You may notice events such as hard parses at the top of the load profile section of the AWR report, indicating that this is what is slowing down the database.

Step 3: As a DBA you might know that a single poorly written adhoc query has potential to bring an entire database to its knees. See if you can identify one or more poorly written SQL that may be leading to the hung database.

Step 4: See if large expdp/impdp operation running in parallel mode consuming all database resources.

Step 5: Check the database for blocking locks and latch contention.

Step 6: Check the server memory usage and CPU usage. Make sure that the sessions are not stalling because of low sized PGA. (not required if u configured AMM)

Step 7:See if it is caused by filling up of all archive log destination. If Archive destination is full, database will hand. Here in this case you can connect as sys and make room for archive log. You can also changed the archive log destination temporarily and then clean up the original destination and then change the archive log destination pointing to original one.

Step 8: Check the FRA. A database hangs when it is unable to write flashback database logs to the recovery area. You can fix this problem by increasing the size of the FRA using the command,
SQL>alter system set db_recovery_file_dest_size=10GB; 


If you are still unable to resolve the reasons for the hung database and you are also unable to connect to the database as SYSDBA then here is another option to use.
Use prelim option. The prelim option does not require a database connection. You can not use prelim option if you are already connected to the database.

[oracle@usha ~]$ sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Wed Apr 25 11:54:02 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

SQL> set _prelim on
SQL> connect / as sysdba
Prelim connection established
SQL>
You can also use
[oracle@usha ~]$ sqlplus -prelim "/ as sysdba"

SQL*Plus: Release 11.2.0.1.0 Production on Wed Apr 25 11:55:39 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Now you can use oradebug hanganalyze command to analyze the hung database.
SQL> oradebug hanganalyze 3

in RAC env

SQL> oradebug setinst all
SQL> oradebug -g def hanganalyze 3


You can repeat the oradebug hanganalyze command mutiple times to generate dump files for varying process states.

When you call Oracle Support about this issue, they may ask you to attach logs for process state dump also called systemstate dump to analyze hung database.

systemstate dump report shows "what processes are doing and the resource currently they are holding"
.
Commands to get systemstate dump (NON RAC ENV)
SQL>oradebug setmypid
SQL>oradebug dump systemstate 266

RAC ENVSQL>oradebug setmypid  --- You can aslo specify the process ID other than yours oradebug setmypid <pid>
SQL>oradebug ulimitSQL>oradebug -g all dump systemstate 266

If you try to issue this command without setting the PID you get ORA-00074.
at No comments:
Subscribe to: Posts (Atom)

Database Options/Management Packs Usage Reporting for Oracle Databases 11.2 and later (Doc ID 1317265.1)

  Database Options/Management Packs Usage Report You can determine whether an option is currently in use in a database by running options_pa...