Steps To Shutdown(stop)/Startup(start) CRS, OHAS, ASM, RDBMS & ACFS Services on a RAC Cluster 11.2 & 12.1 Configuration (Doc ID 1355977.1)

n this Document

Goal

Solution

Additional Steps Required When ACFS filesystem are involved:

Community Discussions

References

---

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.1.0.2 [Release 11.2 to 12.1]
Oracle Database - Enterprise Edition - Version 12.2.0.1 to 12.2.0.1 [Release 12.2]
Information in this document applies to any platform.
***Checked for relevance on 21-Jan-2014***

GOAL

The present document provides in detail the steps to shutdown/startup The CRS, OHAS, ASM & RDBMS Services on a RAC Cluster 11.2 & 12.1 configuration:

SOLUTION

1) Connect to node #1, then please check if the CRS/OHAS & services are enabled to autostart as follow (repeat this step on each node):

# $GRID_ORACLE_HOME/bin/crsctl config crs

2) If not, then you can enable it as follow (repeat this step on each node):

# $GRID_ORACLE_HOME/bin/crsctl enable crs

3) Shutdown the services on each node as follow:

# $GRID_ORACLE_HOME/bin/crsctl stop crs

4) Verify the services were/are down (repeat this step on each node):

# $GRID_ORACLE_HOME/bin/crsctl status resource -t

5) Then start the services on node #1 as follow (only on first node):

# $GRID_ORACLE_HOME/bin/crsctl start crs

6) Wait 1 minute, then validate the services started & diskgroups were mounted (only on first node):

# $GRID_ORACLE_HOME/bin/crsctl status resource -t

7) Then start the services on node #2 as follow:

# $GRID_ORACLE_HOME/bin/crsctl start crs

8) Wait 1 minute, then validate the services started & diskgroups were mounted on node #2:

# $GRID_ORACLE_HOME/bin/crsctl status resource -t

9) If there are more nodes in the RAC, then repeat the same steps (7-8).

10) Then check the status of the clusterware globally as follows:

# crsctl check cluster -all

Sample output:

[root@asmgrid1 ~]# crsctl check cluster -all
**************************************************************
asmgrid1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

asmgrid2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

Note 1: Alternatively, you can stop and start the cluster globally as follows:

Stop:

[root@asmgrid1 ~]# crsctl stop cluster  -all   [-f]

  

Start:

[root@asmgrid1 ~]# crsctl start cluster  -all   

  

Note 2: For 11gR2 RAC ACFS configurations please check the next information:
https://docs.oracle.com/cd/E11882_01/readmes.112/e41331/chapter11204.htm#BABIJJAF

2.9.1Oracle ACFS and Oracle Clusterware Stack Shut DownWhen attempting to shut down Oracle Clusterware, the Oracle Clusterware stack may report that it did not successfully stop on selected nodes (reference Bug 8651848). If the database home is on Oracle ACFS, then you may receive the following error:

CRS-5014: Agent orarootagent.bin timed out starting process acfsmount for action

This error can be ignored.
Alternatively, the Oracle Clusterware stack may report that it did not successfully stop on selected nodes due to the inability to shut down the Oracle ACFS resources. If this occurs, take the following steps:

• Ensure that all file system activity to Oracle ACFS mount points is quiesced by shutting down programs or processes and retry the shutdown.
• If the ora.registry.acfs resource check function times out, or the resource exhibits a state of UNKNOWN or INTERMEDIATE, then this may indicate an inability to access the Oracle Cluster Registry (OCR). The most common cause of this is a network failure. The commands "acfsutil registry" and "ocrcheck" may give you a better indicator of the specific error. Clear this error and attempt to stop Oracle Clusterware again.

For 12cR1
https://docs.oracle.com/database/121/READM/chapter12102.htm#READM135
2.12.1 Oracle ACFS and Oracle Clusterware Stack Shut Down
Some non-Oracle Grid Infrastructure usage of mount points prevents unmounts and volume disables in the kernel (reference Bug 8651848). Examples include:
Network File System (NFS)
Samba/Common Internet File System (CIFS)
If this reflects your situation, ensure that you discontinue usage of these features before trying to initiate a stack shutdown, file system unmount, or volume disable.
Additionally, certain user space processes and system processes may use the file system or volume device in a way that prevents the Oracle Grid Infrastructure stack from shutting down during a patch or upgrade.
If this occurs, use the lsof and fuser commands (Linux and UNIX) or the handle and wmic commands (Windows) to identify processes which are active on the Oracle ACFS file systems and Oracle ADVM volumes. To ensure that these processes are no longer active, dismount all Oracle ACFS file systems or Oracle ADVM volumes and issue an Oracle Clusterware shutdown. Otherwise, errors may be issued during Oracle Clusterware shutdown relating to activity on Oracle ACFS file systems or Oracle ADVM volumes which will stop the successful shutdown of Oracle Clusterware.

Additional Steps Required When ACFS filesystem are involved:

 Note 3:  If ACFS filesystems are associated with this ASM Cluster configuration, then the ACFS filesystems need to be dismounted first, this is because if you try to dismount an ACFS filesystem (or any other regular Unix/Linux filesystem)  that is being used/accessed at that time (same as a regular unix/Linux filesystem)  then you will get a “Resource Busy” error, therefore “crsctl stop crs” statement will fail as expected.

You will need to perform the following additional steps instead:


A) Dismount all the ACFS filesystems running in the cluster as follows (as root user):

# srvctl stop filesystem -d volume_device_name [-n node_name] [-f]

  

Where:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-d  <volume_device_name>  :The Oracle ACFS volume device name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-n  <node_name>  : The name of a node

If you do not specify this option, then the utility stops the volume resource on all active nodes in the cluster.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-f  : This option stops the file system and also stops any databases or other resources that depend on this file system.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Example:

[root@asmgrid1 ~]# df -k | grep asm
/dev/asm/volnew1-347  14680064    159700  14520364   2% /u01/app/grid/acfsmounts/acfsdgnew_volnew1
/dev/asm/vol_oh1-115  35651584   4590104  31061480  13% /u04acfs

#  srvctl stop filesystem -d  /dev/asm/volnew1-347   -f

#  srvctl stop filesystem -d  /dev/asm/vol_oh1-115   -f

  

B) Make sure the ACFS filesystem(s) was/were dismounted on all the nodes:

# df -k | grep asm

  

C) Then perform the steps described in the following document:

=)>     Steps To Shutdown/Startup The CRS, OHAS, ASM & RDBMS Services on RAC 11.2 Configuration. (Doc ID 1355977.1)

  

  

Note: To manually remount the ACFS filesystem back (CRS managed) the next statement needs to be executed as root user:

# srvctl start filesystem -d volume_device_name [-n node_name]

  

  

---

Community Discussions

Still have questions? Use the communities window below to search for similar discussions or start a new discussion on this subject. (Window is the live community not a screenshot)

Click here to open in main browser window

REFERENCES

NOTE:1609127.1 - How To Validate ASM Instances And Diskgroups On A RAC Cluster (When CRS Does Not Start).
NOTE:1355977.1 - Steps To Shutdown(stop)/Startup(start) CRS, OHAS, ASM, RDBMS & ACFS Services on a RAC Cluster 11.2 & 12.1 Configuration

NOTE:1093890.1 - Steps To Shutdown/Startup The Exadata & RDBMS Services and Cell/Compute Nodes On An Exadata Configuration