11gR2Oracle

Oracle Restart – 11gR2

Oracle Restart is the new feature introduced by Oracle in 11gR2 which ensures that various Oracle components gets automatically restarted after hardware or software failures.This feature depends on OHAS (Oracle High Availability Service) daemon and is only for standalone server environment.The various components which gets automatically restarted as part of Oracle Restart are :-
1. Database instance
2. Database listener
3. Database services
4. Oracle ASM instance
5. Oracle ASM disk group
6. Oracle ONS
CRSCTL utility is used to stop and start Oracle Restart.

[[email protected] bin]$ ps -ef | grep has
oracle    2458     1  0 13:00 ?        00:00:01 /u01/app/oracle/grid/11.2.0.2/db_1/bin/ohasd.bin reboot
root      2631     1  0 13:00 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
oracle    8003  3995  0 16:36 pts/2    00:00:00 grep has
[[email protected] bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab
ora.asm
               ONLINE  ONLINE       anand-lab                Started
ora.ons
               OFFLINE OFFLINE      anand-lab
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab
ora.diskmon
      1        ONLINE  ONLINE       anand-lab
ora.evmd
      1        ONLINE  ONLINE       anand-lab
ora.orcl.db
      1        ONLINE  ONLINE       anand-lab                Open
[[email protected] bin]$

SRVCTL utility can be used to start and stop the Oracle Restart components manually.When you start/stop a component with SRVCTL, any components on which this component depends are automatically started/stopped first, and in the proper order.

[[email protected] bin]$ ./srvctl stop asm
PRCR-1065 : Failed to stop resource ora.asm
CRS-2529: Unable to act on 'ora.asm' because that would require stopping or relocating 'ora.orcl.db', but the force option was not specified
[[email protected] bin]$ ./srvctl stop asm -f
[[email protected] bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  OFFLINE      anand-lab
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab
ora.asm
               OFFLINE OFFLINE      anand-lab                Instance Shutdown   
ora.ons
               OFFLINE OFFLINE      anand-lab
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab
ora.diskmon
      1        ONLINE  ONLINE       anand-lab
ora.evmd
      1        ONLINE  ONLINE       anand-lab
ora.orcl.db
      1        ONLINE  OFFLINE                               Instance Shutdown   
[[email protected] bin]$

Though i stopped ASM instance but as Database instance component depends on it, its stopped automatically.How to check the dependencies?

[[email protected] bin]$ ./crsctl stat res ora.orcl.db -p | grep DEPENDENCIES
START_DEPENDENCIES=hard(ora.DATA.dg) weak(type:ora.listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.DATA.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg)
[[email protected] bin]$
[[email protected] bin]$ ./crsctl stat res ora.asm -p | grep DEPENDENCIES
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
STOP_DEPENDENCIES=hard(ora.cssd)
[[email protected] bin]$

HARD :- Specify a hard start dependency for a resource when you want the resource to start only when a particular resource or resource of a particular type starts.Use intermediate to specify that Oracle Clusterware can start this resource if a resource on which it depends is in either the ONLINE or INTERMEDIATE state. If not specified, then resources must be in the ONLINE state for Oracle Clusterware to start this resource.
WEAK :- Specify a weak start dependency for a resource when you want that resource to start despite whether named resources are running, or not. An attempt to start this resource also attempts to start any resources on which this resource depends if they are not running.
PULLUP :- When you specify the pullup start dependency for a resource, then this resource starts as a result of named resources starting.
For more details read
http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/resatt.htm#CHDJHECG
What happens if you shutdown the database using sqlplus, and perform Oracle Restart.Does the instance start?The asnwer to this is NO (in default settings).

[[email protected] bin]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Sun May 1 16:56:14 2011
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
SQL> shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
[[email protected] bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab
ora.asm
               ONLINE  ONLINE       anand-lab                Started
ora.ons
               OFFLINE OFFLINE      anand-lab
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab
ora.diskmon
      1        ONLINE  ONLINE       anand-lab
ora.evmd
      1        ONLINE  ONLINE       anand-lab
ora.orcl.db
      1        OFFLINE OFFLINE                               Instance Shutdown   
[[email protected] bin]$
[[email protected] bin]$
[[email protected] bin]$
[[email protected] bin]$ ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'anand-lab'
....................................
CRS-4133: Oracle High Availability Services has been stopped.
[[email protected] bin]$ ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[[email protected] bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab
ora.asm
               ONLINE  ONLINE       anand-lab                Started
ora.ons
               OFFLINE OFFLINE      anand-lab
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab
ora.diskmon
      1        ONLINE  ONLINE       anand-lab
ora.evmd
      1        ONLINE  ONLINE       anand-lab
ora.orcl.db
      1        OFFLINE OFFLINE                                                   
[[email protected] bin]$

If you shut down the database with SQL*Plus, Oracle Restart does not interpret this as a database failure and does not attempt to restart the database.So, you need to manually startup the database using SQL*Plus which changes both the TARGET and STATE back to ONLINE.
Doing shutdown immediate using SQL*Plus cause the TARGET to be OFFLINE and by default the AUTO_START attribute is set to RESTORE which means, while starting up CRS will restore the resource to the same state that it was in when the server stopped. Oracle Clusterware attempts to restart the resource if the value of TARGET was ONLINE before the server stopped. So, be sure to bring down the database using SRVCTL utility instead of SQL*Plus.

[[email protected] bin]$ ./crsctl stat res ora.orcl.db -p | grep AUTO_START
AUTO_START=restore

From Doc

AUTO_START
Indicates whether Oracle Clusterware automatically starts a resource after a cluster server restart. Valid AUTO_START values are:
always: Restarts the resource when the server restarts regardless of the state of the resource when the server stopped.
restore: Restores the resource to the same state that it was in when the server stopped. Oracle Clusterware attempts to restart the resource if the value of TARGET was ONLINE before the server stopped.
never: Oracle Clusterware never restarts the resource regardless of the state of the resource when the server stopped.

Lets change the AUTO_START attribute to “always”

[[email protected] bin]$ ./crsctl modify resource ora.orcl.db -attr AUTO_START=always
[[email protected] bin]$ ./crsctl stat res ora.orcl.db -p | grep AUTO_START
AUTO_START=always
[[email protected] bin]$
SQL> shu immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
ora.orcl.db
      1        OFFLINE OFFLINE                               Instance Shutdown
[[email protected] bin]$ ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'anand-lab'
CRS-2673: Attempting to stop 'ora.evmd' on 'anand-lab'
.........................
CRS-4133: Oracle High Availability Services has been stopped.

Starting HAS

[[email protected] bin]$ ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[[email protected] bin]$

Check the status of the resource, the ora.orcl.db resource’s TARGET and STATE should be online with db in OPEN mode.

[[email protected] bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab
ora.asm
               ONLINE  ONLINE       anand-lab                Started
ora.ons
               OFFLINE OFFLINE      anand-lab
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab
ora.diskmon
      1        ONLINE  ONLINE       anand-lab
ora.evmd
      1        ONLINE  ONLINE       anand-lab
ora.orcl.db
      1        ONLINE  ONLINE       anand-lab                Open   
[[email protected] bin]$

7 thoughts on “Oracle Restart – 11gR2

  1. Hi Anand,
    Great post on Oracle Restart feature.
    i have a question on this feature-
    If you shut down the database using “srvctl” , it caused the TARGET
    to be OFFLINE and after starting up CRS it didnt bring up the database.
    Is this the correct behaviour?
    Please see the log –>
    $ srvctl stop database -d orcl
    $ crsctl stat res -t
    ——————————————————————————–
    NAME TARGET STATE SERVER STATE_DETAILS
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.DATA.dg
    ONLINE ONLINE temp-dba01
    ora.DG1.dg
    ONLINE ONLINE temp-dba01
    ora.FRA.dg
    ONLINE ONLINE temp-dba01
    ora.LISTENER.lsnr
    ONLINE ONLINE temp-dba01
    ora.asm
    ONLINE ONLINE temp-dba01 Started
    ora.ons
    OFFLINE OFFLINE temp-dba01
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.cssd
    1 ONLINE ONLINE temp-dba01
    ora.diskmon
    1 ONLINE ONLINE temp-dba01
    ora.evmd
    1 ONLINE ONLINE temp-dba01
    ora.orcl.db
    1 OFFLINE OFFLINE Instance Shutdown
    ora.tmp.db
    1 OFFLINE OFFLINE
    $ crsctl stop has
    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.evmd’ on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.DG1.dg’ on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘temp-dba01’
    CRS-2677: Stop of ‘ora.DG1.dg’ on ‘temp-dba01’ succeeded
    CRS-2677: Stop of ‘ora.FRA.dg’ on ‘temp-dba01’ succeeded
    CRS-2677: Stop of ‘ora.DATA.dg’ on ‘temp-dba01’ succeeded
    CRS-2673: Attempting to stop ‘ora.asm’ on ‘temp-dba01’
    CRS-2677: Stop of ‘ora.asm’ on ‘temp-dba01’ succeeded
    CRS-2673: Attempting to stop ‘ora.cssd’ on ‘temp-dba01’
    CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘temp-dba01’ succeeded
    CRS-2677: Stop of ‘ora.cssd’ on ‘temp-dba01’ succeeded
    CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘temp-dba01’
    CRS-2677: Stop of ‘ora.evmd’ on ‘temp-dba01’ succeeded
    CRS-2677: Stop of ‘ora.diskmon’ on ‘temp-dba01’ succeeded
    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘temp-dba01’ has completed
    CRS-4133: Oracle High Availability Services has been stopped.
    $ crsctl start has
    CRS-4123: Oracle High Availability Services has been started.
    $ crsctl stat res -t
    ——————————————————————————–
    NAME TARGET STATE SERVER STATE_DETAILS
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.DATA.dg
    ONLINE OFFLINE temp-dba01
    ora.DG1.dg
    ONLINE OFFLINE temp-dba01
    ora.FRA.dg
    ONLINE OFFLINE temp-dba01
    ora.LISTENER.lsnr
    ONLINE OFFLINE temp-dba01 STARTING
    ora.asm
    ONLINE OFFLINE temp-dba01
    ora.ons
    OFFLINE OFFLINE temp-dba01
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.cssd
    1 ONLINE OFFLINE
    ora.diskmon
    1 ONLINE OFFLINE
    ora.evmd
    1 ONLINE INTERMEDIATE temp-dba01
    ora.orcl.db
    1 OFFLINE OFFLINE
    ora.tmp.db
    1 OFFLINE OFFLINE

  2. You cannot run crsctl stop has for shutting down resources … as this is an abrupt shutdown. Please look into alert log file of database

    1. Yes stopping the OHAS or even CRS using crsctl stop crs on any version of oracle when the instance is up and running causes the instance to abort.As Oracle restart ensures that various Oracle components gets automatically restarted after hardware or software failures, hence showed crsctl stop has to simulate an hardware failure, as in hardware failure oracle instance wouldn’t be shutdown normally.
      Regards,
      Anand

Leave a Reply