Recently i got a chance to upgrade 4-node RAC from 10.2.0.5 to 11.2.0.2 version. Don’t worry the upgrade was successful 😀
In this blog post i just wanted to mention few changes (not all) in Oracle Clusterware which i saw.
1. OHASD –> Oracle High Availability Service Daemon.This is the daemon which starts every other daemon that is part of Oracle Clusterware stack on a node.To check OHASD simply do “ps -ef | grep ohasd”
2. GSD is disabled by default and the service will show as target:offline, status:offline.So don’t panic if you see GSD offline 🙂
3. The voting disk data is automatically backed up in OCR as part of any configuration change and is automatically restored to any voting disk added.
http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/votocr.htm#BABDJDHI
4. To upgrades the configuration of a database and all of its services to the version of the database home

 srvctl upgrade database -d db_unique_name -o 11gR2_oracle_home 

Run the above command with 11gR2 srvctl utility.I am not pretty sure whether “srvctl remove instance” and “srvctl add instance” is required after the above command or not.As i was not sure, i did “srvctl remove instance” and “srvctl add instance”.
5. “crs_stat” has been deprecated and “crsctl stat res -t” is the new command to report “state” details.Lets check the output with crs_stat

$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....ER.lsnr ora....er.type ONLINE    ONLINE    node1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    node3
ora....N2.lsnr ora....er.type ONLINE    ONLINE    node1
ora....N3.lsnr ora....er.type ONLINE    ONLINE    node2
ora.asm        ora.asm.type   OFFLINE   OFFLINE
ora.cvu        ora.cvu.type   OFFLINE   OFFLINE
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    node1
ora.oc4j       ora.oc4j.type  ONLINE    OFFLINE
ora.ons        ora.ons.type   ONLINE    ONLINE    node1
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    node3
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    node1
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    node2
ora....SM1.asm application    OFFLINE   OFFLINE
ora....01.lsnr application    ONLINE    ONLINE    node1
ora.node1.gsd  application    OFFLINE   OFFLINE
ora.node1.ons  application    ONLINE    ONLINE    node1
ora.node1.vip  ora....t1.type ONLINE    ONLINE    node1
ora....SM2.asm application    OFFLINE   OFFLINE
ora....02.lsnr application    ONLINE    ONLINE    node2
ora.node2.gsd  application    OFFLINE   OFFLINE
ora.node2.ons  application    ONLINE    ONLINE    node2
ora.node2.vip  ora....t1.type ONLINE    ONLINE    node2
ora....SM3.asm application    OFFLINE   OFFLINE
ora....03.lsnr application    ONLINE    ONLINE    node3
ora.node3.gsd  application    OFFLINE   OFFLINE
ora.node3.ons  application    ONLINE    ONLINE    node3
ora.node3.vip  ora....t1.type ONLINE    ONLINE    node3
ora....SM4.asm application    OFFLINE   OFFLINE
ora....04.lsnr application    ONLINE    ONLINE    node4
ora.node4.gsd  application    OFFLINE   OFFLINE
ora.node4.ons  application    ONLINE    ONLINE    node4
ora.node4.vip  ora....t1.type ONLINE    ONLINE    node4
ora...._xxd.db ora....se.type ONLINE    ONLINE    node1
ora....com.svc ora....ce.type ONLINE    ONLINE    node1
ora....com.svc ora....ce.type ONLINE    ONLINE    node1
ora....com.svc ora....ce.type ONLINE    ONLINE    node1

From above, the GSD is showing offline on all the nodes.The “ora…._xxd.db” is showing online but only on 1 node i.e., node1.Also, what is missing is “instance” information.In ealier versions you would see

ora....X1.inst application    ONLINE    ONLINE    node1
ora....X2.inst application    ONLINE    ONLINE    node2
......

Looking into the output little more closer we can now see the services extension as “.svc” which used to be “.srv” in previous Oracle database versions.The services is again showing online for only one node.
Lets see the output with “crsctl stat res -t” where stat –> status and res –> resource

$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       node1
               ONLINE  ONLINE       node2
               ONLINE  ONLINE       node3
               ONLINE  ONLINE       node4
ora.asm
               OFFLINE OFFLINE      node1
               OFFLINE OFFLINE      node2
               OFFLINE OFFLINE      node3
               OFFLINE OFFLINE      node4
ora.gsd
               OFFLINE OFFLINE      node1
               OFFLINE OFFLINE      node2
               OFFLINE OFFLINE      node3
               OFFLINE OFFLINE      node4
ora.net1.network
               ONLINE  ONLINE       node1
               ONLINE  ONLINE       node2
               ONLINE  ONLINE       node3
               ONLINE  ONLINE       node4
ora.ons
               ONLINE  ONLINE       node1
               ONLINE  ONLINE       node2
               ONLINE  ONLINE       node3
               ONLINE  ONLINE       node4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       node3
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       node1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       node2
ora.cvu
      1        OFFLINE OFFLINE
ora.oc4j
      1        ONLINE  OFFLINE
ora.scan1.vip
      1        ONLINE  ONLINE       node3
ora.scan2.vip
      1        ONLINE  ONLINE       node1
ora.scan3.vip
      1        ONLINE  ONLINE       node2
ora.node1.vip
      1        ONLINE  ONLINE       node1
ora.node2.vip
      1        ONLINE  ONLINE       node2
ora.node3.vip
      1        ONLINE  ONLINE       node3
ora.node4.vip
      1        ONLINE  ONLINE       node4
ora.XXXXXX.db
      1        ONLINE  ONLINE       node1                   Open
      2        ONLINE  ONLINE       node2                   Open
      3        ONLINE  ONLINE       node3                   Open
      4        ONLINE  ONLINE       node4                   Open
ora.XXXXXX.xyz.svc
      1        ONLINE  ONLINE       node1
      2        ONLINE  ONLINE       node2
      3        ONLINE  ONLINE       node3
      4        ONLINE  ONLINE       node4
ora.XXXXXX.abcdef.svc
      1        ONLINE  ONLINE       node1
      2        ONLINE  ONLINE       node2
      3        ONLINE  ONLINE       node3
      4        ONLINE  ONLINE       node4
ora.XXXXXX.abcd.svc
      1        ONLINE  ONLINE       node1
      2        ONLINE  ONLINE       node2
      3        ONLINE  ONLINE       node3
      4        ONLINE  ONLINE       node4

This gives a more clear picture of the resources, the targets and state.So my services are up and running on all the 4 nodes and the db is open and online on all the 4 nodes 🙂
6. From 11gR2 you can check all the daemons from all nodes belonging to that cluster using the below command

crsctl check cluster -all

$ crsctl check cluster -all
**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node4:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

7. You can start the Oracle Clusterware stack in exclusive mode.All the nodes must be down except for one, from which you can fire the below command as “root” user

# crsctl start crs -excl

8.11gR2 introduced “Single Client Access Name” (SCAN).The SCAN is a single name that resolves to three IP addresses in the public network.For high availability and scalability, Oracle recommends that you configure the SCAN to resolve to three addresses.