11gR2OHASDOLR

Restoring OLR – 11gR2

OLR (Oracle Local Registry) stores the information used by OHASD during the startup sequence of the node.The OLR clusterware configuration file is present on all the nodes of the RAC database.It also stores the Oracle Restart’s metadata for Single instance database in 11gR2.

[[email protected] bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :        152
         Available space (kbytes) :     261968
         ID                       : 1434054292
         Device/File Name         : /u01/app/oracle/grid/11.2.0.2/db_1/cdata/localhost/anand-lab.olr
                                    Device/File integrity check succeeded
         Cluster registry integrity check succeeded
         Logical corruption check bypassed due to non-privileged user

To check for the logical corruption, login as root and do ocrcheck

[[email protected] bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :        152
         Available space (kbytes) :     261968
         ID                       : 1434054292
         Device/File Name         : /u01/app/oracle/grid/11.2.0.2/db_1/cdata/localhost/anand-lab.olr
                                    Device/File integrity check succeeded
         Cluster registry integrity check succeeded
         Logical corruption check succeeded

If the OLR is corrupt or inaccessible, the OHAS doesn’t start.

[[email protected] bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab
ora.DATA02.dg
               ONLINE  ONLINE       anand-lab
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab
ora.LISTENR_TEST.lsnr
               ONLINE  ONLINE       anand-lab
ora.asm
               ONLINE  ONLINE       anand-lab                Started
ora.ons
               OFFLINE OFFLINE      anand-lab
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab
ora.diskmon
      1        ONLINE  ONLINE       anand-lab
ora.evmd
      1        ONLINE  ONLINE       anand-lab
ora.matrix.db
      1        ONLINE  ONLINE       anand-lab                Open
[[email protected] bin]$ 
[[email protected] bin]# ./ocrconfig -local -showbackup
anand-lab     2011/04/29 14:42:14     /u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110429_144214.olr
[[email protected] bin]#
[[email protected] bin]# ./ocrconfig -local -manualbackup   --> Took a new backup
anand-lab     2011/08/20 12:18:06     /u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110820_121806.olr
anand-lab     2011/04/29 14:42:14     /u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110429_144214.olr
[[email protected] bin]#

Lets rename the OLR

[[email protected] localhost]$ ls -lrth
-rw-r----- 1 oracle oinstall 261M Aug 20 12:18 anand-lab.olr
[[email protected] localhost]$ mv anand-lab.olr anand-lab.olr.bkp
[[email protected] localhost]$ ls -lrt
-rw-r----- 1 oracle oinstall 272756736 Aug 20 12:18 anand-lab.olr.bkp
[[email protected] bin]$ ./ocrcheck -local
PROTL-602: Failed to retrieve data from the local registry
PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
[[email protected] client]$ pwd
/u01/app/oracle/grid/11.2.0.2/db_1/log/anand-lab/client
[[email protected] client]$ cat ocrcheck_4189.log
Oracle Database 11g Clusterware Release 11.2.0.2.0 - Production Copyright 1996, 2010 Oracle. All rights reserved.
2011-08-20 12:29:39.216: [OCRCHECK][3086636736]ocrcheck starts...
2011-08-20 12:29:39.217: [  OCROSD][3086636736]utopen:6m': failed in stat OCR file/disk /u01/app/oracle/grid/11.2.0.2/db_1/cdata/localhost/anand-lab.olr, errno=2, os err string=No such file or directory
2011-08-20 12:29:39.217: [  OCROSD][3086636736]utopen:7: failed to open any OCR file/disk, errno=2, os err string=No such file or directory
2011-08-20 12:29:39.217: [  OCRRAW][3086636736]proprinit: Could not open raw device
2011-08-20 12:29:39.217: [ default][3086636736]a_init:7!: Backend init unsuccessful : [26]
2011-08-20 12:29:39.217: [OCRCHECK][3086636736]Failed to access OCR repository. Retval [26] Error [PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]]
2011-08-20 12:29:39.217: [OCRCHECK][3086636736]Failed to initialize ocrchek2
2011-08-20 12:29:39.217: [OCRCHECK][3086636736]Exiting [status=failed]...

Stopped the OHASD using crsctl stop has and started back –

[[email protected] bin]# ./crsctl start has
CRS-4563: Insufficient user privileges.
CRS-4000: Command Start failed, or completed with errors.
[[email protected] localhost]# touch anand-lab.olr
[[email protected] localhost]#
[[email protected] localhost]# ls -lrt
-rw-r----- 1 oracle oinstall 272756736 Aug 20 12:34 anand-lab.olr.bkp
-rw-r--r-- 1 root   root             0 Aug 20 12:45 anand-lab.olr
[[email protected] localhost]# chown oracle:oinstall anand-lab.olr
[[email protected] localhost]# chmod 640 anand-lab.olr
[[email protected] localhost]# ls -lrt
total 2828
-rw-r----- 1 oracle oinstall 272756736 Aug 20 12:34 anand-lab.olr.bkp
-rw-r----- 1 oracle oinstall         0 Aug 20 12:45 anand-lab.olr
[[email protected] localhost]#

Restore OLR from backup –

[[email protected] bin]# ./ocrconfig -local -restore /u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110820_121806.olr
[[email protected] bin]#

Log shows –

[[email protected] client]$ more ocrconfig_5059.log
Oracle Database 11g Clusterware Release 11.2.0.2.0 - Production Copyright 1996, 2010 Oracle. All rights reserved.
2011-08-20 13:05:07.580: [ OCRCONF][3086165696]ocrconfig starts...
2011-08-20 13:05:07.580: [ OCRCONF][3086165696]Started to restore the Oracle Local Registry location from backup file [/u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110820_121806.olr].
2011-08-20 13:05:07.631: [  OCRMSG][3086165696]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2011-08-20 13:05:07.631: [  OCRMSG][3086165696]GIPC error [29] msg [gipcretConnectionRefused]
2011-08-20 13:05:07.632: [  OCRMSG][3086165696]prom_connect: error while waiting for connection complete [24]
2011-08-20 13:05:07.927: [ OCRCONF][3086165696]Successfully restored OCR and set block 0
2011-08-20 13:05:07.927: [ OCRCONF][3086165696]Exiting [status=success]...
[[email protected] bin]# ./crsctl stat res -t
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Status failed, or completed with errors.
Time to start OHASd
[[email protected] bin]# ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[[email protected] bin]#
[[email protected] bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab
ora.DATA02.dg
               ONLINE  ONLINE       anand-lab
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab
ora.LISTENR_TEST.lsnr
               ONLINE  ONLINE       anand-lab
ora.asm
               ONLINE  ONLINE       anand-lab                Started
ora.ons
               OFFLINE OFFLINE      anand-lab
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab
ora.diskmon
      1        ONLINE  ONLINE       anand-lab
ora.evmd
      1        ONLINE  ONLINE       anand-lab
ora.matrix.db
      1        ONLINE  ONLINE       anand-lab                Open
[[email protected] bin]#

2 thoughts on “Restoring OLR – 11gR2

  1. Anand, thank you so much for this article. It helped me to troubleshoot an issue that I was having while upgrading clusterware software from version 12.1.0.1.0 to 12.1.0.2.0.

Leave a Reply