Received a mail from colleague reporting the below error from emgc Metric Collection Errors –
Target crs_node1 Type Cluster Metric Clusterware Status Collection Timestamp Dec 14, 2011 5:35:02 AM Error Type Collection Problem Message WARN|has::Common::hasGetClusterStatus Status verification failed due to cluvfy execution failure for node(s) node3:EFAIL,NODE_STATUS::node2:EFAIL,NODE_STATUS::node1:EFAIL,NODE_STATUS::node4:EFAIL,OVERALL_STATUS:
The first thing i did is ran cluvfy
node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "node1"
Destination Node Reachable?
------------------------------------ ------------------------
node3 yes
node1 yes
node4 yes
node2 yes
Result: Node reachability check passed from node "node1".
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
node3 passed
node2 passed
node1 passed
node4 passed
Result: User equivalence check passed for user "oracle".
ERROR:
The location "/tmp/CVU_10.2.0.5.0.1_dba/" is owned by another user on nodes:
node3,node1,node2
Verification will proceed with nodes:
node4
ERROR:
CRS is not installed on any of the nodes.
Verification cannot proceed.
Post-check for cluster services setup was unsuccessful on all the nodes.
The cluvfy verification failed. Checked for the permission on /tmp/CVU_10.2.0.5.0.1_dba
drwxr----- 3 em dba 4096 May 27 2010 CVU_10.2.0.5.0.1_dba Changed the permission to 770 drwxrwx--- 3 em dba 4096 May 27 2010 CVU_10.2.0.5.0.1_dba
After changing the permission of /tmp/CVU_10.2.0.5.0.1_dba/ on all the nodes ran cluvfy again
node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "node1"
Destination Node Reachable?
------------------------------------ ------------------------
node3 yes
node1 yes
node4 yes
node2 yes
Result: Node reachability check passed from node "node1".
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
node3 passed
node2 passed
node1 passed
node4 passed
Result: User equivalence check passed for user "oracle".
WARNING:
CRS is not installed on nodes:
node4
Verification will proceed with nodes:
node3,node2,node1
.........................
Post-check for cluster services setup was unsuccessful.
Checks did not pass for the following node(s):
node4
olsnodes showed all the 4 nodes, but cluvfy showed CRS is not installed on nodes: node4
To find out more, digged into the cluvfy log in $CRS_HOME/cv/log
[main] [5:36:32:622] [OUIData.readInventoryData:393] ==== CRS home added: Oracle home properties: Name : OraCrs10g_home Type : CRS-HOME Location : /app/oracle/crs Node list: [node1, node2, node3] ............................... [main] [5:43:44:502] [Stage.verify:359] m_currentTaskSet.size=1; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:502] [TaskNodeAppCreation.performTask:157] Performing NodeApp Creation Verification Task... ; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:502] [sVerificationUtil.getInventoryFileLocation:133] Inventory Config File's name is:'/etc/oraInst.loc'; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:503] [sVerificationUtil.getInventoryFileLocation:168] inventory_loc=/app/oracle/crs/oraInventory; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:503] [sVerificationUtil.getInventoryFileLocation:170] Inventory File Location is-->/app/oracle/crs/oraInventory/ContentsXML/inventory.xml; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208] CRS found installed on node: node3; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208] CRS found installed on node: node2; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208] CRS found installed on node: node1; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:504] [TaskNodeAppCreation.performTask:178] ==== Nodes with CRS installed is: 3; Wed Dec 14 05:43:44 CST 2011 ..................
Checked the inventory.xml in /app/oracle/crs/oraInventory/ContentsXML/ which showed
{HOME NAME="OraCrs10g_home" LOC="/app/oracle/crs" TYPE="O" IDX="1" CRS="true"}
{NODE_LIST}
{NODE NAME="node1"/}
{NODE NAME="node2"/}
{NODE NAME="node3"/}
{/NODE_LIST}
{/HOME}
Added NODE NAME “node4” to the inventory.xml using
node1: (matrix1) /app/oracle/crs/oui/bin> ./runInstaller -updateNodeList -silent "CLUSTER_NODES={node1,node2,node3,node4}" ORACLE_HOME=$CRS_HOME CRS=true
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /app/oracle/crs/oraInventory
'UpdateNodeList' was successful.
After it got completed, inventory.xml showed
{HOME_LIST}
{HOME NAME="O]raCrs10g_home" LOC="/app/oracle/crs" TYPE="O" IDX="1" CRS="true"}
{NODE_LIST
{NODE NAME="node1"/}
{NODE NAME="node2"/}
{NODE NAME="node3"/}
{NODE NAME="node4"/}
{/NODE_LIST}
{/HOME}
node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose Performing post-checks for cluster services setup .................. .................. Result: Check passed. Post-check for cluster services setup was successful.