Received a mail from colleague reporting the below error from emgc Metric Collection Errors –
Target crs_node1 Type Cluster Metric Clusterware Status Collection Timestamp Dec 14, 2011 5:35:02 AM Error Type Collection Problem Message WARN|has::Common::hasGetClusterStatus Status verification failed due to cluvfy execution failure for node(s) node3:EFAIL,NODE_STATUS::node2:EFAIL,NODE_STATUS::node1:EFAIL,NODE_STATUS::node4:EFAIL,OVERALL_STATUS:
The first thing i did is ran cluvfy
node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose Performing post-checks for cluster services setup Checking node reachability... Check: Node reachability from node "node1" Destination Node Reachable? ------------------------------------ ------------------------ node3 yes node1 yes node4 yes node2 yes Result: Node reachability check passed from node "node1". Checking user equivalence... Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ ------------------------ node3 passed node2 passed node1 passed node4 passed Result: User equivalence check passed for user "oracle". ERROR: The location "/tmp/CVU_10.2.0.5.0.1_dba/" is owned by another user on nodes: node3,node1,node2 Verification will proceed with nodes: node4 ERROR: CRS is not installed on any of the nodes. Verification cannot proceed. Post-check for cluster services setup was unsuccessful on all the nodes.
The cluvfy verification failed. Checked for the permission on /tmp/CVU_10.2.0.5.0.1_dba
drwxr----- 3 em dba 4096 May 27 2010 CVU_10.2.0.5.0.1_dba Changed the permission to 770 drwxrwx--- 3 em dba 4096 May 27 2010 CVU_10.2.0.5.0.1_dba
After changing the permission of /tmp/CVU_10.2.0.5.0.1_dba/ on all the nodes ran cluvfy again
node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose Performing post-checks for cluster services setup Checking node reachability... Check: Node reachability from node "node1" Destination Node Reachable? ------------------------------------ ------------------------ node3 yes node1 yes node4 yes node2 yes Result: Node reachability check passed from node "node1". Checking user equivalence... Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ ------------------------ node3 passed node2 passed node1 passed node4 passed Result: User equivalence check passed for user "oracle". WARNING: CRS is not installed on nodes: node4 Verification will proceed with nodes: node3,node2,node1 ......................... Post-check for cluster services setup was unsuccessful. Checks did not pass for the following node(s): node4
olsnodes showed all the 4 nodes, but cluvfy showed CRS is not installed on nodes: node4
To find out more, digged into the cluvfy log in $CRS_HOME/cv/log
[main] [5:36:32:622] [OUIData.readInventoryData:393] ==== CRS home added: Oracle home properties: Name : OraCrs10g_home Type : CRS-HOME Location : /app/oracle/crs Node list: [node1, node2, node3] ............................... [main] [5:43:44:502] [Stage.verify:359] m_currentTaskSet.size=1; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:502] [TaskNodeAppCreation.performTask:157] Performing NodeApp Creation Verification Task... ; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:502] [sVerificationUtil.getInventoryFileLocation:133] Inventory Config File's name is:'/etc/oraInst.loc'; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:503] [sVerificationUtil.getInventoryFileLocation:168] inventory_loc=/app/oracle/crs/oraInventory; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:503] [sVerificationUtil.getInventoryFileLocation:170] Inventory File Location is-->/app/oracle/crs/oraInventory/ContentsXML/inventory.xml; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208] CRS found installed on node: node3; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208] CRS found installed on node: node2; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208] CRS found installed on node: node1; Wed Dec 14 05:43:44 CST 2011 [main] [5:43:44:504] [TaskNodeAppCreation.performTask:178] ==== Nodes with CRS installed is: 3; Wed Dec 14 05:43:44 CST 2011 ..................
Checked the inventory.xml in /app/oracle/crs/oraInventory/ContentsXML/ which showed
{HOME NAME="OraCrs10g_home" LOC="/app/oracle/crs" TYPE="O" IDX="1" CRS="true"} {NODE_LIST} {NODE NAME="node1"/} {NODE NAME="node2"/} {NODE NAME="node3"/} {/NODE_LIST} {/HOME}
Added NODE NAME “node4” to the inventory.xml using
node1: (matrix1) /app/oracle/crs/oui/bin> ./runInstaller -updateNodeList -silent "CLUSTER_NODES={node1,node2,node3,node4}" ORACLE_HOME=$CRS_HOME CRS=true Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /app/oracle/crs/oraInventory 'UpdateNodeList' was successful.
After it got completed, inventory.xml showed
{HOME_LIST} {HOME NAME="O]raCrs10g_home" LOC="/app/oracle/crs" TYPE="O" IDX="1" CRS="true"} {NODE_LIST {NODE NAME="node1"/} {NODE NAME="node2"/} {NODE NAME="node3"/} {NODE NAME="node4"/} {/NODE_LIST} {/HOME}
node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose Performing post-checks for cluster services setup .................. .................. Result: Check passed. Post-check for cluster services setup was successful.