How to re-join to cluster after the oracle rac node has been destoryed

Node Destoryed

Install a new operating system.
Delete bad node information from ocr.

[grid@orcl01:/home/grid]$ olsnodes
orcl01
orcl02

--清除重装主机的OCR条目
[root@orcl01:/root]$ /u01/app/11.2.0/grid/bin/crsctl delete node -n orcl02
CRS-4661: Node orcl02 successfully deleted.

参数说明：
-n　　 实例节点名,即主机名

--显示集群节点列表
[root@orcl01:/root]$ su - grid
[grid@orcl01:/home/grid]$ olsnodes
orcl01

Delete bad node vip.

# 从OCR中删除重装主机的VIP信息
[root@orcl01:/root]$ /u01/app/11.2.0/grid/bin/srvctl remove vip -i orcl02 -v -f
Successfully removed VIP orcl02.

# 重启监听
~]$ /u01/app/11.2.0/grid/bin/srvctl stop listener -n rac1
~]$ /u01/app/11.2.0/grid/bin/srvctl stop scan_listener -i 2

~]$ /u01/app/11.2.0/grid/bin/srvctl start listener -n rac1
~]$ /u01/app/11.2.0/grid/bin/srvctl start scan_listener -i 2

delete db/gi inventory.

[grid@orcl01:/u01/app/11.2.0/grid/oui/bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES=orcl01 -silent -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 2047 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

[oracle@orcl01:/u01/app/oracle/product/11.2.0/db/oui/bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES=orcl01 -silent -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 2047 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

[grid@orcl01:/u01/app/11.2.0/grid/oui/bin]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>11.2.0.4.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="orcl01"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/db" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="orcl01"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraHome1" LOC="/ogg213_Micro/ogg213_soft" TYPE="O" IDX="3"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

cvu check.

su - grid
/u01/app/11.2.0/grid/bin/./cluvfy stage -pre nodeadd -n orcl02 -verbose

exec addnode.sh on normal node1 to add node2 gi.

su - grid
cd $ORACLE_HOME/oui/bin
--忽略之前的校验失败选项
export IGNORE_PREADDNODE_CHECKS=Y
--增加节点
./addNode.sh -silent "CLUSTER_NEW_NODES={orcl02}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={orcl02-vip}"

参数说明：
$ORACLE_HOME  即$GRID_HOME
CLUSTER_NEW_VIRTUAL_HOSTNAMES  参数值从幸存节点/etc/hosts中获取或者
olsnodes -n -i  获取vip名称

start crs on new node2.

su - root
/u01/app/oraInventory/orainstRoot.sh
/u01/app/11.2.0/grid/root.sh

su - grid
crsctl stat res -t

exec addnode.sh on normal node1 to add node2 db.

su - oracle
/u01/app/oracle/product/11.2.0/db/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={orcl02}"

参数说明：
CLUSTER_NEW_NODES  参数值从幸存节点/etc/hosts中获取或者olsnodes -n 显示节点编号和节点名称

exec root.sh on new node2.

[root@orcl02:/root]$ /u01/app/oracle/product/11.2.0/db/root.sh
Check /u01/app/oracle/product/11.2.0/db/install/root_orcl02_2024-08-18_06-37-25.log for the output of root script
[root@orcl02:/root]$ tail -f /u01/app/oracle/product/11.2.0/db/install/root_orcl02_2024-08-18_06-37-25.log
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/db
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
Finished product-specific root actions.

start instance on new node2.

su - oracle
cd /u01/app/oracle/product/11.2.0/db/dbs
mv inittopnet1.ora inittopnet2.ora

su - oracle
cd /u01/app/oracle/product/11.2.0/db/dbs
mv orapwtopnet1 orapwtopnet2

su - grid
srvctl start database -d topnet

crsctl stat res -t

Node Reserved

oracle11gasm01(Node01) is the node being removed and oracle11gasm02(Node02) is one of the surviving nodes.
oracle11gasm01(Node01) change hostname to **oracle11grac03.

(Any Nodes, oracle) Remove the DB instance.

~]$ srvctl stop instance -d orcl -i orcl1
~]$ srvctl remove instance -d orcl -i orcl1

(Any Nodes, root) Check the node status and type.

~]# $GRID_HOME/bin/olsnodes -s -t
oracle11gasm01  Active  Unpinned
oracle11gasm02  Active  Unpinned

~]# $GRID_HOME/bin/crsctl unpin css -n oracle11gasm01    # if pinned.

(Node01, root) Remove oracle clusterware.

~]# $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

(Node02, root) Remove node from cluster.

~]# $GRID_HOME/bin/crsctl delete node -n oracle11gasm01

(Node01, grid) Update node list in inventory file.

# login as grid
~]$ $GRID_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/app/11.2.0/grid "CLUSTER_NODES={oracle11gasm01}" CRS=TRUE -silent -local
# login as oracle
~]$ $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 "CLUSTER_NODES={oracle11gasm01}" -silent -local

~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>11.2.0.4.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/dbhome_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="oracle11gasm01"/>
   </NODE_LIST>
</HOME>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="oracle11gasm01"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

(Node01, grid) Deinstall grid.

~]$ $GRID_HOME/deinstall/deinstall –local

(Node02, grid/oracle) Update node list in inventory file.

# login as grid
~]$ $GRID_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/app/11.2.0/grid "CLUSTER_NODES={oracle11gasm02}" CRS=TRUE -silent -local
# login as oracle
~]$ $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 "CLUSTER_NODES={oracle11gasm02}" -silent -local

~]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>11.2.0.4.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraHome1" LOC="/ggs" TYPE="O" IDX="3"/>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="oracle11gasm01"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/dbhome_1" TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="oracle11gasm01"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

(Node02, grid) Post-check for node removal.

~]$ cluvfy stage -post nodedel -n oracle11gasm01 -verbose

(Node01, root) Change hostname.

~]# hostnamectl set-hostname oracle11grac03
~]# sed -i 's/oracle11gasm01/oracle11grac03/' /etc/hosts

(All nodes) Configure user equivalence.

~]# grep -Ev '^#|^$|127.0.0.1|vip|scan|priv|:' /etc/hosts  | awk '{print "ssh " $2 " date;"}' > /tmp/sshTest.sh

# login as grid
~]$ $GRID_HOME/deinstall/sshUserSetup.sh -user grid -hosts "oracle11grac03 oracle11gasm02" -noPromptPassphrase
# login as oracle
~]$ $ORACLE_HOME/deinstall/sshUserSetup.sh -user oracle -hosts "oracle11grac03 oracle11gasm02" -noPromptPassphrase

(Node02, grid) Pre-ckeck for add node.

~]$ cluvfy stage -pre nodeadd -n oracle11grac03 -fixup -fixupdir /tmp -verbose
~]$ cluvfy comp peer -n oracle11grac03

(Node02, grid) Adding node to the grid installation.

~]$ export IGNORE_PREADDNODE_CHECKS=Y
~]$ $GRID_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={oracle11grac03}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac03-vip}"

(Node01, root) Run the $GRID_HOME/root.sh script.

~]# /u01/app/11.2.0/grid/root.sh

(Node01, oracle) Adding node to the oracle installation.

~]$ $ORACLE_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={oracle11grac03}"

(Node02, root) Run the $ORACLE_HOME/root.sh script.

~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh

(Any, oracle) Adding instance.

~]$ srvctl add instance -d orcl -i orcl1 -n oracle11grac03
~]$ srvctl start instance -d orcl -i orcl1