CRS-2674

Oracle Database RAC 11.2.0.4

Symptoms

群集 Node01 修改主机名(oracle11gasm01 -> oracle11grac03),重新添加节点执行 root.sh 脚本时报错。

~]# /u01/app/11.2.0/grid/root.sh

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node oracle11gasm01, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Start of resource "ora.asm" failed
CRS-2672: Attempting to start 'ora.asm' on 'oracle11grac03'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/oracle11grac03/agent/ohasd/oraagent_grid/oraagent_grid.log".
CRS-2674: Start of 'ora.asm' on 'oracle11grac03' failed
CRS-2679: Attempting to clean 'ora.asm' on 'oracle11grac03'
CRS-2681: Clean of 'ora.asm' on 'oracle11grac03' succeeded
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Grid Infrastructure stack
Failed to start ASM at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1339.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

Cause

查看 oraagent_grid.log 日志,发现 ASM 实例不可用。

~]# tail -f /u01/app/11.2.0/grid/log/oracle11grac03/agent/ohasd/oraagent_grid/oraagent_grid.log

2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00ae9760
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] ConnectionPool::resetConnection sid +ASM3 status 2
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] Gimh::check OH /u01/app/11.2.0/grid SID +ASM3
2026-05-28 22:45:08.531: [ AGFW][371975936]{0:0:209} Agent sending reply for: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:705
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0,7 exists
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 0
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet { entry
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet procr_get_conf: retval [0] configured [1] local only [0] error buffer []
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet procr_get_conf: OCR loc [0], Disk Group : [+OCR]
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet m_ocrDgpSet f811bbd8 dgName OCR
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet ocrret 0 found 1
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet ocrDgpSet OCR
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet exit }
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::ocrDgCheck Entry {
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] ConnectionPool::addConnection m_oracleHome:/u01/app/11.2.0/grid m_oracleSid:+ASM3 m_usrOraEnv: pConnxn:c8007740
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/11.2.0/grid Crshome /u01/app/11.2.0/grid
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM3)(ENVS='ORACLE_HOME=/u01/app/11.2.0/grid,ORACLE_SID=+ASM3')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM3)))
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::getConnxn connected
2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] InstConnection::connectInt: server not attached
2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] InstConnection::connectInt (2) Exception OCIException
2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] InstConnection:connect:excp OCIException OCI error 1034
2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] DgpAgent::queryDgStatus excp ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] DgpAgent::queryDgStatus asm inst is down or going down
2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] DgpAgent::queryDgStatus dgName OCR ret 1
2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] (:CLSN00100:)DgpAgent::ocrDgCheck OCR dgName OCR state 1
2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] AsmAgent::check ocrCheck 2 m_OcrOnline 0 m_OcrTimer 0
2026-05-28 22:45:08.543: [ora.asm][339937024]{0:0:209} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1
2026-05-28 22:45:08.543: [ COMMCRS][337835776]clsc_connect: (0x7ff5c80e1620) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

查看 alert_+ASM3.log 日志,实例被终止。

~]# tail -f /u01/app/grid/diag/asm/+asm/+ASM3/trace/alert_+ASM3.log

Fri May 29 15:54:42 2026
MMNL started with pid=21, OS id=11761
lmon registered with NM - instance number 3 (internal mem no 2)
Fri May 29 15:56:39 2026
PMON (ospid: 11717): terminating the instance due to error 481
Fri May 29 15:56:39 2026
System state dump requested by (instance=3, osid=11717 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM3/trace/+ASM3_diag_11729_20260529155639.trc
Fri May 29 15:56:39 2026
ORA-1092 : opitsk aborting process
Dumping diagnostic data in directory=[cdmp_20260529155639], requested by (instance=3, osid=11717 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 11717

查看 orarootagent_root.log 日志,启用了 HAIP,但是无法从 OCR 中获取 HAIP 令牌,因为现存节点禁用了 HAIP。

~]# tail -f /u01/app/11.2.0/grid/log/oracle11grac03/agent/ohasd/orarootagent_root/orarootagent_root.log

2026-05-30 18:43:42.469: [ USRTHRD][1661908736]{0:0:208} Adding 169.254.150.211 on ens224:1
2026-05-30 18:43:42.469: [ USRTHRD][1661908736]{0:0:208} VipActions::startIp }
2026-05-30 18:43:42.469: [ USRTHRD][1661908736]{0:0:208} Assigned IP: 169.254.150.211 on interface ens224
2026-05-30 18:43:44.194: [ USRTHRD][1664009984]{0:0:208} use all detected INF
2026-05-30 18:43:44.194: [ USRTHRD][1664009984]{0:0:208} Completed 1 HAIP assignment, start complete
2026-05-30 18:43:44.404: [ USRTHRD][1664009984]{0:0:208} IptoClean '169.254.0.1', ip '169.254.0.0', mask '255.255.0.0'
2026-05-30 18:43:44.404: [ USRTHRD][1664009984]{0:0:208} USING HAIP[ 0 ]: ens224 - 169.254.150.211
2026-05-30 18:43:44.404: [ora.cluster_interconnect.haip][2157274880]{0:0:208} [start] (:CLSN00107:) clsn_agent::start }
2026-05-30 18:43:44.404: [ AGFW][2157274880]{0:0:208} Command: start for resource: ora.cluster_interconnect.haip 1 1 completed with status: SUCCESS

2026-05-29 15:54:31.529: [ USRTHRD][443913984]{0:0:208} No HAIP info configured in GPNP, using defaults
2026-05-29 15:54:31.529: [ USRTHRD][443913984]{0:0:208} The final CIDR subnet 169.254/16
2026-05-29 15:54:31.529: [ default][443913984]clsvactversion:4: Retrieving Active Version from local storage.
2026-05-29 15:54:31.607: [ USRTHRD][443913984]{0:0:208} HAIP: mbr num is 0.
[ CLWAL][443913984]clsw_Initialize: OLR initlevel [70000]
2026-05-29 15:54:31.637: [ USRTHRD][443913984]{0:0:208} HAIP: initializing to 1 interfaces
2026-05-29 15:54:31.638: [ USRTHRD][443913984]{0:0:208} HAIP: configured to use 1 interfaces
2026-05-29 15:54:31.842: [ USRTHRD][443913984]{0:0:208} use all detected INF
2026-05-29 15:54:31.843: [ USRTHRD][443913984]{0:0:208} HAIP: Updating member info HAIP1;10.10.10.0#0
2026-05-29 15:54:31.843: [ USRTHRD][443913984]{0:0:208} InitializeHaIps[ 0] infList 'inf ens224, ip 10.10.10.152, sub 10.10.10.0'
2026-05-29 15:54:31.844: [ USRTHRD][443913984]{0:0:208} Error in getting Key SYSTEM.network.haip.group.cluster_interconnect.interface.valid in OCR
2026-05-29 15:54:31.846: [ CLSINET][443913984] failed to open OLR HAIP subtype SYSTEM.network.haip.group.cluster_interconnect.interface.valid key, rc=4
2026-05-29 15:54:31.846: [ USRTHRD][443913984]{0:0:208} ipMapsz 0, idxMap sz 0, restart 0, numHaip 1, infListSz 1
2026-05-29 15:54:31.846: [ USRTHRD][443913984]{0:0:208} HAIP reset on new modified startup, ipSize 0 != numInf 1
2026-05-29 15:54:31.846: [ USRTHRD][443913984]{0:0:208} restart 0, haipSize 0, numHaip 1, numSub 0, ipMsz 0
2026-05-29 15:54:31.846: [ USRTHRD][443913984]{0:0:208} HAIP: starting inf 'ens224', suggestedIp '', assignedIp ''

Solution

现存节点禁用了 HAIP 特性,新加入节点也需要禁用 HAIP 特性才能正常加入群集。

Disable HAIP

禁用 HAIP 后,启动群集将此节点配置信息刷新至 OCR 中。

~]# $GRID_HOME/bin/crsctl start crs

停止 CRS,重新执行 root.sh 脚本,查看日志已经不从 OCR 中获取 HAIP 令牌了。

~]# $GRID_HOME/bin/crsctl stop crs
~]# $GRID_HOME/root.sh

但是脚本重新创建节点资源时,依旧会启动 HAIP 功能,需要再次禁用 HAIP。