CRS-2674
Oracle Database RAC 11.2.0.4
Symptoms
群集 Node01 修改主机名(oracle11gasm01 -> oracle11grac03),重新添加节点执行 root.sh 脚本时报错。
~]# /u01/app/11.2.0/grid/root.sh Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory User ignored Prerequisites during installation Installing Trace File Analyzer OLR initialization - successful Adding Clusterware entries to inittab CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node oracle11gasm01, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster Start of resource "ora.asm" failed CRS-2672: Attempting to start 'ora.asm' on 'oracle11grac03' CRS-5017: The resource action "ora.asm start" encountered the following error: ORA-03113: end-of-file on communication channel Process ID: 0 Session ID: 0 Serial number: 0 . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/oracle11grac03/agent/ohasd/oraagent_grid/oraagent_grid.log". CRS-2674: Start of 'ora.asm' on 'oracle11grac03' failed CRS-2679: Attempting to clean 'ora.asm' on 'oracle11grac03' CRS-2681: Clean of 'ora.asm' on 'oracle11grac03' succeeded CRS-4000: Command Start failed, or completed with errors. Failed to start Oracle Grid Infrastructure stack Failed to start ASM at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1339. /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
Cause
查看 oraagent_grid.log 日志,发现 ASM 实例不可用。
~]# tail -f /u01/app/11.2.0/grid/log/oracle11grac03/agent/ohasd/oraagent_grid/oraagent_grid.log 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00ae9760 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] ConnectionPool::resetConnection sid +ASM3 status 2 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] Gimh::check OH /u01/app/11.2.0/grid SID +ASM3 2026-05-28 22:45:08.531: [ AGFW][371975936]{0:0:209} Agent sending reply for : RESOURCE_CLEAN[ora.asm 1 1] ID 4100:705 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0,7 exists 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 0 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet { entry 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet procr_get_conf: retval [0] configured [1] local only [0] error buffer [] 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet procr_get_conf: OCR loc [0], Disk Group : [+OCR] 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet m_ocrDgpSet f811bbd8 dgName OCR 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet ocrret 0 found 1 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet ocrDgpSet OCR 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::initOcrDgpSet exit } 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::ocrDgCheck Entry { 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] ConnectionPool::addConnection m_oracleHome:/u01/app/11.2.0/grid m_oracleSid:+ASM3 m_usrOraEnv: pConnxn:c8007740 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/11.2.0/grid Crshome /u01/app/11.2.0/grid 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM3)(ENVS='ORACLE_HOME=/u01/app/11.2.0/grid,ORACLE_SID=+ASM3' )(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))' ))(CONNECT_DATA=(SID=+ASM3))) 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] DgpAgent::getConnxn connected 2026-05-28 22:45:08.531: [ora.asm][339937024]{0:0:209} [check] InstConnection::connectInt: server not attached 2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] ORA-01034: ORACLE not available ORA-27101: shared memory realm does not exist Linux-x86_64 Error: 2: No such file or directory Process ID: 0 Session ID: 0 Serial number: 0 2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] InstConnection::connectInt (2) Exception OCIException 2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] InstConnection:connect:excp OCIException OCI error 1034 2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] DgpAgent::queryDgStatus excp ORA-01034: ORACLE not available ORA-27101: shared memory realm does not exist Linux-x86_64 Error: 2: No such file or directory Process ID: 0 Session ID: 0 Serial number: 0 2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] DgpAgent::queryDgStatus asm inst is down or going down 2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] DgpAgent::queryDgStatus dgName OCR ret 1 2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] (:CLSN00100:)DgpAgent::ocrDgCheck OCR dgName OCR state 1 2026-05-28 22:45:08.542: [ora.asm][339937024]{0:0:209} [check] AsmAgent::check ocrCheck 2 m_OcrOnline 0 m_OcrTimer 0 2026-05-28 22:45:08.543: [ora.asm][339937024]{0:0:209} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1 2026-05-28 22:45:08.543: [ COMMCRS][337835776]clsc_connect: (0x7ff5c80e1620) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
查看 alert_+ASM3.log 日志,实例被终止。
~]# tail -f /u01/app/grid/diag/asm/+asm/+ASM3/trace/alert_+ASM3.log Fri May 29 15:54:42 2026 MMNL started with pid=21, OS id=11761 lmon registered with NM - instance number 3 (internal mem no 2) Fri May 29 15:56:39 2026 PMON (ospid: 11717): terminating the instance due to error 481 Fri May 29 15:56:39 2026 System state dump requested by (instance=3, osid=11717 (PMON)), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM3/trace/+ASM3_diag_11729_20260529155639.trc Fri May 29 15:56:39 2026 ORA-1092 : opitsk aborting process Dumping diagnostic data in directory=[cdmp_20260529155639], requested by (instance=3, osid=11717 (PMON)), summary=[abnormal instance termination]. Instance terminated by PMON, pid = 11717
查看 orarootagent_root.log 日志,启用了 HAIP,但是无法从 OCR 中获取 HAIP 令牌,因为现存节点禁用了 HAIP。
~]# tail -f /u01/app/11.2.0/grid/log/oracle11grac03/agent/ohasd/orarootagent_root/orarootagent_root.log 2026-05-30 18:43:42.469: [ USRTHRD][1661908736]{0:0:208} Adding 169.254.150.211 on ens224:1 2026-05-30 18:43:42.469: [ USRTHRD][1661908736]{0:0:208} VipActions::startIp } 2026-05-30 18:43:42.469: [ USRTHRD][1661908736]{0:0:208} Assigned IP: 169.254.150.211 on interface ens224 2026-05-30 18:43:44.194: [ USRTHRD][1664009984]{0:0:208} use all detected INF 2026-05-30 18:43:44.194: [ USRTHRD][1664009984]{0:0:208} Completed 1 HAIP assignment, start complete 2026-05-30 18:43:44.404: [ USRTHRD][1664009984]{0:0:208} IptoClean '169.254.0.1', ip '169.254.0.0', mask '255.255.0.0' 2026-05-30 18:43:44.404: [ USRTHRD][1664009984]{0:0:208} USING HAIP[ 0 ]: ens224 - 169.254.150.211 2026-05-30 18:43:44.404: [ora.cluster_interconnect.haip][2157274880]{0:0:208} [start] (:CLSN00107:) clsn_agent::start } 2026-05-30 18:43:44.404: [ AGFW][2157274880]{0:0:208} Command: start for resource: ora.cluster_interconnect.haip 1 1 completed with status: SUCCESS 2026-05-29 15:54:31.529: [ USRTHRD][443913984]{0:0:208} No HAIP info configured in GPNP, using defaults 2026-05-29 15:54:31.529: [ USRTHRD][443913984]{0:0:208} The final CIDR subnet 169.254/16 2026-05-29 15:54:31.529: [ default][443913984]clsvactversion:4: Retrieving Active Version from local storage. 2026-05-29 15:54:31.607: [ USRTHRD][443913984]{0:0:208} HAIP: mbr num is 0. [ CLWAL][443913984]clsw_Initialize: OLR initlevel [70000] 2026-05-29 15:54:31.637: [ USRTHRD][443913984]{0:0:208} HAIP: initializing to 1 interfaces 2026-05-29 15:54:31.638: [ USRTHRD][443913984]{0:0:208} HAIP: configured to use 1 interfaces 2026-05-29 15:54:31.842: [ USRTHRD][443913984]{0:0:208} use all detected INF 2026-05-29 15:54:31.843: [ USRTHRD][443913984]{0:0:208} HAIP: Updating member info HAIP1;10.10.10.0#0 2026-05-29 15:54:31.843: [ USRTHRD][443913984]{0:0:208} InitializeHaIps[ 0] infList 'inf ens224, ip 10.10.10.152, sub 10.10.10.0' 2026-05-29 15:54:31.844: [ USRTHRD][443913984]{0:0:208} Error in getting Key SYSTEM.network.haip.group.cluster_interconnect.interface.valid in OCR 2026-05-29 15:54:31.846: [ CLSINET][443913984] failed to open OLR HAIP subtype SYSTEM.network.haip.group.cluster_interconnect.interface.valid key, rc=4 2026-05-29 15:54:31.846: [ USRTHRD][443913984]{0:0:208} ipMapsz 0, idxMap sz 0, restart 0, numHaip 1, infListSz 1 2026-05-29 15:54:31.846: [ USRTHRD][443913984]{0:0:208} HAIP reset on new modified startup, ipSize 0 != numInf 1 2026-05-29 15:54:31.846: [ USRTHRD][443913984]{0:0:208} restart 0, haipSize 0, numHaip 1, numSub 0, ipMsz 0 2026-05-29 15:54:31.846: [ USRTHRD][443913984]{0:0:208} HAIP: starting inf 'ens224', suggestedIp '', assignedIp ''
Solution
现存节点禁用了 HAIP 特性,新加入节点也需要禁用 HAIP 特性才能正常加入群集。
・ Disable HAIP
禁用 HAIP 后,启动群集将此节点配置信息刷新至 OCR 中。
~]# $GRID_HOME /bin/crsctl start crs
停止 CRS,重新执行 root.sh 脚本,查看日志已经不从 OCR 中获取 HAIP 令牌了。
~]# $GRID_HOME /bin/crsctl stop crs ~]# $GRID_HOME /root.sh
但是脚本重新创建节点资源时,依旧会启动 HAIP 功能,需要再次禁用 HAIP。