2024-02-10 09:17:05.240: [/u01/app/11.2.0/grid/bin/oraagent.bin(27008)]CRS-5019:All OCR locations are on ASM disk groups [OCR], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/oracle11grac03/agent/ohasd/oraagent_grid/oraagent_grid.log". 2024-02-10 09:17:35.246: [/u01/app/11.2.0/grid/bin/oraagent.bin(27008)]CRS-5019:All OCR locations are on ASM disk groups [OCR], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/oracle11grac03/agent/ohasd/oraagent_grid/oraagent_grid.log".
CAUSE
The OCR disk group fails to be mounted.
SOLUTION
Check whether the OCR has a backup.
~]# /u01/app/11.2.0/grid/bin/ocrconfig -showbackup PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
oracle11grac03 2024/02/09 02:41:25 /u01/app/11.2.0/grid/cdata/newrac-cluster/week.ocr PROT-25: Manual backups for the Oracle Cluster Registry are not available
Clear the disk header of a member disk in the OCR disk group.
Start Oracle Clusterware in exclusive mode without starting CRSD.
~]# /u01/app/11.2.0/grid/bin/crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oracle11grac03' CRS-2673: Attempting to stop 'ora.crf' on 'oracle11grac03' CRS-2673: Attempting to stop 'ora.ctssd' on 'oracle11grac03' CRS-2673: Attempting to stop 'ora.evmd' on 'oracle11grac03' CRS-2673: Attempting to stop 'ora.asm' on 'oracle11grac03' CRS-2673: Attempting to stop 'ora.mdnsd' on 'oracle11grac03' CRS-2677: Stop of 'ora.crf' on 'oracle11grac03' succeeded CRS-2677: Stop of 'ora.evmd' on 'oracle11grac03' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'oracle11grac03' succeeded CRS-2677: Stop of 'ora.asm' on 'oracle11grac03' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oracle11grac03' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oracle11grac03' succeeded CRS-2677: Stop of 'ora.ctssd' on 'oracle11grac03' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'oracle11grac03' CRS-2677: Stop of 'ora.cssd' on 'oracle11grac03' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'oracle11grac03' CRS-2677: Stop of 'ora.gipcd' on 'oracle11grac03' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'oracle11grac03' CRS-2677: Stop of 'ora.gpnpd' on 'oracle11grac03' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oracle11grac03' has completed CRS-4133: Oracle High Availability Services has been stopped.
~]# /u01/app/11.2.0/grid/bin/crsctl start crs -excl -nocrs CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'oracle11grac03' CRS-2676: Start of 'ora.mdnsd' on 'oracle11grac03' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'oracle11grac03' CRS-2676: Start of 'ora.gpnpd' on 'oracle11grac03' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oracle11grac03' CRS-2672: Attempting to start 'ora.gipcd' on 'oracle11grac03' CRS-2676: Start of 'ora.cssdmonitor' on 'oracle11grac03' succeeded CRS-2676: Start of 'ora.gipcd' on 'oracle11grac03' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'oracle11grac03' CRS-2672: Attempting to start 'ora.diskmon' on 'oracle11grac03' CRS-2676: Start of 'ora.diskmon' on 'oracle11grac03' succeeded CRS-2676: Start of 'ora.cssd' on 'oracle11grac03' succeeded CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'oracle11grac03' CRS-2672: Attempting to start 'ora.ctssd' on 'oracle11grac03' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'oracle11grac03' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oracle11grac03' CRS-2676: Start of 'ora.ctssd' on 'oracle11grac03' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oracle11grac03' succeeded CRS-2672: Attempting to start 'ora.asm' on 'oracle11grac03' CRS-2676: Start of 'ora.asm' on 'oracle11grac03' succeeded
~]# /u01/app/11.2.0/grid/bin/crsctl replace votedisk +OCR Successful addition of voting disk 957c110f0b724fd2bf93639927c84572. Successfully replaced voting disk group with +OCR. CRS-4266: Voting file(s) successfully replaced
~]# /u01/app/11.2.0/grid/bin/crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 957c110f0b724fd2bf93639927c84572 (ORCL:OCR01) [OCR]
Create spfile to OCR disk group.
~]$ cat > /tmp/asm_spfile.ora << EOF +ASM2.asm_diskgroups='DATA','RECO'#Manual Mount +ASM1.asm_diskgroups='DATA','RECO'#Manual Mount *.asm_power_limit=1 *.diagnostic_dest='/u01/app/grid' *.instance_type='asm' *.large_pool_size=12M *.remote_login_passwordfile='EXCLUSIVE' EOF
SQL> create spfile='+OCR' from pfile='/tmp/asm_spfile.ora';
Restart Oracle Clusterware.
~]# /u01/app/11.2.0/grid/bin/crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oracle11grac03' CRS-2673: Attempting to stop 'ora.ctssd' on 'oracle11grac03' CRS-2673: Attempting to stop 'ora.asm' on 'oracle11grac03' CRS-2673: Attempting to stop 'ora.mdnsd' on 'oracle11grac03' CRS-2677: Stop of 'ora.mdnsd' on 'oracle11grac03' succeeded CRS-2677: Stop of 'ora.asm' on 'oracle11grac03' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oracle11grac03' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oracle11grac03' succeeded CRS-2677: Stop of 'ora.ctssd' on 'oracle11grac03' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'oracle11grac03' CRS-2677: Stop of 'ora.cssd' on 'oracle11grac03' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'oracle11grac03' CRS-2677: Stop of 'ora.gipcd' on 'oracle11grac03' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'oracle11grac03' CRS-2677: Stop of 'ora.gpnpd' on 'oracle11grac03' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oracle11grac03' has completed CRS-4133: Oracle High Availability Services has been stopped.
~]# /u01/app/11.2.0/grid/bin/crsctl start crs CRS-4123: Oracle High Availability Services has been started.
~]# /u01/app/11.2.0/grid/bin/ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2600 Available space (kbytes) : 259520 ID : 1588806050 Device/File Name : +OCR Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Restart Oracle Clusterware on another node.
~]# /u01/app/11.2.0/grid/bin/crsctl start crs CRS-4123: Oracle High Availability Services has been started.
