How to Replace a Hard Drive in an Exadata Storage Cell Server (Predictive Failure)

Storage Drive Status Indicators

There are three status indicators (LEDs) on each drive.

Status Indicator Name	Color	State and Meaning
OK/Activity	Green	- OFF – Power is off or installed drive is not recognized by the system. - STEADY ON – The drive is engaged and is receiving power. - RANDOM BLINK – There is disk activity. Status indicator LED blinks on and off to indicate activity.
Fault-Service Required	Amber	- OFF – The storage drive is operating normally. - STEADY ON – The system detected a fault with the storage drive.
OK to Remove	Blue	- STEADY ON – The storage drive can be removed safely during a hot-plug operation. - OFF – The storage drive is not prepared for removal.

SYMPTOMS

CellCLI> list griddisk attributes name,asmmodestatus,status,asmdeactivationoutcome

 DATAC1_CD_00_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_01_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_02_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_03_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_04_hzsmtceladm04 DROPPED proactive failure Yes
 DATAC1_CD_05_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_06_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_07_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_08_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_09_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_10_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_11_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_00_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_01_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_02_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_03_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_04_hzsmtceladm04 DROPPED proactive failure Yes
 RECOC1_CD_05_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_06_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_07_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_08_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_09_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_10_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_11_hzsmtceladm04 ONLINE  active            Yes

SOLUTION

Check the hard drive status, Cell Node:

CellCLI> list griddisk attributes name,asmmodestatus,status,asmdeactivationoutcome
CellCLI> list physicaldisk attributes name,deviceName,diskType,physicalSize,status
CellCLI> list physicaldisk 20:3 detail
CellCLI> list lun 0_3 detail
CellCLI> list celldisk where lun=0_3 detail
CellCLI> list griddisk where celldisk=CD_03_abcxcel01 detail

Drop for replacement:

~}# imageinfo

# earlier than 11.2.3.3.0
CellCLI> alter physicaldisk 20:3 serviceled on

# 11.2.3.3.0 and later
CellCLI> alter physicaldisk 20:3 drop for replacement

If the hard drive removable led is lit, Remove it. if not, Don't Remove. Wait three minutes, then insert the new drive.
Check the hard drive status, Cell Node:

CellCLI> list griddisk attributes name,asmmodestatus,status,asmdeactivationoutcome
CellCLI> list physicaldisk attributes name,deviceName,diskType,physicalSize,status
CellCLI> list physicaldisk 20:3 detail
CellCLI> list lun 0_3 detail
CellCLI> list celldisk where lun=0_3 detail
CellCLI> list griddisk where celldisk=CD_03_abcxcel01 detail
CellCLI> alter cell validate configuration

Check the ASM disk status, ASM Instance:

SQL> set linesize 200
SQL> col path format a80
SQL> select group_number,path,header_status,mount_status,name from V$ASM_DISK where path like '%_03_abcxcel01';
SQL> select * from gv$asm_operation;
SQL> select group_number,failgroup,mode_status,count(*) from v$asm_disk group by group_number,failgroup,mode_status;

If the new griddisks were not automatically added back into the ASM diskgroup configuration, then locate the disks with group_number=0, and add them back in manually:

SQL> select path,header_status from v$asm_disk where group_number=0;
SQL> alter diskgroup dbfs_dg add disk 'o/192.168.9.9/DBFS_DG_CD_03_abcxcel01' rebalance power 10;
SQL> alter diskgroup data_q1 add disk 'o/192.168.9.9/DATA_Q1_CD_03_abcxcel01' rebalance power 10;
SQL> alter diskgroup reco_q1 add disk 'o/192.168.9.9/RECO_Q1_CD_03_abcxcel01' rebalance power 10;

REFERENCE

・ How to Replace a Hard Drive in an Exadata Storage Cell Server (Predictive Failure)

CASE 1

CellCLI> list griddisk attributes name,asmmodestatus,status,asmdeactivationoutcome

 DATAC1_CD_00_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_01_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_02_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_03_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_04_hzsmtceladm04 DROPPED proactive failure Yes
 DATAC1_CD_05_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_06_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_07_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_08_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_09_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_10_hzsmtceladm04 ONLINE  active            Yes
 DATAC1_CD_11_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_00_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_01_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_02_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_03_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_04_hzsmtceladm04 DROPPED proactive failure Yes
 RECOC1_CD_05_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_06_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_07_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_08_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_09_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_10_hzsmtceladm04 ONLINE  active            Yes
 RECOC1_CD_11_hzsmtceladm04 ONLINE  active            Yes

CellCLI> list physicaldisk attributes name,deviceName,diskType,physicalSize,status

 252:0      /dev/sda     HardDisk  12.474609375T                  normal
 252:1      /dev/sdb     HardDisk  12.474609375T                  normal
 252:2      /dev/sdc     HardDisk  12.474609375T                  normal
 252:3      /dev/sdd     HardDisk  12.474609375T                  normal
 252:4      /dev/sde     HardDisk  12.474609375T                  warning - poor performance
 252:5      /dev/sdf     HardDisk  12.474609375T                  normal
 252:6      /dev/sdg     HardDisk  12.474609375T                  normal
 252:7      /dev/sdh     HardDisk  12.474609375T                  normal
 252:8      /dev/sdi     HardDisk  12.474609375T                  normal
 252:9      /dev/sdj     HardDisk  12.474609375T                  normal
 252:10     /dev/sdk     HardDisk  12.474609375T                  normal
 252:11     /dev/sdl     HardDisk  12.474609375T                  normal
 FLASH_10_1 /dev/nvme2n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_10_2 /dev/nvme3n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_4_1  /dev/nvme4n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_4_2  /dev/nvme5n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_5_1  /dev/nvme6n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_5_2  /dev/nvme7n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_6_1  /dev/nvme0n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_6_2  /dev/nvme1n1 FlashDisk 2.910957656800746917724609375T normal
 M2_SYS_0   /dev/sdm     M2Disk    139.698386669158935546875G     normal
 M2_SYS_1   /dev/sdn     M2Disk    139.698386669158935546875G     normal

CellCLI> list physicaldisk 252:4 detail

 name:                   252:4
 deviceId:               10
 deviceName:             /dev/sde
 diskType:               HardDisk
 enclosureDeviceId:      252
 errOtherCount:          0
 luns:                   0_4
 makeModel:              "WDC     W7214A520ORA014T"
 physicalFirmware:       A420
 physicalInsertTime:     2020-06-23T07:26:51+08:00
 physicalInterface:      sas
 physicalSerial:         N5GZ5C
 physicalSize:           12.474609375T
 slotNumber:             4
 status:                 warning - poor performance

CellCLI> list lun 0_4 detail

 name:                   0_4
 deviceName:             /dev/sde
 diskType:               HardDisk
 id:                     0_4
 isSystemLun:            FALSE
 lunSize:                12.47371959686279296875T
 lunUID:                 0_4
 physicalDrives:         252:4
 raidLevel:              0
 lunWriteCacheMode:      "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU"
 status:                 warning - poor performance

CellCLI> list celldisk where lun=0_4 detail

 name:                   CD_04_hzsmtceladm04
 comment:
 creationTime:           2020-06-23T23:42:51+08:00
 deviceName:             /dev/sde
 devicePartition:        /dev/sde
 diskType:               HardDisk
 freeSpace:              0
 id:                     ada4a55a-b2a2-45a1-8232-1f4ef074a1c7
 physicalDisk:           N5GZ5C
 size:                   12.4737091064453125T
 status:                 proactive failure

CellCLI> list griddisk where celldisk=CD_04_hzsmtceladm04 detail

 name:                   DATAC1_CD_04_hzsmtceladm04
 asmDiskGroupName:       DATAC1
 asmDiskName:            DATAC1_CD_04_HZSMTCELADM04
 asmFailGroupName:       HZSMTCELADM04
 availableTo:
 cachedBy:               FD_03_hzsmtceladm04
 cachingPolicy:          default
 cellDisk:               CD_04_hzsmtceladm04
 comment:                "Cluster smt-cluster diskgroup DATAC1"
 creationTime:           2020-06-23T23:44:10+08:00
 diskType:               HardDisk
 id:                     7bc9452c-2368-430b-a4cd-8e8147877e03
 size:                   9.978515625T
 status:                 proactive failure

 name:                   RECOC1_CD_04_hzsmtceladm04
 asmDiskGroupName:       RECOC1
 asmDiskName:            RECOC1_CD_04_HZSMTCELADM04
 asmFailGroupName:       HZSMTCELADM04
 availableTo:
 cachedBy:               FD_03_hzsmtceladm04
 cachingPolicy:          default
 cellDisk:               CD_04_hzsmtceladm04
 comment:                "Cluster smt-cluster diskgroup RECOC1"
 creationTime:           2020-06-23T23:44:31+08:00
 diskType:               HardDisk
 id:                     b76d5864-09ab-49a0-bc70-87196dee70a9
 size:                   2.495147705078125T
 status:                 proactive failure

CellCLI> alter physicaldisk 252:4 drop for replacement

Physical disk 252:4 was dropped for replacement.

CellCLI> list physicaldisk 252:4 detail
 name:                   252:4
 deviceId:               10
 diskType:               HardDisk
 enclosureDeviceId:      252
 errOtherCount:          0
 makeModel:              "WDC     W7214A520ORA014T"
 physicalFirmware:       A420
 physicalInsertTime:     2020-06-23T07:26:51+08:00
 physicalInterface:      sas
 physicalSerial:         N5GZ5C
 physicalSize:           12.474609375T
 slotNumber:             4
 status:                 warning - poor performance - dropped for replacement

CellCLI> list griddisk attributes name,asmmodestatus,status,asmdeactivationoutcome

 DATAC1_CD_00_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_01_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_02_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_03_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_04_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_05_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_06_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_07_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_08_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_09_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_10_hzsmtceladm04 ONLINE active Yes
 DATAC1_CD_11_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_00_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_01_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_02_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_03_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_04_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_05_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_06_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_07_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_08_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_09_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_10_hzsmtceladm04 ONLINE active Yes
 RECOC1_CD_11_hzsmtceladm04 ONLINE active Yes

CellCLI> list physicaldisk attributes name,deviceName,diskType,physicalSize,status

 252:0      /dev/sda     HardDisk  12.474609375T                  normal
 252:1      /dev/sdb     HardDisk  12.474609375T                  normal
 252:2      /dev/sdc     HardDisk  12.474609375T                  normal
 252:3      /dev/sdd     HardDisk  12.474609375T                  normal
 252:4      /dev/sde     HardDisk  12.474609375T                  normal
 252:5      /dev/sdf     HardDisk  12.474609375T                  normal
 252:6      /dev/sdg     HardDisk  12.474609375T                  normal
 252:7      /dev/sdh     HardDisk  12.474609375T                  normal
 252:8      /dev/sdi     HardDisk  12.474609375T                  normal
 252:9      /dev/sdj     HardDisk  12.474609375T                  normal
 252:10     /dev/sdk     HardDisk  12.474609375T                  normal
 252:11     /dev/sdl     HardDisk  12.474609375T                  normal
 FLASH_10_1 /dev/nvme2n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_10_2 /dev/nvme3n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_4_1  /dev/nvme4n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_4_2  /dev/nvme5n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_5_1  /dev/nvme6n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_5_2  /dev/nvme7n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_6_1  /dev/nvme0n1 FlashDisk 2.910957656800746917724609375T normal
 FLASH_6_2  /dev/nvme1n1 FlashDisk 2.910957656800746917724609375T normal
 M2_SYS_0   /dev/sdm     M2Disk    139.698386669158935546875G     normal
 M2_SYS_1   /dev/sdn     M2Disk    139.698386669158935546875G     normal

CellCLI> alter cell validate configuration

Cell hzsmtceladm04 successfully altered

CellCLI> exit

SQL> select group_number,path,header_status,mount_status,name from V$ASM_DISK where path like '%_04_hzsmtceladm04';

GROUP_NUMBER PATHHEADER_STATU MOUNT_S NAME
------------ -------------------------------------------------- ------------ ------- ------------------------------
   1 o/192.168.10.64;192.168.10.65/DATAC1_CD_04_hzsmtceladm04 MEMBER     CACHED  DATAC1_CD_04_HZSMTCELADM04

   2 o/192.168.10.64;192.168.10.65/RECOC1_CD_04_hzsmtceladm04 MEMBER     CACHED  RECOC1_CD_04_HZSMTCELADM04


SQL> select * from gv$asm_operation;

   INST_ID GROUP_NUMBER OPERA PASSSTAT  POWER     ACTUALSOFAREST_WORK   EST_RATE EST_MINUTES
---------- ------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- -----------
ERROR_CODE CON_ID
-------------------------------------------- ----------
 2      1 REBAL COMPACTWAIT      4
      0

 2      1 REBAL REBALANCE WAIT      4
      0

 2      1 REBAL REBUILDWAIT      4
      0
  ...