Configuring Storage Device Path Persistence Using UDEV

For Oracle Automatic Storage Manager (ASM) to use disks, it needs to be able to identify the devices consistently and for them to have the correct ownership and permissions. In Linux you can use ASMLib to manage these tasks, but it is seen as an additional layer of complexity and has never really gained any popularity. Instead, many people use the Linux device manager "udev" to perform these tasks. This article presents a brief overview of setting up udev rules with respect to disks for use with ASM in Oracle 11g. The examples are all done using Oracle Linux 5, 6, 7 and 8 so they will be consistent with RHEL and CentOS 5, 6, 7 and 8.

  • Identify the Disks (/sbin/scsi_id)
  • Make SCSI Devices Trusted
  • Create UDEV Rules File
  • Load Updated Block Device Partitions (/sbin/partprobe)
  • Test Rules (udevtest)
  • Restart UDEV Service
  • Check Ownership and Permissions

Identify the Disks (/sbin/scsi_id)

We are going to write device-specific rules, so we need to be able to identify each device consistently, irrespective of the order in which Linux discovers it.
To do this we are going to use the SCSI ID for each disk (not the partition), which we get using the scsi_id command. The "-s" option makes the paths relative to the "/sys" directory.

For Oracle Linux 5, use the following command.

# /sbin/scsi_id -g -u -s /block/sdb
SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_

The "-s" is not available in Oracle Linux 6, so you must use the following syntax.

# /sbin/scsi_id -g -u -d /dev/sdb
SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_

The location of the scsi_id command has changed in Oracle Linux 7 and 8, so you must use the following syntax.

# /usr/lib/udev/scsi_id -g -u -d /dev/sdb
SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_

Make SCSI Devices Trusted

Add the options=-g to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn't already exist.

cat >> /etc/scsi_id.config << EOF
options=-g
EOF

Create UDEV Rules File

Create the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file.
The file should contain the following lines for Oracle Linux 5. The PROGRAM parameter must match the command you used to retrieve the SCSI ID, and the RESULT parameter must match the value returned from your disks.

for i in a b c
do
echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -s /block/$parent", RESULT==\"`/sbin/scsi_id -g -u -s /block/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
done

# $parent is used if the device is partitioned, $name is used if the device is not partitioned.

The equivalent for Oracle Linux 6 is shown below.

for i in a b c
do
echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -d /dev/\$parent\", RESULT==\"`/sbin/scsi_id -g -u -d /dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
done

# $parent is used if the device is partitioned, $name is used if the device is not partitioned.

The equivalent for Oracle Linux 7 and 8 is shown below.

for i in a b c
do
echo "KERNEL==\"sd?1\", SUBSYSTEM==\"block\", PROGRAM==\"/usr/lib/udev/scsi_id -g -u -d /dev/\$parent\", RESULT==\"`/usr/lib/udev/scsi_id -g -u -d /dev/sd$i`\", SYMLINK+=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
done

# $parent is used if the device is partitioned, $name is used if the device is not partitioned.

Load Updated Block Device Partitions (/sbin/partprobe)

Load updated block device partition tables.
For OL5, OL6 and OL7

# /sbin/partprobe /dev/sdb1
# /sbin/partprobe /dev/sdc1

For OL8

# /sbin/partx -u /dev/sbc1
# /sbin/partx -u /dev/sdc1

Test Rules (udevtest)

Test the rules are working as expected.
For OL5

# udevtest /block/sdb/sdb1
# udevtest /block/sdc/sdc1

For OL6, OL7 and OL8

# udevadm test /block/sdb/sdb1
# udevadm test /block/sdc/sdc1

The output from the first disk should look something like this.

# udevtest /block/sdb/sdb1
main: looking at device '/block/sdb/sdb1' from subsystem 'block'
udev_rules_get_name: add symlink 'disk/by-id/scsi-SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3-part1'
udev_rules_get_name: add symlink 'disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-part1'
run_program: '/lib/udev/vol_id --export /dev/.tmp-8-17'
run_program: '/lib/udev/vol_id' returned with status 4
run_program: '/sbin/scsi_id -g -u -s /block/sdb/sdb1'
run_program: '/sbin/scsi_id' (stdout) 'SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_'
run_program: '/sbin/scsi_id' returned with status 0
udev_rules_get_name: rule applied, 'sdb1' becomes 'asm-disk1'
udev_device_event: device '/block/sdb/sdb1' already in database, validate currently present symlinks
udev_node_add: creating device node '/dev/asm-disk1', major = '8', minor = '17', mode = '0660', uid = '1100', gid = '1200'
udev_node_add: creating symlink '/dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3-part1' to '../../asm-disk1'
udev_node_add: creating symlink '/dev/disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-part1' to '../../asm-disk1'
main: run: 'socket:/org/kernel/dm/multipath_event'
main: run: 'socket:/org/kernel/udev/monitor'
main: run: '/lib/udev/udev_run_devd'
main: run: 'socket:/org/freedesktop/hal/udev_event'
main: run: '/sbin/pam_console_apply /dev/asm-disk1 /dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3-part1 /dev/disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-part1'

Restart UDEV Service

Restart the UDEV service.
For OL5

# /sbin/udevcontrol reload_rules

For OL5 and OL6 : Not needed for OL7 and OL8

# /sbin/start_udev

For OL6, OL7 and OL8

# /sbin/udevadm control --reload-rules
# /sbin/udevadm trigger

Check Ownership and Permissions

Check the disks are now available with the "asm-disk*" alias and the correct ownership and permissions.

# ls -al /dev/asm-disk*
brw-rw---- 1 oracle dba 8, 17 Apr 8 22:47 asm-disk1
brw-rw---- 1 oracle dba 8, 33 Apr 8 22:47 asm-disk2
brw-rw---- 1 oracle dba 8, 49 Apr 8 22:47 asm-disk3
brw-rw---- 1 oracle dba 8, 65 Apr 8 22:47 asm-disk4

So the ASM_DISKSTRING initialization parameter in the ASM instance can be set to '/dev/asm-disk*' to identify the ASM disks.