Installing QEMU-KVM Pacemaker Corosync Cluster on RHEL 8

RHEL 8.7
pacemaker-2.1.7-5.2.el8_10.x86_64
corosync-3.1.8-1.el8.x86_64

Object	Value
Node	kvm01, kvm02
Node Public IP	192.168.3.211, 192.168.3.212
Node Private IP	100.100.100.211, 100.100.100.212
Cluster Name	vmcluster
BusType	iSCSI, 192.168.3.112
Shared Disk 1	20GB, data
Shared Disk 2	5GB, data
Shared Disk 3	1GB, fence_scsi

Setting Media ISO YUM Repository

・ Configuring Yum Repositories

Enable High Availability Repository

subscription-manager register
subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms

Configuring Network

・ Configuring Network Bonding on RHEL 7

Configuring hosts resolve.

cat >>/etc/hosts <<EOF
# Added by sakura
192.168.3.211    kvm01
192.168.3.212    kvm02
100.100.100.211    kvm01-hb
100.100.100.212    kvm02-hb
EOF

Enabling SSH User Equivalency

ssh-keygen -t rsa
ssh-copy-id kvm01
ssh-copy-id kvm02

Configuring ISCSI

・ Configuring iSCSI Share on Linux

Creating a High Availability Cluster

# Install packages
dnf -y install pcs pacemaker corosync fence-agents-all pcp-zeroconf

firewall-cmd --permanent --add-service=high-availability
firewall-cmd --add-service=high-availability

echo hacluster | passwd hacluster --stdin

systemctl enable --now pcsd.service

# Create a cluster
pcs host auth kvm01-hb kvm02-hb -u hacluster
pcs cluster setup kvmcluster --start kvm01-hb kvm02-hb
pcs cluster enable --all
pcs cluster status

Configuring STONITH

・ Using SCSI Persistent Reservation Fencing (fence_scsi) with pacemaker in a Red Hat High Availability cluster

pcs stonith describe fence_scsi
cat /proc/partitions

ll /dev/disk/by-id/ | grep sdc    # get the /dev/sdc wwn

pcs stonith create scsi-shooter fence_scsi \
  pcmk_host_list="kvm01-hb kvm02-hb" \
  devices="/dev/disk/by-id/wwn-0x60014053a118daeb0374b63a156409c3" \
  pcmk_reboot_action="off" meta provides=unfencing --force

pcs stonith config scsi-shooter

Test fencing.

sg_persist -n -i -r -d /dev/sdc
sg_persist -n -i -k -d /dev/sdc

echo c > /proc/sysrq-trigger

Configruing resources and resource groups

FileSystem resources

・ Configuring LVM Volumes in a Cluster #CLVM-Volumes

CLVM.

subscription-manager repos --enable=rhel-8-for-x86_64-resilientstorage-rpms

dnf -y install lvm2-lockd dlm

sed -i '/^global/a\\tuse_lvmlockd = 1' /etc/lvm/lvm.conf    # set the use_lvmlockd = 1

pcs property set no-quorum-policy=freeze
pcs resource create dlm --group locking ocf:pacemaker:controld \
  op monitor interval=30s on-fail=fence
pcs resource clone locking interleave=true
pcs resource create lvmlockd --group locking ocf:heartbeat:lvmlockd \
  op monitor interval=30s on-fail=fence

Sets the global/use_lvmlockd = 1 in /etc/lvm/lvm.conf.

GFS2.

dnf -y install gfs2-utils

pvcreate /dev/sdb /dev/sdd
vgcreate -Ay --shared vg_ckvm /dev/sdb /dev/sdd
lvcreate -L1G -n lv_ckvm01 vg_ckvm
lvcreate -L20G -n lv_ckvm02 vg_ckvm

mkfs.gfs2 -j3 -p lock_dlm -t kvmcluster:fs_ckvm01 /dev/vg_ckvm/lv_ckvm01
mkfs.gfs2 -j3 -p lock_dlm -t kvmcluster:fs_ckvm02 /dev/vg_ckvm/lv_ckvm02

pcs resource create --disabled --group vg_group lv_ckvm01 \
  ocf:heartbeat:LVM-activate lvname=lv_ckvm01 vgname=vg_ckvm \
  activation_mode=shared vg_access_mode=lvmlockd
pcs resource create --disabled --group vg_group lv_ckvm02 \
  ocf:heartbeat:LVM-activate lvname=lv_ckvm02 vgname=vg_ckvm \
  activation_mode=shared vg_access_mode=lvmlockd

pcs resource clone vg_group interleave=true
pcs constraint order start locking-clone then vg_group-clone
pcs resource enable lv_ckvm01 lv_ckvm02

mkdir -p /vm/config /vm/data
pcs resource create --disabled --group fs_group fs_ckvm01 Filesystem \
  device="/dev/vg_ckvm/lv_ckvm01" directory="/vm/config" \
  fstype="gfs2" options="context=system_u:object_r:virt_etc_rw_t:s0" \
  op monitor interval=10s on-fail=fence
pcs resource create --disabled --group fs_group fs_ckvm02 Filesystem \
  device="/dev/vg_ckvm/lv_ckvm02" directory="/vm/data" \
  fstype="gfs2" options="context=system_u:object_r:svirt_image_t:s0" \
  op monitor interval=10s on-fail=fence

pcs resource clone fs_group interleave=true
pcs constraint order start vg_group-clone then fs_group-clone
pcs resource enable fs_ckvm01 fs_ckvm02

QEMU-KVM resources

・ Installing QEMU-KVM on Linux

Create a VM.

virt-install \
  --virt-type kvm \
  --name vm1 \
  --memory 2048 \
  --vcpus 1 \
  --os-type linux \
  --os-variant centos7.0 \
  --network bridge=br0 \
  --disk /vm/data/vm1.qcow2,size=10,bus=virtio,format=qcow2 \
  --location http://192.168.3.112/assets/tmp/mnt -x console=ttyS0 \
  --graphics none

virsh dumpxml vm1 > /vm/config/vm1.xml
virsh shutdown vm1
virsh undefine vm1
virsh define /vm/config/vm1.xml

Test migration.

virsh migrate --domain vm1 \
  --desturi qemu+ssh://kvm02/system \
  --persistent --undefinesource \
  --unsafe --live

virsh migrate --domain vm1 \
  --desturi qemu+ssh://kvm01/system \
  --persistent --undefinesource \
  --unsafe --live

Create VirtualDomain resources.

pcs resource create vm1 VirtualDomain \
  hypervisor="qemu:///system" \
  config="/vm/config/vm1.xml" \
  migration_transport=ssh \
  meta allow-migrate="true"

pcs constraint order start fs_group-clone then vm1

Troubleshooting

Issue: VM live migration failed due to SELinux.

~]# tail -f /var/log/message
Oct  6 16:26:42 kvm02 journal[835002]: operation failed: migration out job: unexpectedly failed
Oct  6 16:26:43 kvm02 dbus-daemon[1119]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Oct  6 16:26:43 kvm02 systemd[1]: Started SETroubleshoot daemon for processing new SELinux denial logs.
Oct  6 16:26:44 kvm02 dbus-daemon[1119]: [system] Activating service name='org.fedoraproject.SetroubleshootPrivileged' requested by ':1.888' (uid=991 pid=842191 comm="/usr/libexec/platform-python -Es /usr/sbin/setroub" label="system_u:system_r:setroubleshootd_t:s0") (using servicehelper)
Oct  6 16:26:45 kvm02 dbus-daemon[1119]: [system] Successfully activated service 'org.fedoraproject.SetroubleshootPrivileged'
Oct  6 16:26:46 kvm02 setroubleshoot[842191]: SELinux is preventing /usr/libexec/qemu-kvm from read access on the file vm1.qcow2. For complete SELinux messages run: sealert -l 83a2dae1-a4d6-4fe5-949d-15082c01f5b6

~]# sealert -l 83a2dae1-a4d6-4fe5-949d-15082c01f5b6
SELinux is preventing /usr/libexec/qemu-kvm from read access on the file /vm/data/vm1.qcow2.

*****  Plugin qemu_file_image (98.8 confidence) suggests   *******************

If vm1.qcow2 is a virtualization target
Then you need to change the label on vm1.qcow2'
Do
# semanage fcontext -a -t virt_image_t '/vm/data/vm1.qcow2'
# restorecon -v '/vm/data/vm1.qcow2'

*****  Plugin catchall (2.13 confidence) suggests   **************************

If you believe that qemu-kvm should be allowed read access on the vm1.qcow2 file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'live_migration' --raw | audit2allow -M my-livemigration
# semodule -X 300 -i my-livemigration.pp


Additional Information:
Source Context                system_u:system_r:svirt_t:s0:c206,c294
Target Context                system_u:object_r:svirt_image_t:s0:c721,c940
Target Objects                /vm/data/vm1.qcow2 [ file ]
Source                        live_migration
Source Path                   /usr/libexec/qemu-kvm
Port                          <Unknown>
Host                          kvm02
Source RPM Packages
Target RPM Packages
SELinux Policy RPM            selinux-policy-targeted-3.14.3-108.el8.noarch
Local Policy RPM              selinux-policy-targeted-3.14.3-108.el8.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     kvm02
Platform                      Linux kvm02 4.18.0-425.3.1.el8.x86_64 #1 SMP Fri
                              Sep 30 11:45:06 EDT 2022 x86_64 x86_64
Alert Count                   2
First Seen                    2025-10-06 16:26:41 CST
Last Seen                     2025-10-06 16:26:41 CST
Local ID                      83a2dae1-a4d6-4fe5-949d-15082c01f5b6

Raw Audit Messages
type=AVC msg=audit(1759739201.904:2640): avc:  denied  { read } for  pid=839298 comm="worker" path="/vm/data/vm1.qcow2" dev="dm-4" ino=155106 scontext=system_u:system_r:svirt_t:s0:c206,c294 tcontext=system_u:object_r:svirt_image_t:s0:c721,c940 tclass=file permissive=0


Hash: live_migration,svirt_t,svirt_image_t,file,read

Bad file descriptor.

~]# tail -f /var/log/libvirt/qemu/vm1.log
2025-10-06 08:26:40.971+0000: initiating migration
2025-10-06T08:26:41.905381Z qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all() failed (-1)
2025-10-06T08:26:41.906030Z qemu-kvm: Could not reopen qcow2 layer: Could not read qcow2 header: Permission denied
2025-10-06T08:26:42.185453Z qemu-kvm: Unable to read from socket: Bad file descriptor
2025-10-06T08:26:42.185504Z qemu-kvm: Unable to read from socket: Bad file descriptor
2025-10-06T08:26:42.185523Z qemu-kvm: Unable to read from socket: Bad file descriptor

Cause:

SELinux is not supported for GFS2, SELinux labels on GFS2 file systems can get out of sync between cluster nodes because of how they are cached in memory.

Solution:

Specify the mount options when you create a GFS2 file system resource.

options="context=system_u:object_r:svirt_image_t:s0"

Reference

・ Red_Hat_Enterprise_Linux-8-Configuring_and_managing_high_availability_clusters-en-US.pdf |
・ Red_Hat_Enterprise_Linux-8-Configuring_GFS2_file_systems-en-US.pdf |