Installing Pacemaker Corosync Cluster on RHEL 8

RHEL 8.7

Setting Media ISO repository

・ Configuring Yum Repositories

Enable High Availability Repository

subscription-manager register
subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms

Configuring Network

・ Configuring Network Bonding on RHEL 7,8

nmcli con add type bond con-name bond0 ifname bond0 bond.options "mode=active-backup,miimon=1000"
nmcli con add type bond con-name bond1 ifname bond1 bond.options "mode=active-backup,miimon=1000"

nmcli con add type ethernet slave-type bond con-name bond0-port1 ifname eno1 master bond0
nmcli con add type ethernet slave-type bond con-name bond0-port2 ifname eno2 master bond0

nmcli con add type ethernet slave-type bond con-name bond1-port1 ifname eno3 master bond1
nmcli con add type ethernet slave-type bond con-name bond1-port2 ifname eno4 master bond1

# node01
nmcli con modify bond0 ipv4.addresses '192.168.3.211/24' ipv4.gateway '192.168.3.1' ipv4.method manual autoconnect yes
nmcli con modify bond0 connection.autoconnect-slaves 1

nmcli con modify bond1 ipv4.addresses '100.100.100.211/24' ipv4.method manual autoconnect yes
nmcli con modify bond1 connection.autoconnect-slaves 1

# node02
nmcli con modify bond0 ipv4.addresses '192.168.3.212/24' ipv4.gateway '192.168.3.1' ipv4.method manual autoconnect yes
nmcli con modify bond0 connection.autoconnect-slaves 1

nmcli con modify bond1 ipv4.addresses '100.100.100.212/24' ipv4.method manual autoconnect yes
nmcli con modify bond1 connection.autoconnect-slaves 1

Configuring hosts resolve.

cat >>/etc/hosts <<EOF

# for pacemaker cluster
192.168.3.211    node01
192.168.3.212    node01
100.100.100.211    node01-hb
100.100.100.212    node02-hb
192.168.3.213    vip
EOF

Installing cluster software

On each node in the cluster, install the Red Hat High Availability Add-On software packages along with all available fence agents from the High Availability channel.

dnf -y install pcs pacemaker fence-agents-all

If you are running the firewalld daemon, execute the following commands to enable the ports that are required by the Red Hat High Availability Add-On.

firewall-cmd --permanent --add-service=high-availability
firewall-cmd --add-service=high-availability

In order to use pcs to configure the cluster and communicate among the nodes, you must set a password on each node for the user ID hacluster, which is the pcs administration account. It is recommended that the password for user hacluster be the same on each node.

echo hacluster | passwd hacluster --stdin

Before the cluster can be configured, the pcsd daemon must be started and enabled to boot on startup on each node. This daemon works with the pcs command to manage configuration across the nodes in the cluster.

On each node in the cluster, execute the following commands to start the pcsd service and to enable pcsd at system start.

systemctl start pcsd.service
systemctl enable pcsd.service

(Recommended) To install the pcp-zeroconf package, PCP is Red Hat’s recommended resource-monitoring tool for RHEL systems.

dnf -y install pcp-zeroconf

Creating a high availability cluster

Authenticate the pcs user hacluster for each node in the cluster on the node from which you will be running pcs.

pcs host auth node01-hb node02-hb

Execute the following command from node1-hb to create the two-node cluster cluster that consists of nodes node1-hb and node2-hb. This will propagate the cluster configuration files to both nodes in the cluster. This command includes the --start option, which will start the cluster services on both nodes in the cluster.

pcs cluster setup my_cluster --start node01-hb node02-hb

Enable the cluster services to run on each node in the cluster when the node is booted.

pcs cluster enable --all

You can display the current status of the cluster with the pcs cluster status command.

pcs cluster status
pcs cluster setup my_twolink_cluster node01-hb addr=100.100.100.211 addr=100.100.200.211 node02-hb addr=100.100.100.212 addr=100.100.200.212

If you use NIC binding, you do not need to configure multiple links.

Configuring STONITH

Test the IPMI interface is reachable using the default port 623:

nmap -sU -p623 192.168.3.99
nmap -sU -p623 192.168.3.123

Finally you can safely test your configuration by printing the chassis status on each node remotely.

ipmitool -I lanplus -H 192.168.3.99 -U Administrator -P 60142058 -v chassis status
ipmitool -I lanplus -H 192.168.3.123 -U root -P calvin -v chassis status

Set up a password script instead of directly using password.

cat >/usr/local/bin/IPMI_password_node01 <<EOF
#! /bin/bash
echo "60142058"
EOF

cat >/usr/local/bin/IPMI_password_node02 <<EOF
#! /bin/bash
echo "calvin"
EOF

chmod 500 /usr/local/bin/IPMI_password_node*

In a 2-node cluster it may happen that both nodes are unable to contact each other and then each node tries to fence the other one. But can't reboot both nodes at the same time since that will result in downtime and possibly harm cluster integrity. To avoid this need to configure a different delay (e.g., one without delay, and the other one with at least a 10 second delay).

pcs stonith create node01_ilo fence_ilo4 ipaddr="192.168.3.99" \
lanplus="1" login="Administrator" passwd_script="IPMI_password_node01" pcmk_host_list="node01-hb" pcmk_reboot_action="off" power_wait="10" pcmk_delay_base="10"

pcs stonith create node02_idrac fence_idrac ipaddr="192.168.3.123" \
lanplus="1" login="root" passwd_script="IPMI_password_node02" pcmk_host_list="node02-hb" pcmk_reboot_action="off" power_wait="10"

Unwanted fencing might happen also when a node "commit suicide", i.e., shut itself down because it was not able to contact the other node of the cluster. This is an unwanted situation because all nodes of a cluster might be fenced at the same time. To avoid this should set a constraint to prevent a node's stonith resource from running on the cluster node itself:

pcs constraint location node01_ilo avoids node1-hb
pcs constraint location node02_idrac avoids node2-hb

To prevent unwanted fencing in the event of minor network outages, increase the totem token timeout to at least 5 seconds by editing /etc/corosync/corosync.conf as follows:

totem {
    version: 2
    cluster_name: my_cluster
    secauth: off
    transport: udpu
    token: 5000
}

Then sync this config file to all other cluster nodes and reload corosync:

pcs cluster sync
pcs cluster reload corosync

Now that fencing is configured, set the stonith property to true to enable it:

pcs property set stonith-enabled=true
pcs stonith cleanup

Once the fence device has been configured in the cluster with the same options that worked manually and the cluster has been started, test fencing with the pcs stonith fence command from any node (or even multiple times from different nodes), as in the following example. The pcs stonith fence command reads the cluster configuration from the CIB and calls the fence agent as configured to execute the fence action. This verifies that the cluster configuration is correct.

pcs stonith fence node01-hb
pcs stonith fence node02-hb

Create a kdump fence.

systemctl is-enabled kdump

firewall-cmd --add-port=7410/udp
firewall-cmd --add-port=7410/udp --permanent

pcs stonith create kdump fence_kdump pcmk_reboot_action="off" pcmk_host_list="node01-hb node02-hb" pcmk_reboot_action="off"

pcs stonith level add 1 node01-hb kdump
pcs stonith level add 1 node02-hb kdump
pcs stonith level add 2 node01-hb node01_ilo
pcs stonith level add 2 node02-hb node02_idrac

touch /etc/kdump.conf
systemctl restart kdump

lsinitrd /boot/initramfs-$(uname -r)kdump.img | egrep "fence|hosts"

echo c >/proc/sysrq-trigger

・ How can I configure my RHEL High Availability cluster with pacemaker to monitor the link status of a network interface and relocate resources if a failure occurs?

pcs resource create bond0-monitor ethmonitor interface=bond0 --clone
crm_mon -A1
pcs resource create vip IPaddr2 ip=192.168.3.213 cidr_netmask=24 --group ora_grp
pcs constraint location vip rule score=-INFINITY ethmonitor-bond0 ne 1

Creating oracle database resources

・ Oracle Database in Pacemaker

Configuring constraints to prevent resources failling back

・ How can I prevent a managed resource from failing back to its original location automatically when the preferred node rejoins a Red Hat High Availability cluster with Pacemaker?

pcs constraint show --full

pcs constraint location ora_grp prefers node1-hb=2000
pcs constraint location ora_grp prefers node2-hb=1500

pcs property set default-resource-stickiness=1000
pcs resource defaults migration-threshold=3

crm_simulate -sL