Installing Pacemaker Corosync Cluster on RHEL 7

RHEL 7.9
pacemaker-1.1.21-4.el7.x86_64
corosync-2.4.5-4.el7.x86_64

Object	Value
Hostname	nood1, node2
Public NICs	bond0, Mode 1, eno1, eno2
Node Public IP	192.168.3.249, 192.168.3.250
Private NICs	bond1, Mode 1, eno3, eno4
Node Private IP	100.100.100.249, 100.100.100.250
Cluster Name	my_cluster
BusType	iSCSI, 192.168.3.112
Shared Disk 1	5GB
Shared Disk 2	10GB

Setting Media ISO YUM Repository

・ Configuring Yum Repositories

Configuring Network

・ Configuring Network Bonding on RHEL 7,8

nmcli con add type bond con-name bond0 ifname bond0 bond.options "mode=active-backup,miimon=1000"
nmcli con add type bond con-name bond1 ifname bond1 bond.options "mode=active-backup,miimon=1000"

nmcli con add type ethernet slave-type bond con-name bond0-port1 ifname eno1 master bond0
nmcli con add type ethernet slave-type bond con-name bond0-port2 ifname eno2 master bond0

nmcli con add type ethernet slave-type bond con-name bond1-port1 ifname eno3 master bond1
nmcli con add type ethernet slave-type bond con-name bond1-port2 ifname eno4 master bond1

nmcli con modify bond0 ipv4.addresses '192.168.3.249/24' ipv4.gateway '192.168.3.1' ipv4.method manual autoconnect yes
nmcli con modify bond0 connection.autoconnect-slaves 1

nmcli con modify bond1 ipv4.addresses '100.100.100.249/24' ipv4.gateway '100.100.100.1' ipv4.method manual autoconnect yes
nmcli con modify bond1 connection.autoconnect-slaves 1

Configuring hosts resolve.

cat >>/etc/hosts <<EOF
# Added by sakura
192.168.3.249    node1
192.168.3.250    node2
100.100.100.249    node1-hb
100.100.100.250    node2-hb
EOF

Installing the Red Hat High Availability Add-On software

On each node in the cluster, install the Red Hat High Availability Add-On software packages along with all available fence agents from the High Availability channel.
yum -y install pcs pacemaker corosync fence-agents-all
If you are running the firewalld daemon, execute the following commands to enable the ports that are required by the Red Hat High Availability Add-On.
firewall-cmd --permanent --add-service=high-availability
firewall-cmd --add-service=high-availability
In order to use pcs to configure the cluster and communicate among the nodes, you must set a password on each node for the user ID hacluster, which is the pcs administration account. It is recommended that the password for user hacluster be the same on each node.
echo "hacluster" | passwd --stdin hacluster
Before the cluster can be configured, the pcsd daemon must be started and enabled to boot on startup on each node. This daemon works with the pcs command to manage configuration across the nodes in the cluster.

On each node in the cluster, execute the following commands to start the pcsd service and to enable pcsd at system start.
systemctl start pcsd.service
systemctl enable pcsd.service
Authenticate the pcs user hacluster for each node in the cluster on the node from which you will be running pcs.
pcs cluster auth node1-hb node2-hb

Cluster Creation

Execute the following command from node1-hb to create the two-node cluster cluster that consists of nodes node1-hb and node2-hb. This will propagate the cluster configuration files to both nodes in the cluster. This command includes the --start option, which will start the cluster services on both nodes in the cluster.
pcs cluster setup --start --name my_cluster node1-hb node2-hb
Enable the cluster services to run on each node in the cluster when the node is booted.
pcs cluster enable --all
You can display the current status of the cluster with the pcs cluster status command.
pcs cluster status

Configuring Quorum Node

Install corosync-qdevice on the nodes of an existing cluster.

~]# yum install corosync-qdevice

Install pcs and corosync-qnetd on the quorum device host.

~]# sudo cat >>/etc/hosts <<EOF
192.168.3.54    node-quorum
EOF

~]# echo hacluster | passwd hacluster --stdin

~]$ sudo yum -y install pcs corosync-qnetd

~]$ sudo systemctl start pcsd.service
~]$ sudo systemctl enable pcsd.service

~]$ sudo pcs qdevice setup model net --enable --start
Quorum device 'net' initialized
quorum device enabled
Starting quorum device...
quorum device started

~]$ sudo pcs qdevice status net --full
QNetd address:                  *:5403
TLS:                            Supported (client certificate required)
Connected clients:              0
Connected clusters:             0
Maximum send/receive size:      32768/32768 bytes

~]$ sudo firewall-cmd --permanent --add-service=high-availability
~]$ sudo firewall-cmd --add-service=high-availability

Add the quorum device to the cluster.

~]# pcs cluster auth node-quorum
Username: hacluster
Password:
node-quorum: Authorized

~]# pcs quorum config
Options:

~]# pcs quorum status
Quorum information
------------------
Date:             Sun Aug  6 18:33:22 2023
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          1
Ring ID:          1/221
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           1
Flags:            2Node Quorate WaitForAll

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
         1          1         NR node1-hb (local)
         2          1         NR node2-hb

~]# pcs quorum device add model net host=node-quorum algorithm=ffsplit
Setting up qdevice certificates on nodes...
node1-hb: Succeeded
node2-hb: Succeeded
Enabling corosync-qdevice...
node2-hb: not enabling corosync-qdevice: corosync is not enabled
node1-hb: not enabling corosync-qdevice: corosync is not enabled
Sending updated corosync.conf to nodes...
node2-hb: Succeeded
node1-hb: Succeeded
Corosync configuration reloaded
Starting corosync-qdevice...
node2-hb: corosync-qdevice started
node1-hb: corosync-qdevice started

~]# pcs quorum config
Options:
Device:
  votes: 1
  Model: net
    algorithm: ffsplit
    host: node-quorum

~]# pcs quorum status
Quorum information
------------------
Date:             Sun Aug  6 19:05:15 2023
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          1
Ring ID:          1/221
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
         1          1    A,V,NMW node1-hb (local)
         2          1    A,V,NMW node2-hb
         0          1            Qdevice

Configuring STONITH

Test the IPMI interface is reachable using the default port 623:

nmap -sU -p623 192.168.3.99
nmap -sU -p623 192.168.3.123

Finally you can safely test your configuration by printing the chassis status on each node remotely.

ipmitool -I lanplus -H 192.168.3.99 -U Administrator -P 60142058 -v chassis status
ipmitool -I lanplus -H 192.168.3.123 -U root -P calvin -v chassis status

Set up a password script instead of directly using password.

cat > /usr/local/bin/IPMI_password_node1 << EOF
#! /bin/bash
echo "60142058"
EOF

cat > /usr/local/bin/IPMI_password_node2 << EOF
#! /bin/bash
echo "calvin"
EOF

chmod 500 /usr/local/bin/IPMI_password_node*

In a 2-node cluster it may happen that both nodes are unable to contact each other and then each node tries to fence the other one. But can't reboot both nodes at the same time since that will result in downtime and possibly harm cluster integrity. To avoid this need to configure a different delay (e.g., one without delay, and the other one with at least a 10 second delay).

pcs stonith create node1_ilo fence_ilo4 ipaddr="192.168.3.99" \
lanplus="1" login="Administrator" passwd_script="IPMI_password_node1" pcmk_host_list="node1-hb" pcmk_reboot_action="off" power_wait="10" pcmk_delay_base="10"

pcs stonith create node2_idrac fence_idrac ipaddr="192.168.3.123" \
lanplus="1" login="root" passwd_script="IPMI_password_node2" pcmk_host_list="node2-hb" pcmk_reboot_action="off" power_wait="10"

Unwanted fencing might happen also when a node "commit suicide", i.e., shut itself down because it was not able to contact the other node of the cluster. This is an unwanted situation because all nodes of a cluster might be fenced at the same time. To avoid this should set a constraint to prevent a node's stonith resource from running on the cluster node itself:

pcs constraint location fence_node1 avoids node1-hb
pcs constraint location fence_node2 avoids node2-hb

To prevent unwanted fencing in the event of minor network outages, increase the totem token timeout to at least 5 seconds by editing /etc/corosync/corosync.conf as follows:

totem {
    version: 2
    cluster_name: cluster
    secauth: off
    transport: udpu
    token: 5000
}

Then sync this config file to all other cluster nodes and reload corosync:

pcs cluster sync
pcs cluster reload corosync

Now that fencing is configured, set the stonith property to true to enable it:

pcs property set stonith-enabled=true
pcs stonith cleanup

Once the fence device has been configured in the cluster with the same options that worked manually and the cluster has been started, test fencing with the pcs stonith fence command from any node (or even multiple times from different nodes), as in the following example. The pcs stonith fence command reads the cluster configuration from the CIB and calls the fence agent as configured to execute the fence action. This verifies that the cluster configuration is correct.

pcs stonith fence node1-hb
pcs stonith fence node2-hb

Configuring Resources

Monitor the link status of a network interface bond0

・ How can I configure my RHEL High Availability cluster with pacemaker to monitor the link status of a network interface and relocate resources if a failure occurs?

pcs resource create bond0-monitor ethmonitor interface=bond0 --clone
crm_mon -A1
pcs resource create vip1 IPaddr2 ip=192.168.3.248 cidr_netmask=24 --group ora_grp
pcs constraint location vip1 rule score=-INFINITY ethmonitor-bond0 ne 1

Create oracle database resources

・ Oracle Database in Pacemaker

Configuring constraints to prevent resources failling back

・ How can I prevent a managed resource from failing back to its original location automatically when the preferred node rejoins a Red Hat High Availability cluster with Pacemaker?

pcs constraint show --full

pcs constraint location ora_grp prefers node1-hb=2000
pcs constraint location ora_grp prefers node2-hb=1500

pcs property set default-resource-stickiness=1000
pcs resource defaults migration-threshold=3

crm_simulate -sL