Overview of Pacemaker Corosync Cluster

cluster_stack_arch

STONITH Triggering Condition

Under what circumstances will the cluster activate the isolation mechanism STONITH to prevent brain split?

  1. If the heartbeat network between cluster nodes fails, to prevent nodes from competing for cluster resources, the node with fewer votes will be STONITH.
  2. If a certain node fails and the cluster resources cannot be released normally, in order to forcibly release the cluster resources, this failed node will be STONITH.

Network Prerequisites

firewall-cmd --permanent --add-service=high-availability
firewall-cmd --add-service=high-availability
Port When Required
TCP 2224 Required on all nodes (needed by the pcsd Web UI and required for node-to-node communication)

It is crucial to open port 2224 in such a way that pcs from any node can talk to all nodes in the cluster, including itself. When using the Booth cluster ticket manager or a quorum device you must open port 2224 on all related hosts, such as Booth arbiters or the quorum device host.
TCP 3121 Required on all nodes if the cluster has any Pacemaker Remote nodes

Pacemaker's crmd daemon on the full cluster nodes will contact the pacemaker_remoted daemon on Pacemaker Remote nodes at port 3121. If a separate interface is used for cluster communication, the port only needs to be open on that interface. At a minimum, the port should open on Pacemaker Remote nodes to full cluster nodes. Because users may convert a host between a full node and a remote node, or run a remote node inside a container using the host's network, it can be useful to open the port to all nodes. It is not necessary to open the port to any hosts other than nodes.
TCP 5403 Required on the quorum device host when using a quorum device with corosync-qnetd. The default value can be changed with the -p option of the corosync-qnetd command.
UDP 5404 Required on corosync nodes if corosync is configured for multicast UDP
UDP 5405 Required on all corosync nodes (needed by corosync)
TCP 21064 Required on all nodes if the cluster contains any resources requiring DLM (such as clvm or GFS2)
TCP 9929, UDP 9929 Required to be open on all cluster nodes and booth arbitrator nodes to connections from any of those same nodes when the Booth ticket manager is used to establish a multi-site cluster.

Using ocf:pacemaker:ping to monitor network connectivity

pcs resource create ping ocf:pacemaker:ping \
multiplier=1000 host_list=192.168.3.1 --clone

pcs constraint location oraclegroup \
rule score=-INFINITY pingd lt 1 or not_defined pingd

・ About ocf pacemaker ping resource

QEMU-KVM

pcs resource create win2k3a_res VirtualDomain \
  hypervisor="qemu:///system" \
  config="/vm/qemu_config/win2k3a.xml" \
  meta allow-migrate="true" priority="100" \
  migration_transport=ssh \
  op start timeout="120s" \
  op stop timeout="120s" \
  op monitor timeout="30" interval="10" \
  op migrate_from interval="0" timeout="120s" \
  op migrate_to interval="0" timeout="120s"

Troubleshooting

Issue: The resources cannot start normal

When one node in a two-node cluster (without arbitration configuration) fails, the remaining nodes will be unable to start the cluster resources due to insufficient votes (less than 50%).
The pcs status command shows that all resources are in the stopped state.

At this point, the cluster resources can be temporarily forced to start by using the command pcs resource debug-start <resource>.

Issue: Activate or deactivate a volume group manually

If the volume_list parameter is configured in /etc/lvm/lvm.conf but the volume group/Logical Volume cannot be activated, the configuration parameter can be added when executing vgchange to activate it.

vgchange --addtag pacemaker vg_data
vgchange -ay vg_data --config 'activation { volume_list = [ "@pacemaker" ]}'

vgchange -an -vvv vg_data
vgchange --deltag pacemaker vg_data

Issue: Fails with creating new LV

~]# lvcreate  -L 1G -n lv_test vg_data
Ignoring empty string in config file activation/volume_list
    Not activating vg_data/lv_test since it does not pass activation filter.
   Failed to activate new LV.

~]# lvs -a -o +vg_systemid,tags
~]# lvcreate --addtag pacemaker -L 1G -n lv_test vg_data

Issue: The new VG and LV are not visible on other cluster node

~]# lvs --foreign

Reference:

・ Post creating VG and LV on one cluster node, the corresponding VG and LV are not visible on other cluster node.pdf |