Corosync + Pacemaker

pcs – pacemaker/corosync configuration system

Control and configure pacemaker and corosync.

cluster – Configure cluster options and nodes

`start [--all \| <node>... ] [--wait[=<n>]] [--request-timeout=<seconds>]`	Start a cluster on specified node(s). If no nodes are specified then start a cluster on the local node.
`stop [--all \| <node>... ] [--request-timeout=<seconds>]`	Stop a cluster on specified node(s). If no nodes are specified then stop a cluster on the local node.
`status`	View current cluster status (an alias of ‘pcs status cluster’).
`sync`	Sync cluster configuration (files which are supported by all subcommands of this command) to all cluster nodes.
`sync corosync`	Sync corosync configuration to all nodes found from current corosync.conf file.

pcs cluster status

pcs cluster stop node-01
pcs cluster start node-01

resource – Manage cluster resources

`[status [--hide-inactive]]`	Show status of all currently configured resources. If –hide-inactive is specified, only show active resources.
`config [<resource id>]...`	Show options of all currently configured resources or if resource ids are specified show the options for the specified resource ids.
`list [filter] [--nodesc]`	Show list of all available resource agents (if filter is provided then only resource agents matching the filter will be shown). If –nodesc is used then descriptions of resource agents are not printed.
`enable <resource id>... [--wait[=n]]`	Allow the cluster to start the resources.
`disable <resource id>... [--safe [--no-strict]] [--simulate] [--wait[=n]]`	Attempt to stop the resources if they are running and forbid the cluster from starting them again.
`describe [<standard>:[<provider>:]]<type> [--full]`	Show options for the specified resource. If –full is specified, all options including advanced and deprecated ones are shown.
create <resource id> [<standard>:[<provider>:]]<type> [resource options] [op <operation action> <operation options> [<operation action> <operation options>]…] [meta <meta options>…] [clone [<clone options>] \| promotable [<promotable options>] \| –group <group id> [–before <resource id> \| –after <resource id>] \| bundle <bundle id>] [–disabled] [–no-default-ops] [–wait[=n]]	Create specified resource. If clone is used a clone resource is created. If promotable is used a promotable clone resource is created.
`delete <resource id\|group id\|bundle id\|clone id>`	Deletes the resource, group, bundle or clone (and all resources within the group/bundle/clone).
`cleanup [<resource id>] [node=<node>] [operation=<operation> [interval=<interval>]]`	Make the cluster forget failed operations from history of the resource and re-detect its current state. This can be useful to purge knowledge of past failures that have since been resolved.
`restart <resource id> [node] [--wait=n]`	Restart the resource specified. If a node is specified and if the resource is a clone or bundle it will be restarted only on the node specified. If –wait is specified, then we will wait up to ‘n’ seconds for the resource to be restarted and return 0 if the restart was successful or 1 if it was not.
`debug-monitor <resource id> [--full]`	This command will force the specified resource to be monitored on this node ignoring the cluster recommendations and print the output from monitoring the resource. Using –full will give more detailed output. This is mainly used for debugging resources that fail to be monitored.
`move <resource id> [destination node] [--master] [lifetime=<lifetime>] [--wait[=n]]`	Move the resource off the node it is currently running on by creating a -INFINITY location constraint to ban the node. If destination node is specified the resource will be moved to that node by creating an INFINITY location constraint to prefer the destination node.
`clear <resource id> [node] [--master] [--expired] [--wait[=n]]`	Remove constraints created by move and/or ban on the specified resource (and node if specified).
`group list`	Show all currently configured resource groups and their resources.
`group add <group id> <resource id> [resource id] ... [resource id] [--before <resource id> \| --after <resource id>] [--wait[=n]]`	Add the specified resource to the group, creating the group if it does not exist. If the resource is present in another group it is moved to the new group. You can use –before or –after to specify the position of the added resources relatively to some resource already existing in the group.

sudo pcs resource status
sudo pcs resource config san200

pcs resource list systemd:

pcs resource describe ocf:heartbeat:ZFS

pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.122.120 cidr_netmask=24 op monitor interval=30s
pcs resource create san300 ocf:heartbeat:ZFS pool=SAN300
pcs resource create syncoid-timer systemd:syncoid.timer
pcs resource create smbd systemd:smbd --disabled
pcs resource create winbind systemd:winbind --disabled

pcs resource cleanup san200

pcs resource debug-monitor san200

pcs resource move testIP node02

pcs resource group add drives san300 san200 backup sanoid-timer syncoid-timer
pcs resource group add samba smbd winbind

Get the current config of a resource

sudo pcs resource config san200

Update the current config of a resource

sudo pcs resource update san200 op stop timeout=240s

stonith – Manage fence devices

pcs stonith status
pcs stonith config

constraint – Manage resource constraints

`[config \| list \| show] [--full] [--all]`	List all current constraints that are not expired. If –all is specified also show expired constraints. If –full is specified also list the constraint ids.
`location <resource> prefers <node>[=<score>] [<node>[=<score>]]...`	Create a location constraint on a resource to prefer the specified node with score (default score: INFINITY).
`location remove <id>`	Remove a location constraint with the appropriate id.
`colocation [show] [--full]`	List all current colocation constraints (if –full is specified show the internal constraint id’s as well).
`colocation add [<role>] <source resource id> with [<role>] <target resource id> [score] [options] [id=constraint-id]`	Request to run on the same node where pacemaker has determined should run.
`colocation remove <source resource id> <target resource id>`	Remove colocation constraints with specified resources.
`order [show] [--full]`	List all current ordering constraints (if –full is specified show the internal constraint id’s as well).
`order [action] <resource id> then [action] <resource id> [options]`	Add an ordering constraint specifying actions (start, stop, promote, demote) and if no action is specified the default action will be start.
`order remove <resource1> [resourceN]...`	Remove resource from any ordering constraint

List all current constraints

pcs constraint config --full

Others

pcs constraint location remove cli-prefer-sanoid

pcs constraint colocation add WebSite with ClusterIP INFINITY
pcs constraint colocation add syncoid-timer with backup INFINITY

pcs constraint colocation add samba with drives INFINITY

pcs constraint order ClusterIP then WebSite
pcs constraint order start drives then start samba
pcs constraint order stop samba then stop drives

status – View cluster status

`[status] [--full \| --hide-inactive]`	View all information about the cluster and resources (–full provides more details, –hide-inactive hides inactive resources).
`resources [--hide-inactive]`	Show status of all currently configured resources. If –hide-inactive is specified, only show active resources.

pcs status
pcs status --full

config – View and manage cluster configuration

`[show]`	View full cluster configuration.
`backup [filename]`	Creates the tarball containing the cluster configuration files. If filename is not specified the standard output will be used.
`restore [--local] [filename]`	Restores the cluster configuration files on all nodes from the backup. If filename is not specified the standard input will be used. If –local is specified only the files on the current node will be restored.

pcs config show

Tasks

[TOTEM ] Token has not been received in xxx ms

Increase token time in /etc/corosync/corosync.conf

Here token is set to 10000 ms.

totem {
    version: 2
    cluster_name: sancluster
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
    token: 10000
}

Then:

pcs cluster sync
pcs cluster reload corosync

Check:

$ corosync-cmapctl | grep totem.token
runtime.config.totem.token (u32) = 10000
runtime.config.totem.token_retransmit (u32) = 2380
runtime.config.totem.token_retransmits_before_loss_const (u32) = 4
runtime.config.totem.token_warning (u32) = 75
totem.token (u32) = 10000

Doc: https://www.thegeekdiary.com/how-to-change-pacemaker-cluster-heartbeat-timeout-in-centos-rhel-7/

Unsorted

Upgrade from Ubuntu server 20.04 LTS to 22.04 LTS – pcs cannot start

apt install ruby-webrick

Random

pcs host auth node01 addr=IP01 node02 addr=IP02
pcs cluster setup mycluster node01 addr=IP01 node02 addr=IP02

If it’s not working:

pcs cluster destroy

when pcs won’t delete a resource, but crm will

:~$ sudo pcs resource group list
drives: san200 backup
samba: smbd winbind
sanoid: sanoid-timer syncoid-timer
ip: SanIP adminSanIP
:~$ sudo pcs resource delete san300
Error: Resource 'san300' does not exist.
:~$ sudo pcs resource clear san300
Error: 'san300' is not a bundle/clone/group/resource
:~$ sudo crm resource cleanup san300
Cleaned up san300 on ares
Cleaned up san300 on zeus
Waiting for 2 replies from the controller.. OK

https://clusterlabs.org/pacemaker/doc/

ZFS jokes, resource not installed

sudo apt install resource-agents-extra

And if it still does not work, restart and make sure the drive are reachable (multipath etc…).

Also make sure the timeout is enough: to move a lot of ZFS datasets, then have to be unmounted first, the default setting is 30 s and will be way too small. This results in a timeout and all stops so that you cannot migrate your datasets to another node.