Red Hat Enterprise Linux 7

High Availability Add-On Reference

Reference Document for the High Availability Add-On for Red Hat Enterprise Linux 7

Legal Notice

Copyright © 2015 Red Hat, Inc. and others.

This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

Java® is a registered trademark of Oracle and/or its affiliates.

XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.

MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.

Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack® Word Mark and OpenStack Logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Abstract

Red Hat High Availability Add-On Reference provides reference information about installing, configuring, and managing the Red Hat High Availability Add-On for Red Hat Enterprise Linux 7.

⁠Chapter 1. Red Hat High Availability Add-On Configuration and Management Reference Overview

This document provides descriptions of the options and features that the Red Hat High Availability Add-On using Pacemaker supports. For a step-by-step basic configuration example, refer to Red Hat High Availability Add-On Administration.

You can configure a Red Hat High Availability Add-On cluster with the pcs configuration interface or with the pcsd GUI interface.

⁠1.1. New and Changed Features

This section lists features of the Red Hat High Availability Add-On that are new since the initial release of Red Hat Enterprise Linux 7.

⁠1.1.1. New and Changed Features for Red Hat Enterprise Linux 7.1

Red Hat Enterprise Linux 7.1 includes the following documentation and feature updates and changes.

The pcs resource cleanup command can now reset the resource status and failcount for all resources, as documented in Section 5.11, “Cluster Resources Cleanup”.

You can specify a lifetime parameter for the pcs resource move command, as documented in Section 7.1, “Manually Moving Resources Around the Cluster”.

As of Red Hat Enteprise Linux 7.1, you can use the pcs acl command to set permissions for local users to allow read-only or read-write access to the cluster configuration by using access control lists (ACLs). For information on ACLs, see Section 3.3, “Setting User Permissions”.

Section 6.2.3, “Ordered Resource Sets” and Section 6.3, “Colocation of Resources” have been extensively updated and clarified.

Section 5.1, “Resource Creation” documents the disabled parameter of the pcs resource create command, to indicate that the resource being created is not started automatically.

Section 3.1.6, “Configuring Quorum Options” documents the new cluster quorum unblock feature, which prevents the cluster from waiting for all nodes when establishing quorum.

Section 5.1, “Resource Creation” documents the before and after parameters of the pcs resource create command, which can be used to configure resource group ordering.

As of the Red Hat Enterprise Linux 7.1 release, you can backup the cluster configuration in a tarball and restore the cluster configuration files on all nodes from backup with the backup and restore options of the pcs config command. For information on this feature, see Section 2.8, “Backing Up and Restoring a Cluster Configuration”.

Small clarifications have been made throughout this document.

⁠1.1.2. New and Changed Features for Red Hat Enterprise Linux 7.2

Red Hat Enterprise Linux 7.2 includes the following documentation and feature updates and changes.

You can now use the pcs resource relocate run command to move a resource to its preferred node, as determined by current cluster status, constraints, location of resources and other settings. For information on this command, see Section 7.1.2, “Moving a Resource to its Preferred Node”.

Section 8.3, “Event Notification with Monitoring Resources” has been modified and expanded to better document how to configure the ClusterMon resource to execute an external program to determine what to do with cluster notifications.

When configuring fencing for redundant power supplies, you now are only required to define each device once and to specify that both devices are required to fence the node. For information on configuring fencing for redundant power supplies, see Section 4.11, “Configuring Fencing for Redundant Power Supplies”.

This document now provides a procedure for adding a node to an existing cluster in Section 3.2.3, “Adding Cluster Nodes”.

The new resource-discovery location constraint option allows you to indicate whether Pacemaker should perform resource discovery on a node for a specified resource, as documented in Table 6.1, “Location Constraint Options”.

Small clarifications and corrections have been made throughout this document.

⁠1.2. Installing Pacemaker configuration tools

You can use the following yum install command to install the Red Hat High Availability Add-On software packages along with all available: fence agents from the High Availability channel.

# yum install pcs fence-agents-all

Alternately, you can install the Red Hat High Availability Add-On software packages along with only the fence agent that you require with the following command.

# yum install pcs fence-agents-model

The following command displays a listing of the available fence agents.

# rpm -q -a | grep fence
fence-agents-rhevm-4.0.2-3.el7.x86_64
fence-agents-ilo-mp-4.0.2-3.el7.x86_64
fence-agents-ipmilan-4.0.2-3.el7.x86_64
...

The lvm2-cluster and gfs2-utils packages are part of ResilientStorage channel. You can install them, as needed, with the following command.

# yum install lvm2-cluster gfs2-utils

Warning

After you install the Red Hat High Availability Add-On packages, you should ensure that your software update preferences are set so that nothing is installed automatically. Installation on a running cluster can cause unexpected behaviors.

⁠1.3. Configuring the iptables Firewall to Allow Cluster Components

The Red Hat High Availability Add-On requires that the following ports be enabled for incoming traffic:

For TCP: Ports 2224, 3121, 21064

For UDP: Ports 5405

For DLM (if using the DLM lock manager with clvm/GFS2): Port 21064

You can enable these ports by means of the firewalld daemon by executing the following commands.

# firewall-cmd --permanent --add-service=high-availability
# firewall-cmd --add-service=high-availability

⁠1.4. The Cluster and Pacemaker Configuration Files

The configuration files for the Red Hat High Availability add-on are corosync.conf and cib.xml. Do not edit these files directly; use the pcs or pcsd interface instead.

The corosync.conf file provides the cluster parameters used by corosync, the cluster manager that Pacemaker is built on.

The cib.xml file is an XML file that represents both the cluster’s configuration and current state of all resources in the cluster. This file is used by Pacemaker's Cluster Information Base (CIB). The contents of the the CIB are automatically kept in sync across the entire cluster

⁠Chapter 2. The pcs Command Line Interface

The pcs command line interface controls and configures corosync and Pacemaker by providing an interface to the corosync.conf and cib.xml files.

The general format of the pcs command is as follows.

pcs [-f file] [-h] [commands]...

⁠2.1. The pcs Commands

The pcs commands are as follows.

cluster

Configure cluster options and nodes. For information on the pcs cluster command, see Chapter 3, Cluster Creation and Administration.

resource

Create and manage cluster resources. For information on the pcs cluster command, see Chapter 5, Configuring Cluster Resources, Chapter 7, Managing Cluster Resources, and Chapter 8, Advanced Resource types.

stonith

Configure fence devices for use with Pacemaker. For information on the pcs stonith command, see Chapter 4, Fencing: Configuring STONITH.

constraint

Manage resource constraints. For information on the pcs constraint command, see Chapter 6, Resource Constraints.

property

Set Pacemaker properties. For information on seeting properties with the pcs property command, see Chapter 10, Pacemaker Cluster Properties.

status

View current cluster and resource status. For information on the pcs status command, see Section 2.5, “Displaying Status”.

config

Display complete cluster configuration in user-readable form. For information on the pcs config command, see Section 2.6, “Displaying the Full Cluster Configuration”.

⁠2.2. pcs Usage Help Display

You can use the -h option of pcs to display the parameters of a pcs command and a description of those parameters. For example, the following command displays the parameters of the pcs resource command. Only a portion of the output is shown.

# pcs resource -h
Usage: pcs resource [commands]...
Manage pacemaker resources
Commands:
    show [resource id] [--all]
        Show all currently configured resources or if a resource is specified
        show the options for the configured resource.  If --all is specified
        resource options will be displayed

    start <resource id>
        Start resource specified by resource_id
...

⁠2.3. Viewing the Raw Cluster Configuration

Although you should not edit the cluster configuration file directily, you can view the raw cluster configuration with the pcs cluster cib command.

You can save the raw cluster configuration to a specified file with the pcs cluster cib filename as described in Section 2.4, “Saving a Configuration Change to a File”.

⁠2.4. Saving a Configuration Change to a File

When using the pcs command, you can use the -f option to save a configuration change to a file without affecting the active CIB.

If you have previously configured a cluster and there is already an active CIB, you use the following command to save the raw xml a file.

pcs cluster cib filename

For example, the following command saves the raw xml from the CIB into a file name testfile.

pcs cluster cib testfile

The following command creates a resource in the file testfile1 but does not add that resource to the currently running cluster configuration.

# pcs -f testfile1 resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.120 cidr_netmask=24 op monitor interval=30s

You can push the current content of testfile to the CIB with the following command.

pcs cluster cib-push filename

⁠2.5. Displaying Status

You can display the status of the cluster and the cluster resources with the following command.

pcs status commands

If you do not specify a commands parameter, this command displays all information about the cluster and the resources. You display the status of only particular cluster components by specifying resources, groups, cluster, nodes, or pcsd.

⁠2.6. Displaying the Full Cluster Configuration

Use the following command to display the full current cluster configuration.

pcs config

⁠2.7. Displaying The Current pcs Version

The following command displays the current version of pcs that is running.

pcs --version

⁠2.8. Backing Up and Restoring a Cluster Configuration

As of the Red Hat Enterprise Linux 7.1 release, you can back up the cluster configuration in a tarball with the following command. If you do not specify a file name, the standard ouptut will be used.

pcs config backup filename

Use the following command to restore the cluster configuration files on all nodes from the backup. If you do not specify a file name, the standard input will be used. Specifying the --local option restores only the files on the current node.

pcs config restore [--local] [filename]

⁠Chapter 3. Cluster Creation and Administration

This chapter describes how to perform basic cluster administration with Pacemaker, including creating the cluster, managing the cluster components, and displaying cluster status.

⁠3.1. Cluster Creation

To create a running cluster, perform the following steps:

Start the pcsd on each node in the cluster.

Authenticate the nodes that will constitute the cluster.

Configure and sync the cluster nodes.

Start cluster services on the cluster nodes.

The following sections described the commands that you use to perform these steps.

⁠3.1.1. Starting the pcsd daemon

The following commands start the pcsd service and enable pcsd at system start. These commands should be run on each node in the cluster.

# systemctl start pcsd.service
# systemctl enable pcsd.service

⁠3.1.2. Authenticating the Cluster Nodes

The following command authenticates pcs to the pcs daemon on the nodes in the cluster.

The username for the pcs administrator must be hacluster on every node. It is recommended that the password for user hacluster be the same on each node.

If you do not specify username or password, the system will prompt you for those parameters for each node when you execute the command.

If you do not specify any nodes, this command will authenticate pcs on the nodes that are specified with a pcs cluster setup command, if you have previously executed that command.

pcs cluster auth [node] [...] [-u username] [-p password]

For example, the following command authenticates user hacluster on z1.example.com for both of the nodes in the cluster that consist of z1.example.com and z2.example.com. This command prompts for the password for user hacluster on the cluster nodes.

root@z1 ~]# pcs cluster auth z1.example.com z2.example.com
Username: hacluster
Password:
z1.example.com: Authorized
z2.example.com: Authorized

Authorization tokens are stored in the file ~/.pcs/tokens (or /var/lib/pcsd/tokens).

⁠3.1.3. Configuring and Starting the Cluster Nodes

The following command configures the cluster configuration file and syncs the configuration to the specified nodes.

If you specify the --start option, the command will also start the cluster services on the specified nodes. If necessary, you can also start the cluster services with a separate pcs cluster start command.

When you create a cluster with the pcs cluster setup --start command or when you start cluster services with the pcs cluster start command, there may be a slight delay before the cluster is up and running. Before performing any subsequent actions on the cluster and its configuration, it is recommended that you use the pcs cluster status command to be sure that the cluster is up and running.

If you specify the --local option, the command will perform changes on the local node only.

pcs cluster setup [--start] [--local] --name cluster_ name node1 [node2] [...]

The following command starts cluster services on the specified node or nodes.

If you specify the --all option, the command starts cluster services on all nodes.

If you do not specify any nodes, cluster services are started on the local node only.

pcs cluster start [--all] [node] [...]

⁠3.1.4. Configuring Timeout Values for a Cluster

When you create a cluster with the pcs cluster setup command, timeout values for the cluster are set to default values that should be suitable for most cluster configurations. If you system requires different timeout values, however, you can modify these values with the pcs cluster setup options summarized in Table 3.1, “Timeout Options”

⁠

Table 3.1. Timeout Options

Option | Description
--token timeout | Sets time in milliseconds until a token loss is declared after not receiving a token (default 1000 ms)
--join timeout | sets time in milliseconds to wait for join mesages (default 50 ms)
--consensus timeout | sets time in milliseconds to wait for consensus to be achieved before starting a new round of member- ship configuration (default 1200 ms)
--miss_count_const count | sets the maximum number of times on receipt of a token a message is checked for retransmission before a retransmission occurs (default 5 messages)
--fail_recv_const failures | specifies how many rotations of the token without receiving any messages when messages should be received may occur before a new configuration is formed (default 2500 failures)

For example, the following command creates the cluster new_cluster and sets the token timeout value to 10000ms (10 seconds) and the join timeout value to 100ms.

# pcs cluster setup --name new_cluster nodeA nodeB --token 10000 --join 100

⁠3.1.5. Configuring Redundant Ring Protocol (RRP)

When you create a cluster with the pcs cluster setup command, you can configure a cluster with Redundant Ring Protocol by specifying both interfaces for each node. When using the default udpu transport, when you specify the cluster nodes you specify the ring 0 address followed by a ',', then the ring 1 address.

For example, the following command configures a cluster named my_rrp_clusterM with two nodes, node A and node B. Node A has two interfaces, nodeA-0 and nodeA-1. Node B has two interfaces, nodeB-0 and nodeB-1. To configure these nodes as a cluster using RRP, execute the following command.

# pcs cluster setup --name my_rrp_cluster nodeA-0,nodeA-1 nodeB-0,nodeB-1

For information on configuring RRP in a cluster that uses udp transport, see the help screen for the pcs cluster setup command.

⁠3.1.6. Configuring Quorum Options

A Red Hat Enterprise Linux High Availability Add-On cluster uses the votequorum service to avoid split-brain situations. A number of votes is assigned to each system in the cluster, and cluster operations are allowed to proceed only when a majority of votes is present. The service must be loaded into all nodes or none; if it is loaded into a subset of cluster nodes, the results will be unpredictable. For information on the configuration and operation of the votequorum service, see the votequorum(5) man page.

In a situation in which you know that the cluster in inquorate but you want the cluster to proceed with resource management, you can use the following command to prevent the cluster from waiting for all nodes when establishing quorum.

Note

This command should be used with extreme caution. Before issuing this command, it is imperative that you ensure that nodes that are not currently in the cluster are switched off.

# pcs cluster quorum unblock

There are some special features of quorum configuration that you can set when you create a cluster with the pcs cluster setup command. Table 3.2, “Quorum Options” summarizes these options.

⁠

Table 3.2. Quorum Options

Option
Description

--wait_for_all
When enabled, the cluster will be quorate for the first time only after all nodes have been visible at least once at the same time.

--auto_tie_breaker
When enabled, the cluster can suffer up to 50% of the nodes failing at the same time, in a deterministic fashion. The cluster partition, or the set of nodes that are still in contact with the nodeid configured in auto_tie_breaker_node (or lowest nodeid if not set), will remain quorate. The other nodes will be inquorate.

--last_man_standing
When enabled, the cluster can dynamically recalculate expected_votes and quorum under specific circumstances. You must enable wait_for_all and you must specify last_man_standing_window when you enable this option.

--last_man_standing_window
The time, in milliseconds, to wait before recalculating expected_votes and quorum after a cluster loses nodes.

For further information about configuring and using these options, see the votequorum(5) man page.

⁠3.2. Managing Cluster Nodes

The following sections describe the commands you use to manage cluster nodes, including commands to start and stop cluster services and to add and remove cluster nodes.

⁠3.2.1. Stopping Cluster Services

The following command stops cluster services on the specified node or nodes. As with the pcs cluster start, the --all option stops cluster services on all nodes and if you do not specify any nodes, cluster services are stopped on the local node only.

pcs cluster stop [--all] [node] [...]

You can force a stop of cluster services on the local node with the following command, which performs a kill -9 command.

pcs cluster kill

⁠3.2.2. Enabling and Disabling Cluster Services

Use the following command to configure the cluster services to run on startup on the specified node or nodes.

If you specify the --all option, the command enables cluster services on all nodes.

If you do not specify any nodes, cluster services are enabled on the local node only.

pcs cluster enable [--all] [node] [...]

Use the following command to configure the cluster services not to run on startup on the specified node or nodes.

If you specify the --all option, the command disables cluster services on all nodes.

If you do not specify any nodes, cluster services are disables on the local node only.

pcs cluster disable [--all] [node] [...]

⁠3.2.3. Adding Cluster Nodes

Use the following procedure to add a new node to an existing cluster. In this example, the existing cluster nodes are clusternode-01.example.com, clusternode-02.example.com, and clusternode-03.example.com. The new node is newnode.example.com.

On the new node to add to the cluster, perform the following tasks.

Install the cluster packages;

[root@newnode ~]# yum install -y pcs fence-agents-all

If you are running the firewalld daemon, execute the following commands to enable the ports that are required by the Red Hat High Availability Add-On.

# firewall-cmd --permanent --add-service=high-availability
# firewall-cmd --add-service=high-availability

Set a password for the user ID hacluster. It is recommended that you use the same password for each node in the cluster.

[root@newnode ~]# passwd hacluster
Changing password for user hacluster.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Execute the following commands to start the pcsd service and to enable pcsd at system start.

# systemctl start pcsd.service
# systemctl enable pcsd.service

On a node in the existing cluster, perform the following tasks.

Authenticate user hacluster on the new cluster node.

[root@clusternode-01 ~]# pcs cluster auth newnode.example.com
Username: hacluster
Password:
newnode.example.com: Authorized

Add the new node to the existing cluster. This command also syncs the cluster configuration file corosync.conf to all nodes in the cluster, including the new node you are adding.

[root@clusternode-01 ~]# pcs cluster node add newnode.example.com

On the new node to add to the cluster, perform the following tasks.

Authenticate user hacluster on the new node for all nodes in the cluster.

[root@newnode ~]# pcs cluster auth
Username: hacluster
Password:
clusternode-01.example.com: Authorized
clusternode-02.example.com: Authorized
clusternode-03.example.com: Authorized
newnode.example.com: Already authorized

Start and enable cluster services on the new node.

[root@newnode ~]# pcs cluster start
Starting Cluster...
[root@newnode ~]# pcs cluster enable

Ensure that you configure a fencing device for the new cluster node. For information on configuring fencing devices, see Chapter 4, Fencing: Configuring STONITH.

⁠3.2.4. Removing Cluster Nodes

The following command shuts down the specified node and removes it from the cluster configuration file, corosync.conf, on all of the other nodes in the cluster. For information on removing all information about the cluster from the cluster nodes entirely, thereby destroying the cluster permanently, refer to Section 3.4, “Removing the Cluster Configuration”.

pcs cluster node remove node

⁠3.2.5. Standby Mode

The following command puts the specified node into standby mode. The specified node is no longer able to host resources. Any resources currently active on the node will be moved to another node. If you specify the --all, this command puts all nodes into standby mode.

You can use this command when updating a resource's packages. You can also use this command when testing a configuration, to simulate recovery without actually shutting down a node.

pcs cluster standby node | --all

The following command removes the specified node from standby mode. After running this command, the specified node is then able to host resources. If you specify the --all, this command removes all nodes from standby mode.

pcs cluster unstandby node | --all

Note that when you execute the pcs cluster standby command, this adds constraints to the resources to prevent them from running on the indicated node. When you execute the pcs cluster unstandby command, this removes the constraints. This does not necessarily move the resources back to the indicated node; where the resources can run at that point depends on how you have configured your resources initially. For information on resource constraints, refer to Chapter 6, Resource Constraints.

⁠3.3. Setting User Permissions

By default, the root user and any user who is a member of the group haclient has full read/write access to the cluster configuration. As of Red Hat Enteprise Linux 7.1, you can use the pcs acl command to set permissions for local users to allow read-only or read-write access to the cluster configuration by using access control lists (ACLs).

Setting permissions for local users is a two-step process:

Execute the pcs acl role create... command to create a role which defines the permissions for that role.

Assign the role you created to a user with the pcs acl user create command.

The following example procedure provides read-only access for a cluster configuration to a local user named rouser.

This procedure requires that the user rouser exists on the local system and that the user rouser is a member of the group haclient.

# adduser rouser
# usermod -a -G haclient rouser

Enable Pacemaker ACLs with the enable-acl cluster property.

# pcs property set enable-acl=true --force

Create a role named read-only with read-only permissions for the cib.

# pcs acl role create read-only description="Read access to cluster" read xpath /cib

Create the user rouser in the pcs ACL system and assign that user the read-only role.

# pcs acl user create rouser read-only

View the current ACLs.

# pcs acl
User: rouser
  Roles: read-only
Role: read-only
  Description: Read access to cluster
  Permission: read xpath /cib (read-only-read)

The following example procedure provides write access for a cluster configuration to a local user named wuser.

This procedure requires that the user wuser exists on the local system and that the user wuser is a member of the group haclient.

# adduser wuser
# usermod -a -G haclient wuser

Enable Pacemaker ACLs with the enable-acl cluster property.

# pcs property set enable-acl=true --force

Create a role named write-access with write permissions for the cib.

# pcs acl role create write-access description="Full access" write xpath /cib

Create the user wuser in the pcs ACL system and assign that user the write-access role.

# pcs acl user create wuser write-access

View the current ACLs.

# pcs acl
User: rouser
  Roles: read-only
User: wuser
  Roles: write-access
Role: read-only
  Description: Read access to cluster
  Permission: read xpath /cib (read-only-read)
Role: write-access
  Description: Full Access
  Permission: write xpath /cib (write-access-write)

For further information about cluster ACLs, see the help screen for the pcs acl command.

⁠3.4. Removing the Cluster Configuration

To remove all cluster configuration files and stop all cluster services, thus permanently destroying a cluster, use the following command.

Warning

This command permanently removes any cluster configuration that has been created. It is recommended that you run pcs cluster stop before destroying the cluster.

pcs cluster destroy

⁠3.5. Displaying Cluster Status

The following command displays the current status of the cluster and the cluster resources.

pcs status

You can display a subset of information about the current status of the cluster with the following commands.

The following command displays the status of the cluster, but not the cluster resources.

pcs cluster status

The following command displays the status of the cluster resources.

pcs status resources

⁠Chapter 4. Fencing: Configuring STONITH

STONITH is an acronym for Shoot-The-Other-Node-In-The-Head and it protects your data from being corrupted by rogue nodes or concurrent access.

Just because a node is unresponsive, this does not mean it is not accessing your data. The only way to be 100% sure that your data is safe, is to fence the node using STONITH so we can be certain that the node is truly offline, before allowing the data to be accessed from another node.

STONITH also has a role to play in the event that a clustered service cannot be stopped. In this case, the cluster uses STONITH to force the whole node offline, thereby making it safe to start the service elsewhere.

⁠4.1. Available STONITH (Fencing) Agents

Use the following command to view of list of all available STONITH agents. You specify a filter, then this command displays only the STONITH agents that match the filter.

pcs stonith list [filter]

⁠4.2. General Properties of Fencing Devices

Note

To disable a fencing device/resource, you can set the target-role as you would for a normal resource.

Note

To prevent a specific node from using a fencing device, you can configure location constraints for the fencing resource.

Table 4.1, “General Properties of Fencing Devices” describes the general properties you can set for fencing devices. Refer to Section 4.3, “Displaying Device-Specific Fencing Options” for information on fencing properties you can set for specific fencing devices.

Note

For information on more advanced fencing configuration properties, refer to Section 4.9, “Additional Fencing Configuration Options”

⁠

Table 4.1. General Properties of Fencing Devices

Field | Type | Default | Description
priority | integer | 0 | The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
pcmk_host_map | string |  | A mapping of host names to ports numbers for devices that do not support host names. For example: node1:1;node2:2,3 tells the cluster to use port 1 for node1 and ports 2 and 3 for node2
pcmk_host_list | string |  | A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
pcmk_host_check | string | dynamic-list | How to determine which machines are controlled by the device. Allowed values: dynamic-list (query the device), static-list (check the pcmk_host_list attribute), none (assume every device can fence every machine)

⁠4.3. Displaying Device-Specific Fencing Options

Use the following command to view the options for the specified STONITH agent.

pcs stonith describe stonith_agent

For example, the following command displays the options for the fence agent for APC over telnet/SSH.

# pcs stonith describe fence_apc
Stonith options for: fence_apc
  ipaddr (required): IP Address or Hostname
  login (required): Login Name
  passwd: Login password or passphrase
  passwd_script: Script to retrieve password
  cmd_prompt: Force command prompt
  secure: SSH connection
  port (required): Physical plug number or name of virtual machine
  identity_file: Identity file for ssh
  switch: Physical switch number on device
  inet4_only: Forces agent to use IPv4 addresses only
  inet6_only: Forces agent to use IPv6 addresses only
  ipport: TCP port to use for connection with device
  action (required): Fencing Action
  verbose: Verbose mode
  debug: Write debug information to given file
  version: Display version information and exit
  help: Display help and exit
  separator: Separator for CSV created by operation list
  power_timeout: Test X seconds for status change after ON/OFF
  shell_timeout: Wait X seconds for cmd prompt after issuing command
  login_timeout: Wait X seconds for cmd prompt after login
  power_wait: Wait X seconds after issuing ON/OFF
  delay: Wait X seconds before fencing is started
  retry_on: Count of attempts to retry power on

⁠4.4. Creating a Fencing Device

The following command creates a stonith device.

pcs stonith create stonith_id stonith_device_type [stonith_device_options]

# pcs stonith create MyStonith fence_virt pcmk_host_list=f1 op monitor interval=30s

If you use a single fence device for several nodes, using a different port of each node, you do not need to create a device separately for each node. Instead you can use the pcmk_host_map option to define which port goes to which node. For example, the following command creates a single fencing device called myapc-west-13 that uses an APC powerswitch called west-apc and uses port 15 for node west-13.

# pcs stonith create myapc-west-13 fence_apc pcmk_host_list="west-13" ipaddr="west-apc" login="apc" passwd="apc" port="15"

The following example, however, uses the APC powerswitch named west-apc to fence nodes west-13 using port 15, west-14 using port 17, west-15 using port 18, and west-16 using port 19.

# pcs stonith create myapc fence_apc pcmk_host_list="west-13,west-14,west-15,west-16" pcmk_host_map="west-13:15;west-14:17;west-15:18;west-16:19" ipaddr="west-apc" login="apc" passwd="apc"

⁠4.5. Configuring Storage-Based Fence Devices with unfencing

When creating a SAN/storage fence device (that is, one that uses a non-power based fencing agent), you must set the meta option provides=unfencing when creating the stonith device. This ensures that a fenced node is unfenced before the node is rebooted and the cluster services are started on the node.

Setting the provides=unfencing meta option is not necessary when configuring a power-based fence device, since the device itself is providing power to the node in order for it to boot (and attempt to rejoin the cluster). The act of booting in this case implies that unfencing occurred.

The following command configures a stonith device named my-scsi-shooter that uses the fence_scsi fence agent, enabling unfencing for the device.

pcs stonith create my-scsi-shooter fence_scsi devices=/dev/sda meta provides=unfencing

⁠4.6. Displaying Fencing Devices

The following command shows all currenly configured fencing devices. If a stonith_id is specified, the command shows the options for that configured stonith device only. If the --full option is specified, all configured stonith options are displayed.

pcs stonith show [stonith_id] [--full]

⁠4.7. Modifying and Deleting Fencing Devices

Use the following command to modify or add options to a currently configured fencing device.

pcs stonith update stonith_id [stonith_device_options]

Use the following command to remove a fencing device from the current configuration.

pcs stonith delete stonith_id

⁠4.8. Managing Nodes with Fence Devices

You can fence a node manually with the following command. If you specify --off this will use the off API call to stonith which will turn the node off instead of rebooting it.

pcs stonith fence node [--off]

You can confirm whether a specified node is currently powered off with the following command.

Warning

If the node you specify is still running the cluster software or services normally controlled by the cluster, data corruption/cluster failure will occur.

pcs stonith confirm node

⁠4.9. Additional Fencing Configuration Options

Table 4.2, “Advanced Properties of Fencing Devices”. summarizes additional properties you can set for fencing devices. Note that these properties are for advanced use only.

⁠

Table 4.2. Advanced Properties of Fencing Devices

Field
Type
Default
Description

pcmk_host_argument
string
port
An alternate parameter to supply instead of port. Some devices do not support the standard port parameter or may provide additional ones. Use this to specify an alternate, device-specific, parameter that should indicate the machine to be fenced. A value of none can be used to tell the cluster not to supply any additional parameters.

pcmk_reboot_action
string
reboot
An alternate command to run instead of reboot. Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the reboot action.

pcmk_reboot_timeout
time
60s
Specify an alternate timeout to use for reboot actions instead of stonith-timeout. Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for reboot actions.

pcmk_reboot_retries
integer
2
The maximum number of times to retry the reboot command within the timeout period. Some devices do not support multiple connections. Operations may fail if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries reboot actions before giving up.

pcmk_off_action
string
off
An alternate command to run instead of off. Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the off action.

pcmk_off_timeout
time
60s
Specify an alternate timeout to use for off actions instead of stonith-timeout. Some devices need much more or much less time to complete than normal. Use this to specify an alternate, device-specific, timeout for off actions.

pcmk_off_retries
integer
2
The maximum number of times to retry the off command within the timeout period. Some devices do not support multiple connections. Operations may fail if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries off actions before giving up.

pcmk_list_action
string
list
An alternate command to run instead of list. Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the list action.

pcmk_list_timeout
time
60s
Specify an alternate timeout to use for list actions instead of stonith-timeout. Some devices need much more or much less time to complete than normal. Use this to specify an alternate, device-specific, timeout for list actions.

pcmk_list_retries
integer
2
The maximum number of times to retry the list command within the timeout period. Some devices do not support multiple connections. Operations may fail if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries list actions before giving up.

pcmk_monitor_action
string
monitor
An alternate command to run instead of monitor. Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the monitor action.

pcmk_monitor_timeout
time
60s
Specify an alternate timeout to use for monitor actions instead of stonith-timeout. Some devices need much more or much less time to complete than normal. Use this to specify an alternate, device-specific, timeout for monitor actions.

pcmk_monitor_retries
integer
2
The maximum number of times to retry the monitor command within the timeout period. Some devices do not support multiple connections. Operations may fail if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries monitor actions before giving up.

pcmk_status_action
string
status
An alternate command to run instead of status. Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the status action.

pcmk_status_timeout
time
60s
Specify an alternate timeout to use for status actions instead of stonith-timeout. Some devices need much more or much less time to complete than normal. Use this to specify an alternate, device-specific, timeout for status actions.

pcmk_status_retries
integer
2
The maximum number of times to retry the status command within the timeout period. Some devices do not support multiple connections. Operations may fail if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries status actions before giving up.

⁠4.10. Configuring Fencing Levels

Pacemaker supports fencing nodes with multiple devices through a feature called fencing topologies. To implement topologies, create the individual devices as you normally would and then define one or more fencing levels in the fencing-topology section in the configuration.

Each level is attempted in ascending numeric order, starting at 1.

If a device fails, processing terminates for the current level. No further devices in that level are exercised and the next level is attempted instead.

If all devices are successfully fenced, then that level has succeeded and no other levels are tried.

The operation is finished when a level has passed (success), or all levels have been attempted (failed).

Use the following command to add a fencing level to a node. The devices are given as a comma-separated list of stonith ids, which are attempted for the node at that level.

pcs stonith level add level node devices

The following command lists all of the fencing levels that are currently configured.

pcs stonith level

In the following example, there are two fence devices configured for node rh7-2: an ilo fence device called my_ilo and an apc fence device called my_apc. These commands sets up fence levels so that if the device my_ilo fails and is unable to fence the node, then Pacemaker will attempt to use the device my_apc. This example also shows the output of the pcs stonith level command after the levels are configured.

# pcs stonith level add 1 rh7-2 my_ilo
# pcs stonith level add 2 rh7-2 my_apc
# pcs stonith level
 Node: rh7-2
  Level 1 - my_ilo
  Level 2 - my_apc

The following command removes the fence level for the specified node and devices. If no nodes or devices are specified then the fence level you specify is removed from all nodes.

pcs stonith level remove level [node_id] [stonith_id] ... [stonith_id]

The following command clears the fence levels on the specified node or stonith id. If you do not specify a node or stonith id, all fence levels are cleared.

pcs stonith level clear [node|stonith_id(s)]

If you specify more than one stonith id, they must be separated by a comma and no spaces, as in the following example.

# pcs stonith level clear dev_a,dev_b

The following command verifies that all fence devices and nodes specified in fence levels exist.

pcs stonith level verify

⁠4.11. Configuring Fencing for Redundant Power Supplies

When configuring fencing for redundant power supplies, the cluster must ensure that when attempting to reboot a host, both power supplies are turned off before either power supply is turned back on.

If the node never completely loses power, the node may not release its resources. This opens up the possibility of nodes accessing these resources simultaneously and corrupting them.

Prior to Red Hat Enterprise Linux 7.2, you needed to explicitly configure different versions of the devices which used either the 'on' or 'off' actions. Since Red Hat Enterprise Linux 7.2, it is now only required to define each device once and to specify that both are required to fence the node, as in the following example.

# pcs stonith create apc1 fence_apc_snmp ipaddr=apc1.example.com login=user passwd='7a4D#1j!pz864' pcmk_host_map="node1.example.com:1,node2.example.com:2"

# pcs stonith create apc2 fence_apc_snmp ipaddr=apc2.example.com login=user passwd='7a4D#1j!pz864' pcmk_host_map="node1.example.com:1,node2.example.com:2"

# pcs stonith level add 1 node1.example.com apc1,apc2
# pcs stonith level add 1 node2.example.com apc1,apc2

⁠Chapter 5. Configuring Cluster Resources

This chapter provides information on configuring resources in a cluster.

⁠5.1. Resource Creation

Use the following command to create a cluster resource.

pcs resource create resource_id standard:provider:type|type [resource options]
[op operation_action operation_options [operation_action  operation_options]...]
[meta meta_options...] [--clone clone_options |
--master master_options | --group group_name
[--before  resource_id | --after resource_id]] [--disabled]

When you specify the --group option, the resource is added to the resource group named. If the group does not exist, this creates the group and adds this resource to the group. For information on resource groups, refer to Section 5.5, “Resource Groups”.

The --before and --after options specify the position of the added resource relative to a resource that already exists in a resource group.

Specifying the --disabled option indicates that the resource is not started automatically.

The following command creates a resource with the name VirtualIP of standard ocf, provider heartbeat, and type IPaddr2. The floating address of this resource is 192.168.0.120, the system will check whether the resource is running every 30 seconds.

# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.120 cidr_netmask=24 op monitor interval=30s

Alternately, you can omit the standard and provider fields and use the following command. This will default to a standard of ocf and a provider of heartbeat.

# pcs resource create VirtualIP IPaddr2 ip=192.168.0.120 cidr_netmask=24 op monitor interval=30s

Use the following command to delete a configured resource.

pcs resource delete resource_id

For example, the following command deletes an existing resource with a resource ID of VirtualIP

# pcs resource delete VirtualIP

For information on the resource_id, standard, provider, and type fields of the pcs resource create command, refer to Section 5.2, “Resource Properties”.

For information on defining resource parameters for individual resources, refer to Section 5.3, “Resource-Specific Parameters”.

For information on defining resource meta options, which are used by the cluster to decide how a resource should behave, refer to Section 5.4, “Resource Meta Options”.

For information on defining the operations to perform on a resource, refer to Section 5.6, “Resource Operations”.

Specifying the --clone creates a clone resource. Specifying the --master creates a master/slave resource. For information on resource clones and resources with multiple modes, refer to Chapter 8, Advanced Resource types.

⁠5.2. Resource Properties

The properties that you define for a resource tell the cluster which script to use for the resource, where to find that script and what standards it conforms to. Table 5.1, “Resource Properties” describes these properties.

⁠

Table 5.1. Resource Properties

Field
Description

resource_id

Your name for the resource

standard

The standard the script conforms to. Allowed values: ocf, service, upstart, systemd, lsb, stonith

type

The name of the Resource Agent you wish to use, for example IPaddr or Filesystem

provider

The OCF spec allows multiple vendors to supply the same ResourceAgent. Most of the agents shipped by Red Hat use heartbeat as the provider.

Table 5.2, “Commands to Display Resource Properties”. summarizes the commands that display the available resource properties.

⁠

Table 5.2. Commands to Display Resource Properties

pcs Display Command | Output
pcs resource list | Displays a list of all available resources.
pcs resource standards | Displays a list of available resources agent standards.
pcs resource providers | Displays a list of available resources agent providers.
pcs resource list string | Displays a list of available resources filtered by the specified string. You can use this command to display resources filtered by the name of a standard, a provider, or a type.

⁠5.3. Resource-Specific Parameters

For any individual resource, you can use the following command to display the parameters you can set for that resource.

# pcs resource describe standard:provider:type|type

For example, the following command displays the parameters you can set for a resource of type LVM.

# pcs resource describe LVM
Resource options for: LVM
  volgrpname (required): The name of volume group.
  exclusive: If set, the volume group will be activated exclusively.
  partial_activation: If set, the volume group will be activated even
  only partial of the physical volumes available. It helps to set to
  true, when you are using mirroring logical volumes.

⁠5.4. Resource Meta Options

In addition to the resource-specific parameters, you can configure additional resource options for any resource. These options are used by the cluster to decide how your resource should behave. Table 5.3, “Resource Meta Options” describes this options.

⁠

Table 5.3. Resource Meta Options

Field
Default
Description

priority

0

If not all resources can be active, the cluster will stop lower priority resources in order to keep higher priority ones active.

target-role

Started

What state should the cluster attempt to keep this resource in? Allowed values:

* Stopped - Force the resource to be stopped

* Started - Allow the resource to be started (In the case of multistate resources, they will not promoted to master)

* Master - Allow the resource to be started and, if appropriate, promoted

is-managed

true

Is the cluster allowed to start and stop the resource? Allowed values: true, false

resource-stickiness

0

Value to indicate how much the resource prefers to stay where it is.

requires

Calculated

Indicates under what conditions the resource can be started.

Defaults to fencing except under the conditions noted below. Possible values:

* nothing - The cluster can always start the resource.

* quorum - The cluster can only start this resource if a majority of the configured nodes are active. This is the default value if stonith-enabled is false or the the resource's standard is stonith.

* fencing - The cluster can only start this resource if a majority of the configured nodes are active and any failed or unknown nodes have been powered off.

* unfencing - The cluster can only start this resource if a majority of the configured nodes are active and any failed or unknown nodes have been powered off and only on nodes that have been unfenced. This is the default value if the provides=unfencing stonith meta option has been set for a fencing device. For information on the provides=unfencing stonith meta option, see Section 4.5, “Configuring Storage-Based Fence Devices with unfencing”.

migration-threshold

INFINITY (disabled)

How many failures may occur for this resource on a node, before this node is marked ineligible to host this resource. For information on configuring the migration-threshold option, refer to Section 7.2, “Moving Resources Due to Failure”.

failure-timeout

0 (disabled)

Used in conjunction with the migration-threshold option, indicates how many seconds to wait before acting as if the failure had not occurred, and potentially allowing the resource back to the node on which it failed. For information on configuring the failure-timeout option, refer to Section 7.2, “Moving Resources Due to Failure”.

multiple-active

stop_start

What should the cluster do if it ever finds the resource active on more than one node. Allowed values:

* block - mark the resource as unmanaged

* stop_only - stop all active instances and leave them that way

* stop_start - stop all active instances and start the resource in one location only

To change the default value of a resource option, use the following command.

pcs resource defaults options

For example, the following command resets the default value of resource-stickiness to 100.

# pcs resource defaults resource-stickiness=100

Omitting the options parameter from the pcs resource defaults displays a list of currently configured default values for resource options. The following example shows the output of this command after you have reset the default value of resource-stickiness to 100.

# pcs resource defaults
resource-stickiness:100

Whether you have reset the default value of a resource meta option or not, you can set a resource option for a particular resource to a value other than the default when you create the resource. The following shows the format of the pcs resource create command you use when specifying a value for a resource meta option.

pcs resource create resource_id standard:provider:type|type [resource options] [meta meta_options...]

For example, the following command creates a resource with a resource-stickiness value of 50.

# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.120 cidr_netmask=24 meta resource-stickiness=50

You can also set the value of a resource meta option for an existing resource, group, cloned resource, or master resource with the following command.

pcs resource meta resource_id | group_id | clone_id | master_id  meta_options

In the following example, there is an existing resource named dummy_resource. This command sets the failure-timeout meta option to 20 seconds, so that the resource can attempt to restart on the same node in 20 seconds.

# pcs resource meta dummy_resource failure-timeout=20s

After executing this command, you can display the values for the resource to verity that failure-timeout=20s is set.

# pcs resource show dummy_resource
 Resource: dummy_resource (class=ocf provider=heartbeat type=Dummy)
  Meta Attrs: failure-timeout=20s
  Operations: start interval=0s timeout=20 (dummy_resource-start-timeout-20)
              stop interval=0s timeout=20 (dummy_resource-stop-timeout-20)
              monitor interval=10 timeout=20 (dummy_resource-monitor-interval-10)

For information on resource clone meta options, see Section 8.1, “Resource Clones”. For information on resource master meta options, see Section 8.2, “Multi-State Resources: Resources That Have Multiple Modes”.

⁠5.5. Resource Groups

One of the most common elements of a cluster is a set of resources that need to be located together, start sequentially, and stop in the reverse order. To simplify this configuration, Pacemaker supports the concept of groups.

You create a resource group with the following command, specifying the resources to include in the group. If the group does not exist, this command creates the group. If the group exists, this command adds additional resources to the group. The resources will start in the order you specify them with this command, and will stop in the reverse order of their starting order.

pcs resource group add group_name resource_id [resource_id] ... [resource_id]
[--before resource_id | --after resource_id

You can use the --before and --after options of this command to specify the position of the added resources relative to a resource that already exists in the group.

You can also add a new resource to an existing group when you create the resource, using the following command. The resource you create is added to the group named group_name.

pcs resource create resource_id standard:provider:type|type [resource_options] [op operation_action operation_options] --group group_name

You remove a resource from a group with the following command. If there are no resources in the group, this command removes the group itself.

pcs resource group remove group_name resource_id...

The following command lists all currently configured resource groups.

pcs resource group list

The following example creates a resource group named shortcut that contains the existing resources IPaddr and Email.

# pcs resource group add shortcut IPaddr Email

There is no limit to the number of resources a group can contain. The fundamental properties of a group are as follows.

Resources are started in the order in which you specify them (in this example, IPaddr first, then Email).

Resources are stopped in the reverse order in which you specify them. (Email first, then IPaddr).

If a resource in the group cannot run anywhere, then no resource specified after that resource is allowed to run.

If IPaddr cannot run anywhere, neither can Email.

If Email cannot run anywhere, however, this does not affect IPaddr in any way.

Obviously as the group grows bigger, the reduced configuration effort of creating resource groups can become significant.

⁠5.5.1. Group Options

A resource group inherits the following options from the resources that it contains: priority, target-role, is-managed For information on resource options, refer to Table 5.3, “Resource Meta Options”.

⁠5.5.2. Group Stickiness

Stickiness, the measure of how much a resource wants to stay where it is, is additive in groups. Every active resource of the group will contribute its stickiness value to the group’s total. So if the default resource-stickiness is 100, and a group has seven members, five of which are active, then the group as a whole will prefer its current location with a score of 500.

⁠5.6. Resource Operations

To ensure that resources remain healthy, you can add a monitoring operation to a resource's definition. If you do not specify a monitoring operation for a resource, by default the pcs command will create a monitoring operation, with an interval that is determined by the resource agent. If the resource agent does not provide a default monitoring interval, the pcs command will create a monitoring operation with an interval of 60 seconds.

Table 5.4, “Properties of an Operation” summarizes the properties of a resource monitoring operation.

⁠

Table 5.4. Properties of an Operation

Field
Description

id

Unique name for the action. The system assigns this when you configure an operation.

name

The action to perform. Common values: monitor, start, stop

interval

How frequently (in seconds) to perform the operation. Default value: 0, meaning never.

timeout

How long to wait before declaring the action has failed. If you find that your system includes a resource that takes a long time to start or stop or perform a non-recurring monitor action at startup, and requires more time than the system allows before declaring that the start action has failed, you can increase this value from the default of 20 or the value of timeout in "op defaults".

on-fail

The action to take if this action ever fails. Allowed values:

* ignore - Pretend the resource did not fail

* block - Do not perform any further operations on the resource

* stop - Stop the resource and do not start it elsewhere

* restart - Stop the resource and start it again (possibly on a different node)

* fence - STONITH the node on which the resource failed

* standby - Move all resources away from the node on which the resource failed

The default for the stop operation is fence when STONITH is enabled and block otherwise. All other operations default to restart.

enabled

If false, the operation is treated as if it does not exist. Allowed values: true, false

You can configure monitoring operations when you create a resource, using the following command.

pcs resource create resource_id standard:provider:type|type [resource_options] [op operation_action operation_options [operation_type operation_options]...]

For example, the following command creates an IPaddr2 resource with a monitoring operation. The new resource is called VirtualIP with an IP address of 192.168.0.99 and a netmask of 24 on eth2. A monitoring operation will be performed every 30 seconds.

# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 cidr_netmask=24 nic=eth2 op monitor interval=30s

Alternately, you can add a monitoring operation to an existing resource with the following command.

pcs resource op add resource_id operation_action [operation_properties]

Use the following command to delete a configured resource operation.

pcs resource op remove resource_id operation_name operation_properties

Note

You must specify the exact operation properties to properly remove an existing operation.

To change the values of a monitoring option, you remove the existing operation, then add the new operation. For example, you can create a VirtualIP with the following command.

# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 cidr_netmask=24 nic=eth2

By default, this command creates these operations.

Operations: start interval=0s timeout=20s (VirtualIP-start-timeout-20s)
            stop interval=0s timeout=20s (VirtualIP-stop-timeout-20s)
            monitor interval=10s timeout=20s (VirtualIP-monitor-interval-10s)

The change the stop timeout operation, execute the following commands.

# pcs resource op remove VirtualIP stop interval=0s timeout=20s
# pcs resource op add VirtualIP stop interval=0s timeout=40s

# pcs resource show VirtualIP
 Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=192.168.0.99 cidr_netmask=24 nic=eth2
  Operations: start interval=0s timeout=20s (VirtualIP-start-timeout-20s)
              monitor interval=10s timeout=20s (VirtualIP-monitor-interval-10s)
              stop interval=0s timeout=40s (VirtualIP-name-stop-interval-0s-timeout-40s)

To set global default values for monitoring operations, use the following command.

pcs resource op defaults [options]

For example, the following command sets a global default of a timeout value of 240s for all monitoring operations.

# pcs resource op defaults timeout=240s

To display the currently configured default values for monitoring operations, do not specify any options when you execute the pcs resource op defaults commmand.

For example, following command displays the default monitoring operation values for a cluster which has been configured with a timeout value of 240s.

# pcs resource op defaults
timeout: 240s

⁠5.7. Displaying Configured Resources

To display a list of all configured resources, use the following command.

pcs resource show

For example, if your system is configured with a resource named VirtualIP and a resource named WebSite, the pcs resource show command yields the following output.

# pcs resource show
 VirtualIP	(ocf::heartbeat:IPaddr2):	Started
 WebSite	(ocf::heartbeat:apache):	Started

To display a list of all configured resources and the parameters configured for those resources, use the --full option of the the pcs resource show command, as in the following example.

# pcs resource show --full
 Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat)
  Attributes: ip=192.168.0.120 cidr_netmask=24
  Operations: monitor interval=30s
 Resource: WebSite (type=apache class=ocf provider=heartbeat)
  Attributes: statusurl=http://localhost/server-status configfile=/etc/httpd/conf/httpd.conf
  Operations: monitor interval=1min

To display the configured parameters for a resource, use the following command.

pcs resource show resource_id

For example, the following command displays the currently configured parameters for resource VirtualIP.

# pcs resource show VirtualIP
 Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat)
  Attributes: ip=192.168.0.120 cidr_netmask=24
  Operations: monitor interval=30s

⁠5.8. Modifying Resource Parameters

To modify the parameters of a configured resource, use the following command.

pcs resource update resource_id [resource_options]

The following sequence of commands show the initial values of the configured parameters for resource VirtualIP, the command to change the value of the ip parameter, and the values following the update command.

# pcs resource show VirtualIP
 Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat)
  Attributes: ip=192.168.0.120 cidr_netmask=24
  Operations: monitor interval=30s
# pcs resource update VirtualIP ip=192.169.0.120
# pcs resource show VirtualIP
 Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat)
  Attributes: ip=192.169.0.120 cidr_netmask=24
  Operations: monitor interval=30s

⁠5.9. Multiple Monitoring Operations

You can configure a single resource with as many monitor operations as a resource agent supports. In this way you can do a superficial health check every minute and progressively more intense ones at higher intervals.

Note

When configuring multiple monitor operations, you must ensure that no two operations are performed at the same interval.

To configure additional monitoring operations for a resource that supports more in-depth checks at different levels, you add an OCF_CHECK_LEVEL=n option.

For example, if you configure the following IPaddr2 resource, by default this creates a monitoring operation with an interval of 10 seconds and a timeout value of 20 seconds.

# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 cidr_netmask=24 nic=eth2

If the Virtual IP supports a different check with a depth of 10, the following command causes Packemaker to perform the more advanced monitoring check every 60 seconds in addition to the normal Virtual IP check every 10 seconds. (As noted, you should not configure the additional monitoring operation with a 10-second interval as well.)

# pcs resource op add VirtualIP monitor interval=60s OCF_CHECK_LEVEL=10

⁠5.10. Enabling and Disabling Cluster Resources

The following command enables the resource specified by resource_id.

pcs resource enable resource_id

The following command disables the resource specified by resource_id.

pcs resource disable resource_id

⁠5.11. Cluster Resources Cleanup

If a resource has failed, a failure message appears when you display the cluster status. If you resolve that resource, you can clear that failure status with the pcs resource cleanup command. This command resets the resource status and failcount, telling the cluster to forget the operation history of a resource and re-detect its current state.

The following command cleans up the resource specified by resource_id.

pcs resource cleanup resource_id

If you do not specify a resource_id, this command resets the resource status and failcount for all resources.

⁠Chapter 6. Resource Constraints

You can determine the behavior of a resource in a cluster by configuring constraints for that resource. You can configure the following categories of constraints:

location constraints — A location constraint determines which nodes a resource can run on. Location constraints are described in Section 6.1, “Location Constraints”.

order constraints — An order constraint determines the order in which the resources run. Order constraints are described in Section 6.2, “Order Constraints”.

colocation constraints — A colocation constraint determines where resources will be placed relative to other resources. Colocation constraints are described in Section 6.3, “Colocation of Resources”.

As a shorthand for configuring a set of constraints that will locate a set of resources together and ensure that the resources start sequentially and stop in reverse order, Pacemaker supports the concept of resource groups. For information on resource groups, see Section 5.5, “Resource Groups”.

⁠6.1. Location Constraints

Location constraints determine which nodes a resource can run on. You can configure location constraints to determine whether a resource will prefer or avoid a specified node.

Table 6.1, “Location Constraint Options”. summarizes the options for configuring location constraints.

⁠

Table 6.1. Location Constraint Options

Field
Description

rsc

A resource name

node

A node’s name

score

Value to indicate the preference for whether a resource should run on or avoid a node.

A Value of INFINITY changes "should" to "must"; INFINITY is the default score value for a resource location constraint.

resource-discovery

Value to indicate the preference for whether Pacemaker should perform resource discovery on this node for the specified resource. Limiting resource discovery to a subset of nodes the resource is physically capable of running on can significantly boost performance when a large set of nodes is present. When pacemaker_remote is in use to expand the node count into the hundreds of nodes range, this option should be considered. Possible values include:

always: Always perform resource discovery for the specified resource on this node.

never: Never perform resource discovery for the specified resource on this node.

exclusive: Perform resource discovery for the specified resource only on this node (and other nodes similarly marked as exclusive). Multiple location constraints using exclusive discovery for the same resource across different nodes creates a subset of nodes resource-discovery is exclusive to. If a resource is marked for exclusive discovery on one or more nodes, that resource is only allowed to be placed within that subset of nodes.

Note that setting this option to never or exclusive allows the possibility for the resource to be active in those locations without the cluster’s knowledge. This can lead to the resource being active in more than one location if the service is started outside the cluster's control (for example, by systemd or by an administrator). This can also occur if the resource-discovery property is changed while part of the cluster is down or suffering split-brain, or if the resource-discovery property is changed for a resource and node while the resource is active on that node. For this reason, using this option is appropriate only when you have more than eight nodes and there is a way to guarantee that the resource can run only in a particular location (for example, when the required software is not installed anywhere else).

always is the default resource-discovery value for a resource location constraint.

The following command creates a location constraint for a resource to prefer the specified node or nodes.

pcs constraint location rsc prefers node[=score] ...

The following command creates a location constraint for a resource to avoid the specified node or nodes.

pcs constraint location rsc avoids node[=score] ...

There are two alternative strategies for specifying which nodes a resources can run on:

Opt-In Clusters — Configure a cluster in which, by default, no resource can run anywhere and then selectively enable allowed nodes for specific resources. The procedure for configuring an opt-in cluster is described in Section 6.1.1, “Configuring an "Opt-In" Cluster”.

Opt-Out Clusters — Configure a cluster in which, by default, all resources an run anywhere and then create location constraints for resources that are not allowed to run on specific nodes. The procedure for configuring an opt-out cluster is described in Section 6.1.2, “Configuring an "Opt-Out" Cluster”.

Whether you should choose to configure an opt-in or opt-out cluster depends both on your personal preference and the make-up of your cluster. If most of your resources can run on most of the nodes, then an opt-out arrangement is likely to result in a simpler configuration. On the other-hand, if most resources can only run on a small subset of nodes an opt-in configuration might be simpler.

⁠6.1.1. Configuring an "Opt-In" Cluster

To create an opt-in cluster, set the symmetric-cluster cluster property to false to prevent resources from running anywhere by default.

# pcs property set symmetric-cluster=false

Enable nodes for individual resources. The following commands configure location constraints so that the resource Webserver prefers node example-1, the resource Database prefers node example-2, and both resources can fail over to node example-3 if their preferred node fails.

# pcs constraint location Webserver prefers example-1=200
# pcs constraint location Webserver prefers example-3=0
# pcs constraint location Database prefers example-2=200
# pcs constraint location Database prefers example-3=0

⁠6.1.2. Configuring an "Opt-Out" Cluster

To create an opt-out cluster, set the symmetric-cluster cluster property to true to allow resources to run everywhere by default.

# pcs property set symmetric-cluster=true

The following commands will then yield a configuration that is equivalent to the example in Section 6.1.1, “Configuring an "Opt-In" Cluster”. Both resources can fail over to node example-3 if their preferred node fails, since every node has an implicit score of 0.

# pcs constraint location Webserver prefers example-1=200
# pcs constraint location Webserver avoids example-2=INFINITY
# pcs constraint location Database avoids example-1=INFINITY
# pcs constraint location Database prefers example-2=200

Note that it is not necessary to specify a score of INFINITY in these commands, since that is the default value for the score.

⁠6.2. Order Constraints

Order constraints determine the order in which the resources run. You can configure an order constraint to determine the order in which resources start and stop.

Use the following command to configure an order constraint.

pcs constraint order [action] resource_id then [action] resource_id [options]

Table 6.2, “Properties of an Order Constraint”. summarizes the properties and options for configuring order constraints.

⁠

Table 6.2. Properties of an Order Constraint

Field
Description

resource_id

The name of a resource on which an action is performed.

action

The action to perform on a resource. Possible values of the action property are as follows:

* start - Start the resource.

* stop - Stop the resource.

* promote - Promote the resource from a slave resource to a master resource.

* demote - Demote the resource from a master resource to a slave resource.

If no action is specified, the default action is start. For information on master and slave resources, refer to Section 8.2, “Multi-State Resources: Resources That Have Multiple Modes”.

kind option

How to enforce the constraint. The possible values of the kind option are as follows:

* Optional - Only applies if both resources are starting and/or stopping. For information on optional ordering, refer to Section 6.2.2, “Advisory Ordering”.

* Mandatory - Always (default value). If the first resource you specified is stopping or cannot be started, the second resource you specified must be stopped. For information on mandatory ordering, refer to Section 6.2.1, “Mandatory Ordering”.

* Serialize - Ensure that no two stop/start actions occur concurrently for a set of resources.

symmetrical option

If true, which is the default, stop the resources in the reverse order. Default value: true

⁠6.2.1. Mandatory Ordering

A mandatory constraints indicates that the second resource you specify cannot run without the first resource you specify being active. This is the default value of the kind option. Leaving the default value ensures that the second resource you specify will react when the first resource you specify changes state.

If the first resource you specified resource was running and is stopped, the second resource you specified will also be stopped (if it is running).

If the first resource you specified resource was not running and cannot be started, the second resource you specified will be stopped (if it is running).

If the first resource you specified is (re)started while the second resource you specified is running, the second resource you specified will be stopped and restarted.

⁠6.2.2. Advisory Ordering

When the kind=Optional option is specified for an order constraint, the constraint is considered optional and only has an effect when both resources are stopping and/or starting. Any change in state of the first resource you specified has no effect on the second resource you specified.

The following command configures an advisory ordering constraint for the resources named VirtualIP and dummy_resource,

# pcs constraint order VirtualIP then dummy_resource kind=Optional

⁠6.2.3. Ordered Resource Sets

A common situation is for an administrator to create a chain of ordered resources, where, for example, resource A starts before resource B which starts before resource C. You can configure a chain of ordered resources with the following command. The resources will start in the specified order.

You can create an order constraint on a set or sets of resources with the pcs constraint order set command.

You can set the following options for a set of resources following the options parameter of the pcs constraint order set command.

sequential, which can be set to true or false to indicate whether the set of colocated resources is an ordered set.

require-all, which can be set to true or false to indicate whether all of the resources in the set must be up.

action, which can be set to start, promote, demote or stop, as described in Table 6.2, “Properties of an Order Constraint”.

role, which can be set to Stopped, Started, Master, or Slave. For information on multi-state resources, see Section 8.2, “Multi-State Resources: Resources That Have Multiple Modes”.

You can set the following constraint options for a set of resources following the setoptions parameter of the pcs constraint order set command.

id, to provide a name for the constraint you are defining.

score, to indicate the degree of preference for this constraint. For information on this option, see Table 6.3, “Properties of a Colocation Constraint”.

pcs constraint order set resource1 resource2 [resourceN]... [options] [set resourceX resourceY ... [options]] [setoptions [constraint_options]]

If you have three resources named D1, D2, and D3, the following command configures them as an ordered resource set.

# pcs constraint order set D1 D2 D3

⁠6.2.4. Removing Resources From Ordering Constraints

Use the following command to remove resources from any ordering constraint.

pcs constraint order remove resource1 [resourceN]...

⁠6.3. Colocation of Resources

A colocation constraint determines that the location of one resource depends on the location of another resource.

There is an important side effect of creating a colocation constraint between two resources: it affects the order in which resources are assigned to a node. This is because you cannot place resource A relative to resource B unless you know where resource B is. So when you are creating colocation constraints, it is important to consider whether you should colocate resource A with resource B or resource B with resource A.

Another thing to keep in mind when creating colocation constraints is that, assuming resource A is collocated with resource B, the cluster will also take into account resource A's preferences when deciding which node to choose for resource B.

The following command creates a colocation constraint.

pcs constraint colocation add [master|slave] source_resource with [master|slave] target_resource [score] [options]

For information on master and slave resources, see Section 8.2, “Multi-State Resources: Resources That Have Multiple Modes”.

Table 6.3, “Properties of a Colocation Constraint”. summarizes the properties and options for configuring colocation constraints.

⁠

Table 6.3. Properties of a Colocation Constraint

Field
Description

source_resource

The colocation source. If the constraint cannot be satisfied, the cluster may decide not to allow the resource to run at all.

target_resource

The colocation target. The cluster will decide where to put this resource first and then decide where to put the source resource.

score

Positive values indicate the resource should run on the same node. Negative values indicate the resources should not run on the same node. A value of + INFINITY, the default value, indicates that the source_resource must run on the same node as the target_resource. A value of - INFINITY indicates that the source_resource must not run on the same node as the target_resource.

⁠6.3.1. Mandatory Placement

Mandatory placement occurs any time the constraint's score is +INFINITY or -INFINITY. In such cases, if the constraint cannot be satisfied, then the source_resource is not permitted to run. For score=INFINITY, this includes cases where the target_resource is not active.

If you need myresource1 to always run on the same machine as myresource2, you would add the following constraint:

# pcs constraint colocation add myresource1 with myresource2 score=INFINITY

Because INFINITY was used, if myresource2 cannot run on any of the cluster nodes (for whatever reason) then myresource1 will not be allowed to run.

Alternatively, you may want to configure the opposite, a cluster in which myresource1 cannot run on the same machine as myresource2. In this case use score=-INFINITY

# pcs constraint colocation add myresource1 with myresource2 score=-INFINITY

Again, by specifying -INFINITY, the constraint is binding. So if the only place left to run is where myresource2 already is, then myresource1 may not run anywhere.

⁠6.3.2. Advisory Placement

If mandatory placement is about "must" and "must not", then advisory placement is the "I'd prefer if" alternative. For constraints with scores greater than -INFINITY and less than INFINITY, the cluster will try to accommodate your wishes but may ignore them if the alternative is to stop some of the cluster resources. Advisory colocation constraints can combine with other elements of the configuration to behave as if they were mandatory.

⁠6.3.3. Colocating Sets of Resources

You can create a colocation constraint on a set or sets of resources with the pcs constraint colocation set command.

You can set the following options for a set of resources following the options parameter of the pcs constraint colocation set command.

sequential, which can be set to true or false to indicate whether the set of colocated resources is an ordered set.

require-all, which can be set to true or false to indicate whether all of the resources in the set must be up.

action, which can be set to start, promote, demote or stop, as described in Table 6.2, “Properties of an Order Constraint”.

role, which can be set to Stopped, Started, Master, or Slave. For information on multi-state resources, see Section 8.2, “Multi-State Resources: Resources That Have Multiple Modes”.

You can set the following constraint options for a set of resources following the setoptions parameter of the pcs constraint colocation set command.

kind, to indicate how to enforce the constraint. For information on this option, see Table 6.2, “Properties of an Order Constraint”.

symmetrical, to indicate the order in which to stop the resourses. If true, which is the default, stop the resources in the reverse order. Default value: true

id, to provide a name for the constraint you are defining.

The following command creates a colocation constraint on a set or sets of resources.

pcs constraint colocation set resource1 resource2 [resourceN]... [options] [set resourceX resourceY ... [options]] [setoptions [constraint_options]]

⁠6.3.4. Removing Colocation Constraints

Use the following command to remove colocation constraints with source_resource.

pcs constraint colocation remove source_resource target_resource

⁠6.4. Displaying Constraints

There are a several commands you can use to display constraints that have been configured.

The following command lists all current location, order, and colocation constraints.

pcs constraint list|show

The following command lists all current location constraints.

If resources is specified, location constraints are displayed per resource. This is the default behavior.

If nodes is specified, location constraints are displayed per node.

If specific resources or nodes are specified, then only information about those resources or nodes is displayed.

pcs constraint location [show resources|nodes [specific nodes|resources]] [--full]

The following command lists all current ordering constraints. If the --full option is specified, show the internal constraint IDs.

pcs constraint order show [--full]

The following command lists all current colocation constraints. If the --full option is specified, show the internal constraint IDs.

pcs constraint colocation show [--full]

The following command lists the constraints that reference specific resources.

pcs constraint ref resource ...

⁠Chapter 7. Managing Cluster Resources

This chapter describes various commands you can use to manage cluster resources. It provides information on the following procedures.

Section 7.1, “Manually Moving Resources Around the Cluster”

Section 7.2, “Moving Resources Due to Failure”

Section 7.4, “Enabling, Disabling, and Banning Cluster Resources”

Section 7.5, “Disabling a Monitor Operations”

⁠7.1. Manually Moving Resources Around the Cluster

You can override the cluster and force resources to move from their current location. There are two occasions when you would want to do this:

When a node is under maintenance, and you need to move all resources running on that node to a different node

When individually specified resources needs to be moved

To move all resources running on a node to a different node, you put the node in standby mode. For information on putting a cluster node in standby node, refer to Section 3.2.5, “Standby Mode”.

You can move individually specified resources in either of the following ways.

You can use the pcs resource move command to move a resource off a node on which it is currently running, as described in Section 7.1.1, “Moving a Resource from its Current Node”.

You can use the pcs resource relocate run command to move a resource to its preferred node, as determined by current cluster status, constraints, location of resources and other settings. For information on this command, see Section 7.1.2, “Moving a Resource to its Preferred Node”.

⁠7.1.1. Moving a Resource from its Current Node

To move a resource off the node on which it is currently running, use the following command, specifying the resource_id of the node as defined. Specify the destination_node. if you want to indicate on which node to run the resource that you are moving.

pcs resource move resource_id [destination_node] [--master] [lifetime=lifetime]

Note

When you execute the pcs resource move command, this adds a constraint to the resource to prevent it from running on the node on which it is currently running. You can execute the pcs resource clear or the pcs constraint delete command to remove the constraint. This does not necessarily move the resources back to the original node; where the resources can run at that point depends on how you have configured your resources initially.

If you specify the --master parameter of the pcs resource ban command, the scope of the constraint is limited to the master role and you must specify master_id rather than resource_id.

You can optionally configure a lifetime parameter for the pcs resource move command to indicate a period of time the constraint should remain. You specify the units of a lifetime parameter according to the format defined in ISO 8601, which requires that you specify the unit as a capital letter such as Y (for years), M (for months), W (for weeks), D (for days), H (for hours), M (for minutes), and S (for seconds).

To distinguish a unit of minutes(M) from a unit of months(M), you must specify PT before indicating the value in minutes. For example, a lifetime parameter of 5M indicates an interval of five months, while a lifetime parameter of PT5M indicates an interval of five minutes.

The lifetime parameter is checked at intervals defined by the cluster-recheck-interval cluster property. By default this value is 15 minutes. If your configuration requires that you check this parameter more frequently, you can reset this value with the following command.

pcs property set cluster-recheck-interval=value

You can optionally configure a --wait[=n] parameter for the pcs resource ban command to indicate the number of seconds to wait for the resource to start on the destination node before returning 0 if the resource is starated or 1 if the resource has not yet started. If you do not specify n, the default resource timeout willb e used.

The following command moves the resource resource1 to node example-node2 and prevents it from moving back to the node on which it was originally running for one hour and thirty minutes.

pcs resource move resource1 example-node2 lifetime=PT1H30M

The following command moves the resource resource1 to node example-node2 and prevents it from moving back to the node on which it was originally running for thirty minutes.

pcs resource move resource1 example-node2 lifetime=PT30M

For information on resource constraints, refer to Chapter 6, Resource Constraints.

⁠7.1.2. Moving a Resource to its Preferred Node

After a resource has moved, either due to a failover or to an administrator manually moving the node, it will not necessarily move back to its original node even after the circumstances that caused the failover have been corrected. To relocate resources to their preferred node, use the following command. A preferred node is determined by the current cluster status, constraints, resource location, and other settings and may change over time.

pcs resource relocate run [resource1] [resource2] ...

If you do not specify any resources, all resource are relocated to their preferred nodes.

This command calculates the preferred node for each resource while ignoring resource stickiness. After calculating the preferred node, it creates location constraints which will cause the resources to move to their preferred nodes. Once the resources have been moved, the constraints are deleted auto matically. To remove all constraints created by the pcs resource relocate run command, you can run the pcs resource relocate clear command. To display the current status of resources and their optimal node ignoring resource stickiness, run the pcs resource relocate show command.

⁠7.2. Moving Resources Due to Failure

When you create a resource, you can configure the resource so that it will move to a new node after a defined number of failures by setting the migration-threshold option for that resource. Once the threshold has been reached, this node will no longer be allowed to run the failed resource until:

The administrator manually resets the resource's failcount using the pcs resource failcount command.

The resource's failure-timeout value is reached.

There is no threshold defined by default.

Note

Setting a migration-threshold for a resource is not the same as configuring a resource for migration, in which the resource moves to another location without loss of state.

The following example adds a migration threshold of 10 to the resource named dummy_resource, which indicates that the resource will move to a new node after 10 failures.

# pcs resource meta dummy_resource migration-threshold=10

You can add a migration threshold to the defaults for the whole cluster with the following command.

# pcs resource defaults migration-threshold=10

To determine the resource's current failure status and limits, use the pcs resource failcount command.

There are two exceptions to the migration threshold concept; they occur when a resource either fails to start or fails to stop. Start failures cause the failcount to be set to INFINITY and thus always cause the resource to move immediately.

Stop failures are slightly different and crucial. If a resource fails to stop and STONITH is enabled, then the cluster will fence the node in order to be able to start the resource elsewhere. If STONITH is not enabled, then the cluster has no way to continue and will not try to start the resource elsewhere, but will try to stop it again after the failure timeout.

⁠7.3. Moving Resources Due to Connectivity Changes

Setting up the cluster to move resources when external connectivity is lost is a two-step process.

Add a ping resource to the cluster. The ping resource uses the system utility of the same name to test if a list of machines (specified by DNS hostname or IPv4/IPv6 address) are reachable and uses the results to maintain a node attribute called pingd.

Configure a location constraint for the resource that will move the resource to a different node when connectivity is lost.

Table 5.1, “Resource Properties” describes the properties you can set for a ping resource.

⁠

Table 7.1. Properties of a ping resources

Field
Description

dampen

The time to wait (dampening) for further changes to occur. This prevents a resource from bouncing around the cluster when cluster nodes notice the loss of connectivity at slightly different times.

multiplier

The number of connected ping nodes gets multiplied by this value to get a score. Useful when there are multiple ping nodes configured.

host_list

The machines to contact in order to determine the current connectivity status. Allowed values include resolvable DNS host names, IPv4 and IPv6 addresses. The entries in the host list are space separated.

The following example command creates a ping resource that verifies connectivity to gateway.example.com. In practice, you would verify connectivity to your network gateway/router. You configure the ping resource as a clone so that the resource will run on all cluster nodes.

# pcs resource create ping ocf:pacemaker:ping dampen=5s multiplier=1000 host_list=gateway.example.com --clone

The following example configures a location constraint rule for the existing resource named Webserver. This will cause the Webserver resource to move to a host that is able to ping www.example.com if the host that it is currently running on can not ping www.example.com

# pcs constraint location Webserver rule score=-INFINITY pingd lt 1 or not_defined pingd

⁠7.4. Enabling, Disabling, and Banning Cluster Resources

In addition to the pcs resource move and pcs resource relocate commands described in Section 7.1, “Manually Moving Resources Around the Cluster”, there are a variety of other commands you can use to control the behavior of cluster resources.

You can manually stop a running resource and prevent the cluster from starting it again with the following command. Depending on the rest of the configuration (constraints, options, failures, etc), the resource may remain started. If you specify the --wait option, pcs will wait up to 30 seconds (or 'n' seconds, as specified) for the resource to stop and then return 0 if the resource is stopped or 1 if the resource has not stopped.

pcs resource disable resource_id [--wait[=n]]

You can use the following command to allow the cluster to start a resource. Depending on the rest of the configuration, the resource may remain stopped. If you specify the --wait option, pcs will wait up to 30 seconds (or 'n' seconds, as specified) for the resource to start and then return 0 if the resource is started or 1 if the resource has not started.

pcs resource enable resource_id [--wait[=n]]

Use the following command to prevent a resource from running on a specified node, or on the current node if no node is specified.

pcs resource ban resource_id [node] [--master] [lifetime=lifetime]

Note that when you execute the pcs resource ban command, this adds a -INFINITY location constraint to the resource to prevent it from running on the indicated node. You can execute the pcs resource clear or the pcs constraint delete command to remove the constraint. This does not necessarily move the resources back to the indicated node; where the resources can run at that point depends on how you have configured your resources initially. For information on resource constraints, refer to Chapter 6, Resource Constraints.

If you specify the --master parameter of the pcs resource ban command, the scope of the constraint is limited to the master role and you must specify master_id rather than resource_id.

You can optionally configure a lifetime parameter for the pcs resource ban command to indicate a period of time the constraint should remain. For information on specifying units for the lifetime parameter and on specifying the intervals at which the lifetime parameter should be checked, see Section 7.1, “Manually Moving Resources Around the Cluster”.

You can optionally configure a --wait[=n] parameter for the pcs resource ban command to indicate the number of seconds to wait for the resource to start on the destination node before returning 0 if the resource is starated or 1 if the resource has not yet started. If you do not specify n, the default resource timeout willb e used.

You can use the debug-start parameter of the pcs resource command to force a specified resource to start on the current node, ignoring the cluster recommendations and printing the output from starting the resource. This is mainly used for debugging resources; starting resources on a cluster is (almost) always done by Pacemaker and not directly with a pcs command. If your resource is not starting, it is usually due to either a misconfiguration of the resource (which you debug in the system log), constraints that the resource from starting, or the resource being disabled. You can use this command to test resource configuration, but it should not normally be used to start resources in a cluster.

The format of the debug-start command is as follows.

pcs resource debug-start resource_id

⁠7.5. Disabling a Monitor Operations

The easiest way to stop a recurring monitor is to delete it. However, there can be times when you only want to disable it temporarily. In such cases, add enabled="false" to the operation’s definition. When you want to reinstate the monitoring operation, set enabled="true" to the operation's definition.

⁠7.6. Managed Resources

You can set a resource to unmanaged mode, which indicates that the resource is still in the configuration but Pacemaker does not manage the resource.

The following command sets the indicated resources to unmanaged mode.

pcs resource unmanage resource1  [resource2] ...

The following command sets resources to managed mode, which is the default state.

pcs resource manage resource1  [resource2] ...

You can specify the name of a resource group with the pcs resource manage or pcs resource unmanage command. The command will act on all of the resources in the group, so that you can manage or unmanage all of the resource in a group with a single command and then manage the contained resources individually.

⁠Chapter 8. Advanced Resource types

This chapter describes advanced resource types that Pacemaker supports.

⁠8.1. Resource Clones

You can clone a resource so that the resouce can be active on multiple nodes. For example, you can use cloned resources to configure multiple instances of an IP resource to distribute throughout a cluster for node balancing. You can clone any resource provided the resource agent supports it. A clone consists of one resource or one resource group.

Note

Only resources that can be active on multiple nodes at the same time are suitable for cloning. For example, a Filesystem resource mounting a non-clustered file system such as ext4 from a shared memory device should not be cloned. Since the ext4 partition is not cluster aware, this file system is not suitable for read/write operations occurring from multiple nodes at the same time.

⁠8.1.1. Creating and Removing a Cloned Resource

You can create a resource and a clone of that resource at the same time with the following command.

pcs resource create resource_id standard:provider:type|type [resource options]  \
clone  [meta clone_options]

The name of the clone will be resource_id-clone.

You cannot create a resource group and a clone of that resource group in a single command.

Alternately, you can create a clone of a previously-created resource or resource group with the following command.

pcs resource clone resource_id | group_name [clone_options]...

The name of the clone will be resource_id-clone or group_name-clone.

Note

You need to configure resource configuration changes on one node only.

Note

When configuring constraints, always use the name of the group or clone.

When you create a clone of a resource, the clone takes on the name of the resource with -clone appended to the name. The following commands creates a resource of type apache named webfarm and a clone of that resource named webfarm-clone.

# pcs resource create webfarm apache clone

Use the following command to remove a clone of a resource or a resource group. This does not remove the resource or resource group itself.

pcs resource unclone resource_id | group_name

For information on resource options, refer to Section 5.1, “Resource Creation”.

Table 8.1, “Resource Clone Options” describes the options you can specify for a cloned resource.

⁠

Table 8.1. Resource Clone Options

Field
Description

priority, target-role, is-managed

Options inherited from resource that is being cloned, as described in. Table 5.3, “Resource Meta Options”.

clone-max

How many copies of the resource to start. Defaults to the number of nodes in the cluster.

clone-node-max

How many copies of the resource can be started on a single node; the default value is 1.

notify

When stopping or starting a copy of the clone, tell all the other copies beforehand and when the action was successful. Allowed values: false, true. The default value is false.

globally-unique

Does each copy of the clone perform a different function? Allowed values: false, true

If the value of this option is false, these resources behave identically everywhere they are running and thus there can be only one copy of the clone active per machine.

If the value of this option is true, a copy of the clone running on one machine is not equivalent to another instance, whether that instance is running on another node or on the same node. The default value is true if the value of clone-node-max is greater than one; otherwise the default value is false.

ordered

Should the copies be started in series (instead of in parallel). Allowed values: false, true. The default value is false.

interleave

Changes the behavior of ordering constraints (between clones/masters) so that copies of the first clone can start or stop as soon as the copy on the same node of the second clone has started or stopped (rather than waiting until every instance of the second clone has started or stopped). Allowed values: false, true. The default value is false.

⁠8.1.2. Clone Constraints

In most cases, a clone will have a single copy on each active cluster node. You can, however, set clone-max for the resource clone to a value that is less than the total number of nodes in the cluster. If this is the case, you can indicate which nodes the cluster should preferentially assign copies to with resource location constraints. These constraints are written no differently to those for regular resources except that the clone's id must be used.

The following command creates a location constraint for the cluster to preferentially assign resource clone webfarm-clone to node1.

# pcs constraint location webfarm-clone prefers node1

Ordering constraints behave slightly differently for clones. In the example below, webfarm-stats will wait until all copies of webfarm-clone that need to be started have done so before being started itself. Only if no copies of webfarm-clone can be started then webfarm-stats will be prevented from being active. Additionally, webfarm-clone will wait for webfarm-stats to be stopped before stopping itself.

# pcs constraint order start webfarm-clone then webfarm-stats

Colocation of a regular (or group) resource with a clone means that the resource can run on any machine with an active copy of the clone. The cluster will choose a copy based on where the clone is running and the resource's own location preferences.

Colocation between clones is also possible. In such cases, the set of allowed locations for the clone is limited to nodes on which the clone is (or will be) active. Allocation is then performed as normally.

The following command creates a colocation constraint to ensure that the resource webfarm-stats runs on the same node as an active copy of webfarm-clone.

# pcs constraint colocation add webfarm-stats with webfarm-clone

⁠8.1.3. Clone Stickiness

To achieve a stable allocation pattern, clones are slightly sticky by default. If no value for resource-stickiness is provided, the clone will use a value of 1. Being a small value, it causes minimal disturbance to the score calculations of other resources but is enough to prevent Pacemaker from needlessly moving copies around the cluster.

⁠8.2. Multi-State Resources: Resources That Have Multiple Modes

Multi-state resources are a specialization of Clone resources. They allow the instances to be in one of two operating modes; these are called Master and Slave. The names of the modes do not have specific meanings, except for the limitation that when an instance is started, it must come up in the Slave state.

You can create a resource as a master/slave clone with the following single command.

pcs resource create resource_id standard:provider:type|type [resource options] \
--master [meta master_options]

The name of the master/slave clone will be resource_id-master.

Alternately, you can create a master/slave resource from a previously-created resource or resource group with the following command: When you use this command, you can specify a name for the master/slave clone. If you do not specify a name, the name of the master/slave clone will be resource_id-master or group_name-master.

pcs resource master master/slave_name resource_id|group_name [master_options]

For information on resource options, refer to Section 5.1, “Resource Creation”.

Table 8.2, “Properties of a Multi-State Resource” describes the options you can specify for a multi-state resource.

⁠

Table 8.2. Properties of a Multi-State Resource

Field
Description

id

Your name for the multi-state resource

priority, target-role, is-managed

See Table 5.3, “Resource Meta Options”.

clone-max, clone-node-max, notify, globally-unique, ordered, interleave

See Table 8.1, “Resource Clone Options”.

master-max

How many copies of the resource can be promoted to master status; default 1.

master-node-max

How many copies of the resource can be promoted to master status on a single node; default 1.

⁠8.2.1. Monitoring Multi-State Resources

To add a monitoring operation for the master resource only, you can add an additional monitor operation to the resource. Note, however, that every monitor operation on a resource must have a different interval.

The following example configures a monitor operation with an interval of 11 seconds on the master resource for ms_resource. This monitor operation is in addition to the default monitor operation with the default monitor interval of 10 seconds.

# pcs resource op add ms_resource interval=11s role=Master

⁠8.2.2. Multi-state Constraints

In most cases, a multi-state resources will have a single copy on each active cluster node. If this is not the case, you can indicate which nodes the cluster should preferentially assign copies to with resource location constraints. These constraints are written no differently than those for regular resources.

For information on resource location constraints, see Section 6.1, “Location Constraints”.

You can create a colocation constraint which specifies whether the resources are master or slave resources. The following command creates a resource colocation constraint.

pcs constraint colocation add [master|slave] source_resource with [master|slave] target_resource [score] [options]

For information on colocation constraints, see Section 6.3, “Colocation of Resources”.

When configuring an ordering constraint that includes multi-state resources, one of the actions that you can specify for the resources is promote, indicating that the resource be promoted from slave to master. Additionally, you can specify an action of demote, indicated that the resource be demoted from master to slave.

The command for configuring an order constraint is as follows.

pcs constraint order [action] resource_id then [action] resource_id [options]

For information on resource order constraints, see Section 6.2, “Order Constraints”.

⁠8.2.3. Multi-state Stickiness

To achieve a stable allocation pattern, multi-state resources are slightly sticky by default. If no value for resource-stickiness is provided, the multi-state resource will use a value of 1. Being a small value, it causes minimal disturbance to the score calculations of other resources but is enough to prevent Pacemaker from needlessly moving copies around the cluster.

⁠8.3. Event Notification with Monitoring Resources

A Pacemaker cluster is an event-driven system, where an event might be a resource failure or configuration change. The ocf:pacemaker:ClusterMon resource can monitor the cluster status and trigger alerts on each cluster event. This resource runs the crm_mon command in the background at regular intervals.

By default, the crm_mon command listens for resource events only; to enable listing for fencing events you can provide the --watch-fencing option to the command when you configure the ClusterMon resource. The crm_mon command does not monitor for membership issues but will print a message when fencing is started and when monitoring is started for that node, which would imply that a member just joined the cluster.

The ClusterMon resource can execute an external program to determine what to do with cluster notifications by means of the extra_options parameter. Table 8.3, “Environment Variables Passed to the External Monitor Program” lists the environment variables that are passed to that program, which describe the type of cluster event that occurred.

⁠

Table 8.3. Environment Variables Passed to the External Monitor Program

Environment Variable
Description

CRM_notify_recipient

The static external-recipient from the resource definition

CRM_notify_node

The node on which the status change happened

CRM_notify_rsc

The name of the resource that changed the status

CRM_notify_task

The operation that caused the status change

CRM_notify_desc

The textual output relevant error code of the operation (if any) that caused the status change

CRM_notify_rc

The return code of the operation

CRM_target_rc

The expected return code of the operation

CRM_notify_status

The numerical representation of the status of the operation

The following example configures a ClusterMon resource that executes the external program crm_logger.sh which will log the event notifications specified in the program.

The following procedure creates the crm_logger.sh program that this resource will use.

On one node of the cluster, create the program that will log the event notifications.

# cat <<-END >/usr/local/bin/crm_logger.sh
#!/bin/sh
logger -t "ClusterMon-External" "${CRM_notify_node} ${CRM_notify_rsc} \
${CRM_notify_task} ${CRM_notify_desc} ${CRM_notify_rc} \
${CRM_notify_target_rc} ${CRM_notify_status} ${CRM_notify_recipient}";
exit;
END

Set the ownership and permissions for the program.

# chmod 700 /usr/local/bin/crm_logger.sh
# chown root.root /usr/local/bin/crm_logger.sh

Use the scp command to copy the crm_logger.sh program to the other nodes of the cluster, putting the program in the same location on those nodes and setting the same ownership and permissions for the program.

The following example configures the ClusterMon resource, named ClusterMon-External, that runs the program /usr/local/bin/crm_logger.sh. The ClusterMon resource outputs the cluster status to an html file, which is /var/www/html/cluster_mon.html in this example. The pidfile detects whether ClusterMon is already running; in this example that file is /var/run/crm_mon-external.pid. This resource is created as a clone so that it will run on every node in the cluster. The watch-fencing is specified to enable monitoring of fencing events in addition to resource events, unclding the start/stop/monitor, start/monitor. and stop of the fencing resource.

# pcs resource create ClusterMon-External ClusterMon user=root \
update=10 extra_options="-E /usr/local/bin/crm_logger.sh --watch-fencing" \
htmlfile=/var/www/html/cluster_mon.html \
pidfile=/var/run/crm_mon-external.pid clone

Note

The crm_mon command that this resource executes and which could be run manually is as follows:

# /usr/sbin/crm_mon -p /var/run/crm_mon-manual.pid -d -i 5 \
-h /var/www/html/crm_mon-manual.html -E "/usr/local/bin/crm_logger.sh" \
--watch-fencing

The following example shows the format of the output of the monitoring notifications that this example yields.

Aug  7 11:31:32 rh6node1pcmk ClusterMon-External: rh6node2pcmk.examplerh.com ClusterIP st_notify_fence Operation st_notify_fence requested by rh6node1pcmk.examplerh.com for peer rh6node2pcmk.examplerh.com: OK (ref=b206b618-e532-42a5-92eb-44d363ac848e) 0 0 0 #177
Aug  7 11:31:32 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com ClusterIP start OK 0 0 0
Aug  7 11:31:32 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com ClusterIP monitor OK 0 0 0
Aug  7 11:33:59 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com fence_xvms monitor OK 0 0 0
Aug  7 11:33:59 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com ClusterIP monitor OK 0 0 0
Aug  7 11:33:59 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com ClusterMon-External start OK 0 0 0
Aug  7 11:33:59 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com fence_xvms start OK 0 0 0
Aug  7 11:33:59 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com ClusterIP start OK 0 0 0
Aug  7 11:33:59 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com ClusterMon-External monitor OK 0 0 0
Aug  7 11:34:00 rh6node1pcmk crmd[2887]:   notice: te_rsc_command: Initiating action 8: monitor ClusterMon-External:1_monitor_0 on rh6node2pcmk.examplerh.com
Aug  7 11:34:00 rh6node1pcmk crmd[2887]:   notice: te_rsc_command: Initiating action 16: start ClusterMon-External:1_start_0 on rh6node2pcmk.examplerh.com
Aug  7 11:34:00 rh6node1pcmk ClusterMon-External: rh6node1pcmk.examplerh.com ClusterIP stop OK 0 0 0
Aug  7 11:34:00 rh6node1pcmk crmd[2887]:   notice: te_rsc_command: Initiating action 15: monitor ClusterMon-External_monitor_10000 on rh6node2pcmk.examplerh.com
Aug  7 11:34:00 rh6node1pcmk ClusterMon-External: rh6node2pcmk.examplerh.com ClusterMon-External start OK 0 0 0
Aug  7 11:34:00 rh6node1pcmk ClusterMon-External: rh6node2pcmk.examplerh.com ClusterMon-External monitor OK 0 0 0
Aug  7 11:34:00 rh6node1pcmk ClusterMon-External: rh6node2pcmk.examplerh.com ClusterIP start OK 0 0 0
Aug  7 11:34:00 rh6node1pcmk ClusterMon-External: rh6node2pcmk.examplerh.com ClusterIP monitor OK 0 0 0

⁠8.4. The pacemaker_remote Service

The pacemaker_remote service allows nodes not running corosync to integrate into the cluster and have the cluster manage their resources just as if they were real cluster nodes. This means that Pacemaker clusters are now capable of managing virtual environments (KVM/LXC) as well as the resources that live within those virtual environments without requiring the virtual environments to run pacemaker or corosync.

The following terms are used to describe the pacemaker_remote service.

cluster node - A node running the High Availability services (pacemaker and corosync).

remote node — A node running pacemaker_remote to remotely integrate into the cluster without requiring corosync cluster membership.

container — A Pacemaker resource that contains additional resources. For example, a KVM virtual machine resource that contains a webserver resource.

container remote node — A virtual guest remote node running the pacemaker_remote service. This describes a specific remote node use case where a virtual guest resource managed by the cluster is both started by the cluster and integrated into the cluster as a remote node.

pacemaker_remote — A service daemon capable of performing remote application management within guest nodes (KVM and LXC) in both Pacemaker cluster environments and standalone (non-cluster) environments. This service is an enhanced version of Pacemaker’s local resource manage daemon (LRMD) that is capable of managing and monitoring LSB, OCF, upstart, and systemd resources on a guest remotely. It also allows pcs to work on remote nodes.

LXC — A Linux Container defined by the libvirt-lxc Linux container driver.

A Pacemaker cluster running the pacemaker_remote service has the following characteristics.

The virtual remote nodes run the pacemaker_remote service (with very little configuration required on the virtual machine side).

The cluster stack (pacemaker and corosync), running on the cluster nodes, launches the virtual machines and immediately connects to the pacemaker_remote service, allowing the virtual machines to integrate into the cluster.

The key difference between the virtual machine remote nodes and the cluster nodes is that the remote nodes are not running the cluster stack. This means the remote nodes do not take place in quorum. On the other hand, this also means that remote nodes are not bound to the scalability limits associated with the cluster stack. Other than the quorum limitation, the remote nodes behave just like cluster nodes in respect to resource management. The cluster is fully capable of managing and monitoring resources on each remote node. You can build constraints against remote nodes, put them in standby, or perform any other action you perform on cluster nodes. Remote nodes appear in cluster status output just as cluster nodes do.

⁠8.4.1. Container Remote Node Resource Options

When configuring a virtual machine or LXC resource to act as a remote node, you create a VirtualDomain resource, which manages the virtual machine. For descriptions of the options you can set for a VirtualDomain resource, use the following command.

# pcs resource describe VirtualDomain

In addition to the VirtualDomain resource options, you can configure metadata options to both enable the resource as a remote node and define the connection parameters. Table 8.4, “Metadata Options for Configuring KVM/LXC Resources as Remote Nodes” describes these metadata options.

⁠

Table 8.4. Metadata Options for Configuring KVM/LXC Resources as Remote Nodes

Field
Default
Description

remote-node

<none>

The name of the remote node this resource defines. This both enables the resource as a remote node and defines the unique name used to identify the remote node. If no other parameters are set, this value will also be assumed as the hostname to connect to at port 3121. WARNING: This value cannot overlap with any resource or node IDs.

remote-port

3121

Configures a custom port to use for the guest connection to pacemaker_remote.

remote-addr

remote-node value used as hostname

The IP address or hostname to connect to if remote node’s name is not the hostname of the guest

remote-connect-timeout

60s

Amount of time before a pending guest connection will time out

⁠8.4.2. Host and Guest Authentication

Authentication and encryption of the connection between cluster-nodes and remote nodes is achieved using TLS with PSK encryption/authentication on TCP port 3121. This means both the cluster node and the remote node must share the same private key. By default this key must be placed at /etc/pacemaker/authkey on both cluster nodes and remote nodes.

⁠8.4.3. Changing Default pacemaker_remote Options

If you need to change the default port or authkey location for either Pacemaker or pacemaker_remote, there are environment variables you can set that affect both of those daemons. These environment variables can be enabled by placing them in the /etc/sysconfig/pacemaker file as follows.

#==#==# Pacemaker Remote
# Use a custom directory for finding the authkey.
PCMK_authkey_location=/etc/pacemaker/authkey
#
# Specify a custom port for Pacemaker Remote connections
PCMK_remote_port=3121

⁠8.4.4. Configuration Overview: KVM Remote Node

This section provides a high-level summary overview of the steps to perform to have Pacemaker launch a virtual machine and to integrate that machine as a remote node, using libvirt and KVM virtual guests.

After installing the virtualization software and enabling the libvirtd service on the cluster nodes, put an authkey with the path /etc/pacemaker/authkey on every cluster node and virtual machine. This secures remote communication and authentication.

The following command creates an authkey.

# dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1

On every virtual machine, install pacemaker_remote packages, start the pacemaker_remote service and enable it to run on startup, and allow TCP port 3121 through the firewall.

# yum install pacemaker-remote resource-agents
# systemctl start pacemaker_remote.service
# systemctl enable pacemaker_remote.service
# firewall-cmd --add-port 3121/tcp --permanent

Give each virtual machine a static network address and unique hostname.

To create the VirtualDomain resource agent for the management of the virtual machine, Pacemaker requires the virtual machine's xml config file to be dumped to a file on disk. For example, if you created a virtual machine named guest1, dump the xml to a file somewhere on the host using the following command.

# virsh dumpxml guest1 > /virtual_machines/guest1.xml

Create the VirtualDomain resource, configuring the remote-note resource meta option to indicate that the virtual machine is a remote node capable of running resources.

In the example below, the meta-attribute remote-node=guest1 tells pacemaker that this resource is a remote node with the hostname guest1 that is capable of being integrated into the cluster. The cluster will attempt to contact the virtual machine’s pacemaker_remote service at the hostname guest1 after it launches.

# pcs resource create vm-guest1 VirtualDomain hypervisor="qemu:///system" config="/virtual_machines/vm-guest1.xml" meta remote-node=guest1

After creating the VirtualDomain resource, you can treat the remote node just as you would treat any other node in the cluster. For example, you can create a resource and place a resource constraint on the resource to run on the remote node.

# pcs resource create webserver apache params configfile=/etc/httpd/conf/httpd.conf op monitor interval=30s
# pcs constraint location webserver prefers guest1

Once a remote node is integrated into the cluster, you can execute pcs commands from the remote node itself, just as if the remote node was running Pacemaker.

⁠Chapter 9. Pacemaker Rules

Rules can be used to make your configuration more dynamic. One common example is to set one value for resource-stickiness during working hours, to prevent resources from being moved back to their most preferred location, and another on weekends when no-one is around to notice an outage.

Another use of rules might be to assign machines to different processing groups (using a node attribute) based on time and to then use that attribute when creating location constraints.

Each rule can contain a number of expressions, date-expressions and even other rules. The results of the expressions are combined based on the rule's boolean-op field to determine if the rule ultimately evaluates to true or false. What happens next depends on the context in which the rule is being used.

⁠

Table 9.1. Properties of a Rule

Field
Description

role

Limits the rule to apply only when the resource is in that role. Allowed values: Started, Slave, and Master. NOTE: A rule with role="Master" can not determine the initial location of a clone instance. It will only affect which of the active instances will be promoted.

score

The score to apply if the rule evaluates to true. Limited to use in rules that are part of location constraints.

score-attribute

The node attribute to look up and use as a score if the rule evaluates to true. Limited to use in rules that are part of location constraints.

boolean-op

How to combine the result of multiple expression objects. Allowed values: and and or. The default value is and.

⁠9.1. Node Attribute Expressions

Node attribute expressions are used to control a resource based on the attributes defined by a node or nodes.

⁠

Table 9.2. Properties of an Expression

Field
Description

value

User supplied value for comparison

attribute

The node attribute to test

type

Determines how the value(s) should be tested. Allowed values: string, integer, version

operation

The comparison to perform. Allowed values:

* lt - True if the node attribute’s value is less than value

* gt - True if the node attribute’s value is greater than value

* lte - True if the node attribute’s value is less than or equal to value

* gte - True if the node attribute’s value is greater than or equal to value

* eq - True if the node attribute’s value is equal to value

* ne - True if the node attribute’s value is not equal to value

* defined - True if the node has the named attribute

* not_defined - True if the node does not have the named attribute

⁠9.2. Time/Date Based Expressions

Date expressions are used to control a resource or cluster option based on the current date/time. They can contain an optional date specification.

⁠

Table 9.3. Properties of a Date Expression

Field
Description

start

A date/time conforming to the ISO8601 specification.

end

A date/time conforming to the ISO8601 specification.

operation

Compares the current date/time with the start and/or end date, depending on the context. Allowed values:

* gt - True if the current date/time is after start

* lt - True if the current date/time is before end

* in-range - True if the current date/time is after start and before end

* date-spec - performs a cron-like comparison to the current date/time

⁠9.3. Date Specifications

Date specificiations are used to create cron-like expressions relating to time. Each field can contain a single number or a single range. Instead of defaulting to zero, any field not supplied is ignored.

For example, monthdays="1" matches the first day of every month and hours="09-17" matches the hours between 9am and 5pm (inclusive). However, you cannot specify weekdays="1,2" or weekdays="1-2,5-6" since they contain multiple ranges.

⁠

Table 9.4. Properties of a Date Specification

Field
Description

id

A unique name for the date

hours

Allowed values: 0-23

monthdays

Allowed values: 0-31 (depending on month and year)

weekdays

Allowed values: 1-7 (1=Monday, 7=Sunday)

yeardays

Allowed values: 1-366 (depending on the year)

months

Allowed values: 1-12

weeks

Allowed values: 1-53 (depending on weekyear)

years

Year according the Gregorian calendar

weekyears

May differ from Gregorian years; for example, 2005-001 Ordinal is also 2005-01-01 Gregorian is also 2004-W53-6 Weekly

moon

Allowed values: 0-7 (0 is new, 4 is full moon).

⁠9.4. Durations

Durations are used to calculate a value for end when one is not supplied to in_range operations. They contain the same fields as date_spec objects but without the limitations (ie. you can have a duration of 19 months). Like date_specs, any field not supplied is ignored.

⁠9.5. Configuring Rules with pcs

To configure a rule, use the following command. If score is omitted, it defaults to INFINITY. If id is omitted, one is generated from the constraint_id. The rule_type should be expression or date_expression.

pcs constraint rule add constraint_id [rule_type] [score=score [id=rule_id] expression|date_expression|date_spec options

To remove a rule, use the following. If the rule that you are removing is the last rule in its constraint, the constraint will be removed.

pcs constraint rule remove rule_id

⁠9.6. Sample Time Based Expressions

The following command configures an expression that is true if now is any time in the year 2005.

# pcs constraint location Webserver rule score=INFINITY date-spec years=2005

The following command configures an expression that is true from 9am to 5pm, Monday through Friday. Note that the hours value of 16 matches up to 16:59:59, as the numeric value (hour) still matches.

# pcs constraint location Webserver rule score=INFINITY date-spec hours="9-16" weekdays="1-5"

The following command configures an expression that is true when there is a full moon on Friday the 13th.

# pcs constraint location Webserver rule date-spec weekdays=5 monthdays=13 moon=4

⁠9.7. Using Rules to Determine Resource Location

You can use a rule to determine a resource's location with the following command.

pcs constraint location resource_id rule [rule_id] [role=master|slave] [score=score expression]

The expression can be one of the following:

defined|not_defined attribute

attribute lt|gt|lte|gte|eq|ne value

date [start=start] [end=end] operation=gt|lt|in-range

date-spec date_spec_options

⁠Chapter 10. Pacemaker Cluster Properties

Cluster properties control how the cluster behaves when confronted with situations that may occur during cluster operation.

Table 10.1, “Cluster Properties” describes the cluster properties options.

Section 10.2, “Setting and Removing Cluster Properties” describes how to set cluster properties.

Section 10.3, “Querying Cluster Property Settings” describes how to list the currently set cluster properties.

⁠10.1. Summary of Cluster Properties and Options

Table 10.1, “Cluster Properties” summaries the Pacemaker cluster properties, showing the default values of the properties and the possible values you can set for those properties.

Note

In addition to the properties described in this table, there are additional cluster properties that are exposed by the cluster software. For these properties, it is recommended that you not change their values from their defaults.

⁠

Table 10.1. Cluster Properties

Option
Default
Description

batch-limit
30

The number of jobs that the transition engine (TE) is allowed to execute in parallel. The "correct" value will depend on the speed and load of your network and cluster nodes.

migration-limit
-1 (unlimited)

The number of migration jobs that the TE is allowed to execute in parallel on a node.

no-quorum-policy
stop

What to do when the cluster does not have quorum. Allowed values:

* ignore - continue all resource management

* freeze - continue resource management, but do not recover resources from nodes not in the affected partition

* stop - stop all resources in the affected cluster partition

* suicide - fence all nodes in the affected cluster partition

symmetric-cluster
true

Indicates whether resources can run on any node by default.

stonith-enabled
true

Indicates that failed nodes and nodes with resources that can not be stopped should be fenced. Protecting your data requires that you set this true.

If true, or unset, the cluster will refuse to start resources unless one or more STONITH resources have been configured also.

stonith-action
reboot

Action to send to STONITH device. Allowed values: reboot, off. The value poweroff is also allowed, but is only used for legacy devices.

cluster-delay
60s

Round trip delay over the network (excluding action execution). The "correct" value will depend on the speed and load of your network and cluster nodes.

stop-orphan-resources
true

Indicates whether deleted resources should be stopped.

stop-orphan-actions
true

Indicates whether deleted actions should be cancelled.

start-failure-is-fatal
true

Indicates whether a failure to start is treated as fatal for a resource. When set to false, the cluster will instead use the resource's failcount and value for migration-threshold. For information on setting the migration-threshold option for a resource, see Section 7.2, “Moving Resources Due to Failure”.

pe-error-series-max
-1 (all)

The number of PE inputs resulting in ERRORs to save. Used when reporting problems.

pe-warn-series-max
-1 (all)

The number of PE inputs resulting in WARNINGs to save. Used when reporting problems.

pe-input-series-max
-1 (all)

The number of "normal" PE inputs to save. Used when reporting problems.

cluster-infrastructure

The messaging stack on which Pacemaker is currently running. Used for informational and diagnostic purposes; not user-configurable.

dc-version

Version of Pacemaker on the cluster's Designated Controller (DC). Used for diagnostic purposes; not user-configurable.

last-lrm-refresh

Last refresh of the Local Resource Manager, given in units of seconds since epoca. Used for diagnostic purposes; not user-configurable.

cluster-recheck-interval
15min

Polling interval for time-based changes to options, resource parameters and constraints. Allowed values: Zero disables polling, positive values are an interval in seconds (unless other SI units are specified, such as 5min).

default-action-timeout
20s

Timeout value for a Pacemaker action. The setting for an operation in a resource itself always takes precedence over the default value set as a cluster option.

maintenance-mode
false

Maintenance Mode tells the cluster to go to a "hands off" mode, and not start or stop any services until told otherwise. When maintanence mode is completed, the cluster does a sanity check of the current state of any services, and then stops or starts any that need it.

shutdown-escalation
20min

The time after which to give up trying to shut down gracefully and just exit. Advanced use only.

stonith-timeout
60s

How long to wait for a STONITH action to complete.

stop-all-resources
false

Should the cluster stop all resources.

default-resource-stickiness
5000

Indicates how much a resource prefers to stay where it is. It is recommended that you set this value as a resource/operation default rather than as a cluster option.

is-managed-default
true

Indicates whether the cluster is allowed to start and stop a resource. It is recommended that you set this value as a resource/operation default rather than as a cluster option.

enable-acl
false

(Red Hat Enterprise Linux 7.1 and later) Indicates whether the cluster can use access control lists, as set with the pcs acl command.

⁠10.2. Setting and Removing Cluster Properties

To set the value of a cluster property, use the following pcs command.

pcs property set property=value

For example, to set the value of symmetric-cluster to false, use the following command.

# pcs property set symmetric-cluster=false

You can remove a cluster property from the configuration with the following command.

pcs property unset property

Alternately, you can remove a cluster property from a configuration by leaving the value field of the pcs property set command blank. This restores that property to its default value. For example, if you have previously set the symmetric-cluster property to false, the following command removes the value you have set from the configuration and restores the value of symmetric-cluster to true, which is its default value.

# pcs property set symmetic-cluster=

⁠10.3. Querying Cluster Property Settings

In most cases, when you use the pcs command to display values of the various cluster components, you can use pcs list or pcs show interchangeably. In the following examples, pcs list is the format used to display an entire list of all settings for more than one property, while pcs show is the format used to display the values of a specific property.

To display the values of the property settings that have been set for the cluster, use the following pcs command.

pcs property list

To display all of the values of the property settings for the cluster, including the default values of the property settings that have not been explicitly set, use the following command.

pcs property list --all

To display the current value of a specific cluster property, use the following command.

pcs property show property

For example, to display the current value of the cluster-infrastructure property, execute the following command:

# pcs property show cluster-infrastructure
Cluster Properties:
 cluster-infrastructure: cman

For informational purposes, you can display a list of all of the default values for the properties, whether they have been set to a value other than the default or not, by using the following command.

pcs property [list|show] --defaults

⁠Chapter 11. The pcsd Web UI

This chapter provides an overview of configuring a Red Hat High Availability cluster with the pcsd Web UI.

⁠11.1. pcsd Web UI Setup

To set up your system to use the pcsd Web UI to configure a cluster, use the following procedure.

Install the Pacemaker configuration tools, as described in Section 1.2, “Installing Pacemaker configuration tools”.

On each node that will be part of the cluster, use the passwd command to set the password for user hacluster, using the same password on each node.

Start and enable the pcsd daemon on each node:

# systemctl start pcsd.service
# systemctl enable pcsd.service

On each node, authenticate the nodes that will constitute the cluster with the following command. After executing this command, you will be prompted for a Username and a Password. Specify hacluster as the Username.

# pcs cluster auth node1 node2 ... nodeN

On any system, open a browser to the following URL, specifying one of the nodes you have authorized (note that this uses the https protocol). This brings up the pcsd Web UI login screen.

https://nodename:2224

Log in as user hacluster. This brings up the Manage Clusters page.

⁠11.2. Managing Clusters with the pcsd Web UI

From the Manage Clusters page, you can create a new cluster or add an existing cluster to the Web UI. Once you have created a cluster, the cluster name is displayed on this page. Moving the cursor over the cluster's name displays information about the cluster.

To create a cluster, click on Create New and enter the name of the cluster to create and the nodes that constitute the cluster. After entering this information, click Create Cluster. This displays the cluster just created on the Manage Clusters screen, showing the cluster nodes.

To add an existing cluster to the Web UI, click on Add Existing and enter the hostname or IP address of a node in the cluster that you would like to manage with the Web UI.

To manage a cluster, click on the name of the cluster. This brings up a page that allows you to configure the nodes, resources, fence devices, and cluster properties for that cluster.

Note

When using the pcsd Web UI to configure a cluster, you can move your mouse over the text describing many of the options to see longer descriptions of those options as a tooltip display.

⁠11.3. Cluster Nodes

Selecting the Nodes option from the menu along the top of the cluster management page displays the currently configured nodes and the status of the currently-selected node. You can add or remove nodes from this page, and you can start, stop, restart, or put a node in standby mode. For information on standby mode, see Section 3.2.5, “Standby Mode”.

You can also configure fence devices directly from this page, as described in Section 11.4, “Fence Devices”. by selecting Configure Fencing.

⁠11.4. Fence Devices

Selecting the Fence Devices option from the menu along the top of the cluster management page displays Fence Devices screen, showing the currently configured fence devices.

To add a new fence device to the cluster, click Add. The brings up the Add Fence Device screen. When you select a fence device type from the drop-down Type menu, the arguments you must specify for that fence device appear in the menu. You can click on Optional Arguments to display additional arguments you can specify for the fence device you are defining. After entering the parameters for the new fence device, click Create Fence Instance.

For information on configuring fence devices with Pacemaker, see Chapter 4, Fencing: Configuring STONITH.

⁠11.5. Cluster Resources

Selecting the Resources option from the menu along the top of the cluster management page displays the currently configured resources for the cluster as well as the configuration parameters of the currently selected resource. If a resource is part of a resource group, the name of the resource group will be displayed in parenthesis along with the resource name.

You can add or remove resources and you can edit the configuration of existing resources.

To add a new resource to the cluster, click Add. The brings up the Add Resource screen. When you select a resource type from the drop-down Type menu, the arguments you must specify for that resource appear in the menu. You can click Optional Arguments to display additional arguments you can specify for the resource you are defining. After entering the parameters for the resource you are creating, click Create Resource.

When configuring the arguments for a resource, a brief description of the argument appears in the menu. If you move the cursor to the field, a longer help description of that argument is displayed.

You can define as resource as a cloned resource, or as a master/slave resource. For information on these resource types, see Chapter 8, Advanced Resource types.

Once you have created at least one resource, you can create resource a resource group. For information on resource groups, see Section 5.5, “Resource Groups”.

To create a resource group, select a resource that will be part of the group from the Resources screen, then click Create Group. This displays the Create Group screen. Enter a group name and click Create Group. This returns you to the Resources screen, which now displays the group name for the resource. After you have created a resource group, you can indicate that group name as a resource parameter when you create or modify additional resources.

⁠11.6. Cluster Properties

Selecting the Cluster Properties option from the menu along the top of the cluster management page displays the cluster properties and allows you to modify these properties from their default values. For information on the Pacemaker cluster properties, see Chapter 10, Pacemaker Cluster Properties.

⁠Appendix A. Cluster Creation in Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7

Configuring a Red Hat High Availability Cluster in Red Hat Enterprise Linux 7 with Pacemaker requires a different set of configuration tools with a different administrative interface than configuring a cluster in Red Hat Enterprise Linux 6 with rgmanager. Section A.1, “Cluster Creation with rgmanager and with Pacemaker” summarizes the configuration differences between the various cluster components.

The Red Hat Enterprise Linux 6.5 release supports cluster configuration with Pacemaker, using the pcs configuration tool. Section A.2, “Cluster Creation with Pacemaker in Red Hat Enterprise Linux 6.5 and Red Hat Enterprise Linux 7” summarizes some small configuration differences between pcs support in Red Hat Enterprise Linux 6.5 and pcs support in Red Hat Enterprise Linux 7.0.

⁠A.1. Cluster Creation with rgmanager and with Pacemaker

Table A.1, “Comparison of Cluster Configuration with rgmanager and with Pacemaker” provides a comparative summary of how you configure the components of a cluster with rgmanager in Red Hat Enterprise Linux 6 and with Pacemaker in Red Hat Enterprise Linux 7.

⁠

Table A.1. Comparison of Cluster Configuration with rgmanager and with Pacemaker

Configuration Component
rgmanager
Pacemaker

Cluster configuration file

The cluster configuration file on each node is cluster.conf file, which can can be edited directly if desired. Otherwise, use the luci or ccs interface to define the cluster configuration.

The cluster and Pacemaker configuration files are corosync.conf and cib.xml. Do not edit these files directly; use the pcs or pcsd interface instead.

Network setup

Configure IP addresses and SSH before configuring the cluster.

Configure IP addresses and SSH before configuring the cluster.

Cluster Configuration Tools

luci, ccs command, manual editing of cluster.conf file.

pcs or pcsd.

Installation

Install rgmanager (which pulls in all dependencies, including ricci, luci, and the resource and fencing agents). If needed, install lvm2-cluster and gfs2-utils.

Install pcs, and the fencing agents you require. If needed, install lvm2-cluster and gfs2-utils.

Starting cluster services

Start and enable cluster services with the following procedure:

Start rgmanager, cman, and, if needed, clvmd and gfs2.

Start ricci, and start luci if using the luci interface.

Run chkconfig on for the needed services so that they start at each runtime.

Alternately, you can run ccs --start to start and enable the cluster services.

Start and enable cluster services with the following procedure:

On every node, execute systemctl start pcsd.service, then systemctl enable pcsd.service to enable pcsd to start at runtime.

On one node in the cluster, run pcs cluster start --all to start corosync and pacemaker.

Controlling access to configuration tools

For luci, the root user or a user with luci permissions can access luci. All access requires the ricci password for the node.

The pcsd gui requires that you authenticate as user hacluster, which is the common system user. The root user can set the password for hacluster.

Cluster creation

Name the cluster and define which nodes to include in the cluster with luci or ccs, or directly edit the cluster.conf file.

Name the cluster and include nodes with pcs cluster setup command or with the pcsd Web UI. You can add nodes to an existing cluster with the pcs cluster node add command or with the pcsd Web UI.

Propagating cluster configuration to all nodes

When configuration a cluster with luci, propagation is automatic. With ccs, use the --sync option. You can also use the cman_tool version -r command.

Propagation of the cluster and Pacemaker configuration files, corosync.conf and cib.xml, is automatic on cluster setup or when adding a node or resourcea

Global cluster properties

The following feature are supported with rgmanager in Red Hat Enterprise Linux 6:

* You can configure the system so that the system chooses which multicast address to use for IP multicasting in the cluster network.

* If IP multicasting is not available, you can use UDP Unicast transport mechanism.

* You can configure a cluster to use RRP protocol.

Pacemaker in RHEL 7 supports the following features for a cluster:

* You can set no-quorum-policy for the cluster to specify what the system should do when the cluster does not have quorum.

* For additional cluster properties you can set, refer to Table 10.1, “Cluster Properties”.

Logging

You can set global and daemon-specific logging configuration.

See the file /etc/sysconfig/pacemaker for information on how to configure logging manually.

Validating the cluster

Cluster validation is automatic with luci and with ccs, using the cluster schema. The cluster is automatically validated on startup.

The cluster is automatically validated on startup, or you can validate the cluster with pcs cluster verify.

Quorum in 2-node clusters

With a two-node cluster, you can configure how the system determines quorum:

* Configure a quorum disk

* Use ccs or edit the cluster.conf file to set two_node=1 and expected_votes=1 to allow a single node to maintain quorum.

pcs automatically adds the necessary options for a two-node cluster to corosync.

Cluster status

On luci, the current status of the cluster is visible in the various components of the interface, which can be refreshed. You can use the --getconf option of the ccs command to see current the configuration file. You can use the clustat command to display cluster status.

You can display the current cluster status with the pcs status command.

Resources

You add resources of defined types and configure resource-specific properties with luci or the ccs command, or by editing the cluster.conf configuration file.

You add resources of defined types and configure resource-specific properties with the pcs resource create command or with the pcsd Web UI. For general information on configuring cluster resources with Pacemaker refer to Chapter 5, Configuring Cluster Resources.

Resource behavior, grouping, and start/stop order

Define cluster services to configure how resources interact.

With Pacemaker, you use resource groups as a shorthand method of defining a set of resources that need to be located together and started and stopped sequentially. In addition, you define how resources behave and interact in the following ways:

* You set some aspects of resource behavior as resource options.

* You use location constraints to determine which nodes a resource can run on.

* You use order constraints to determine the order in which resources run.

* You use colocation constraints to determine that the location of one resource depends on the location of another resource.

For more complete information on these topics, refer to Chapter 5, Configuring Cluster Resources and Chapter 6, Resource Constraints.

Resource administration: Moving, starting, stopping resources

With luci, you can manage clusters, individual cluster nodes, and cluster services. With the ccs command, you can manage cluster. You can use the clusvadm to manage cluster services.

You can temporarily disable a node so that it can not host resources with the pcs cluster standby command, which causes the resources to migrate. You can stop a resource with the pcs resource disable command.

Removing a cluster configuration completely

With luci, you can select all nodes in a cluster for deletion to delete a cluster entirely. You can also remove the cluster.conf from each node in the cluster.

You can remove a cluster configuration with the pcs cluster destroy command.

Resources active on multiple nodes, resources active on multiple nodes in multiple modes

No equivalent.

With Pacemaker, you can clone resources so that they can run in multiple nodes, and you can define cloned resources as master and slave resources so that they can run in multiple modes. For information on cloned resources and master/slave resources, refer to Chapter 8, Advanced Resource types.

Fencing -- single fence device per node

Create fencing devices globally or locally and add them to nodes. You can define post-fail delay and post-join delay values for the cluster as a whole.

Create a fencing device for each node with the pcs stonith create command or with the pcsd Web UI. For devices that can fence multiple nodes, you need to define them only once rather than separately for each node. You can also define pcmk_host_map to configure fencing devices for all nodes with a single command; for information on pcmk_host_map refer to Table 4.1, “General Properties of Fencing Devices”. You can define the stonith-timeout value for the cluster as a whole.

Multiple (backup) fencing devices per node

Define backup devices with luci or the ccs command, or by editing the cluster.conf file directly.

Configure fencing levels.

⁠A.2. Cluster Creation with Pacemaker in Red Hat Enterprise Linux 6.5 and Red Hat Enterprise Linux 7

The Red Hat Enterprise Linux 6.5 release supports cluster configuration with Pacemaker, using the pcs configuration tool. There are, however, some differences in cluster installation and creation between Red Hat Enterprise Linux 6.5 and Red Hat Enterprise Linux 7 when using Pacemaker. This section provides a brief listing of the command differences between these two releases. For further information on installation and cluster creation in Red Hat Enterprise Linux 7, see Chapter 1, Red Hat High Availability Add-On Configuration and Management Reference Overview and Chapter 3, Cluster Creation and Administration.

⁠A.2.1. Pacemaker Installation in Red Hat Enterprise Linux 6.5 and Red Hat Enterprise Linux 7

The following commands install the Red Hat High Availability Add-On software packages that Pacemaker requires in Red Hat Enterprise Linux 6.5 and prevent corosync from starting without cman. You must run these commands on each node in the cluster.

[root@rhel6]# yum install pacemaker cman
[root@rhel6]# yum install pcs
[root@rhel6]# chkconfig corosync off

In Red Hat Enterprise Linux 7, in addition to installing the Red Hat High Availability Add-On software packages that Pacemaker requires, you set up a password for the pcs administration account named hacluster, and you start and enable the pcsd service, You also authenticate the administration account for the nodes of the cluster.

In Red Hat Enterprise Linux 7, run the following commands on each node in the cluster.

[root@rhel7]# yum install pcs fence-agents-all
[root@rhel7]# passwd hacluster
[root@rhel7]# systemctl start pcsd.service
[root@rhel7]# systemctl enable pcsd.service

In Red Hat Enterprise Linux 7, run the following command on one node in the cluster.

[root@rhel7]# pcs cluster auth [node] [...] [-u username] [-p password]

⁠A.2.2. Cluster Creation with Pacemaker in Red Hat Enterprise Linux 6.5 and Red Hat Enterprise Linux 7

To create a Pacemaker cluster in Red Hat Enterprise Linux 6.5, you must create the cluster and start the cluster services on each node in the cluster. For example, to create a cluster named my_cluster that consists of nodes z1.example.com and z2.example.com and start cluster services on those nodes, run the following commands from both z1.example.com and z2.example.com.

[root@rhel6]# pcs cluster setup --name my_cluster z1.example.com z2.example.com
[root@rhel6]# pcs cluster start

In Red Hat Enterprise Linux 7, you run the cluster creation command from one node of the cluster. The following command, run from one node only, creates the cluster named my_cluster that consists of nodes z1.example.com and z2.example.com and starts cluster services on those nodes.

[root@rhel7]# pcs cluster setup --start --name my_cluster z1.example.com z2.example.com

⁠Appendix B. Revision History

Revision History

Revision 2.1-9
Tue Dec 8 2015
Steven Levine

Resolves #1281638

Corrects typo in table of resource display commands

Revision 2.1-8
Mon Nov 9 2015
Steven Levine

Preparing document for 7.2 GA publication

Revision 2.1-7
Fri Oct 16 2015
Steven Levine

Resolves #1129854

Documents updated pcs resource cleanup command

Resolves #1267443

Fixes small errors and typos

Resolves #1245806

Documents pcs resource relocate run command

Resolves #1245809

Clarifies behavior of pcs resource move command

Resolves #1244872

Updates and expands documentation on event monitoring

Resolves #1129713

Adds warning regarding using quorum unblock

Resolves #1229826

Adds note regarding time delay following pcs cluster create

Resolves #1202956

Adds procedure for adding new nodes to a running cluster

Resolves #1211674

Removes incorrect reference to stonith-timeout veing valid as a per-device attribute

Resolves #129451

Documents resouce-discovery location constraint option

Resolves #1129851

Documents use of tooltips when using the pcsd Web UI

Revision 2.1-5
Mon Aug 24 2015
Steven Levine

Preparing document for 7.2 Beta publication.

Revision 1.1-9
Mon Feb 23 2015
Steven Levine

Version for 7.1 GA release

Revision 1.1-7
Thu Dec 11 2014
Steven Levine

Version for 7.1 Beta release

Revision 1.1-2
Tue Dec 2 2014
Steven Levine

Resolves #1129854

Documents updated pcs resource cleanup command.

Resolves #1129837

Documents lifetime parameter for pcs resource move command.

Resolves #1129862, #1129461

Documents support for access control lists.

Resolves #1069385

Adds port number to enable for DLM.

Resolves #1121128, #1129738

Updates and clarifies descriptions of set colocation and ordered sets.

Resolves #1129713

Documents cluster quorum unblock feature.

Resolves #1129722

Updates description of auto_tie_breaker quorum option.

Resolves #1129737

Documents disabled parameter for pcs resource create command.

Resolves #1129859

Documents pcs cluster configuration backup and restore.

Resolves #1098289

Small clarification to description of port requirements.

Revision 1.1-1
Tue Nov 18 2014
Steven Levine

Draft updates for 7.1 release

Revision 0.1-41
Mon Jun 2 2014
Steven Levine

Version for 7.0 GA release

Revision 0.1-39
Thu May 29 2014
Steven Levine

Resolves: #794494

Documents quorum support

Resolves: #1088465

Documents unfencing

Resolves: #987087

Documents pcs updates

Revision 0.1-23
Wed Apr 9 2014
Steven Levine

Updated 7.0 Beta draft

Revision 0.1-15
Fri Dec 6 2013
Steven Levine

Beta document

Revision 0.1-2
Thu May 16 2013
Steven Levine

First printing of initial draft

⁠Index

, Cluster Creation

A

Action

Property

enabled, Resource Operations

id, Resource Operations

interval, Resource Operations

name, Resource Operations

on-fail, Resource Operations

timeout, Resource Operations

Action Property, Resource Operations

attribute, Node Attribute Expressions

Constraint Expression, Node Attribute Expressions

Attribute Expression, Node Attribute Expressions

attribute, Node Attribute Expressions

operation, Node Attribute Expressions

type, Node Attribute Expressions

value, Node Attribute Expressions

B

batch-limit, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

boolean-op, Pacemaker Rules

Constraint Rule, Pacemaker Rules

C

Clone

Option

clone-max, Creating and Removing a Cloned Resource

clone-node-max, Creating and Removing a Cloned Resource

globally-unique, Creating and Removing a Cloned Resource

interleave, Creating and Removing a Cloned Resource

notify, Creating and Removing a Cloned Resource

ordered, Creating and Removing a Cloned Resource

Clone Option, Creating and Removing a Cloned Resource

Clone Resources, Resource Clones

clone-max, Creating and Removing a Cloned Resource

Clone Option, Creating and Removing a Cloned Resource

clone-node-max, Creating and Removing a Cloned Resource

Clone Option, Creating and Removing a Cloned Resource

Clones, Resource Clones

Cluster

Option

batch-limit, Summary of Cluster Properties and Options

cluster-delay, Summary of Cluster Properties and Options

cluster-infrastructure, Summary of Cluster Properties and Options

cluster-recheck-interval, Summary of Cluster Properties and Options

dc-version, Summary of Cluster Properties and Options

default-action-timeout, Summary of Cluster Properties and Options

default-resource-stickiness, Summary of Cluster Properties and Options

enable-acl, Summary of Cluster Properties and Options

is-managed-default, Summary of Cluster Properties and Options

last-lrm-refresh, Summary of Cluster Properties and Options

maintenance-mode, Summary of Cluster Properties and Options

migration-limit, Summary of Cluster Properties and Options

no-quorum-policy, Summary of Cluster Properties and Options

pe-error-series-max, Summary of Cluster Properties and Options

pe-input-series-max, Summary of Cluster Properties and Options

pe-warn-series-max, Summary of Cluster Properties and Options

shutdown-escalation, Summary of Cluster Properties and Options

start-failure-is-fatal, Summary of Cluster Properties and Options

stonith-action, Summary of Cluster Properties and Options

stonith-enabled, Summary of Cluster Properties and Options

stonith-timeout, Summary of Cluster Properties and Options

stop-all-resources, Summary of Cluster Properties and Options

stop-orphan-actions, Summary of Cluster Properties and Options

stop-orphan-resources, Summary of Cluster Properties and Options

symmetric-cluster, Summary of Cluster Properties and Options

Querying Properties, Querying Cluster Property Settings

Removing Properties, Setting and Removing Cluster Properties

Setting Properties, Setting and Removing Cluster Properties

Cluster Option, Summary of Cluster Properties and Options

Cluster Properties, Setting and Removing Cluster Properties, Querying Cluster Property Settings

cluster status

display, Displaying Cluster Status

cluster-delay, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

cluster-infrastructure, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

cluster-recheck-interval, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

Colocation, Colocation of Resources

Constraint

Attribute Expression, Node Attribute Expressions

attribute, Node Attribute Expressions

operation, Node Attribute Expressions

type, Node Attribute Expressions

value, Node Attribute Expressions

Date Specification, Date Specifications

hours, Date Specifications

id, Date Specifications

monthdays, Date Specifications

months, Date Specifications

moon, Date Specifications

weekdays, Date Specifications

weeks, Date Specifications

weekyears, Date Specifications

yeardays, Date Specifications

years, Date Specifications

Date/Time Expression, Time/Date Based Expressions

end, Time/Date Based Expressions

operation, Time/Date Based Expressions

start, Time/Date Based Expressions

Duration, Durations

Rule, Pacemaker Rules

boolean-op, Pacemaker Rules

role, Pacemaker Rules

score, Pacemaker Rules

score-attribute, Pacemaker Rules

Constraint Expression, Node Attribute Expressions, Time/Date Based Expressions

Constraint Rule, Pacemaker Rules

Constraints

Colocation, Colocation of Resources

Location

id, Location Constraints

score, Location Constraints

Order, Order Constraints

kind, Order Constraints

D

dampen, Moving Resources Due to Connectivity Changes

Ping Resource Option, Moving Resources Due to Connectivity Changes

Date Specification, Date Specifications

hours, Date Specifications

id, Date Specifications

monthdays, Date Specifications

months, Date Specifications

moon, Date Specifications

weekdays, Date Specifications

weeks, Date Specifications

weekyears, Date Specifications

yeardays, Date Specifications

years, Date Specifications

Date/Time Expression, Time/Date Based Expressions

end, Time/Date Based Expressions

operation, Time/Date Based Expressions

start, Time/Date Based Expressions

dc-version, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

default-action-timeout, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

default-resource-stickiness, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

Determine by Rules, Using Rules to Determine Resource Location

Determine Resource Location, Using Rules to Determine Resource Location

disabling

resources, Enabling and Disabling Cluster Resources

Duration, Durations

E

enable-acl, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

enabled, Resource Operations

Action Property, Resource Operations

enabling

resources, Enabling and Disabling Cluster Resources

end, Time/Date Based Expressions

Constraint Expression, Time/Date Based Expressions

F

failure-timeout, Resource Meta Options

Resource Option, Resource Meta Options

features, new and changed, New and Changed Features

G

globally-unique, Creating and Removing a Cloned Resource

Clone Option, Creating and Removing a Cloned Resource

Group Resources, Resource Groups

Groups, Resource Groups, Group Stickiness

H

host_list, Moving Resources Due to Connectivity Changes

Ping Resource Option, Moving Resources Due to Connectivity Changes

hours, Date Specifications

Date Specification, Date Specifications

I

id, Resource Properties, Resource Operations, Date Specifications

Action Property, Resource Operations

Date Specification, Date Specifications

Location Constraints, Location Constraints

Multi-State Property, Multi-State Resources: Resources That Have Multiple Modes

Resource, Resource Properties

interleave, Creating and Removing a Cloned Resource

Clone Option, Creating and Removing a Cloned Resource

interval, Resource Operations

Action Property, Resource Operations

is-managed, Resource Meta Options

Resource Option, Resource Meta Options

is-managed-default, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

K

kind, Order Constraints

Order Constraints, Order Constraints

L

last-lrm-refresh, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

Location

Determine by Rules, Using Rules to Determine Resource Location

score, Location Constraints

Location Constraints, Location Constraints

Location Relative to other Resources, Colocation of Resources

M

maintenance-mode, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

master-max, Multi-State Resources: Resources That Have Multiple Modes

Multi-State Option, Multi-State Resources: Resources That Have Multiple Modes

master-node-max, Multi-State Resources: Resources That Have Multiple Modes

Multi-State Option, Multi-State Resources: Resources That Have Multiple Modes

migration-limit, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

migration-threshold, Resource Meta Options

Resource Option, Resource Meta Options

monthdays, Date Specifications

Date Specification, Date Specifications

months, Date Specifications

Date Specification, Date Specifications

moon, Date Specifications

Date Specification, Date Specifications

Moving, Manually Moving Resources Around the Cluster

Resources, Manually Moving Resources Around the Cluster

Multi-state, Multi-State Resources: Resources That Have Multiple Modes

Multi-State, Multi-state Stickiness

Option

master-max, Multi-State Resources: Resources That Have Multiple Modes

master-node-max, Multi-State Resources: Resources That Have Multiple Modes

Property

id, Multi-State Resources: Resources That Have Multiple Modes

Multi-State Option, Multi-State Resources: Resources That Have Multiple Modes

Multi-State Property, Multi-State Resources: Resources That Have Multiple Modes

multiple-active, Resource Meta Options

Resource Option, Resource Meta Options

multiplier, Moving Resources Due to Connectivity Changes

Ping Resource Option, Moving Resources Due to Connectivity Changes

N

name, Resource Operations

Action Property, Resource Operations

no-quorum-policy, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

notify, Creating and Removing a Cloned Resource

Clone Option, Creating and Removing a Cloned Resource

O

on-fail, Resource Operations

Action Property, Resource Operations

operation, Node Attribute Expressions, Time/Date Based Expressions

Constraint Expression, Node Attribute Expressions, Time/Date Based Expressions

Option

batch-limit, Summary of Cluster Properties and Options

clone-max, Creating and Removing a Cloned Resource

clone-node-max, Creating and Removing a Cloned Resource

cluster-delay, Summary of Cluster Properties and Options

cluster-infrastructure, Summary of Cluster Properties and Options

cluster-recheck-interval, Summary of Cluster Properties and Options

dampen, Moving Resources Due to Connectivity Changes

dc-version, Summary of Cluster Properties and Options

default-action-timeout, Summary of Cluster Properties and Options

default-resource-stickiness, Summary of Cluster Properties and Options

enable-acl, Summary of Cluster Properties and Options

failure-timeout, Resource Meta Options

globally-unique, Creating and Removing a Cloned Resource

host_list, Moving Resources Due to Connectivity Changes

interleave, Creating and Removing a Cloned Resource

is-managed, Resource Meta Options

is-managed-default, Summary of Cluster Properties and Options

last-lrm-refresh, Summary of Cluster Properties and Options

maintenance-mode, Summary of Cluster Properties and Options

master-max, Multi-State Resources: Resources That Have Multiple Modes

master-node-max, Multi-State Resources: Resources That Have Multiple Modes

migration-limit, Summary of Cluster Properties and Options

migration-threshold, Resource Meta Options

multiple-active, Resource Meta Options

multiplier, Moving Resources Due to Connectivity Changes

no-quorum-policy, Summary of Cluster Properties and Options

notify, Creating and Removing a Cloned Resource

ordered, Creating and Removing a Cloned Resource

pe-error-series-max, Summary of Cluster Properties and Options

pe-input-series-max, Summary of Cluster Properties and Options

pe-warn-series-max, Summary of Cluster Properties and Options

priority, Resource Meta Options

requires, Resource Meta Options

resource-stickiness, Resource Meta Options

shutdown-escalation, Summary of Cluster Properties and Options

start-failure-is-fatal, Summary of Cluster Properties and Options

stonith-action, Summary of Cluster Properties and Options

stonith-enabled, Summary of Cluster Properties and Options

stonith-timeout, Summary of Cluster Properties and Options

stop-all-resources, Summary of Cluster Properties and Options

stop-orphan-actions, Summary of Cluster Properties and Options

stop-orphan-resources, Summary of Cluster Properties and Options

symmetric-cluster, Summary of Cluster Properties and Options

target-role, Resource Meta Options

Order

kind, Order Constraints

Order Constraints, Order Constraints

symmetrical, Order Constraints

ordered, Creating and Removing a Cloned Resource

Clone Option, Creating and Removing a Cloned Resource

Ordering, Order Constraints

overview

features, new and changed, New and Changed Features

P

pe-error-series-max, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

pe-input-series-max, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

pe-warn-series-max, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

Ping Resource

Option

dampen, Moving Resources Due to Connectivity Changes

host_list, Moving Resources Due to Connectivity Changes

multiplier, Moving Resources Due to Connectivity Changes

Ping Resource Option, Moving Resources Due to Connectivity Changes

priority, Resource Meta Options

Resource Option, Resource Meta Options

Property

enabled, Resource Operations

id, Resource Properties, Resource Operations, Multi-State Resources: Resources That Have Multiple Modes

interval, Resource Operations

name, Resource Operations

on-fail, Resource Operations

provider, Resource Properties

standard, Resource Properties

timeout, Resource Operations

type, Resource Properties

provider, Resource Properties

Resource, Resource Properties

Q

Querying

Cluster Properties, Querying Cluster Property Settings

Querying Options, Querying Cluster Property Settings

R

Removing

Cluster Properties, Setting and Removing Cluster Properties

Removing Properties, Setting and Removing Cluster Properties

requires, Resource Meta Options

Resource, Resource Properties

Constraint

Attribute Expression, Node Attribute Expressions

Date Specification, Date Specifications

Date/Time Expression, Time/Date Based Expressions

Duration, Durations

Rule, Pacemaker Rules

Constraints

Colocation, Colocation of Resources

Order, Order Constraints

Location

Determine by Rules, Using Rules to Determine Resource Location

Location Relative to other Resources, Colocation of Resources

Moving, Manually Moving Resources Around the Cluster

Option

failure-timeout, Resource Meta Options

is-managed, Resource Meta Options

migration-threshold, Resource Meta Options

multiple-active, Resource Meta Options

priority, Resource Meta Options

requires, Resource Meta Options

resource-stickiness, Resource Meta Options

target-role, Resource Meta Options

Property

id, Resource Properties

provider, Resource Properties

standard, Resource Properties

type, Resource Properties

Start Order, Order Constraints

Resource Option, Resource Meta Options

resource-stickiness, Resource Meta Options

Groups, Group Stickiness

Multi-State, Multi-state Stickiness

Resource Option, Resource Meta Options

Resources, Manually Moving Resources Around the Cluster

Clones, Resource Clones

Groups, Resource Groups

Multi-state, Multi-State Resources: Resources That Have Multiple Modes

resources

cleanup, Cluster Resources Cleanup

disabling, Enabling and Disabling Cluster Resources

enabling, Enabling and Disabling Cluster Resources

role, Pacemaker Rules

Constraint Rule, Pacemaker Rules

Rule, Pacemaker Rules

boolean-op, Pacemaker Rules

Determine Resource Location, Using Rules to Determine Resource Location

role, Pacemaker Rules

score, Pacemaker Rules

score-attribute, Pacemaker Rules

S

score, Location Constraints, Pacemaker Rules

Constraint Rule, Pacemaker Rules

Location Constraints, Location Constraints

score-attribute, Pacemaker Rules

Constraint Rule, Pacemaker Rules

Setting

Cluster Properties, Setting and Removing Cluster Properties

Setting Properties, Setting and Removing Cluster Properties

shutdown-escalation, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

standard, Resource Properties

Resource, Resource Properties

start, Time/Date Based Expressions

Constraint Expression, Time/Date Based Expressions

Start Order, Order Constraints

start-failure-is-fatal, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

status

display, Displaying Cluster Status

stonith-action, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

stonith-enabled, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

stonith-timeout, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

stop-all-resources, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

stop-orphan-actions, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

stop-orphan-resources, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

symmetric-cluster, Summary of Cluster Properties and Options

Cluster Option, Summary of Cluster Properties and Options

symmetrical, Order Constraints

Order Constraints, Order Constraints

T

target-role, Resource Meta Options

Resource Option, Resource Meta Options

Time Based Expressions, Time/Date Based Expressions

timeout, Resource Operations

Action Property, Resource Operations

type, Resource Properties, Node Attribute Expressions

Constraint Expression, Node Attribute Expressions

Resource, Resource Properties

V

value, Node Attribute Expressions

Constraint Expression, Node Attribute Expressions

W

weekdays, Date Specifications

Date Specification, Date Specifications

weeks, Date Specifications

Date Specification, Date Specifications

weekyears, Date Specifications

Date Specification, Date Specifications

Y

yeardays, Date Specifications

Date Specification, Date Specifications

years, Date Specifications

Date Specification, Date Specifications