DM-Multipath Administration and Troubleshooting
- Resizing an Online Multipath Device
- Moving root File Systems from a Single Path Device to a Multipath Device
- Moving swap File Systems from a Single Path Device to a Multipath Device
- The Multipath Daemon
- Issues with queue_if_no_path
- Multipath Command Output
- Multipath Queries with multipath Command
- Multipath Command Options
- Determining Device Mapper Entries with dmsetup Command
- Troubleshooting with the multipathd interactive console
Resizing an Online Multipath Device
If you need to resize an online multipath device, use the following procedure
-
Resize your physical device. This is storage platform specific.
-
Use the following command to find the paths to the LUN:
# multipath -l
-
Resize your paths. For SCSI devices, writing 1 to the rescan file for the device causes the SCSI driver to rescan, as in the following command:
# echo 1 > /sys/block/device_name/device/rescan
-
Resize your multipath device by running the multipathd resize command:
# multipathd -k 'resize map mpatha'
-
Resize the file system (assuming no LVM or DOS partitions are used):
# resize2fs /dev/mapper/mpatha
Moving root File Systems from a Single Path Device to a Multipath Device
This is dramatically simplified by the use of UUIDs to identify devices as an intrinsic label. Simply install multipath-tools-boot and reboot. This will rebuild the initial ramdisk and afford multipath the opportunity to build it's paths before the root file system is mounted by UUID.
Whenever multipath.conf is updated, so should the initrd by executing update-initramfs -u -k all. The reason being is multipath.conf is copied to the ramdisk and is integral to determining the available devices for grouping via it's blacklist and device sections.
Moving swap File Systems from a Single Path Device to a Multipath Device
The procedure is exactly the same as illustrated in the previous section called Moving root File Systems from a Single Path to a Multipath Device.
The Multipath Daemon
If you find you have trouble implementing a multipath configuration, you should ensure the multipath daemon is running as described in "Setting up DM-Multipath". The multipathd daemon must be running in order to use multipathd devices. Also see section Troubleshooting with the multipathd interactive console concerning interacting with multipathd as a debugging aid.
Issues with queue_if_no_path
If features "1 queue_if_no_path" is specified in the /etc/multipath.conf file, then any process that uses I/O will hang until one or more paths are restored. To avoid this, set the no_path_retry N parameter in the /etc/multipath.conf.
When you set the no_path_retry parameter, remove the features "1 queue_if_no_path" option from the /etc/multipath.conf file as well. If, however, you are using a multipathed device for which the features "1 queue_if_no_path" option is set as a compiled in default, as it is for many SAN devices, you must add features "0" to override this default. You can do this by copying the existing devices section, and just that section (not the entire file), from /usr/share/doc/multipath-tools/examples/multipath.conf.annotated.gz into /etc/multipath.conf and editing to suit your needs.
If you need to use the features "1 queue_if_no_path" option and you experience the issue noted here, use the dmsetup command to edit the policy at runtime for a particular LUN (that is, for which all the paths are unavailable). For example, if you want to change the policy on the multipath device mpathc from "queue_if_no_path" to "fail_if_no_path", execute the following command.
# dmsetup message mpathc 0 "fail_if_no_path"
You must specify the mpathN alias rather than the path
Multipath Command Output
When you create, modify, or list a multipath device, you get a printout of the current device setup. The format is as follows. For each multipath device:
action_if_any: alias (wwid_if_different_from_alias) dm_device_name_if_known vendor,product size=size features='features' hwhandler='hardware_handler' wp=write_permission_if_known
For each path group:
-+- policy='scheduling_policy' prio=prio_if_known status=path_group_status_if_known
For each path:
`- host:channel:id:lun devnode major:minor dm_status_if_known path_status online_status
For example, the output of a multipath command might appear as follows:
3600d0230000000000e13955cc3757800 dm-1 WINSYS,SF2372 size=269G features='0' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=1 status=active | `- 6:0:0:0 sdb 8:16 active ready running `-+- policy='round-robin 0' prio=1 status=enabled `- 7:0:0:0 sdf 8:80 active ready running
If the path is up and ready for I/O, the status of the path is ready or ghost. If the path is down, the status is faulty or shaky. The path status is updated periodically by the multipathd daemon based on the polling interval defined in the /etc/multipath.conf file.
The dm status is similar to the path status, but from the kernel's point of view. The dm status has two states: failed, which is analogous to faulty, and active which covers all other path states. Occasionally, the path state and the dm state of a device will temporarily not agree.
The possible values for online_status are running and offline. A status of offline means that the SCSI device has been disabled.
When a multipath device is being created or modified , the path group status, the dm device name, the write permissions, and the dm status are not known. Also, the features are not always correct
Multipath Queries with multipath Command
You can use the -l and -ll options of the multipath command to display the current multipath configuration. The -l option displays multipath topology gathered from information in sysfs and the device mapper. The -ll option displays the information the -l displays in addition to all other available components of the system.
When displaying the multipath configuration, there are three verbosity levels you can specify with the -v option of the multipath command. Specifying -v0 yields no output. Specifying -v1 outputs the created or updated multipath names only, which you can then feed to other tools such as kpartx. Specifying -v2 prints all detected paths, multipaths, and device maps.
The default verbosity level of multipath is 2 and can be globally modified by defining the verbosity attribute in the defaults section of multipath.conf.
The following example shows the output of a multipath -l command.
# multipath -l 3600d0230000000000e13955cc3757800 dm-1 WINSYS,SF2372 size=269G features='0' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=1 status=active | `- 6:0:0:0 sdb 8:16 active ready running `-+- policy='round-robin 0' prio=1 status=enabled `- 7:0:0:0 sdf 8:80 active ready running
The following example shows the output of a multipath -ll command.
# multipath -ll 3600d0230000000000e13955cc3757801 dm-10 WINSYS,SF2372 size=269G features='0' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=1 status=enabled | `- 19:0:0:1 sdc 8:32 active ready running `-+- policy='round-robin 0' prio=1 status=enabled `- 18:0:0:1 sdh 8:112 active ready running 3600d0230000000000e13955cc3757803 dm-2 WINSYS,SF2372 size=125G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 19:0:0:3 sde 8:64 active ready running `- 18:0:0:3 sdj 8:144 active ready running
Multipath Command Options
Table Useful multipath Command Options describes some options of the multipath command that you might find useful.
Option | Description |
---|---|
-l | Display the current multipath configuration gathered from sysfs and the device mapper. |
-ll | Display the current multipath configuration gathered from sysfs, the device mapper, and all other available components on the system. |
-f device | Remove the named multipath device. |
-F | Remove all unused multipath devices. |
Determining Device Mapper Entries with dmsetup Command
You can use the dmsetup command to find out which device mapper entries match the multipathed devices.
The following command displays all the device mapper devices and their major and minor numbers. The minor numbers determine the name of the dm device. For example, a minor number of 3 corresponds to the multipathed device /dev/dm-3.
# dmsetup ls mpathd (253, 4) mpathep1 (253, 12) mpathfp1 (253, 11) mpathb (253, 3) mpathgp1 (253, 14) mpathhp1 (253, 13) mpatha (253, 2) mpathh (253, 9) mpathg (253, 8) VolGroup00-LogVol01 (253, 1) mpathf (253, 7) VolGroup00-LogVol00 (253, 0) mpathe (253, 6) mpathbp1 (253, 10) mpathd (253, 5)
Troubleshooting with the multipathd interactive console
The multipathd -k command is an interactive interface to the multipathd daemon. Entering this command brings up an interactive multipath console. After entering this command, you can enter help to get a list of available commands, you can enter a interactive command, or you can enter CTRL-D to quit.
The multipathd interactive console can be used to troubleshoot problems you may be having with your system. For example, the following command sequence displays the multipath configuration, including the defaults, before exiting the console. See the IBM article "Tricks with Multipathd" for more examples.
# multipathd -k > > show config > > CTRL-D
The following command sequence ensures that multipath has picked up any changes to the multipath.conf,
# multipathd -k > > reconfigure > > CTRL-D
Use the following command sequence to ensure that the path checker is working properly.
# multipathd -k > > show paths > > CTRL-D
Commands can also be streamed into multipathd using stdin like so:
# echo 'show config' | multipathd -k