Dell EMC VMAX iSCSI and FC drivers¶

Masking view names¶

Masking views are dynamically created by the VMAX FC and iSCSI drivers using the following naming conventions. [protocol] is either I for volumes attached over iSCSI or F for volumes attached over Fiber Channel.

VMAX All Flash and Hybrid

OS-[shortHostName]-[SRP]-[SLO]-[workload]-[protocol]-MV

Initiator group names¶

For each host that is attached to VMAX volumes using the drivers, an initiator group is created or re-used (per attachment type). All initiators of the appropriate type known for that host are included in the group. At each new attach volume operation, the VMAX driver retrieves the initiators (either WWNNs or IQNs) from OpenStack and adds or updates the contents of the Initiator Group as required. Names are of the following format. [protocol] is either I for volumes attached over iSCSI or F for volumes attached over Fiber Channel.

OS-[shortHostName]-[protocol]-IG

FA port groups¶

VMAX array FA ports to be used in a new masking view are chosen from the list provided in the EMC configuration file.

Storage group names¶

As volumes are attached to a host, they are either added to an existing storage group (if it exists) or a new storage group is created and the volume is then added. Storage groups contain volumes created from a pool (either single-pool or FAST-controlled), attached to a single host, over a single connection type (iSCSI or FC). [protocol] is either I for volumes attached over iSCSI or F for volumes attached over Fiber Channel.

VMAX All Flash and Hybrid

OS-[shortHostName]-[SRP]-[SLO]-[Workload]-[protocol]-SG

Interval and Retries¶

By default, Interval and Retries are 10 seconds and 60 retries respectively. These determine how long (Interval) and how many times (Retries) a user is willing to wait for a single SMIS call, 10*60=300seconds. Depending on usage, these may need to be overriden by the user in the XML file. For example, if performance is a factor, then the Interval should be decreased to check the job status more frequently, and if multiple concurrent provisioning requests are issued then Retries should be increased so calls will not timeout prematurely.

In the example below, the driver checks every 5 seconds for the status of the job. It will continue checking for 120 retries before it times out.

Add the following lines to the XML file:

VMAX All Flash and Hybrid

<?xml version="1.0" encoding="UTF-8" ?>
<EMC>
  <EcomServerIp>1.1.1.1</EcomServerIp>
  <EcomServerPort>00</EcomServerPort>
  <EcomUserName>user1</EcomUserName>
  <EcomPassword>password1</EcomPassword>
  <PortGroups>
    <PortGroup>OS-PORTGROUP1-PG</PortGroup>
    <PortGroup>OS-PORTGROUP2-PG</PortGroup>
  </PortGroups>
  <Array>111111111111</Array>
  <Pool>SRP_1</Pool>
  <Interval>5</Interval>
  <Retries>120</Retries>
</EMC>

Scenario 1¶

EMCMaxSubscriptionPercent is 200 and the user defined max_over_subscription_ratio is 2.5, the latter is ignored. Oversubscription is 200%.

Scenario 2¶

EMCMaxSubscriptionPercent is 200 and the user defined max_over_subscription_ratio is 1.5, 1.5 equates to 150% and is less than the value set on the pool. Oversubscription is 150%.

Scenario 3¶

EMCMaxSubscriptionPercent is 0. This means there is no upper limit on the pool. The user defined max_over_subscription_ratio is 1.5. Oversubscription is 150%.

Scenario 4¶

EMCMaxSubscriptionPercent is 0. max_over_subscription_ratio is not set by the user. We recommend to default to upper limit, this is 150%.

Scenario 5¶

EMCMaxSubscriptionPercent is 200 on one pool. It is 300 on another pool. The user defined max_over_subscription_ratio is 2.5. Oversubscription is 200% on the first pool and 250% on the other.

USE CASE 1 - Default values¶

Prerequisites - VMAX

• Host I/O Limit (MB/Sec) - No Limit
• Host I/O Limit (IO/Sec) - No Limit
• Set Dynamic Distribution - N/A

1. Create QoS Specs with the prerequisite values above:

   $ openstack volume qos create --property maxIOPS=4000 maxMBPS=4000 DistributionType=Always SILVER
   

2. Associate QoS specs with specified volume type:

   $ openstack volume qos associate SILVER VOLUME_TYPE
   

3. Create volume with the volume type indicated above:

   $ openstack volume create --size 1 --type VOLUME_TYPE TEST_VOLUME
   

Outcome - VMAX (storage group)

• Host I/O Limit (MB/Sec) - 4000
• Host I/O Limit (IO/Sec) - 4000
• Set Dynamic Distribution - Always

Outcome - Block Storage (cinder)

Volume is created against volume type and QoS is enforced with the parameters above.

USE CASE 2 - Preset limits¶

Prerequisites - VMAX

• Host I/O Limit (MB/Sec) - 2000
• Host I/O Limit (IO/Sec) - 2000
• Set Dynamic Distribution - Never

1. Create QoS specifications with the prerequisite values above:

   $ openstack volume qos create --property maxIOPS=4000 maxMBPS=4000 DistributionType=Always SILVER
   

2. Associate QoS specifications with specified volume type:

   $ openstack volume qos associate SILVER VOLUME_TYPE
   

3. Create volume with the volume type indicated above:

   $ openstack volume create --size 1 --type VOLUME_TYPE TEST_VOLUME
   

Outcome - VMAX (storage group)

• Host I/O Limit (MB/Sec) - 4000
• Host I/O Limit (IO/Sec) - 4000
• Set Dynamic Distribution - Always

Outcome - Block Storage (cinder)

Volume is created against volume type and QoS is enforced with the parameters above.

USE CASE 3 - Preset limits¶

Prerequisites - VMAX

• Host I/O Limit (MB/Sec) - No Limit
• Host I/O Limit (IO/Sec) - No Limit
• Set Dynamic Distribution - N/A

1. Create QoS specifications with the prerequisite values above:

   $ openstack volume qos create --property DistributionType=Always SILVER
   

2. Associate QoS specifications with specified volume type:

   $ openstack volume qos associate SILVER VOLUME_TYPE
   

3. Create volume with the volume type indicated above:

   $ openstack volume create --size 1 --type VOLUME_TYPE TEST_VOLUME
   

Outcome - VMAX (storage group)

• Host I/O Limit (MB/Sec) - No Limit
• Host I/O Limit (IO/Sec) - No Limit
• Set Dynamic Distribution - N/A

Outcome - Block Storage (cinder)

Volume is created against volume type and there is no QoS change.

USE CASE 4 - Preset limits¶

Prerequisites - VMAX

• Host I/O Limit (MB/Sec) - No Limit
• Host I/O Limit (IO/Sec) - No Limit
• Set Dynamic Distribution - N/A

1. Create QoS specifications with the prerequisite values above:

   $ openstack volume qos create --property DistributionType=OnFailure SILVER
   

2. Associate QoS specifications with specified volume type:

   $ openstack volume qos associate SILVER VOLUME_TYPE
   

3. Create volume with the volume type indicated above:

   $ openstack volume create --size 1 --type VOLUME_TYPE TEST_VOLUME
   

Outcome - VMAX (storage group)

• Host I/O Limit (MB/Sec) - No Limit
• Host I/O Limit (IO/Sec) - No Limit
• Set Dynamic Distribution - N/A

Outcome - Block Storage (cinder)

Volume is created against volume type and there is no QoS change.

Multipath configuration file¶

The multipath configuration file may be edited for better management and performance. Log in as a privileged user and make the following changes to /etc/multipath.conf on the Compute (nova) node(s).

devices {
# Device attributed for EMC VMAX
    device {
            vendor "EMC"
            product "SYMMETRIX"
            path_grouping_policy multibus
            getuid_callout "/lib/udev/scsi_id --page=pre-spc3-83 --whitelisted --device=/dev/%n"
            path_selector "round-robin 0"
            path_checker tur
            features "0"
            hardware_handler "0"
            prio const
            rr_weight uniform
            no_path_retry 6
            rr_min_io 1000
            rr_min_io_rq 1
    }
}

You may need to reboot the host after installing the MPIO tools or restart iSCSI and multipath services.

On Ubuntu:

# service open-iscsi restart
# service multipath-tools restart

On openSUSE, SUSE Linux Enterprise Server, Red Hat Enterprise Linux, and CentOS:

# systemctl restart open-iscsi
# systemctl restart multipath-tools

$ lsblk
NAME                                       MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                          8:0    0     1G  0 disk
..360000970000196701868533030303235 (dm-6) 252:6    0     1G  0 mpath
sdb                                          8:16   0     1G  0 disk
..360000970000196701868533030303235 (dm-6) 252:6    0     1G  0 mpath
vda                                        253:0    0     1T  0 disk

OpenStack configurations¶

On Compute (nova) node, add the following flag in the [libvirt] section of /etc/nova/nova.conf:

iscsi_use_multipath = True

On cinder controller node, set the multipath flag to true in /etc/cinder/cinder.conf:

use_multipath_for_image_xfer = True

Restart nova-compute and cinder-volume services after the change.

Verify you have multiple initiators available on the compute node for I/O¶

1. Create a 3GB VMAX volume.

2. Create an instance from image out of native LVM storage or from VMAX storage, for example, from a bootable volume

3. Attach the 3GB volume to the new instance:

   $ multipath -ll
   mpath102 (360000970000196700531533030383039) dm-3 EMC,SYMMETRIX
   size=3G features='1 queue_if_no_path' hwhandler='0' wp=rw
   '-+- policy='round-robin 0' prio=1 status=active
   33:0:0:1 sdb 8:16 active ready running
   '- 34:0:0:1 sdc 8:32 active ready running
   

4. Use the lsblk command to see the multipath device:

   $ lsblk
   NAME                                       MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   sdb                                          8:0    0     3G  0 disk
   ..360000970000196700531533030383039 (dm-6) 252:6    0     3G  0 mpath
   sdc                                          8:16   0     3G  0 disk
   ..360000970000196700531533030383039 (dm-6) 252:6    0     3G  0 mpath
   vda
   

Operations¶

• Create a Consistency Group:

  cinder --os-volume-api-version 2 consisgroup-create [--name <name>]
  [--description <description>] [--availability-zone <availability-zone>]
  <volume-types>
  

  $ cinder --os-volume-api-version 2 consisgroup-create --name bronzeCG2 volume_type_1
  

• List Consistency Groups:

  cinder consisgroup-list [--all-tenants [<0|1>]]
  

  $ cinder consisgroup-list
  

• Show a Consistency Group:

  cinder consisgroup-show <consistencygroup>
  

  $ cinder consisgroup-show 38a604b7-06eb-4202-8651-dbf2610a0827
  

• Update a consistency Group:

  cinder consisgroup-update [--name <name>] [--description <description>]
  [--add-volumes <uuid1,uuid2,......>] [--remove-volumes <uuid3,uuid4,......>]
  <consistencygroup>
  

  Change name:

  $ cinder consisgroup-update --name updated_name 38a604b7-06eb-4202-8651-dbf2610a0827
  

  Add volume(s) to a Consistency Group:

  $ cinder consisgroup-update --add-volumes af1ae89b-564b-4c7f-92d9-c54a2243a5fe 38a604b7-06eb-4202-8651-dbf2610a0827
  

  Delete volume(s) from a Consistency Group:

  $ cinder consisgroup-update --remove-volumes af1ae89b-564b-4c7f-92d9-c54a2243a5fe 38a604b7-06eb-4202-8651-dbf2610a0827
  

• Create a snapshot of a Consistency Group:

  cinder cgsnapshot-create [--name <name>] [--description <description>]
  <consistencygroup>
  

  $ cinder cgsnapshot-create 618d962d-2917-4cca-a3ee-9699373e6625
  

• Delete a snapshot of a Consistency Group:

  cinder cgsnapshot-delete <cgsnapshot> [<cgsnapshot> ...]
  

  $ cinder cgsnapshot-delete 618d962d-2917-4cca-a3ee-9699373e6625
  

• Delete a Consistency Group:

  cinder consisgroup-delete [--force] <consistencygroup> [<consistencygroup> ...]
  

  $ cinder consisgroup-delete --force 618d962d-2917-4cca-a3ee-9699373e6625
  

• Create a Consistency group from source:

  cinder consisgroup-create-from-src [--cgsnapshot <cgsnapshot>]
  [--source-cg <source-cg>] [--name <name>] [--description <description>]
  

  $ cinder consisgroup-create-from-src --source-cg 25dae184-1f25-412b-b8d7-9a25698fdb6d
  

  $ cinder consisgroup-create-from-src --cgsnapshot 618d962d-2917-4cca-a3ee-9699373e6625
  

• You can also create a volume in a consistency group in one step:

  $ openstack volume create [--consistency-group consistency-group>]
    [--description <description>]  [--type <volume-type>]
    [--availability-zone <availability-zone>]  [--size <size>] <name>
  

  $ openstack volume create --type volume_type_1 ----consistency-group \
    1de80c27-3b2f-47a6-91a7-e867cbe36462 --size 1 cgBronzeVol
  

Enabling WLP on Unisphere¶

1. To enable WLP on Unisphere, click on the array‣Performance‣Settings.
2. Set both the Real Time and the Root Cause Analysis.
3. Click Register.

Using TestSmiProvider to add statistics access point¶

After enabling WLP you must then enable SMI-S to gain access to the WLP data:

1. Connect to the SMI-S Provider using TestSmiProvider.

2. Navigate to the Active menu.

3. Type reg and enter the noted responses to the questions:

   (EMCProvider:5989) ? reg
   Current list of statistics Access Points: ?
   Note: The current list will be empty if there are no existing Access Points.
   Add Statistics Access Point {y|n} [n]: y
   HostID [l2se0060.lss.emc.com]: ?
   Note: Enter the Unisphere for VMAX location using a fully qualified Host ID.
   Port [8443]: ?
   Note: The Port default is the Unisphere for VMAX default secure port. If the secure port
   is different for your Unisphere for VMAX setup, adjust this value accordingly.
   User [smc]: ?
   Note: Enter the Unisphere for VMAX username.
   Password [smc]: ?
   Note: Enter the Unisphere for VMAX password.
   

4. Type reg again to view the current list:

   (EMCProvider:5988) ? reg
   Current list of statistics Access Points:
   HostIDs:
   l2se0060.lss.emc.com
   PortNumbers:
   8443
   Users:
   smc
   Add Statistics Access Point {y|n} [n]: n
   

Enable attach and detach snapshot functionality¶

1. Ensure that the cinder-backup service is running.

2. The backup driver for the swift back end performs a volume backup to an object storage system. To enable the swift backup driver, include the following option in the cinder.conf file:

   backup_driver = cinder.backup.drivers.swift
   

3. In order to force the volume to run attach and detach on the snapshot and not the volume you need to put the following key-value pair in the [DEFAULT] section of the cinder.conf:

   backup_use_same_host = True
   

rpc_response_timeout = 240

Use case 1 - Create a volume backup when the volume is in-use¶

1. Create a bootable volume and launch it so the volume status is in-use.

2. Create a backup of the volume, where VOLUME is the volume name or volume ID. This will initiate a snapshot attach and a snapshot detach on a temporary snapshot:

   openstack backup create --force VOLUME
   

3. For example:

   openstack backup create --force cba1ca83-b857-421a-87c3-df81eb9ea8ab
   

Use case 2 - Restore a backup of a volume¶

1. Restore the backup from Use case 1, where BACKUP_ID is the identifier of the backup from Use case 1.

   openstack backup restore BACKUP_ID
   

2. For example:

   openstack backup restore ec7e17ec-ae3c-4495-9ee6-7f45c9a89572
   

Once complete, launch the back up as an instance, and it should be a bootable volume.

Use case 3 - Create a backup of a snapshot¶

1. Create a volume.

2. Create a snapshot of the volume.

3. Create a backup of the snapshot, where VOLUME is the volume name or volume ID, SNAPSHOT_ID is the ID of the volume’s snapshot. This will initiate a snapshot attach and a snapshot detach on the snapshot.

   openstack backup create [--snapshot SNAPSHOT_ID} VOLUME
   

4. For example:

   openstack backup create --snapshot 6ab440c2-80ef-4f16-ac37-2d9db938732c 9fedfc4a-5f25-4fa1-8d8d-d5bec91f72e0
   

Use case 4 - Restore backup of a snapshot¶

1. Restore the backup where BACKUP_ID is the identifier of the backup from Use case 3.

   openstack backup restore BACKUP_ID
   

2. For example:

   openstack backup restore ec7e17ec-ae3c-4495-9ee6-7f45c9a89572
   

Use case 1 - Compression disabled create, attach, detach, and delete volume¶

1. Create a new volume type called VMAX_COMPRESSION_DISABLED.
2. Set an extra spec volume_backend_name.
3. Set a new extra spec storagetype:disablecompression = True.
4. Create a new volume.
5. Check in Unisphere or symcli to see if the volume exists in storage group OS-<srp>-<servicelevel>-<workload>-CD-SG, and compression is disabled on that storage group.
6. Attach the volume to an instance. Check in Unisphere or symcli to see if the volume exists in storage group OS-<shorthostname>-<srp>-<servicelevel>-<workload>-CD-SG, and compression is disabled on that storage group.
7. Detach volume from instance. Check in Unisphere or symcli to see if the volume exists in storage group OS-<srp>-<servicelevel>-<workload>-CD-SG, and compression is disabled on that storage group.
8. Delete the volume. If this was the last volume in the OS-<srp>-<servicelevel>-<workload>-CD-SG storage group, it should also be deleted.

Use case 2 - Compression disabled create, delete snapshot and delete volume¶

1. Repeat steps 1-5 of Use case 1.
2. Create a snapshot. The volume should now exist in OS-<srp>-<servicelevel>-<workload>-CD-SG.
3. Delete the snapshot. The volume should be removed from OS-<srp>-<servicelevel>-<workload>-CD-SG.
4. Delete the volume. If this volume is the last volume in OS-<srp>-<servicelevel>-<workload>-CD-SG, it should also be deleted.

Use case 3 - Retype from compression disabled to compression enabled¶

1. Repeat steps 1-4 of Use case 1.
2. Create a new volume type. For example VMAX_COMPRESSION_ENABLED.
3. Set extra spec volume_backend_name as before.
4. Set the new extra spec’s compression as storagetype:disablecompression = False or DO NOT set this extra spec.
5. Retype from volume type VAX_COMPRESSION_DISABLED to VMAX_COMPRESSION_ENABLED.
6. Check in Unisphere or symcli to see if the volume exists in storage group OS-<srp>-<servicelevel>-<workload>-SG, and compression is enabled on that storage group.

Configure the source and target arrays¶

1. Configure a synchronous SRDF group between the chosen source and target arrays for the VMAX cinder driver to use. The source array must correspond with the <Array> entry in the VMAX XML file.

2. Select both the director and the ports for the SRDF emulation to use on both sides. Bear in mind that network topology is important when choosing director endpoints. Currently, the only supported mode is Synchronous.

   Note

   For full failover functionality, the source and target VMAX arrays must be discovered and managed by the same SMI-S/ECOM server, locally connected for example. This SMI-S/ ECOM server cannot be embedded - it can be installed on a physical server or a VM hosted by an ESX server only.

   Note

   With both arrays being managed by the one SMI-S server, it is the cloud storage administrators responsibility to account for a DR scenario where the management (SMI-S) server goes down as well as the primary array. In that event, the details and credentials of a back-up SMI-S server can be passed in to the XML file, and the VMAX cinder driver can be rebooted. It would be advisable to have the SMI-S server at a third location (separate from both arrays) if possible.

   Note

   If the source and target arrays are not managed by the same management server (that is, the target array is remotely connected to server), in the event of a full disaster scenario (for example, the primary array is completely lost and all connectivity to it is gone), the SMI-S server would no longer be able to contact the target array. In this scenario, the volumes would be automatically failed over to the target array, but administrator intervention would be required to either; configure the target (remote) array as local to the current SMI-S server, or enter the details to the XML file of a second SMI-S server, which is locally connected to the target array, and restart the cinder volume service.

3. Enable replication in /etc/cinder/cinder.conf. To enable the replication functionality in VMAX cinder driver, it is necessary to create a replication volume-type. The corresponding back-end stanza in the cinder.conf for this volume-type must then include a replication_device parameter. This parameter defines a single replication target array and takes the form of a list of key value pairs.

   enabled_backends = VMAX_FC_REPLICATION
   [VMAX_FC_REPLICATION]
   volume_driver = cinder.volume.drivers.emc.emc_vmax_FC.EMCVMAXFCDriver
   cinder_emc_config_file = /etc/cinder/cinder_emc_config_VMAX_FC_REPLICATION.xml
   volume_backend_name = VMAX_FC_REPLICATION
   replication_device = target_device_id:000197811111, remote_port_group:os-failover-pg, remote_pool:SRP_1, rdf_group_label: 28_11_07, allow_extend:False
   

   • target_device_id is a unique VMAX array serial number of the target array. For full failover functionality, the source and target VMAX arrays must be discovered and managed by the same SMI-S/ECOM server. That is, locally connected. Follow the instructions in the SMI-S release notes.
   • remote_port_group is the name of a VMAX port group that has been pre-configured to expose volumes managed by this backend in the event of a failover. Make sure that this portgroup contains either all FC or all iSCSI port groups (for a given back end), as appropriate for the configured driver (iSCSI or FC).
   • remote_pool is the unique pool name for the given target array.
   • rdf_group_label is the name of a VMAX SRDF group (Synchronous) that has been pre-configured between the source and target arrays.
   • allow_extend is a flag for allowing the extension of replicated volumes. To extend a volume in an SRDF relationship, this relationship must first be broken, both the source and target volumes are then independently extended, and then the replication relationship is re-established. As the SRDF link must be severed, due caution should be exercised when performing this operation. If not explicitly set, this flag defaults to False.

   Note

   Service Level and Workload: An attempt will be made to create a storage group on the target array with the same service level and workload combination as the primary. However, if this combination is unavailable on the target (for example, in a situation where the source array is a Hybrid, the target array is an All Flash, and an All Flash incompatible SLO like Bronze is configured), no SLO will be applied.

   Note

   The VMAX cinder drivers can support a single replication target per back-end, that is we do not support Concurrent SRDF or Cascaded SRDF. Ensure there is only a single .replication_device. entry per back-end stanza.

4. Create a replication-enabled volume type. Once the replication_device parameter has been entered in the VMAX backend entry in the cinder.conf, a corresponding volume type needs to be created replication_enabled property set. See above Setup VMAX drivers for details.

   $ openstack volume type set --property replication_enabled = ``<is> True`` VMAX_FC_REPLICATION
   

Volume replication interoperability with other features¶

Most features are supported, except for the following:

• There is no OpenStack Consistency Group or Generic Volume Group support for replication-enabled VMAX volumes.

• Storage-assisted retype operations on replication-enabled VMAX volumes (moving from a non-replicated type to a replicated-type and vice-versa. Moving to another SLO/workload combination, for example) are not supported.

• The image volume cache functionality is supported (enabled by setting image_volume_cache_enabled = True), but one of two actions must be taken when creating the cached volume:

  • The first boot volume created on a backend (which will trigger the cached volume to be created) should be the smallest necessary size. For example, if the minimum size disk to hold an image is 5GB, create the first boot volume as 5GB.
  • Alternatively, ensure that the allow_extend option in the replication_device parameter is set to True.

  This is because the initial boot volume is created at the minimum required size for the requested image, and then extended to the user specified size.

Failover host¶

In the event of a disaster, or where there is required downtime, upgrade of the primary array for example, the administrator can issue the failover host command to failover to the configured target:

$ cinder failover-host cinder_host@VMAX_FC_REPLICATION#Diamond+SRP_1+000192800111

If the primary array becomes available again, you can initiate a failback using the same command and specifying --backend_id default:

$ cinder failover-host \
  cinder_host@VMAX_FC_REPLICATION#Diamond+SRP_1+000192800111 \
  --backend_id default