Optimizing Storage | Scaling and Performance Guide

Overview

Optimizing storage helps to minimize storage use across all resources. By optimizing storage, administrators help ensure that existing storage resources are working in an efficient manner.

General Storage Guidelines

The following table lists the available persistent storage technologies for OpenShift Origin.

Table 1. Available storage options
Storage type	Description	Examples
Block	Presented to the operating system (OS) as a block device Suitable for applications that need full control of storage and operate at a low level on files bypassing the file system Also referred to as a Storage Area Network (SAN) Non-shareable, which means that only one client at a time can mount an endpoint of this type	CNS/CRS GlusterFS ^[1] iSCSI, Fibre Channel, Ceph RBD, OpenStack Cinder, AWS EBS ^[1], Dell/EMC Scale.IO, VMware vSphere Volume, GCE Persistent Disk ^[1], Azure Disk
File	Presented to the OS as a file system export to be mounted Also referred to as Network Attached Storage (NAS) Concurrency, latency, file locking mechanisms, and other capabilities vary widely between protocols, implementations, vendors, and scales.	CNS/CRS GlusterFS ^[1], RHEL NFS, NetApp NFS ^[2] , Azure File, Vendor NFS, Vendor GlusterFS ^[3], Azure File, AWS EFS
Object	Accessible through a REST API endpoint Configurable for use in the OpenShift Origin Registry Applications must build their drivers into the application and/or container.	CNS/CRS GlusterFS ^[1], Ceph Object Storage (RADOS Gateway), OpenStack Swift, Aliyun OSS, AWS S3, Google Cloud Storage, Azure Blob Storage, Vendor S3 ^[3], Vendor Swift ^[3]

As of OpenShift Origin 3.6.1, Container-Native Storage (CNS) GlusterFS (a hyperconverged or cluster-hosted storage solution) and Container-Ready Storage (CRS) GlusterFS (an externally hosted storage solution) provides interfaces for block, file, and object storage for the purpose of the OpenShift Origin registry, logging, and metrics.

Storage Recommendations

The following table summarizes the recommended and configurable storage technologies for the given OpenShift Origin cluster application.

Table 2. Recommended and configurable storage technology
Storage type	ROX ^[4]	RWX ^[5]	Registry	Scaled registry	Metrics	Logging	Apps
Block	Yes ^[6]	No	Configurable	Not configurable	Recommended	Recommended	Recommended
File	Yes ^[6]	Yes	Configurable	Configurable	Configurable	Configurable	Recommended
Object	Yes	Yes	Recommended	Recommended	Not configurable	Not configurable	Not configurable ^[7]

A scaled registry is an OpenShift Origin registry where three or more pod replicas are running.

Specific Application Storage Recommendations

Registry

In a non-scaled/high-availability (HA) OpenShift Origin registry cluster deployment:

The preferred storage technology is object storage followed by block storage. The storage technology does not need to support RWX access mode.
The storage technology must ensure read-after-write consistency. All NAS storage (excluding CNS/CRS GlusterFS as it uses an object storage interface) are not recommended for OpenShift Origin Registry cluster deployment with production workloads.
While hostPath volumes are configurable for a non-scaled/HA OpenShift Origin Registry, they are not recommended for cluster deployment.

Corruption may occur when using NFS to back OpenShift Origin registry with production workloads.

Scaled Registry

In a scaled/HA OpenShift Origin registry cluster deployment:

The preferred storage technology is object storage. The storage technology must support RWX access mode and must ensure read-after-write consistency.
File storage and block storage are not recommended for a scaled/HA OpenShift Origin registry cluster deployment with production workloads.
All NAS storage (excluding CNS/CRS GlusterFS as it uses an object storage interface) are not recommended for OpenShift Origin Registry cluster deployment with production workloads.

Corruption may occur when using NFS to back OpenShift Origin scaled/HA registry with production workloads.

Metrics

In an OpenShift Origin hosted metrics cluster deployment:

The preferred storage technology is block storage.
It is not recommended to use NAS storage (excluding CNS/CRS GlusterFS as it uses a block storage interface from iSCSI) for a hosted metrics cluster deployment with production workloads.

Corruption may occur when using NFS to back a hosted metrics cluster deployment with production workloads.

Logging

In an OpenShift Origin hosted logging cluster deployment:

The preferred storage technology is block storage.
It is not recommended to use NAS storage (excluding CNS/CRS GlusterFS as it uses a block storage interface from iSCSI) for a hosted metrics cluster deployment with production workloads.

Corruption may occur when using NFS to back hosted logging with production workloads.

Applications

Application use cases vary from application to application, as described in the following examples:

Storage technologies that support dynamic PV provisioning have low mount time latencies, and are not tied to nodes to support a healthy cluster.
NFS does not guarantee read-after-write consistency and is not recommended for applications which require it.
Applications that depend on writing to the same, shared NFS export may experience issues with production workloads.

Other Specific Application Storage Recommendations

OpenShift Origin Internal etcd: For the best etcd reliability, the lowest consistent latency storage technology is preferable.
OpenStack Cinder: OpenStack Cinder tends to be adept in ROX access mode use cases.
Databases: Databases (RDBMSs, NoSQL DBs, etc.) tend to perform best with dedicated block storage.

Choosing a Docker Graph Driver

Docker stores images and containers in a graph driver (a pluggable storage backend), such as Device Mapper, Overlay, and Btrfs. Each have advantages and disadvantages. For example, Overlay is faster than Device Mapper at starting and stopping containers, but is not POSIX compliant because of the architectural limitations of a union file system, and does not yet support SELinux.

For more information about Overlay, including supportability and usage caveats, see the RHEL 7.3 Release Notes.

In production environments, using a LVM thin pool on top of regular block devices (not loop devices) for container images and container root file systems storage is recommended.

Using a Loop device back-end can affect performance issues. While you can still continue to use it, Docker logs a warning message. For example:

devmapper: Usage of loopback devices is strongly discouraged for production use.
Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to
dm.thinpooldev section.

To ease Docker backend storage configuration, use the docker-storage-setup utility, which automates much of the configuration details:

If you had a separate disk drive dedicated to Docker storage (for example, /dev/xvdb), add the following to the /etc/sysconfig/docker-storage-setup file:
```
DEVS=/dev/xvdb
VG=docker_vg
```
Restart the docker-storage-setup service:
```
# systemctl restart docker-storage-setup
```
After the restart, docker-storage-setup sets up a volume group named docker_vg and creates a thin pool logical volume. Documentation for thin provisioning on RHEL is available in the LVM Administrator Guide. View the newly created volumes with the lsblk command:
```
# lsblk /dev/xvdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvdb 202:16 0 20G 0 disk
└─xvdb1 202:17 0 10G 0 part
  ├─docker_vg-docker--pool_tmeta 253:0 0 12M 0 lvm
  │ └─docker_vg-docker--pool 253:2 0 6.9G 0 lvm
  └─docker_vg-docker--pool_tdata 253:1 0 6.9G 0 lvm
  └─docker_vg-docker--pool 253:2 0 6.9G 0 lvm
```
Thin-provisioned volumes are not mounted and have no file system (individual containers do have an XFS file system), thus they will not show up in “df” output.

To verify that Docker is using a LVM thin pool, and to monitor disk space utilization, use the docker info command. The Pool Name will correspond with the VG you specified in /etc/sysconfig/docker-storage-setup:

# docker info | egrep -i 'storage|pool|space|filesystem'
Storage Driver: devicemapper
 Pool Name: docker_vg-docker--pool
 Pool Blocksize: 524.3 kB
 Backing Filesystem: xfs
 Data Space Used: 62.39 MB
 Data Space Total: 6.434 GB
 Data Space Available: 6.372 GB
 Metadata Space Used: 40.96 kB
 Metadata Space Total: 16.78 MB
 Metadata Space Available: 16.74 MB

By default, a thin pool is configured to use 40% of the underlying block device. As you use the storage, LVM automatically extends the thin pool up to 100%. This is why the Data Space Total value does not match the full size of the underlying LVM device. This auto-extend technique was used to unify the storage approach taken in both Red Hat Enterprise Linux and Red Hat Atomic Host, which only uses a single partition.

In development, Docker in Red Hat distributions defaults to a loopback mounted sparse file. To see if your system is using the loopback mode:

# docker info|grep loop0
 Data file: /dev/loop0

Red Hat strongly recommends using the Device Mapper storage driver in thin pool mode for production workloads.

Overlay is also supported for Docker use cases as of Red Hat Enterprise Linux 7.2, and provides faster start up time and page cache sharing, which can potentially improve density by reducing overall memory utilization.

1. CNS/CRS GlusterFS, Ceph RBD, OpenStack Cinder, AWS EBS, Azure Disk, GCE persistent disk, and VMware vSphere support dynamic persistent volume (PV) provisioning natively in OpenShift Origin.

2. NetApp NFS supports dynamic PV provisioning when using the Trident plugin.

3. Vendor GlusterFS, Vendor S3, and Vendor Swift supportability and configurability may vary.

4. ReadOnlyMany

5. ReadWriteMany

6. This does not apply to physical disk, VM physical disk, VMDK, loopback over NFS, AWS EBS, and Azure Disk.

7. Object storage is not consumed through OpenShift Origin’s PVs/persistent volume claims (PVCs). Apps must integrate with the object storage REST API.