1. Documentation
×

Example: Containerized Heketi for managing dedicated GlusterFS storage

Overview

This example provides information about the integration, deployment, and management of GlusterFS containerized storage nodes by using Heketi running on OpenShift Container Platform.

This example:

  • Shows how to install and configure a Heketi server on OpenShift to perform dynamic provisioning.

  • Assumes you have familiarity with Kubernetes and the Kubernetes Persistent Storage model.

  • Assumes you have access to an existing, dedicated GlusterFS cluster that has raw devices available for consumption and management by a Heketi server. If you do not have this, you can create a three node cluster using your virtual machine solution of choice. Ensure sure you create a few raw devices and give plenty of space (at least 100GB recommended). See Red Hat Gluster Storage Installation Guide.

Environment and Prerequisites

This example uses the following environment and prerequisites:

  • GlusterFS cluster running Red Hat Gluster Storage (RHGS) 3.1. Three nodes, each with at least two 100GB RAW devices:

    • gluster23.rhs (192.168.1.200)

    • gluster24.rhs (192.168.1.201)

    • gluster25.rhs (192.168.1.202)

  • This example uses an all-in-one OpenShift Origin cluster (master and node on a single host), though it can work using a standard, multi-node cluster as well.

    • k8dev2.rhs (192.168.1.208)

Installing and Configuring Heketi

Heketi is used to manage the Gluster cluster storage (adding volumes, removing volumes, etc.). Download deploy-heketi-template to install Heketi on OpenShift.

This template file places the database in an EmptyDir volume. Adjust the database accordingly for a reliable persistent storage.

  1. Create a new project:

    $ oc new-project <project-name>

  2. Enable privileged containers in the new project:

    $ oc adm policy add-scc-to-user privileged -z default

  3. Register the deploy-heketi template:

    $ oc create -f <template-path>/deploy-heketi-template

  4. Deploy the bootstrap Heketi container:

    $ oc process deploy-heketi -v \
         HEKETI_KUBE_NAMESPACE=<project-name> \
         HEKETI_KUBE_APIHOST=<master-url-and-port> \
         HEKETI_KUBE_INSECURE=y \
         HEKETI_KUBE_USER=<cluster-admin-username> \
         HEKETI_KUBE_PASSWORD=<cluster-admin-password> | oc create -f -

  5. Wait until the deploy-heketi pod starts and all services are running. Then get Heketi service details:

    $ oc get svc
NAME            CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
deploy-heketi   172.30.96.173   <none>        8080/TCP   2m

  6. Check if Heketi services are running properly, it must return Hello from Heketi.

    $ curl http://<cluster-ip>:8080/hello
Hello from Heketi

  7. Set an environment variable for the Heketi server:

    $ export HEKETI_CLI_SERVER=http://<cluster-ip>:8080

Loading Topology

Topology is used to tell Heketi about the environment and what nodes and devices it will manage.

Heketi is currently limited to managing raw devices only. If a device is already a Gluster volume, it is skipped and ignored.

  1. Create and load the topology file. There is a sample file located in /usr/share/heketi/topology-sample.json by default, or /etc/heketi depending on how it was installed.

    Depending upon your method of installation this file may not exist. If it is missing, manually create the topology-sample.json file, as shown in the following example.

    		 
    {
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster23.rhs"
              ],
              "storage": [
                "192.168.1.200"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster24.rhs"
              ],
              "storage": [
                "192.168.1.201"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster25.rhs"
              ],
              "storage": [
                "192.168.1.202"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        }
      ]
    }
  ]
}

  2. Run the following command to load the topology of your environment.

    $ heketi-cli topology load --json=topology-sample.json

            Found node gluster23.rhs on cluster bdf9d8ca3fa269ff89854faf58f34b9a
                   Adding device /dev/sde ... OK
                    Adding device /dev/sdf ... OK
            Creating node gluster24.rhs ... ID: 8e677d8bebe13a3f6846e78a67f07f30
                    Adding device /dev/sde ... OK
                    Adding device /dev/sdf ... OK
...

  3. Create a Gluster volume to verify Heketi:

    $ heketi-cli volume create --size=50

  4. View the volume information from one of the the Gluster nodes:

    $ gluster volume info

        Volume Name: vol_335d247ac57ecdf40ac616514cc6257f (1)
        Type: Distributed-Replicate
        Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3
        Status: Started
...
    1 Volume created by heketi-cli.

Dynamically Provision a Volume

If you installed OpenShift Origin by using the OpenShift Ansible example inventory configuration files for either native or external GlusterFS instance, the GlusterFS StorageClass automatically gets created during the installation. For such cases you can skip the following storage class creation steps and directly proceed with creating persistent volume claim instruction.

  1. Create a StorageClass object definition. The following definition is based on the minimum requirements needed for this example to work with OpenShift Origin. See Dynamic Provisioning and Creating Storage Classes for additional parameters and specification definitions.

    kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gluster-dyn
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://glusterclient2.rhs:8080" (1)
  restauthenabled: "false" (2)
    1 The Heketi server from the HEKETI_CLI_SERVER environment variable.
    2 Since authentication is not turned on in this example, set to false.

  2. From the OpenShift Origin master host, create the storage class:

    $ oc create -f glusterfs-storageclass1.yaml
storageclass "gluster-dyn" created

  3. Create a persistent volume claim (PVC), requesting the newly-created storage class. For example:

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: gluster-dyn-pvc
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
        storage: 30Gi
 storageClassName: gluster-dyn

  4. From the OpenShift Origin master host, create the PVC:

    $ oc create -f glusterfs-pvc-storageclass.yaml
persistentvolumeclaim "gluster-dyn-pvc" created

  5. View the PVC to see that the volume was dynamically created and bound to the PVC:

    $ oc get pvc
NAME                  STATUS        VOLUME                                                 CAPACITY           ACCESSMODES           STORAGECLASS           AGE
gluster-dyn-pvc Bound        pvc-78852230-d8e2-11e6-a3fa-0800279cf26f           30Gi                   RWX               gluster-dyn        42s

  6. Verify and view the new volume on one of the Gluster nodes:

    $ gluster volume info

        Volume Name: vol_335d247ac57ecdf40ac616514cc6257f (1)
        Type: Distributed-Replicate
        Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3
        Status: Started
        ...
        Volume Name: vol_f1404b619e6be6ef673e2b29d58633be (2)
        Type: Distributed-Replicate
        Volume ID: 7dc234d0-462f-4c6c-add3-fb9bc7e8da5e
        Status: Started
        Number of Bricks: 2 x 2 = 4
        ...
    1 Volume created by heketi-cli.
    2 New dynamically created volume triggered by Kubernetes and the storage class.

Creating a NGINX Pod That Uses the PVC

At this point, you have a dynamically created GlusterFS volume bound to a PVC. You can now now utilize this PVC in a pod. In this example, create a simple NGINX pod.

  1. Create the pod object definition:

    apiVersion: v1
kind: Pod
metadata:
  name: gluster-pod1
  labels:
    name: gluster-pod1
spec:
  containers:
  - name: gluster-pod1
    image: gcr.io/google_containers/nginx-slim:0.8
    ports:
    - name: web
      containerPort: 80
    securityContext:
      privileged: true
    volumeMounts:
    - name: gluster-vol1
      mountPath: /usr/share/nginx/html
  volumes:
  - name: gluster-vol1
    persistentVolumeClaim:
      claimName: gluster-dyn-pvc (1)
    1 The name of the PVC created in the previous step.

  2. From the OpenShift Origin master host, create the pod:

    $ oc create -f nginx-pod.yaml
pod "gluster-pod1" created

  3. View the pod. Give it a few minutes, as it might need to download the image if it does not already exist:

    $ oc get pods -o wide
NAME                               READY     STATUS    RESTARTS   AGE       IP               NODE
gluster-pod1                       1/1       Running   0          9m        10.38.0.0        node1

  4. Now remote into the container with oc exec and create an index.html file:

    $ oc exec -ti gluster-pod1 /bin/sh
$ cd /usr/share/nginx/html
$ echo 'Hello World from GlusterFS!!!' > index.html
$ ls
index.html
$ exit

  5. Now curl the URL of the pod:

    $ curl http://10.38.0.0
Hello World from GlusterFS!!!