1. Documentation
×

Backing Docker Registry with GlusterFS Storage

Overview

This topic reviews how to attach a GlusterFS persistent volume to the Docker Registry.

It is assumed that the Docker registry service has already been started and the Gluster volume has been created.

Prerequisites

All oc commands are executed on the master node as the admin user.

Create the Gluster Persistent Volume

First, make the Gluster volume available to the registry.

$ oc create -f gluster-endpoints-service.yaml
$ oc create -f gluster-endpoints.yaml
$ oc create -f gluster-pv.yaml
$ oc create -f gluster-pvc.yaml

Check to make sure the PV and PVC were created and bound successfully. The expected output should resemble the following. Note that the PVC status is Bound, indicating that it has bound to the PV.

$ oc get pv
NAME         LABELS    CAPACITY   ACCESSMODES   STATUS      CLAIM     REASON    AGE
gluster-pv   <none>    1Gi        RWX           Available                       37s
$ oc get pvc
NAME            LABELS    STATUS    VOLUME       CAPACITY   ACCESSMODES   AGE
gluster-claim   <none>    Bound     gluster-pv   1Gi        RWX           24s

If either the PVC or PV failed to create or the PVC failed to bind, refer back to the GlusterFS Persistent Storage guide. Do not proceed until they initialize and the PVC status is Bound.

Attach the PVC to the Docker Registry

Before moving forward, ensure that the docker-registry service is running.

$ oc get svc
NAME              CLUSTER_IP       EXTERNAL_IP   PORT(S)                 SELECTOR                  AGE
docker-registry   172.30.167.194   <none>        5000/TCP                docker-registry=default   18m

If either the docker-registry service or its associated pod is not running, refer back to the docker-registry setup instructions for troubleshooting before continuing.

Then, attach the PVC:

$ oc volume deploymentconfigs/docker-registry --add --name=registry-storage -t pvc \
     --claim-name=gluster-claim --overwrite

Deploying a Docker Registry provides more information on using the Docker registry.

Known Issues

Pod Cannot Resolve the Volume Host

In non-production cases where the dnsmasq server is located on the same node as the OpenShift Origin master service, pods might not resolve to the host machines when mounting the volume, causing errors in the docker-registry-1-deploy pod. This can happen when dnsmasq.service fails to start because of a collision with OpenShift Origin DNS on port 53. To run the DNS server on the master host, some configurations needs to be changed.

In /etc/dnsmasq.conf, add:

# Reverse DNS record for master
host-record=master.example.com,<master-IP>
# Wildcard DNS for OpenShift Applications - Points to Router
address=/apps.example.com/<master-IP>
# Forward .local queries to SkyDNS
server=/local/127.0.0.1#8053
# Forward reverse queries for service network to SkyDNS.
# This is for default OpenShift SDN - change as needed.
server=/17.30.172.in-addr.arpa/127.0.0.1#8053

With these settings, dnsmasq will pull from the /etc/hosts file on the master node.

Add the appropriate host names and IPs for all necessary hosts.

In master-config.yaml, change bindAddress to:

dnsConfig:
 bindAddress: 127.0.0.1:8053

When pods are created, they receive a copy of /etc/resolv.conf, which typically contains only the master DNS server so they can resolve external DNS requests. To enable internal DNS resolution, insert the dnsmasq server at the top of the server list. This way, dnsmasq will attempt to resolve requests internally first.

In /etc/resolv.conf all scheduled nodes:

nameserver 192.168.1.100  (1)
nameserver 192.168.1.1    (2)
1 Add the internal DNS server.
2 Pre-existing external DNS server.

Once the configurations are changed, restart the OpenShift Origin master and dnsmasq services.

$ systemctl restart origin-master-api origin-master-controllers
$ systemctl restart dnsmasq