$ oc get svc/docker-registry -o yaml | grep clusterIP:
OpenShift Origin refers to the integrated registry by its service IP address, so if you decide to delete and recreate the docker-registry service, you can ensure a completely transparent transition by arranging to re-use the old IP address in the new service. If a new IP address cannot be avoided, you can minimize cluster disruption by rebooting only the masters.
To re-use the IP address, you must save the IP address of the old docker-registry service prior to deleting it, and arrange to replace the newly assigned IP address with the saved one in the new docker-registry service.
Make a note of the clusterIP for the service:
$ oc get svc/docker-registry -o yaml | grep clusterIP:
Delete the service:
$ oc delete svc/docker-registry dc/docker-registry
Create the registry definition in registry.yaml, replacing <options>
with, for example, those used in step 3 of the instructions in the
Non-Production Use section:
$ oc adm registry <options> -o yaml > registry.yaml
Edit registry.yaml, find the Service there,
and change its clusterIP to the address noted in step 1.
Create the registry using the modified registry.yaml:
$ oc create -f registry.yaml
If you are unable to re-use the IP address, any operation that uses a pull specification that includes the old IP address will fail. To minimize cluster disruption, you must reboot the masters:
# systemctl restart origin-master-api origin-master-controllers
This ensures that the old registry URL, which includes the old IP address, is cleared from the cache.
| We recommend against rebooting the entire cluster because that incurs unnecessary downtime for pods and does not actually clear the cache. |
You can specify a whitelist of docker registries, allowing you to curate a set of images and templates that are available for download by OpenShift Origin users. This curated set can be placed in one or more docker registries, and then added to the whitelist. When using a whitelist, only the specified registries are accessible within OpenShift Origin, and all other registries are denied access by default.
To configure a whitelist:
Edit the /etc/sysconfig/docker file to block all registries:
BLOCK_REGISTRY='--block-registry=all'
You may need to uncomment the BLOCK_REGISTRY line.
In the same file, add registries to which you want to allow access:
ADD_REGISTRY='--add-registry=<registry1> --add-registry=<registry2>'
ADD_REGISTRY='--add-registry=registry.access.redhat.com'
This example would restrict access to images available on the Red Hat Customer Portal.
Once the whitelist is configured, if a user tries to pull from a docker registry that is not on the whitelist, they will receive an error message stating that this registry is not allowed.
You can configure the hostname and port the registry is known by for both internal and external references. By doing this, image streams will provide hostname based push and pull specifications for images, allowing consumers of the images to be isolated from changes to the registry service ip and potentially allowing image streams and their references to be portable between clusters.
To set the hostname used to reference the registry from within the cluster, set the internalRegistryHostname
in the imagePolicyConfig section of the master configuration file. The external hostname is controlled by setting the externalRegistryHostname value in the same location.
imagePolicyConfig:
internalRegistryHostname: docker-registry.default.svc.cluster.local:5000
externalRegistryHostname: docker-registry.mycompany.comThe registry itself must be configured with the same internal hostname value. This can be accomplished by setting the
REGISTRY_OPENSHIFT_SERVER_ADDR environment variable on the registry deployment configuration,
or by setting the value in the OpenShift section of the registry configuration.
|
If you have enabled TLS for your registry the server certificate must include the hostnames by which you expect the registry to be referenced. See securing the registry for instructions on adding hostnames to the server certificate. |
You can override the integrated registry’s default configuration, found by default at /config.yml in a running registry’s container, with your own custom configuration.
|
Upstream configuration options in this file may also be overridden using environment variables. The middleware section is an exception as there are just a few options that can be overridden using environment variables. Learn how to override specific configuration options. |
To enable management of the registry configuration file directly and deploy an updated configuration using
a ConfigMap:
Edit the registry configuration file locally as needed. The initial YAML file deployed on the registry is provided below. Review supported options.
version: 0.1
log:
level: debug
http:
addr: :5000
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /registry
delete:
enabled: true
auth:
openshift:
realm: openshift
middleware:
registry:
- name: openshift
repository:
- name: openshift
options:
acceptschema2: true
pullthrough: true
enforcequota: false
projectcachettl: 1m
blobrepositorycachettl: 10m
storage:
- name: openshift
openshift:
version: 1.0
metrics:
enabled: false
secret: <secret>Create a ConfigMap holding the content of each file in this directory:
$ oc create configmap registry-config \
--from-file=</path/to/custom/registry/config.yml>/
Add the registry-config ConfigMap as a volume to the registry’s deployment configuration to mount the custom configuration file at /etc/docker/registry/:
$ oc volume dc/docker-registry --add --type=configmap \
--configmap-name=registry-config -m /etc/docker/registry/
Update the registry to reference the configuration path from the previous step by adding the following environment variable to the registry’s deployment configuration:
$ oc set env dc/docker-registry \
REGISTRY_CONFIGURATION_PATH=/etc/docker/registry/config.yml
This may be performed as an iterative process to achieve the desired configuration. For example, during troubleshooting, the configuration may be temporarily updated to put it in debug mode.
To update an existing configuration:
|
This procedure will overwrite the currently deployed registry configuration. |
Edit the local registry configuration file, config.yml.
Delete the registry-config secret:
$ oc delete secret registry-config
Recreate the secret to reference the updated configuration file:
$ oc secrets new registry-config config.yml=</path/to/custom/registry/config.yml>
Redeploy the registry to read the updated configuration:
$ oc rollout latest docker-registry
|
Maintain configuration files in a source control repository. |
There are many configuration options available in the upstream docker distribution library. Not all configuration options are supported or enabled. Use this section as a reference when overriding the registry configuration.
|
Upstream configuration options in this file may also be overridden using environment variables. However, the middleware section may not be overridden using environment variables. Learn how to override specific configuration options. |
Upstream options are supported.
Example:
log:
level: debug
formatter: text
fields:
service: registry
environment: stagingThis section lists the supported registry storage drivers.
The following list includes storage drivers that need to be configured in the registry’s configuration file:
Filesystem. Filesystem is the default and does not need to be configured.
S3. Learn more about CloudFront configuration.
General registry storage configuration options are supported.
The following storage options need to be configured through the filesystem driver:
|
For more information on supported persistent storage drivers, see Configuring Persistent Storage and Persistent Storage Examples. |
storage:
delete:
enabled: true (1)
redirect:
disable: false
cache:
blobdescriptor: inmemory
maintenance:
uploadpurging:
enabled: true
age: 168h
interval: 24h
dryrun: false
readonly:
enabled: false| 1 | This entry is mandatory for image pruning to work properly. |
Auth options should not be altered. The openshift extension is the only supported option.
auth:
openshift:
realm: openshiftThe repository middleware extension allows to configure OpenShift Origin middleware responsible for interaction with OpenShift Origin and image proxying.
middleware:
registry:
- name: openshift (1)
repository:
- name: openshift (1)
options:
acceptschema2: true (2)
pullthrough: true (3)
mirrorpullthrough: true (4)
enforcequota: false (5)
projectcachettl: 1m (6)
blobrepositorycachettl: 10m (7)
storage:
- name: openshift (1)| 1 | These entries are mandatory. Their presence ensures required components are loaded. These values should not be changed. |
| 2 | Allows you to store manifest schema v2 during a push to the registry. See below for more details. |
| 3 | Allows the registry to act as a proxy for remote blobs. See below for more details. |
| 4 | Allows the registry cache blobs to be served from remote registries for fast access later. The mirroring starts when the blob is accessed for the first time. The option has no effect if the pullthrough is disabled. |
| 5 | Prevents blob uploads exceeding the size limit, which are defined in the targeted project. |
| 6 | An expiration timeout for limits cached in the registry. The lower the value, the less time it takes for the limit changes to propagate to the registry. However, the registry will query limits from the server more frequently and, as a consequence, pushes will be slower. |
| 7 | An expiration timeout for remembered associations between blob and repository. The higher the value, the higher probability of fast lookup and more efficient registry operation. On the other hand, memory usage will raise as well as a risk of serving image layer to user, who is no longer authorized to access it. |
The CloudFront middleware extension can be added to support AWS, CloudFront CDN storage provider. CloudFront middleware speeds up distribution of image content internationally. The blobs are distributed to several edge locations around the world. The client is always directed to the edge with the lowest latency.
|
The CloudFront middleware extension can be only used with S3 storage. It is utilized only during blob serving. Therefore, only blob downloads can be speeded up, not uploads. |
The following is an example of minimal configuration of S3 storage driver with a CloudFront middleware:
version: 0.1
log:
level: debug
http:
addr: :5000
storage:
cache:
blobdescriptor: inmemory
delete:
enabled: true
s3: (1)
accesskey: BJKMSZBRESWJQXRWMAEQ
secretkey: 5ah5I91SNXbeoUXXDasFtadRqOdy62JzlnOW1goS
region: us-east-1
bucket: docker.myregistry.com
auth:
openshift:
realm: openshift
middleware:
registry:
- name: openshift
repository:
- name: openshift
storage:
- name: cloudfront (2)
options:
baseurl: https://jrpbyn0k5k88bi.cloudfront.net/ (3)
privatekey: /etc/docker/cloudfront-ABCEDFGHIJKLMNOPQRST.pem (4)
keypairid: ABCEDFGHIJKLMNOPQRST (5)
- name: openshift| 1 | The S3 storage must be configured the same way regardless of CloudFront middleware. |
| 2 | The CloudFront storage middleware needs to be listed before OpenShift middleware. |
| 3 | The CloudFront base URL. In the AWS management console, this is listed as Domain Name of CloudFront distribution. |
| 4 | The location of your AWS private key on the filesystem. This must be not confused with Amazon EC2 key pair. Please refer to AWS documentation on creating CloudFront key pairs for your trusted signers. The file needs to be mounted as a secret secret into the registry pod. |
| 5 | The ID of your Cloudfront key pair. |
The middleware section cannot be overridden using environment variables. There are a few exceptions, however. For example:
middleware:
repository:
- name: openshift
options:
acceptschema2: true (1)
pullthrough: true (2)
mirrorpullthrough: true (3)
enforcequota: false (4)
projectcachettl: 1m (5)
blobrepositorycachettl: 10m (6)| 1 | A configuration option that can be overridden by the boolean environment
variable REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2, which
allows for the ability to accept manifest schema v2 on manifest put requests.
Recognized values are true and false (which applies to all the other
boolean variables below). |
| 2 | A configuration option that can be overridden by the boolean environment
variable REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_PULLTHROUGH, which
enables a proxy mode for remote repositories. |
| 3 | A configuration option that can be overridden by the boolean environment
variable REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_MIRRORPULLTHROUGH, which
instructs registry to mirror blobs locally if serving remote blobs. |
| 4 | A configuration option that can be overridden by the boolean environment
variable REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA, which
allows the ability to turn quota enforcement on or off. By default, quota
enforcement is off. |
| 5 | A configuration option that can be overridden by the environment variable
REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_PROJECTCACHETTL, specifying an
eviction timeout for project quota objects. It takes a valid time duration
string (for example, 2m). If empty, you get the default timeout. If zero
(0m), caching is disabled. |
| 6 | A configuration option that can be overridden by the environment variable
REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_BLOBREPOSITORYCACHETTL, specifying
an eviction timeout for associations between blob and containing repository.
The format of the value is the same as in projectcachettl case. |
If enabled, the registry will attempt to fetch requested blob from a remote registry unless the blob exists locally. The remote candidates are calculated from DockerImage entries stored in status of the image stream, a client pulls from. All the unique remote registry references in such entries will be tried in turn until the blob is found.
Pullthrough will only occur if an image stream tag exists for the image being
pulled. For example, if the image being pulled is
docker-registry.default.svc:5000/yourproject/yourimage:prod then the
registry will look for an image stream tag named yourimage:prod in the
project yourproject. If it finds one, it will attempt to pull the image
using the dockerImageReference associated with that image stream tag.
When performing pullthrough, the registry will use pull credentials found in the project associated with the image stream tag that is being referenced. This capability also makes it possible for you to pull images that reside on a registry they do not have credentials to access, as long as you have access to the image stream tag that references the image.
Note that by default image stream tags use a reference policy type of Source
which means that when the image stream reference is resolved to an image pull
specification, the specification used will point to the source of the image.
For images hosted on external registries, this will be the external registry and
as a result the resource will reference and pull the image by the external
registry. For example, registry.access.redhat.com/openshift3/jenkins-2-rhel7 and
pullthrough will not apply. To ensure that resources referencing image streams
use a pull specification that points to the internal registry, the image stream
tag should use a reference policy type of Local. More information is available on
Reference Policy.
This feature is on by default. However, it can be disabled using a configuration option.
By default, all the remote blobs served this way are stored locally for
subsequent faster access unless mirrorpullthrough is disabled. The downside
of this mirroring feature is an increased storage usage.
|
The mirroring starts when a client tries to fetch at least a single byte of the blob. To pre-fetch a particular image into integrated registry before it is actually needed, you can run the following command: $ oc get imagestreamtag/${IS}:${TAG} -o jsonpath='{ .image.dockerImageLayers[*].name }' | \
xargs -n1 -I {} curl -H "Range: bytes=0-1" -u user:${TOKEN} \
http://${REGISTRY_IP}:${PORT}/v2/default/mysql/blobs/{}
|
|
This OpenShift Origin mirroring feature should not be confused with the upstream registry pull through cache feature, which is a similar but distinct capability. |
Each image has a manifest describing its blobs, instructions for running it and additional metadata. The manifest is versioned, with each version having different structure and fields as it evolves over time. The same image can be represented by multiple manifest versions. Each version will have different digest though.
The registry currently supports manifest v2 schema 1 (schema1) and manifest v2 schema 2 (schema2). The former is being obsoleted but will be supported for an extended amount of time.
You should be wary of compatibility issues with various Docker clients:
Docker clients of version 1.9 or older support only schema1. Any manifest this client pulls or pushes will be of this legacy schema.
Docker clients of version 1.10 support both schema1 and schema2. And by default, it will push the latter to the registry if it supports newer schema.
The registry, storing an image with schema1 will always return it unchanged to the client. Schema2 will be transferred unchanged only to newer Docker client. For the older one, it will be converted on-the-fly to schema1.
This has significant consequences. For example an image pushed to the registry by a newer Docker client cannot be pulled by the older Docker by its digest. That’s because the stored image’s manifest is of schema2 and its digest can be used to pull only this version of manifest.
For this reason, the registry is configured by default not to store schema2. This ensures that any docker client will be able to pull from the registry any image pushed there regardless of client’s version.
Once you’re confident that all the registry clients support schema2, you’ll be safe to enable its support in the registry. See the middleware configuration reference above for particular option.
This section reviews the configuration of global settings for features specific
to OpenShift Origin. In a future release, openshift-related settings in the
Middleware section will
be obsoleted.
Currently, this section allows you to configure registry metrics collection:
openshift:
version: 1.0 (1)
server:
addr: docker-registry.default.svc (2)
metrics:
enabled: false (3)
secret: <secret> (4)
requests:
read:
maxrunning: 10 (5)
maxinqueue: 10 (6)
maxwaitinqueue 2m (7)
write:
maxrunning: 10 (8)
maxinqueue: 10 (9)
maxwaitinqueue 2m (10)
| 1 | A mandatory entry specifying configuration version of this section. The only
supported value is 1.0. |
| 2 | The hostname of the registry. Should be set to the same value configured on the
master. It can be overridden by the environment variable REGISTRY_OPENSHIFT_SERVER_ADDR. |
| 3 | Can be set to true to enable metrics collection. It can be overridden by the boolean
environment variable REGISTRY_OPENSHIFT_METRICS_ENABLED. |
| 4 | A secret used to authorize client requests. Metrics clients must use it as a
bearer token in Authorization header. It can be overridden by the environment
variable REGISTRY_OPENSHIFT_METRICS_SECRET. |
| 5 | Maximum number of simultaneous pull requests. It can be overridden by the
environment variable REGISTRY_OPENSHIFT_REQUESTS_READ_MAXRUNNING. Zero indicates no limit. |
| 6 | Maximum number of queued pull requests. It can be overridden by the
environment variable REGISTRY_OPENSHIFT_REQUESTS_READ_MAXINQUEUE. Zero indicates no limit. |
| 7 | Maximum time a pull request can wait in the queue before being rejected.
It can be overridden by the environment variable REGISTRY_OPENSHIFT_REQUESTS_READ_MAXWAITINQUEUE.
Zero indicates no limit. |
| 8 | Maximum number of simultaneous push requests. It can be overridden by the
environment variable REGISTRY_OPENSHIFT_REQUESTS_WRITE_MAXRUNNING. Zero indicates no limit. |
| 9 | Maximum number of queued push requests. It can be overridden by the
environment variable REGISTRY_OPENSHIFT_REQUESTS_WRITE_MAXINQUEUE. Zero indicates no limit. |
| 10 | Maximum time a push request can wait in the queue before being rejected. It can be overridden by the
environment variable REGISTRY_OPENSHIFT_REQUESTS_WRITE_MAXWAITINQUEUE.
Zero indicates no limit. |
See Accessing Registry Metrics for usage information.
Upstream options are supported. Learn how to alter these settings via environment variables. Only the tls section should be altered. For example:
http:
addr: :5000
tls:
certificate: /etc/secrets/registry.crt
key: /etc/secrets/registry.keyUpstream options are supported. The REST API Reference provides more comprehensive integration options.
Example:
notifications:
endpoints:
- name: registry
disabled: false
url: https://url:port/path
headers:
Accept:
- text/plain
timeout: 500
threshold: 5
backoff: 1000Upstream options are supported. The registry deployment configuration provides an integrated health check at /healthz.
Proxy configuration should not be enabled. This functionality is provided by the OpenShift Origin repository middleware extension, pullthrough: true.