1. Documentation
×

Pruning Objects

Overview

Over time, API objects created in OpenShift Origin can accumulate in the etcd data store through normal user operations, such as when building and deploying applications.

As an administrator, you can periodically prune older versions of objects from your OpenShift Origin instance that are no longer needed. For example, by pruning images you can delete older images and layers that are no longer in use, but are still taking up disk space.

Basic Prune Operations

The CLI groups prune operations under a common parent command.

$ oadm prune <object_type> <options>

This specifies:

  • The <object_type> to perform the action on, such as builds, deployments, or images.

  • The <options> supported to prune that object type.

Pruning Deployments

In order to prune deployments that are no longer required by the system due to age and status, administrators may run the following command:

$ oadm prune deployments [<options>]
Table 1. Prune Deployments CLI Configuration Options
Option Description

--confirm

Indicate that pruning should occur, instead of performing a dry-run.

--orphans

Prune all deployments whose deployment config no longer exists, status is complete or failed, and replica count is zero.

--keep-complete=<N>

Per deployment config, keep the last N deployments whose status is complete and replica count is zero. (default 5)

--keep-failed=<N>

Per deployment config, keep the last N deployments whose status is failed and replica count is zero. (default 1)

--keep-younger-than=<duration>

Do not prune any object that is younger than <duration> relative to the current time. (default 60m) Valid units of measurement include nanoseconds (ns), microseconds (us), milliseconds (ms), seconds (s), minutes (m), and hours (h).

To see what a pruning operation would delete:

$ oadm prune deployments --orphans --keep-complete=5 --keep-failed=1 \
    --keep-younger-than=60m

To actually perform the prune operation:

$ oadm prune deployments --orphans --keep-complete=5 --keep-failed=1 \
    --keep-younger-than=60m --confirm

Pruning Builds

In order to prune builds that are no longer required by the system due to age and status, administrators may run the following command:

$ oadm prune builds [<options>]
Table 2. Prune Builds CLI Configuration Options
Option Description

--confirm

Indicate that pruning should occur, instead of performing a dry-run.

--orphans

Prune all builds whose build config no longer exists, status is complete, failed, error, or canceled.

--keep-complete=<N>

Per build config, keep the last N builds whose status is complete. (default 5)

--keep-failed=<N>

Per build config, keep the last N builds whose status is failed, error, or canceled (default 1)

--keep-younger-than=<duration>

Do not prune any object that is younger than <duration> relative to the current time. (default 60m)

To see what a pruning operation would delete:

$ oadm prune builds --orphans --keep-complete=5 --keep-failed=1 \
    --keep-younger-than=60m

To actually perform the prune operation:

$ oadm prune builds --orphans --keep-complete=5 --keep-failed=1 \
    --keep-younger-than=60m --confirm

Pruning Images

In order to prune images that are no longer required by the system due to age, status, or exceed limits, administrators may run the following command:

$ oadm prune images [<options>]

Currently, to prune images you must first log in to the CLI as a user with an access token. The user must also have the cluster role system:image-pruner or greater (for example, cluster-admin).

Pruning images removes data from the integrated registry. For this operation to work properly, ensure your registry is configured with storage:delete:enabled set to true.

Pruning images with the --namespace flag does not remove images, only image streams. Images are non-namespaced resources. Therefore, limiting pruning to a particular namespace makes it impossible to calculate their current usage.
Table 3. Prune Images CLI Configuration Options
Option Description

--all

Include images that were not pushed to the registry, but have been mirrored by pullthrough. This is on by default. To limit the pruning to images that were pushed to the integrated registry, pass --all=false.

--certificate-authority

The path to a certificate authority file to use when communicating with the OpenShift Origin-managed registries. Defaults to the certificate authority data from the current user’s configuration file. If provided, a secure connection will be initiated.

--confirm

Indicate that pruning should occur, instead of performing a dry-run. This requires a valid route to the integrated Docker registry. If this command is run outside of the cluster network, the route needs to be provided using --registry-url.

--force-insecure

Use caution with this option. Allow an insecure connection to the Docker registry that is hosted via HTTP or has an invalid HTTPS certificate. See Using Secure or Insecure Connections for more information.

--keep-tag-revisions=<N>

For each image stream, keep up to at most N image revisions per tag. (default 3)

--keep-younger-than=<duration>

Do not prune any image that is younger than <duration> relative to the current time. Do not prune any image that is referenced by any other object that is younger than <duration> relative to the current time. (default 60m)

--prune-over-size-limit

Prune each image that exceeds the smallest limit defined in the same project. This flag cannot be combined with --keep-tag-revisions nor --keep-younger-than.

--registry-url

The address to use when contacting the registry. The command will attempt to use a cluster-internal URL determined from managed images and image streams. In case it fails (the registry cannot be resolved or reached), use this flag to provide an alternative route.

Image Prune Conditions

  • Remove any image "managed by OpenShift Origin" (images with the annotation openshift.io/image.managed) that was created at least --keep-younger-than minutes ago and is not currently referenced by:

    • any pod created less than --keep-younger-than minutes ago.

    • any image stream created less than --keep-younger-than minutes ago.

    • any running pods.

    • any pending pods.

    • any replication controllers.

    • any deployment configurations.

    • any build configurations.

    • any builds.

    • the --keep-tag-revisions most recent items in stream.status.tags[].items.

  • Remove any image "managed by OpenShift Origin" (images with the annotation openshift.io/image.managed) that is exceeding the smallest limit defined in the same project and is not currently referenced by:

    • any running pods.

    • any pending pods.

    • any replication controllers.

    • any deployment configurations.

    • any build configurations.

    • any builds.

  • There is no support for pruning from external registries.

  • When an image is pruned, all references to the image are removed from all image streams that have a reference to the image in status.tags.

  • Image layers that are no longer referenced by any images are removed as well.

--prune-over-size-limit cannot be combined with --keep-tag-revisions nor --keep-younger-than flags. Doing so will return an information that this operation is not allowed.

To see what a pruning operation would delete:

  1. Keeping up to three tag revisions, and keeping resources (images, image streams and pods) younger than sixty minutes:

    $ oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m

  2. Pruning every image that exceeds defined limits:

    $ oadm prune images --prune-over-size-limit

To actually perform the prune operation for the previously mentioned options accordingly:

$ oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm

$ oadm prune images --prune-over-size-limit --confirm

Using Secure or Insecure Connections

The secure connection is the preferred and recommended approach. It is done over HTTPS protocol with a mandatory certificate verification. The prune command always attempts to use it if possible. If not possible, in some cases it can fall-back to insecure connection, which is dangerous. In this case, either certificate verification is skipped or plain HTTP protocol is used.

The fall-back to insecure connection is allowed in the following cases unless --certificate-authority is specified:

  1. The prune command is run with the --force-insecure option.

  2. The provided registry-url is a local-link address or localhost.

  3. The configuration of the current user allows for an insecure connection. This may be caused by the user either logging in using --insecure-skip-tls-verify or choosing the insecure connection when prompted.

If the registry is secured by a certificate authority different from the one used by OpenShift Origin, it needs to be specified using the --certificate-authority flag. Otherwise, the prune command will fail with an error similar to those listed in Using the Wrong Certificate Authority or Using an Insecure Connection Against a Secured Registry.

Image Pruning Problems

Images Not Being Pruned

If your images keep accumulating and the prune command removes just a small portion of what you expect, ensure that you understand the conditions that must apply for an image to be considered a candidate for pruning.

Especially ensure that images you want removed occur at higher positions in each tag history than your chosen tag revisions threshold. For example, consider an old and obsolete image named sha:abz. By running the following command in namespace N, where the image is tagged, you will see the image is tagged three times in a single image stream named myapp:

$ image_name="sha:abz"
$ oc get is -n N -o go-template='{{range $isi, $is := .items}}{{range $ti, $tag := $is.status.tags}}'\
  '{{range $ii, $item := $tag.items}}{{if eq $item.image "'"${image_name}"\
  $'"}}{{$is.metadata.name}}:{{$tag.tag}} at position {{$ii}} out of {{len $tag.items}}\n'\
  '{{end}}{{end}}{{end}}{{end}}'
myapp:v2 at position 4 out of 5
myapp:v2.1 at position 2 out of 2
myapp:v2.1-may-2016 at position 0 out of 1

When default options are used, the image will not ever be pruned because it occurs at position 0 in a history of myapp:v2.1-may-2016 tag. For an image to be considered for pruning, the administrator must either:

  1. Specify --keep-tag-revisions=0 with the oadm prune images command.

    This action will effectively remove all the tags from all the namespaces with underlying images, unless they are younger or they are referenced by objects younger than the specified threshold.

  2. Delete all the istags where the position is below the revision threshold, which means myapp:v2.1 and myapp:v2.1-may-2016.

  3. Move the image further in the history, either by running new builds pushing to the same istag, or by tagging other image. Unfortunately, this is not always desirable for old release tags.

Tags having a date or time of a particular image’s build in their names should be avoided, unless the image needs to be preserved for undefined amount of time. Such tags tend to have just one image in its history, which effectively prevents them from ever being pruned. Learn more about istag naming.

Using a Secure Connection Against Insecure Registry

If you see a message similar to the following in the output of the oadm prune images, then your registry is not secured and the oadm prune images client attempted to use secure connection:

error: error communicating with registry: Get https://172.30.30.30:5000/healthz: http: server gave HTTP response to HTTPS client

  1. The recommended solution is to secure the registry. If that is not desired, you can force the client to use an insecure connection by appending --force-insecure to the command (not recommended).

Using an Insecure Connection Against a Secured Registry

If you see one of the following errors in the output of the oadm prune images command, it means that your registry is secured using a certificate signed by a certificate authority other than the one used by oadm prune images client for connection verification.

error: error communicating with registry: Get http://172.30.30.30:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
error: error communicating with registry: [Get https://172.30.30.30:5000/healthz: x509: certificate signed by unknown authority, Get http://172.30.30.30:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"]

By default, the certificate authority data stored in user’s configuration file are used — the same for communication with the master API.

Use the --certificate-authority option to provide the right certificate authority for the Docker registry server.

Using the Wrong Certificate Authority

The following error means that the certificate authority used to sign the certificate of the secured Docker registry is different than the authority used by the client.

error: error communicating with registry: Get https://172.30.30.30:5000/: x509: certificate signed by unknown authority

Make sure to provide the right one with the flag --certificate-authority.

As a work-around, the --force-insecure flag can be added instead (not recommended).