Controlling Pod Placement - Scheduling | Cluster Administration

Overview

As a cluster administrator, you can set a policy to prevent application developers with certain roles from targeting specific nodes when scheduling pods.

The Pod Node Constraints admission controller ensures that pods are deployed onto only specified node hosts using labels] and prevents users without a specific role from using the nodeSelector field to schedule pods.

Constraining Pod Placement Using Node Name

Use the Pod Node Constraints admission controller to ensure a pod is deployed onto only a specified node host by assigning it a label and specifying this in the nodeName setting in a pod configuration.

Ensure you have the desired labels (see Updating Labels on Nodes for details) and node selector set up in your environment.

For example, make sure that your pod configuration features the nodeName value indicating the desired label:
```
apiVersion: v1
kind: Pod
spec:
  nodeName: <value>
```

Modify the master configuration file (/etc/origin/master/master-config.yaml) in two places:

Add PodNodeConstraints to the admissionConfig section:

...
admissionConfig:
  pluginConfig:
    PodNodeConstraints:
      configuration:
        apiversion: v1
        kind: PodNodeConstraintsConfig
...

Then, add the same to the kubernetesMasterConfig section:

...
kubernetesMasterConfig:
  admissionConfig:
    pluginConfig:
      PodNodeConstraints:
        configuration:
          apiVersion: v1
          kind: PodNodeConstraintsConfig
...

Restart OpenShift Origin for the changes to take effect.
```
# systemctl restart origin-master
```

Constraining Pod Placement Using a Node Selector

Using node selectors, you can ensure that pods are only placed onto nodes with specific labels. As a cluster administrator, you can use the Pod Node Constraints admission controller to set a policy that prevents users without the pods/binding permission from using node selectors to schedule pods.

The nodeSelectorLabelBlacklist field of a master configuration file gives you control over the labels that certain roles can specify in a pod configuration’s nodeSelector field. Users, service accounts, and groups that have the pods/binding permission role can specify any node selector. Those without the pods/binding permission are prohibited from setting a nodeSelector for any label that appears in nodeSelectorLabelBlacklist.

For example, an OpenShift Origin cluster might consist of five data centers spread across two regions. In the U.S., "us-east", "us-central", and "us-west"; and in the Asia-Pacific region (APAC), "apac-east" and "apac-west". Each node in each geographical region is labeled accordingly. For example, region: us-east.

See Updating Labels on Nodes for details on assigning labels.

As a cluster administrator, you can create an infrastructure where application developers should be deploying pods only onto the nodes closest to their geographical location. You can create a node selector, grouping the U.S. data centers into superregion: us and the APAC data centers into superregion: apac.

To maintain an even loading of resources per data center, you can add the desired region to the nodeSelectorLabelBlacklist section of a master configuration. Then, whenever a developer located in the U.S. creates a pod, it is deployed onto a node in one of the regions with the superregion: us label. If the developer tries to target a specific region for their pod (for example, region: us-east), they receive an error. If they try again, without the node selector on their pod, it can still be deployed onto the region they tried to target, because superregion: us is set as the project-level node selector, and nodes labeled region: us-east are also labeled superregion: us.

Ensure you have the desired labels (see Updating Labels on Nodes for details) and node selector set up in your environment.

For example, make sure that your pod configuration features the nodeSelector value indicating the desired label:
```
apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    <key>: <value>
...
```

Modify the master configuration file (/etc/origin/master/master-config.yaml) in two places:

Add nodeSelectorLabelBlacklist to the admissionConfig section with the labels that are assigned to the node hosts you want to deny pod placement:

...
admissionConfig:
  pluginConfig:
    PodNodeConstraints:
      configuration:
        apiversion: v1
        kind: PodNodeConstraintsConfig
        nodeSelectorLabelBlacklist:
          - kubernetes.io/hostname
          - <label>
...

Then, add the same to the kubernetesMasterConfig section to restrict direct pod creation:

...
kubernetesMasterConfig:
  admissionConfig:
    pluginConfig:
      PodNodeConstraints:
        configuration:
          apiVersion: v1
          kind: PodNodeConstraintsConfig
          nodeSelectorLabelBlacklist:
            - kubernetes.io/hostname
            - <label_1>
...

Restart OpenShift Origin for the changes to take effect.
```
# systemctl restart origin-master
```

Control Pod Placement to Projects

The Pod Node Selector admission controller allows you to force pods onto nodes associated with a specific project and prevent pods from being scheduled in those nodes.

The Pod Node Selector admission controller determines where a pod can be placed using labels on projects and node selectors specified in pods. A new pod will be placed on a node associated with a project only if the node selectors in the pod match the labels in the project.

After the pod is created, the node selectors are merged into the pod so that the pod specification includes the labels originally included in the specification and any new labels from the node selectors. The example below illustrates the merging effect.

The Pod Node Selector admission controller also allows you to create a list of labels that are permitted in a specific project. This list acts as a whitelist that lets developers know what labels are acceptable to use in a project and gives administrators greater control over labeling in a cluster.

To activate the Pod Node Selector admission controller:

Configure the Pod Node Selector admission controller and whitelist, using one of the following methods:

Add the following to the master configuration file (/etc/origin/master/master-config.yaml):

admissionConfig:
  pluginConfig:
    PodNodeSelector:
      configuration:
        podNodeSelectorPluginConfig: (1)
          clusterDefaultNodeSelector: "k3=v3" (2)
          ns1: region=west,env=test,infra=fedora,os=fedora (2)

1 Adds the Pod Node Selector admission controller plug-in.

2 Creates default labels for all nodes.

Creates a whitelist of permitted labels in the specified project. Here, the project is ns1 and the labels are the key=value pairs that follow.

Create a file containing the admission controller information:
```
podNodeSelectorPluginConfig:
    clusterDefaultNodeSelector: "k3=v3"
     ns1: region=west,env=test,infra=fedora,os=fedora
```
Then, reference the file in the master configuration:
```
admissionConfig:
  pluginConfig:
    PodNodeSelector:
      location: <path-to-file>
```
If a project does not have a node selectors specified, the pods associated with that project will be merged using the default node selector (clusterDefaultNodeSelector).

Restart OpenShift Origin for the changes to take effect.
```
# systemctl restart origin-master
```

Create a project object that includes the scheduler.alpha.kubernetes.io/node-selector annotation and labels.

{
    "kind": "Namespace",
    "apiVersion": "v1",
    "metadata": {
        "name": "ns1",
        "annotations": {
            "scheduler.alpha.kubernetes.io/node-selector": "env=test,infra=fedora" (1)
        }
    },
    "spec": {},
    "status": {}
}

1	Annotation to create the labels to match the project label selector. Here, the key/value labels are `env=test` and `infra=fedora`.

Create a pod specification that includes the labels in the node selector, for example:

apiVersion: v1
kind: Pod
metadata:
  labels:
    name: hello-pod
  name: hello-pod
spec:
  containers:
    - image: "docker.io/ocpqe/hello-pod:latest"
      imagePullPolicy: IfNotPresent
      name: hello-pod
      ports:
        - containerPort: 8080
          protocol: TCP
      resources: {}
      securityContext:
        capabilities: {}
        privileged: false
      terminationMessagePath: /dev/termination-log
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  nodeSelector: (1)
    env: test
    os: fedora
  serviceAccount: ""
status: {}

1	Node selectors to match project labels.

Create the pod in the project:
```
oc create -f pod.yaml --namespace=ns1
```
Check that the node selector labels were added to the pod configuration:
```
get pod pod1 --namespace=ns1 -o json

nodeSelector": {
 "env": "test",
 "infra": "fedora",
 "os": "fedora"
}
```
The node selectors are merged into the pod and the pod should be scheduled in the appropriate project.

If you create a pod with a label that is not specified in the project specification, the pod is not scheduled on the node.

For example, here the label env: production is not in any project specification:

nodeSelector:
 "env: production"
 "infra": "fedora",
 "os": "fedora"

If there is a node that does not have a node selector annotation, the pod will be scheduled there.