Configuring for AWS | Installation and Configuration

Overview

OpenShift Origin can be configured to access an AWS EC2 infrastructure, including using AWS volumes as persistent storage for application data. After AWS is configured properly, some additional configurations will need to be completed on the OpenShift Origin hosts.

Permissions

Configuring AWS for OpenShift Origin requires the following permissions:

Table 1. Master Permissions
Elastic Compute Cloud(EC2)	`ec2:DescribeVolume`, `ec2:CreateVolume`, `ec2:CreateTags`, `ec2:DescribeInstance`, `ec2:AttachVolume`, `ec2:DetachVolume`, `ec2:DeleteVolume`, `ec2:DescribeSubnets`, `ec2:CreateSecurityGroup`, `ec2:DescribeSecurityGroups`, `ec2:DescribeRouteTables`, `ec2:AuthorizeSecurityGroupIngress`
Elastic Load Balancing	`elasticloadbalancing:DescribeTags`, `elasticloadbalancing:CreateLoadBalancerListeners`, `elasticloadbalancing:ConfigureHealthCheck`, `elasticloadbalancing:DeleteLoadBalancerListeners`, `elasticloadbalancing:RegisterInstancesWithLoadBalancer`, `elasticloadbalancing:DescribeLoadBalancers`, `elasticloadbalancing:CreateLoadBalancer`, `elasticloadbalancing:DeleteLoadBalancer`, `elasticloadbalancing:ModifyLoadBalancerAttributes`, `elasticloadbalancing:DescribeLoadBalancerAttributes`

Table 2. Node Permissions
Elastic Compute Cloud(EC2)	`ec2:DescribeInstance*`

Every master, node, and subnet must have the KubernetesCluster: value tag.
One security group, preferably the one linked to the nodes, must have the KubernetesCluster: value tag.
- Do not tag all security groups with the KubernetesCluster: value tag or the Elastic Load Balancing (ELB) will not be able to create a load balancer.

Configuring AWS Variables

To set the required AWS variables, create a /etc/aws/aws.conf file with the following contents on all of your OpenShift Origin hosts, both masters and nodes:

[Global]
Zone = us-east-1c (1)

1	This is the Availability Zone of your AWS Instance and where your EBS Volume resides; this information is obtained from the AWS Managment Console.

Configuring OpenShift Origin Masters for AWS

You can set the AWS configuration on your OpenShift Origin master hosts in two ways:

Configuring OpenShift Origin for AWS with Ansible

During advanced installations, AWS can be configured using the openshift_cloudprovider_aws_access_key, openshift_cloudprovider_aws_secret_key, openshift_cloudprovider_kind, openshift_clusterid parameters, which are configurable in the inventory file.

Example AWS Configuration with Ansible

# Cloud Provider Configuration
#
# Note: You may make use of environment variables rather than store
# sensitive configuration within the ansible inventory.
# For example:
#openshift_cloudprovider_aws_access_key="{{ lookup('env','AWS_ACCESS_KEY_ID') }}"
#openshift_cloudprovider_aws_secret_key="{{ lookup('env','AWS_SECRET_ACCESS_KEY') }}"
#
#openshift_clusterid=unique_identifier_per_availablility_zone
#
# AWS (Using API Credentials)
#openshift_cloudprovider_kind=aws
#openshift_cloudprovider_aws_access_key=aws_access_key_id
#openshift_cloudprovider_aws_secret_key=aws_secret_access_key
#
# AWS (Using IAM Profiles)
#openshift_cloudprovider_kind=aws
# Note: IAM roles must exist before launching the instances.

When Ansible configures AWS, the following files are created for you:

/etc/aws/aws.conf
/etc/origin/master/master-config.yaml
/etc/origin/node/node-config.yaml
/etc/sysconfig/atomic-openshift-master
/etc/sysconfig/atomic-openshift-master-api
/etc/sysconfig/atomic-openshift-master-controllers
/etc/sysconfig/atomic-openshift-node

Manually Configuring OpenShift Origin Masters for AWS

Edit or create the master configuration file on all masters (/etc/origin/master/master-config.yaml by default) and update the contents of the apiServerArguments and controllerArguments sections:

kubernetesMasterConfig:
  ...
  apiServerArguments:
    cloud-provider:
      - "aws"
    cloud-config:
      - "/etc/aws/aws.conf"
  controllerArguments:
    cloud-provider:
      - "aws"
    cloud-config:
      - "/etc/aws/aws.conf"

Currently, the nodeName must match the instance name in AWS in order for the cloud provider integration to work properly. The name must also be RFC1123 compliant.

When triggering a containerized installation, only the directories of /etc/origin and /var/lib/origin are mounted to the master and node container. Therefore, aws.conf should be in /etc/origin/ instead of /etc/.

Manually Configuring OpenShift Origin Nodes for AWS

Edit or create the node configuration file on all nodes (/etc/origin/node/node-config.yaml by default) and update the contents of the kubeletArguments section:

kubeletArguments:
  cloud-provider:
    - "aws"
  cloud-config:
    - "/etc/aws/aws.conf"

Setting Key Value Access Pairs

Make sure the following environment variables are set in the /etc/sysconfig/origin-master-api file and /etc/sysconfig/origin-master-containers file on masters and the /etc/sysconfig/origin-node file on nodes:

AWS_ACCESS_KEY_ID=<key_ID>
AWS_SECRET_ACCESS_KEY=<secret_key>

Access keys are obtained when setting up your AWS IAM user.

Applying Configuration Changes

Start or restart OpenShift Origin services on all master and node hosts to apply your configuration changes, see Restarting OpenShift Origin services:

# systemctl restart origin-master-api origin-master-controllers
# systemctl restart origin-node

Switching from not using a cloud provider to using a cloud provider produces an error message. Adding the cloud provider tries to delete the node because the node switches from using the hostname as the externalID (which would have been the case when no cloud provider was being used) to using the AWS instance-id (which is what the AWS cloud provider specifies). To resolve this issue:

Check and back up existing node labels:

$ oc describe node <node_name> | grep -Poz '(?s)Labels.*\n.*(?=Taints)'

Delete the nodes:
```
$ oc delete node <node_name>
```
On each node host, restart the OpenShift Origin service.
```
# systemctl restart origin-node
```
Add back any labels on each node that you previously had.

Labeling Clusters for AWS

Starting with OpenShift Origin version 3.7 of the atomic-openshift-installer, if you configured AWS provider credentials, you must also ensure that all instances are labeled. This topic describes how to label an existing OpenShift Origin cluster running on AWS.

To correctly identify which resources are associated with a cluster, tag resources with the key kubernetes.io/cluster/<name>,Value=<clusterid>, where:

<name> is a unique name for the cluster.
<clusterid> is a cluster identifier unique to the AWS Availability Zone.

Tagging all resources with the kubernetes.io/cluster/<name>,Value=<clusterid> tag avoids potential issues with multiple zones or multiple clusters.

In versions prior to OpenShift Origin version 3.6, this was Key=KubernetesCluster,Value=clusterid.

See Pods and Services to learn more about labeling and tagging in OpenShift Origin.

Resources That Need Tags

There are four types of resources that need to be tagged:

Instances
Security Groups
Load Balancers
EBS Volumes

Tagging an Existing Cluster

A cluster uses the value of the kubernetes.io/cluster/<name>,Value=<clusterid> tag to determine which resources belong to the cluster. Therefore, you must tag all resources with the key kubernetes.io/cluster/<name>,Value=<clusterid> and have the same value for that key.

Tag all instances with kubernetes.io/cluster/<name>,Value=<clusterid> and a value to be used as the cluster ID.
Tag any security groups with kubernetes.io/cluster/<name>,Value=<clusterid> and the same value used for the instances.
Tag any load balancers with kubernetes.io/cluster/<name>,Value=<clusterid> and the same value used for the instances.
Tag all EBS volumes with kubernetes.io/cluster/<name>,Value=<clusterid> and the same value used for the instances. The EBS Volumes that need to be tagged can found with:
```
$ oc get pv -o json|jq '.items[].spec.awsElasticBlockStore.volumeID'
```
Restart the master services on the master and the node service on all nodes, see Applying Configuration Section.

Configuring for Amazon Web Services (AWS)