Cloud Controller

Page last updated: July 31, 2015

The Cloud Controller is written in Ruby and provides REST API endpoints for clients to access the system. The Cloud Controller maintains a database with tables for orgs, spaces, apps, services, service instances, user roles, and more.

Droplet Execution Agent (DEA) Placement Algorithm

Whenever Cloud Foundry needs to spin up a new instance of an app, the cloud controller is responsible for selecting a droplet execution agent (DEA) to run it.

DEAs broadcast their availability to the cloud controller via a NATS message called an advertisement, which contains a stats hash with information about their available memory, available disk, and the stack which the DEA is running, as well as an expiration time.

The cloud controller collects these advertisements in a construct called a pool. When the cloud controller is called upon to find a DEA to run an app, it runs through the following steps, using criteria (minimum thresholds for disk, memory, etc.) specific to the app that the chosen DEA will run:

  1. It removes the expired DEA advertisements from the pool.

  2. It filters the remaining advertisements to include only those:

    • with adequate disk
    • with adequate memory
    • running the required stack (linux or windows)

  3. It then narrows its search to DEAs running in the availability zone with the fewest running instances (according to the information provided by the advertisements in the pool).

  4. It then narrows its search to the DEAs with the fewest running instances.

  5. It then narrows its search to the top half of the DEAs, sorted by memory.

  6. It then randomly selects one of the remaining DEAs.

Considerations

It is important to note that the cloud controller uses this algorithm to balance new app instances between DEAs when the new app instances are created, but do not balance already-running apps.

For example, suppose a set of apps are running on DEAs in two AZs, and one AZ temporarily goes down. While the second AZ is down, all instances will be placed on the remaining AZ. After the second AZ comes back online, new instances will be allocated to DEAs there, since the algorithm favors DEAs in the zone with the fewest running instances. However, instances running on the first AZ will not be moved to the other AZ, so the imbalance will persist.

An imbalance may also result from a deploy where DEAs have had a change to their source code or stemcell.

It is possible to rebalance the DEAs between AZs in two ways:

  • Restarting the app, which may result in a brief down-time.
  • Terminating and restarting half of the instances one by one.

Database (CC_DB)

The Cloud Controller database has been tested with Postgres.

Blob Store

The Cloud Controller manages a blob store for the following:

  • Resources: Files that are uploaded to the Cloud Controller with a unique SHA such that they can be reused without re-uploading the file

  • App Packages: Unstaged files that represent an application

  • Droplets: Result of taking an app package, staging it by processing a buildpack, and preparing it to run

The blob store uses the Fog Ruby gem such that it can use abstractions like Amazon S3 or an NFS-mounted file system for storage.

NATS Messaging

The Cloud Controller interacts with other core components of the Cloud Foundry platform using the NATS message bus. For example, the Cloud Controller uses NATS when performing the following interactions:

  • Instructing a DEA to stage an application by processes a buildpack for the app and to prepare it to run
  • Instructing a DEA to start or stop an application
  • Receiving information from the Health Manager about applications
  • Subscribing to Service Gateways that advertise available services
  • Instructing Service Gateways to handle provisioning, unprovisioning, binding, and unbinding operations for services

Testing

By default rspec runs a test suite with the SQLite in-memory database. Specify a connection string using the DB_CONNECTION environment variable to test against Postgres and MySQL. For example:

    DB_CONNECTION="postgres://postgres@localhost:5432/ccng" rspec
    DB_CONNECTION="mysql2://root:password@localhost:3306/ccng" rspec

Travis currently runs three build jobs against SQLite, Postgres, and MySQL.

Logs

Cloud Controller manages its logs using Steno. Each log entry includes a source field to designate from which code module the entry originates. Some of the possible sources are cc.app, cc.app_stager, cc.dea.client and cc.healthmanager.client.

Configuration

The Cloud Controller uses a YAML configuration file. For an example, see config/cloud_controller.yml. Some of the keys the Cloud Controller reads from this configuration file include:

  • logging: A Steno configuration hash
  • bulk_api: - Basic auth credentials for the application state bulk API. In Cloud Foundry, the health manager uses this endpoint to retrieve the expected state of every user application.
  • uaa: URL and credentials for connecting to the UAA, the Cloud Foundry OAuth 2.0 server.