Chapter 5. Scaling

Determining the scalability of your cloud and how to improve it is an exercise with many variables to balance. No one solution meets everyone's scalability goals. However, it is helpful to track a number of metrics. Since you can define virtual hardware templates called "flavors" in OpenStack, you can start to make scaling decisions based on the flavors you'll provide. These templates define sizes for memory in RAM, root disk size, amount of ephemeral data disk space available, and number of cores for starters.

The default OpenStack flavors are:

The starting point for most is the core count of your cloud. By applying some ratios, you can gather information about:

You can use these ratios to determine how much additional infrastructure you need to support your cloud.

Here is an example using the ratios for gathering scalability information for the number of VMs expected as well as the storage needed. The following numbers support (200 / 2) × 16 = 1600 VM instances and require 80 TB of storage for /var/lib/nova/instances:

However, you need more than the core count alone to estimate the load that the API services, database servers, and queue servers are likely to encounter. You must also consider the usage patterns of your cloud.

As a specific example, compare a cloud that supports a managed web-hosting platform with one running integration tests for a development project that creates one VM per code commit. In the former, the heavy work of creating a VM happens only every few months, whereas the latter puts constant heavy load on the cloud controller. You must consider your average VM lifetime, as a larger number generally means less load on the cloud controller.

Aside from the creation and termination of VMs, you must consider the impact of users accessing the service—particularly on nova-api and its associated database. Listing instances garners a great deal of information and, given the frequency with which users run this operation, a cloud with a large number of users can increase the load significantly. This can occur even without their knowledge—leaving the OpenStack dashboard instances tab open in the browser refreshes the list of VMs every 30 seconds.

After you consider these factors, you can determine how many cloud controller cores you require. A typical eight core, 8 GB of RAM server is sufficient for up to a rack of compute nodes — given the above caveats.

You must also consider key hardware specifications for the performance of user VMs. You must consider both budget and performance needs. Examples include: storage performance (spindles/core), memory availability (RAM/core), network bandwidth (Gbps/core), and overall CPU performance (CPU/core).

	Tip
	For a discussion of metric tracking, including how to extract metrics from your cloud, see Chapter 13, Logging and Monitoring.

You can facilitate the horizontal expansion of your cloud by adding nodes. Adding compute nodes is straightforward—they are easily picked up by the existing installation. However, you must consider some important points when you design your cluster to be highly available.

Recall that a cloud controller node runs several different services. You can install services that communicate only using the message queue internally—nova-scheduler and nova-console—on a new server for expansion. However, other integral parts require more care.

You should load balance user-facing services such as Dashboard, nova-api, or the Object Storage proxy. Use any standard HTTP load-balancing method (DNS round robin, hardware load balancer, software such as Pound or HAProxy). One caveat with Dashboard is the VNC proxy, which uses the WebSocket protocol—something that an L7 load balancer might struggle with. See also Horizon session storage (http://docs.openstack.org/developer/horizon/topics/deployment.html#session-storage).

You can configure some services, such as nova-api and glance-api, to use multiple processes by changing a flag in their configuration file—allowing them to share work between multiple cores on the one machine.

	Tip
	Several options are available for MySQL load balancing, and the supported AMQP brokers have in-built clustering support. Information on how to configure these and many of the other services can be found in Part II: Operations.

When you want to offer users different regions to provide legal considerations for data storage, redundancy across earthquake fault lines, or for low-latency API calls, you segregate your cloud. Use one of the following OpenStack methods to segregate your cloud: cells, regions, zones, and host aggregates.

Each method provides different functionality, and can be best divided into two groups:

Table 5.1, “OpenStack Segregation Methods” provides a comparison view of each segregation method currently provided by OpenStack Compute.

 Availability Zones and Host Aggregates

You can use availability zones, host aggregates, or both to partition a nova deployment.

Availability zones are implemented through and configured in a similar way to host aggregates.

However, you use them for different reasons:

• Availability zone. Enables you to arrange OpenStack Compute hosts into logical groups, and provides a form of physical isolation and redundancy from other availability zones, such as by using a separate power supply or network equipment.

  You define the availability zone in which a specified Compute host resides locally on each server. An availability zone is commonly used to identify a set of servers that have a common attribute. For instance, if some of the racks in your data center are on a separate power source, you can put servers in those racks in their own availability zone. Availability zones can also help separate different classes of hardware.

  When users provision resources, they can specify from which availability zone they want their instance to be built. This allows cloud consumers to ensure that their application resources are spread across disparate machines to achieve high availability in the event of hardware failure.

• Host aggregates. Enable you to partition OpenStack Compute deployments into logical groups for load balancing and instance distribution. You can use host aggregates to further partition an availability zone. For example, you might use host aggregates to partition an availability zone into groups of hosts that either share common resources, such as storage and network, or have a special property, such as trusted computing hardware.

  A common use of host aggregates is to provide information for use with the nova-scheduler. For example, you might use a host aggregate to group a set of hosts that share specific flavors or images.

  The general case for this is setting key-value pairs in the aggregate metadata and matching key-value pairs in instance type extra specs. The AggregateInstanceExtraSpecsFilter in the filter scheduler will enforce that instances be scheduled only on hosts in aggregates that define the same key to the same value.

  An advanced use of this general concept allows different instance types to run with different CPU and RAM allocation rations so that high-intensity computing loads and low-intensity development and testing systems can share the same cloud without either starving the high-use systems or wasting resources on low-utilization systems. This works by setting metadata in your host aggregates and matching extra_specs in your instance types.

  The first step is setting the aggregate metadata keys cpu_allocation_ratio and ram_allocation_ration to a floating point value. The filter schedulers AggregateCoreFilter and AggregateRamFilter will use those values rather than the global defaults in nova.conf when scheduling to hosts in the aggregate. It is important to be cautious when using this feature since each host can be in multiple aggregates but should have only one allocation ratio for each resources. It is up to you to avoid putting a host in multiple aggregates that define different values for the same resource.

  This is the first half of the equation. To get instance types that are guaranteed a particular ratio, you must set the extra_specs in the instance type to the key value pair you want to match in the aggregate. For example if you define extra specs cpu_allocation_ratio to "1.0", then instances of that type will run in aggregates only where the metadata key cpu_allocation_ratio is also defined as "1.0." In practice, it is better to define an additional key-value pair in the aggregate metadata to match on rather than match directly on cpu_allocation_ratio or core_allocation_ratio. This allows better abstraction. For example, by defining a key overcommit and setting a value of "high," "medium," or "low," you could then tune the numeric allocation ratios in the aggregates without also needing to change all instance types relating to them.

Note

Previously, all services had an availability zone. Currently, only the nova-compute service has its own availability zone. Services such as nova-scheduler, nova-network, and nova-conductor have always spanned all availability zones. When you run any of the following operations, the services appear in their own internal availability zone (CONF.internal_service_availability_zone): nova host-list (os-hosts) euca-describe-availability-zones verbose nova-manage service list The internal availability zone is hidden in euca-describe-availability_zones (non-verbose). CONF.node_availability_zone has been renamed to CONF.default_availability_zone and is used only by the nova-api and nova-scheduler services. CONF.node_availability_zone still works but is deprecated.

While several resources already exist to help with deploying and installing OpenStack, it's very important to make sure that you have your deployment planned out ahead of time. This guide presumes that you have at least set aside a rack for the OpenStack cloud but also offers suggestions for when and what to scale.