Administration Guide

If you are familiar with JSON, you will understand the format of this output. What actually happened is a new document was created in the broker's MongoDB database. To view this document inside of the database, execute the following (substitute MongoDB access parameters from broker.conf if needed):

This will drop you into the MongoDB shell where you can perform commands against the broker database. To list all of the available collections in the openshift_broker_dev database, you can issue the following command:

You should see the following collections returned:

We can now query the districts collection to verify the creation of our small district:

The output should be similar to:

Exit the MongoDB shell using the exit command:

Now we can add our node host, node.example.com, to the small_district that we created above:

It is important to note that the server identity (node.example.com here) is the node’s hostname as configured on that node, which could be different from the PUBLIC_HOSTNAME configured in /etc/openshift/node.conf on the node. The PUBLIC_HOSTNAME is used in CNAME records and must resolve to the host via DNS; the hostname could be something completely different and may not resolve in DNS at all.

The hostname is recorded in MongoDB both in the district and with any gears that are hosted on the node, so changing the node’s hostname will disrupt the broker’s ability to use the node. In general, it’s wisest to use the hostname as the DNS name and not change either after install.

You should see output like the following from the node addition:

Repeat the steps above to query the database for information about districts. Notice that the servers array now contains the following information:

If you continued to add additional nodes to this district, the servers array would show all the node hosts that are assigned to the district.

This command line tool can be used just to display district information. Simply run the command with no arguments to view the JSON records in the MongoDB database for all districts:

For a node host, the maximum number of active gears allowed per node is specified with the max_active_gears value in /etc/openshift/resource_limits.conf; by default it is 100, but most administrators will need to modify this. Note that stopped or idle gears are not counted toward this limit; it is possible for a node to have any number of inactive gears, bounded only by storage. It is also possible to exceed the limit by starting inactive gears after the limit has been reached - nothing prevents or corrects this; reaching the limit simply exempts the node from future gear placement by the broker.

Determining the max_active_gears limit to use involves a certain amount of prognostication on the part of an administrator. The safest way to calculate the limit is to consider the resource most likely to be exhausted first (typically RAM) and divide the amount of available resource by the resource limit per gear.

So, for example, if a node host has 7.5 GB of RAM available and gears are constrained to .5 GB RAM:

However, in practice, most gears will not consume their entire resource quota, so this conservative limit would leave a lot of wasted resources. Most administrators will want to overcommit at least some of their systems by allowing more gears than would fit if all used all their resources; and this is where prognostication (or better, experimentation) is required. Based on the types of cartridges and applications expected in the installation and how much RAM (or other scarce resources - CPU, network bandwidth, processes, inodes…) they actually use, administrators should determine an overcommit percent by which to increase their limits.

There is no harm in changing max_active_gears after installation. It may be wisest to begin with conservative limits and adjust them upwards after empirical evidence of usage is available. It is easier to add more active gears than to move them away.

Due to current constraints, each district can only contain 6000 gears. It is important not to put too many node hosts in a district, because once a district’s UID pool is exhausted, nodes in that district will not receive any more gears, even if they have plenty of capacity; therefore, resources will be wasted. It is possible to remove excess nodes from a district by deactivating them and moving all of their gears away (known as "compacting" the district); but this should be avoided if possible to minimize disruption to the gears, and because mass moves of gears are slow and failure-prone at this time.

Districts exist to facilitate gear movement; the only advantage to having more than two or three nodes in a district is that there are fewer districts to keep track of. It is easy to add nodes to a district, and difficult to remove them. Therefore, adding nodes to districts very conservatively is wise, and it would be simplest to just plan on districts having two or three nodes.

With perfect knowledge, we could calculate how many node hosts to put in each district. It is a function of the following values:

However, on nodes, we do not limit G; we want to make sure we are filling the capacity for active gears:

For deployments that use the idler to idle inactive gears, or that stop many applications for any other reason, the percentage of active gears in the long run may be very low. It is important to take this into account because the broker will keep filling the nodes to the active limit as gears are stopped or idled, but the district capacity must also contain all those inactive gears. We can project roughly how many gears a "full" node will have in the long run by determining (guessing, at first, then adjusting):

Then our estimate of G is simply C * 100 / A, and thus the number of nodes per district should be:

For example, if only 10% of gears are active over time, and max_active_gears is 50, then 6000 * 10 / (100 * 50) = 12 (round down if needed) nodes should be added per district.

In performing this calculation with imperfect knowledge, however, it is best to be conservative by guessing a low value for A and a high value for C. Adding nodes later is much better than compacting districts.

There is a tool for viewing gear usage across nodes and districts; it can be invoked on the broker:

Consult the man page or the output of oo-stats -h for script arguments. By default, this tool summarizes gear usage by districts and profiles in a human-readable format. It can also produce several computer-readable formats for use by automation or monitoring.

Notably absent from the config file is any sort of access control. There is no concept of an OpenShift administrative role. Either a visitor can browse to the admin console or not, so the place to control access is with proxy configuration. Keep in mind that the current admin console is informational only and any actions to be taken require logging in to an OpenShift host.

The front page of the admin console provides a visual and numeric summary of the capacity and usage of the entire installation. It can also be configured to provide suggestions for when an administrator should adjust capacity. As no two OpenShift environments are quite alike, the default is not to set any thresholds, and thus to make no capacity suggestions. Configuring the capacity planning settings in the config file enables suggestions that can help draw administrator attention to current or impending capacity problems: for example, where to add nodes to ensure a particular profile can continue to create gears, or where capacity is being wasted.

Please reference the main capacity planning section in this document to understand the information the admin console is displaying here and the significance of the settings. Suggestions for adding and removing capacity are based on both the settings as well as the existing data, with a bias toward being conservative in putting nodes in districts. In particular, in making that calculation, if the observed active gear percent is lower than expected, the observed percent will be used, and if the nodes do not all have the same max_active_gears limit, the largest will be used.

Note that the capacity data and suggestions are generated and cached (for one hour unless configured otherwise). If changes you expect to see haven’t shown up, you likely just need to refresh the data by clicking on the refresh icon in any page.

The admin console uses the same Admin Stats library used by oo-stats to collect capacity data. In fact, you can record YAML or JSON output from oo-stats and use this directly instead of the actual system data:

Then copy this file to where you have the admin-console loaded, configure it as STATS_FROM_FILE in the configuration file, adjust its context as described below, and restart the broker. Capacity views and suggestions will all be based on the loaded data (although navigation will still only work for entities actually present).

You need to ensure that the broker can actually read the data file. Because SELinux limits what the broker application can read (for example, it cannot ordinarily read /tmp entries), the file’s context will likely need adjustment as follows:

This script checks that broker setup is valid and functional. It is run without options on a broker.

If there are no errors, it simply prints "PASS" and exits with return code 0 (unless the -v option is added, in which case it also prints the checks that it is performing).

If there are errors, they are printed, and the return code is the number of errors.

This is a good monitoring script to make sure nothing has gone wrong with a broker host.

This script checks that application records in the MongoDB datastore are consistent with gear presence on the node hosts. It is a good sanity check for proper system operation.

Typical output:

(Without -v you just get the "Success" line.)

If this does not run cleanly, consult the Troubleshooting Guide for hints.

A utility for updating DNS A records in BIND (generally for a broker or node host, though could be other infrastructure hosts. Do not use this to change DNS records for apps/gears, as those are CNAME records). It just wraps an nsupdate command.

This is a utility for all district operations. See the full explanation in a later section. The options are as follows:

Used to move a gear from one node in a district to another, or even outside its district.

This is used to administer what a user is allowed to use on the system, mainly number and type of gears. Note that this is not the full usage listing as we won’t discuss subaccounts until a later installment of the documentation.

Used to query and control a user’s domain (AKA namespace). This reports basically all of the information there is to know about the user’s properties in OpenShift, as well as allowing domain and sshkey updates.

The "info" command is default if no other is provided. The output is very detailed YAML.

Used to administratively run commands against an app.

This script checks that node setup is valid and functional and its gears are in good condition. It is run without options on a node.

If there are no errors, it simply prints "PASS" and exits with return code 0.

If there are errors, they are printed, and the return code is the number of errors. Here are the items that it checks (can be used with -v to show these details; otherwise you see just errors and end result):

This is a good monitoring script to make sure nothing has gone wrong with a node host.

Good overview of gear statistics in general (not necessarily related to idling). No options, just returns a single line of stats about the gears on the node.

Node system script for stopping/starting gears on a node. This is used by the openshift-gears service at boot time to activate existing gears. It can also be used directly by an administrator.

Usage is like a service script:

The idler is a tool for shutting down gears that haven’t been used recently in order to reclaim their resources and overcommit the node host’s resources.

See https://github.com/openshift/origin-server/blob/master/node-util/README-Idler.md for more details about the idler.