OpenShift Origin Comprehensive Deployment Guide

Configure the openshift-dependencies RPM repository:

Configure the openshift-origin RPM repository:

The hosts should be running the latest "minimal install" packages, and should be configured to use NTP for clock synchronization.

At a typical network site, a DNS infrastructure will already be in place. However, this section describes a known good DNS configuration that will ensure that name resolution works properly.

Server used:

Tools used:

If you configured and started a BIND server per this document, or you are working against a BIND server that was already in place, you now need to add a record for your broker node (or host) to BIND’s database. To accomplish this task, you will use the nsupdate command, which opens an interactive shell. Replace "broker.example.com" with your preferred hostname and "10.0.0.1" with the ip address of the broker:

Press <ctrl-D> to exit from the interactive session.

In order to verify that you have successfully added your broker node to your DNS server, you can perform:

and it should resolve to the local machine that you are working on. You can also perform a dig request using the following command:

Server used:

Tools used:

In order to use MongoDB, you will need to install the mongod server:

This will pull in a number of RPM dependencies and install them automatically.

MongoDB uses a configuration file for its settings. This file can be found at /etc/mongodb.conf. You will need to make a few changes to this file to ensure that MongoDB handles authentication correctly and that is enabled to use small files.

If MongoDB is setup on a machine that is not running the broker, you will need to ensure that the MongoDB is configured to listen on the external IP and that the firewall allows MongoDB connections to pass-through.

Edit the mongodb.conf file and update the bind_ip setting.

Enable MongoDB access on the firewall.

MongoDB is an essential part of the OpenShift Origin platform. Because of this, you must ensure that mongod is configured to start on system boot:

By default, when you install mongod via the yum command, the service is not started. You can verify this with the following:

This should return "mongod is stopped". In order to start the service, simply issue:

Now verify that mongod was installed and configured correctly. To do this, use the mongo shell client tool. If you are familiar with MySQL or Postgres, this is similar to the mysql client’s interactive SQL shell. However, because MongoDB is a NoSQL database, it does not respond to traditional SQL-style commands.

In order to start the mongo shell, enter the following command:

You should see a confirmation message that you are using MongoDB shell version: x.x.x and that you are connecting to the admin database. Authenticate against the database with the user you created above.

To verify even further, you can list all of the available databases that the database currently has:

You will then be presented with a list of valid databases that are currently available to the mongod service.

To exit the Mongo shell, you can simply type exit:

Installing ActiveMQ on Fedora is a fairly easy process as the packages are included in the rpm repositories that are already configured on your broker node. You need to install both the server and client packages by using the following command:

ActiveMQ uses an XML configuration file that is located at /etc/activemq/activemq.xml. This installation guide is accompanied by a template version of activemq.xml that you can use to replace this file. But first: back up the original file:

Copy the basic configuration template in to /etc/activemq/activemq.xml.

Copy the jetty template in to /etc/activemq/jetty.xml.

Copy the jetty auth template in to /etc/activemq/jetty-realm.properties.

Once you have the configuration template in place, you will need to make a few minor changes to the configuration.

First, replace the hostname provided (activemq.example.com) to the FQDN of your broker host. For example, the following line:

Should become:

The second change is to provide your own credentials for authentication. The authentication information is stored inside of the __ block of code. Make the changes that you desire to the following code block:

Next modify the /etc/activemq/jetty-realm.properties and set a password for the admin user

The broker host firewall rules must be adjusted to allow MCollective to communicate on port 61613:

Finally, you need to enable the ActiveMQ service to start on boot as well as start the service for the first time.

Now that ActiveMQ has been installed, configured, and started, verify that the web console is working as expected. The ActiveMQ web console should be running and listening on port 8161. In order to verify that everything worked correctly, load the following URL in a web browser:

For example, the following command adds a rule to your firewall to allow connections to the ActiveMQ console.

Execute the following on the ActiveMQ host.

Alternatively, the following command creates a SSH tunnel, so that if you connect to port 8161 on your local host, the connection will be forwarded to port 8161 on the remote host, where the the ActiveMQ console is listening.

Execute the following on your local machine.

Note: Use the username and password from the jetty-realm.properties file to log into the console.

In order to use MCollective, first install it via yum:

Replace the contents of the /opt/rh/ruby193/root/etc/mcollective/client.cfg with the following information:

Update the plugin.activemq.pool.1.password password to match what you set up in the active configuration.

Now you have configured the MCollective client to connect to ActiveMQ running on the local host. In a typical deployment, you will configure MCollective to connect to ActiveMQ running on a remote server by putting the appropriate hostname for the plugin.activemq.pool.1.host setting.

In order for users to interact with the OpenShift Origin platform, they will typically use client tools or the web console. These tools communicate with the broker via a REST API that is also accessible for writing third party applications and tools. In order to use the broker application, we need to install several packages from the OpenShift Origin repository.

The broker application requires a number of services to be running in order to function properly. Configure them to start at boot time:

Additionally, modify the firewall rules to ensure that the traffic for these services is accepted:

Now you will need to generate access keys that will allow some of the services (Jenkins for example) to communicate to the broker.

You will also need to generate a ssh key pair to allow communication between the broker host and any nodes that you have configured. For example, the broker host will use this key in order to transfer data between nodes when migrating a gear from one node host to another.

In order to generate this SSH keypair, perform the following commands:

Press <enter> for the passphrase. This generates a passwordless key which is convenient for machine-to-machine authentication but is inherently less secure than other alternatives. Finally, copy the private and public key files to the openshift directory:

Later, during configuration of the node hosts, you will copy this newly created key to each node host.

SELinux has several variables that we want to ensure are set correctly. These variables include the following:

In order to set all of these variables correctly, enter the following:

You will also need to set several files and directories with the proper SELinux contexts. Issue the following commands:

The fixfiles command updates SELinux’s database that associates pathnames with SELinux contexts. The restorecon command uses this database to update the SELinux contexts of the specified files on the file system itself so that those contexts will be in effect when the kernel enforces policy. See the manual pages of the fixfiles and restorecon commands for further details.

To begin, copy the .example files to actual configuration files that will be used by OpenShift Origin:

The broker application will check the plugins.d directory for files ending in .conf. The presence of .conf file enables the corresponding plug-in. Thus, for example, copying the openshift-origin-auth-remote-user.conf.example file to openshift-origin-auth-remote-user.conf enables the auth-remote-user plug-in.

If you installed a DNS server on the same host as the broker, you can create a DNS configuration file using the cat command instead of starting with the example DNS configuration file. You can do that by taking advantage of the $domain and $keyfile environment variables that you created during that process. If you no longer have these variables set, you can recreate them with the following commands:

To verify that your variables were recreated correctly, echo the contents of your keyfile and verify your $KEY variable is set correctly:

If you performed the above steps correctly, you should see output similar to this:

and

Now that you have your variables setup correctly, you can create the openshift-origin-dns-bind.conf file. Ensure that you are still in the /etc/openshift/plugins.d directory and issue the following command:

After running this command, cat the contents of the file and ensure they look similar to the following:

OpenShift Origin supports various different authentication systems for authorizing a user. In a production environment, you will probably want to use LDAP, kerberos, or some other enterprise class authorization and authentication system. For this reference system we will use a system called Basic Auth that relies on a htpasswd file to configure authentication. OpenShift Origin provides three example authentication configuration files in the /var/www/openshift/broker/httpd/conf.d/ directory:

Using Basic Auth, you need to copy the sample configuration file to the actual configuration file:

This configuration file specifies that the AuthUserFile is located at /etc/openshift/htpasswd. At this point, that file doesn’t exist, so you will need to create it and add a user named demo.

After entering the above command, you will be prompted for a password for the user demo. Once you have provided that password, view the contents of the htpasswd file to ensure that the user was added correctly. Make a note of the password as you will need it later.

If the operation was a success, you should see output similar to the following:

The optional OpenShift Origin administrative console (a.k.a. "admin console") enables OpenShift administrators an at-a-glance view of an OpenShift deployment, in order to search and navigate OpenShift entities and make reasonable inferences about adding new capacity.

The broker will load the plugin if the gem is installed and its configuration file is placed at /etc/openshift/plugins.d/openshift-origin-admin-console.conf (or ...-dev.conf for specifying development-mode settings) which the RPM does by default. Edit the configuration file as needed; options are commented, and discussed more fully in the Administration Guide.

The last step in configuring our broker application is to ensure that all of the necessary services are started and that they are configured to start upon system boot.

This will ensure that the broker starts upon next system boot. However, you also need to start the broker application to run now.

In order to verify that the REST API is functioning for the broker host, you can use the following curl command:

You should see the following output:

Start the apache server on the node to proxy web traffic to the broker.

In order to verify that the REST API is functioning for the broker host, you can use the following curl command:

At this point you have a fully functional Broker. In order to work with it, proceed through the Web Console installation.

The installation of the web console can be performed with a simple yum install command, but note that it will pull in many dependencies from the Ruby programming language. At the time of this writing, executing the following command installed 77 additional packages.

If you are building the reference configuration described in this document, then you have configured the broker application for Basic Authentication. What you actually configured was authentication for the Broker REST API. The console application uses a separate authentication scheme for authenticating users to the web console. This will enable you to restrict which users you want to have access to the REST API and keep that authentication separate from the web based user console.

The openshift-console package created some sample authentication files for us. These files are located in the /var/www/openshift/console/httpd/conf.d directory. For this reference configuration, you will use the same htpasswd file that you created when you set up authentication for the Broker application. In order to do this, issue the following commands:

SELinux has several variables that we want to ensure are set correctly. These variables include the following:

In order to set all of these variables correctly, enter the following:

You will also need to set several files and directories with the proper SELinux contexts. Issue the following commands:

The fixfiles command updates SELinux’s database that associates pathnames with SELinux contexts. The restorecon command uses this database to update the SELinux contexts of the specified files on the file system itself so that those contexts will be in effect when the kernel enforces policy. See the manual pages of the fixfiles and restorecon commands for further details.

Start the service and ensure it starts on boot:

Once completed, the console will prompt the user to provide their login credentials as specified in the /etc/openshift/htpasswd file.

SSH to your broker application host and set a variable that points to your keyfile. The following command should work after you replace "example.com" with the domain that you are going to use.

In order to configure your DNS to resolve your node host, we need to tell our BIND server about the host. Run the following command and replace the IP address with the correct IP address of your node.

Execute the following on the broker host:

Now that you have added your node host to the DNS server, the broker application host should be able to resolve the node host by referring to it by name. Let’s test this:

This should resolve to the 10.4.59.y IP address that you specified for the node host in the oo-register-node command.

While on the broker application host, you need to copy the SSH key that you previously created over to the node. This will enable operations to work from inside of OpenShift Origin without requiring a password.

Once you connect to the broker host, copy the key with the following command:

Execute the following on the broker host:

Once you enter that command, you will be prompted to authenticate to the node host.

At this point, you need to login to your node host to add the newly copied key to our authorized_keys. SSH into your node host and run the following:

Execute the following on the node host:

Now that your key has been copied from your broker application host to your node host, let’s verify that is copied correctly and was added to the authorized_keys file. Once you issue the following command, you should be authenticated to the node host without having to specify the root user password.

Verify the key by executing the following on the broker host:

Now you need to configure the node host to use the BIND server that was installed and configured on the broker application host. This is a fairly straightforward process of adding the IP address of the DNS server to the /etc/resolv.conf on the node host.

Edit this file and add the following line, making sure to use the correct IP address of your broker host:

Perform this change on the node host:

You can skip this section if you are building an all-in-one environment.

On the node host, configure your system settings to prepend the DNS server to the resolv.conf file on system boot. This will allow the node host to resolve references to broker.example.com to ensure that all pieces of OpenShift Origin can communicate with one another. This process is similar to setting up the dhclient-eth0.conf configuration file for the broker application.

Edit the /etc/dhcp/dhclient-eth0.conf file, or add it if it doesn’t exist, and add the following information ensuring that you replace the IP address with the correct IP of your broker application host:

Update your network scripts to use the DNS server. Update /etc/sysconfig/network-scripts/ifcfg-<eth device> file and add the following information making sure to substitute the IP address of the broker host:

Now set the hostname for node host to correctly reflect node.example.com.

Edit the /etc/sysconfig/network file and change the HOSTNAME entry to the following:

Finally, set the hostname for your current session by issuing the hostname command at the command prompt.

Verify that the hostname was set correctly by running the hostname command. If the hostname was set correctly, you should see node.example.com as the output of the hostname command.

Server used:

Tools used:

MCollective is the tool that OpenShift Origin uses to send and receive messages via the ActiveMQ messaging server. In order for the node host to send and receive messages with the broker application, you need to install and configure MCollective on the node host to communicate with the broker application.

Configure the MCollective server to communicate with the broker application service. In order to accomplish this, replace the contents of the MCollective server.cfg configuration file to point to your correct stomp host. Edit the /opt/rh/ruby193/root/etc/mcollective/server.cfg file and add the following information. If you used a different hostname for your broker application host, ensure that you provide the correct stomp host. You also need to ensure that you use the same username and password that you specified in your ActiveMQ configuration.

Update the plugin.activemq.pool.1.password password to match what you set up in the active configuration.

Now ensure that MCollective is set to start on boot and also start the service for our current session.

At this point, MCollective on the node host should be able to communicate with the broker application host. You can verify this by running the mco ping command on the broker.example.com host.

If MCollective was installed and configured correctly, you should see node.example.com in the output from the previous command.

Server used:

Tools used:

Just as we installed specific packages that provide the source code and functionality for the broker application to work correctly, the node host also has a set of packages that need to be installed to properly identify the host as a node that will contain application gears.

The node host will need to allow HTTP, HTTPS, and SSH traffic to flow through the firewall. We also want to ensure that the httpd, network, and sshd services are set to start on boot.

Augeas is a very useful toolset to perform scripted updates to configuration files. Run the following to install it:

The pam_namespace PAM module sets up a private namespace for a session with polyinstantiated directories. A polyinstantiated directory provides a different instance of itself based on user name, or when using SELinux, user name, security context or both. OpenShift Origin ships with its own PAM configuration and we need to configure the node to use the configuration.

Cgroups enable you to allocate resources—such as CPU time, system memory, network bandwidth, or combinations of these resources—among user-defined groups of tasks (processes) running on a system. You can monitor the cgroups you configure, deny cgroups access to certain resources, and even reconfigure your cgroups dynamically on a running system.

In order for cgroups to work correctly, you need to ensure that services are started in the correct order.

When a consumer of OpenShift Origin creates an application gear, you will need to be able to control and set the amount of disk space that the gear can consume. This configuration is located in the /etc/openshift/resource_limits.conf file. The two settings of interest are quota_files and quota_blocks. The quota_files setting specifies the total number of files that a gear / user is allowed to own. The quota_blocks is the actual amount of disk storage that the gear is allowed to consume — where 1 block is equal to 1024 bytes.

In order to enable usrquota on the filesystem, you will need to add the usrquota option in the /etc/fstab for the mount of /var/lib/openshift. In this chapter, the /var/lib/openshift directory is mounted as part of the root filesystem. The corresponding line in the /etc/fstab file looks like

In order to add the usrquota option to this mount point, change the entry to the following:

For the usrquota option to take effect, you can reboot the node host or simply remount the filesystem:

And then generate user quota info for the mount point:

Server used:

Tools used:

Server used:

Tools used:

Configure traffic control to measure and control the amount of outgoing and incoming traffic used by each gear:

Update the minimum UID and GID for the machine to match GEAR_MIN_UID from node.conf. This value is 500 by default.

Facter generates metadata files for MCollective and is normally run by cron. Run the following command to execute facter immediately to create the initial database and ensure that it runs properly:

In order to verify that all services were installed and configured correctly, restart the node to ensure that all services start on boot as described.

As of OpenShift Origin v4, Nodes must belong to a district in order to work properly. Adding a Node to a district after the Node already has hosted apps running on it is very difficult, so this is a very important pre-deployment task. For a discussion of what districts are, refer to the Administration Guide.

Define one or more districts
From a Broker host, run the following command to define a new district:

Add one or more Nodes to the district(s)
To perform a blank assignment of all Nodes to a district, run:

Otherwise add Nodes one at a time with:

From a Broker host, run the following command to poll a Node for available cartridge information:

This will automatically register the cartridges with the Broker and make them available to users for new hosted applications.

If you are using Linux, the process for updating your name server is straightforward. Simply edit the /etc/resolv.conf configuration file and add the IP address of your broker node as the first entry. For example, add the following at the top of the file, replacing the 10.4.59.x IP address with the correct address of your broker node:

Once you have added the above nameserver, you should be able to communicate with your OpenShift Origin PaaS by using the server hostname. To test this out, ping the broker and node hosts from your local machine:

If you are using OSX, you will notice that the operating has a /etc/resolv.conf configuration file. However, the operating system does not respect this file and requires users to edit the DNS servers via the System Preferences tool.

Open up the System Preferences tool and select the Network utility:

On the bottom left hand corner of the Network utility, ensure that the lock button is unlocked to enable user modifications to the DNS configuration. Once you have unlocked the system for changes, locate the Ethernet device that is providing connectivity for your machine and click the advanced button:

Select the DNS tab at the top of the window:

Click the + button to add a new DNS server and enter the 10.4.59.x IP address of your broker host.

After you have applied the changes, we can now test that name resolution is working correctly. To test this out, ping the broker and node hosts from your local machine:

Broker clustering is accomplished in the Puppet module by using HAProxy and a virtual host / IP address as a front for an arbitrary number of Broker instances. One Broker host also serves as the HAProxy host, and calls to the "virtual Broker" are resolved here and directed via HAProxy to one of the real Brokers. Be aware, however, that the HAProxy instance in this scenario represents a single point of failure. As a failsafe, individual Broker instances can still be addressed using their own hostnames and IP addresses.

The backend MongoDB instance that the Broker uses to manage hosted applications can be replicated using MongoDB’s native support. One MongoDB service is identified as the master instance and the others must be registered to it as slave instances.

In the Broker config file, located on each Broker host at /etc/openshift/broker.conf, the following two settings must be modified to make OpenShift aware of the replication:

For message server redundancy, OpenShift makes use of ActiveMQ’s native clustering capability. This does not require any special configuration in OpenShift. Refer to ActiveMQ’s documentation for details on setting this up.