openMosix Cluster on Gentoo

1. Introduction

This HOWTO will help you create an openMosix cluster. It will be based around the Gentoo Linux distribution. We intend to make this as user friendly as possible and cater to the Linux newbie. While an experienced user could easily tie the multiple HOWTOs available on openMosix, diskless nodes and networking together, we hope that this HOWTO can ease the procedure for less-experienced users.

Note: If you plan on setting up your openMosix cluster using diskless nodes, you should read the kernel section below to learn how to compile a kernel with openMosix support, then read our diskless HOWTO, and come back here to proceed with the actual cluster installation.

About openMosix

OpenMosix is a patch to the Linux kernel that allows multiple hosts to act as a single system image (SSI). This results in multiple hosts appearing as one large multiprocessor host. At the time of writing the latest release of the openMosix kernel patch is version 2.4.22, and the openMosix user tools has version 0.3.4. There is a wide variety of information about openMosix at http://openmosix.sourceforge.net. It is not easy to cluster different versions of patched kernel sources, and we have found that most patches are not backwards compatible. OpenMosix migrates heavy weight processes explicitly when executing a.out or ELF binaries or when a heavy weight process forks. It will not migrate light weight processes such as p-threads, or heavyweight processes that use shared memory.

For more information about openMosix visit their home page.

About the cluster

Our cluster will be composed of individual computers (nodes) sharing computational resources in an effort to increase the computational power of all nodes. Not all nodes need to be of the same architecture but that makes the task of clustering them much easier.

Before you start

You should have Gentoo installed on the computers you wish to bring into your cluster. Additionally you should have the openMosix kernel source which has been conveniently patched by Gentoo.

To get this source simply use emerge:

Code Listing 1.1: Getting patched kernel source

# emerge openmosix-sources

2. Configuring the kernel

About kernels

The kernel is the software that sits between your hardware and all other software you have loaded on your machine, essentially the heart of a kernel based operating system. When your computer is started, the BIOS executes the instructions found at the reserved boot space of your hard drive. These instructions are typically a boot loader that loads your kernel. After your kernel has been loaded all processes are handled by the kernel.

For more information on kernels and kernel configuration you might want to check out the kernel HOWTO.

Configuring the kernel

The kernel can be as large and as customized as you would like but there are a few required kernel options you need to check.

Important: It is extremely important to link /usr/src/linux to the openmosix-sources as shown below.

Code Listing 2.1: Setting /usr/src/linux to openmosix

# cd /usr/src
# rm linux
# ln -s linux-2.4.22-openmosix linux

Then go into your kernel configuration:

Code Listing 2.2: Editing the kernel configuration

# cd /usr/src/linux
# make menuconfig

You should get a grey and blue GUI that offers a safe alternative to manually editing the /usr/src/linux/.config file. If your kernel is currently functioning well you might want to backup the current configuration file:

Code Listing 2.3: Backing up the kernel configuration

# cp .config .config_working

The topmost menu item should say openMosix --->. If it doesn't, you need to emerge the kernel source with the openMosix patch (see Before you start). Go into the following sub-menus and make sure the following items are checked as built-in (and NOT as module):

Code Listing 2.4: Menuconfig options

openMosix --->
    [*] openMosix process migration support
    [ ] Support clusters with a complex network topology
    [*] Stricter security on openMosix ports
    (1) Level of process-identity disclosure (0-3)
    [*] openMosix File-System
    [ ] Poll/Select exceptions on pipes
    [ ] Disable OOM Killer
    [ ] Load Limit

Code maturity level options  --->
    [*] Prompt for development and/or incomplete code/drivers

Networking options --->
    <*> Packet socket
    [ ]   Packet socket: mmapped IO
    < > Netlink device emulation
    [ ] Network packet filtering (replaces ipchains)
    [ ] Socket Filtering
    <*> Unix domain sockets
    [*] TCP/IP networking
    [*]   IP: multicasting

File systems --->
    [*] /proc file system support
    [*] /dev file system support (EXPERIMENTAL)
    [*]   Automatically mount at boot

Note: These kernel configuration options should only be appended to your system specific configuration options and are not meant to completely replace those options.

After you have re-configured the kernel you will want to rebuild it:

Code Listing 2.5: Recompiling the kernel and modules

# make dep
# make clean bzImage modules modules_install
(Make sure /boot is mounted before copying to it)
# cp arch/i386/boot/bzImage /boot/bzImage-openmosix

Then add an entry for that new kernel into lilo.conf or grub.conf (depending on which bootloader you are using) and make the new kernel the default one. Now that the new bzImage has been copied into your /boot directory all you will have to do is reboot the system.

Missing Options

If you have missing options in your kernel configuration make sure you check:

Code maturity level options --->
- Prompt for development and/or incomplete code/drivers

3. Configuring of the openMosix nodes

Installing openMosix user tools

In order for the cluster to be able to migrate processes, a few user-land binaries need to be installed. Additionally, an openMosix server needs to be started in order for a node to join a cluster and make full use of the openMosix capabilities. To get those binaries and files, use emerge:

Code Listing 3.1: Installing openMosix userland utilities

# emerge openmosix-user

Configuring openMosix nodes

Editing or creating /etc/openmosix.map is no longer needed. The newer versions of the userland utilities have an autodiscovery daemon available that automatically detects all nodes in your network.

Create a directory /mfs on every node and mount it if you want to use the openMosix filesystem (highly recommended) that allows you to access all your nodes.

Code Listing 3.2: mkdir /mfs

# mkdir /mfs

Code Listing 3.3: Sample entry in /etc/fstab

none                    /mfs            mfs             noauto,dfsa=1          0 0

Note: Not all nodes need to be up and running for openMosix to function correctly.

Starting openMosix

Starting openMosix is really simple. The following command will enable openMosix functionality in your kernel and start the autodiscovery daemon, that keeps track of all available nodes.

Code Listing 3.4: Starting openMosix

# /etc/init.d/openmosix start

4. openMosix userland utilities

Command line utilities

openMosix-user installs several useful tools on your system. To name a few:

mosmon - openMosix monitor. Allows you to see the state of all your nodes including cpu utilization, installed memory, used memory, etc.
mtop - Enhanced version of top that shows you on which node a process is actually running.
mps - Enhanced version of ps. Shows nodenumber too.
mosctl whois - This is very useful as mosmon and the other tools only tell you the node-number. With mosctl whois nodenumber you can figure out the ip or hostname of that node.

The openmosix stress test can be used to test if everything on your cluster works as it should and how it behaves under load. This package is called openmosixtest:

Code Listing 4.1: Emerge openmosixtest