Mini-Wulf

A small Beowulf Cluster running FreeBSD

(With apologies to Verne Troyer and The GIMP)

FreeBSD Beowulf cluster prototype/test system. Six PCs:


	Master node: "Master"
		Pentium 133
		64M RAM
		8GB hard drive
		2 10bT Ethernet NICs
			3Com
			Addtron

	Slave node: "Alpha"
		Pentium 133
		32M RAM
		2 1.5GB hard drives
		3Com 10bT NIC

	Slave node: "Bravo"
		K6/2 333
		150M RAM
		3GB hard drive
		3Com 10/100bT NIC

	Slave node: "Charlie"
		Pentium 133
		64M RAM
		2GB hard drive
		6GB hard drive
		3Com 10bT NIC

	Slave node: "Delta"
		Pentium 133
		32M RAM
		1GB hard drive
		3Com 10bT NIC

	Slave node: "Echo"
		Pentium 100
		64M RAM
		1GB hard drive
		4GB hard drive
		SMC 10bT NIC

	Network:
		Addtron 8-port 10bT ethernet hub
		Cat 5 ethernet cables

	Software:
		OS: FreeBSD 4.7
			http://www.freebsd.org
		Message passing: MPICH
			http://www-unix.mcs.anl.gov/mpi/mpich/
		Message passing: LAM/MPI
			http://www.lam-mpi.org

	Model:
		'Klingon Bird of Prey' cluster
			http://phoenix.physast.uga.edu/klingon/


"Mini-Wulf" Beowulf cluster
The story:
Mini-wulf is my first attempt at building a true Beowulf cluster computer. While I had experimented with using MPICH on other workstations on our LAN before, I ran into file system problems and network security issues, and abandoned it until I had the hardware to build a cluster specifically for this use.

Purpose:
The reason for building mini-wulf is education and experience. I'd only played around with MPI on a Cray T3E a little bit, and had never used a workstation cluster system before. My boss asked me to spec out a Beowulf cluster system for running an atmospheric model. I figured it would be a good idea to actually build a system out of surplus hardware first to 'get my feet wet' before spending any money on a more state-of-the-art cluster.
The hardware I used for mini-wulf was systems and network hardware that had been taken off line during upgrades. My goal was to use hardware that I had sitting around in my spares bins and obsolete systems. Thus, the cost of mini-wulf would be free in terms of hardware. Time, on the other hand...

Background:
A Beowulf cluster is a collection of computers on a separate local area network that can act as one large parallel processing computer [see figure 1]. This is done using software that implements message-passing between copies of the same program, which is run on each of the computers ("nodes") on the network. Each node works on a separate section of the problem, and when done they send the results back to the master node, which assembles the results. Message passing can be done on just about any networked computers, as long as they can talk to each other. What differentiates a Beowulf cluster from other clusters is that the Beowulf has its own local area network for the nodes to communicate with each other, and all the nodes (except the master) are not normal workstations but completely dedicated to being processor nodes for the cluster.

Figure 1.

Since they are on their own LAN, the processor nodes can be configured differently in terms of network security settings than regular workstations. This allows faster communications and less overhead processing on the nodes. It also allows the cluster to appear as one single computer to users. The users are only required to log on to the master node to use the entire cluster. This master node has all the user account information, compilers, disk storage, and message-passing software that the cluster uses. Processor nodes are little more than slave CPUs, and can be more stripped-down in terms of disk space and storage.
There are different network topologies used for Beowulf clusters, but the bottom line seems to be: the faster, the better. As the number of nodes in a cluster grows, so does the load on the local area network connecting them, and this LAN bandwidth can be the limiting factor when the number of nodes exceeds a certain point. In the 'typical' Beowulf, the master node has two network interface cards (NICs), one attached to the external network (the internet), and one attached to the cluster LAN. The external NIC is usually allocated a static IP address assigned by the network administrator. Internal NICs also use static IPs, but since they are only for use on the Beowulf LAN, they can be anything you like. Usually a set of IP addresses from an RFC 1918 pool is chosen, since these IPs are non-routable and therefore won't cause problems if a node is accidentally attached to an outside network. The processor or slave nodes usually have only one NIC. All the processor nodes and the inside NIC of the master node are all connected together using a fast switch. Hubs can be used, but will cause slow-downs due to collisions. Some switches can support channel-bonding, which allows multiple NICs in processor nodes to act as a single NIC of larger bandwidth. This is beyond the scope of this article, but more information can be found on the web.

Building Mini-wulf:
Note: Mini's hardware has been changed a few times, so this description no longer matches what currently comprises the cluster. See the component listing at the top of this document for the current setup.
Of the three computers I had at my disposal to build Mini-wulf, I chose the Pentium 90 as the master node. I chose the P90 because it had the largest hard drive, and the most memory. Please note that when building a Beowulf cluster you usually use the fastest, most powerful workstation for the master, not the weakest. The master node is where all the compiling takes place, as well as carrying a lot of NFS traffic and X if it's running. My choice of the P90 for the master node was based only on storage and physical memory, and since mini-wulf is more of an educational machine than a production number cruncher, I could get away with it.
I began construction by adding a second NIC to the P90. One caveat here using old hardware: don't try to use two ISA NICs in the same machine! My 3Com plug-and-play ISA boards did not get along all that well. Switching one to a SMC PCI NIC fixed the problem. I installed FreeBSD 4.5 on the system and configured the outside NIC for the assigned IP of the machine. After I got that working properly, I looked through the /var/log/messages file, found the device name of the second NIC, and modified the /etc/rc.conf file to assign it an RFC 1918 IP of 192.168.1.1. For an internal IP schema, I elected to use the RFC 1918 pool of 192.168.1.x.
I then installed FreeBSD on each of the two processor nodes. These I assigned IPs of 192.168.1.10 and 192.168.1.11. I left the numbering gap in case I wanted to add any auxiliary nodes later, such as a name server or additional NFS server. Since I didn't feel like typing in all those numbers every time I wanted to access these other nodes, I assigned names to the boxes as well. 192.168.1.1 I called 'master', 192.168.1.10 'alpha', and 192.168.1.11 'bravo'. Since these names aren't in any DNS databases, I hard-coded them in to each machine by adding them to the /etc/hosts file along with their IP numbers.
At this point I hooked all the internal NICs to an Addtron 5-port 10bT hub, the outside master NIC to our network, and rebooted all three boxes. All were then talking to each other (I could ssh to each box from all the others). I created a login account for myself on each of the nodes, using the same userid and number on each.

Synchronizing the clocks:
The next step was to get all the boxes' clocks synchronized. According to some of the Beowulf documentation I found, synching the clocks is important, and the easiest way to do it is have your master node act as a network time protocol (NTP) server. First I had the master node sync its clock to an internet NTP server by modifying the /etc/ntp.conf file and HUPing the ntpd daemon. On each node the /etc/ntp.conf file has 'server 192.168.1.1' as its' first line. This should cause each node to synch its' clock to the master node.

NFS home directories:
The message-passing software I used expects the home directories on all the processor nodes to be the same as the master. This I accomplished by mounting the home directory from the master to the nodes via NFS. I added an entry to the /etc/exports file on master to share out the /usr/home directory to the processor nodes. On the processor nodes, I mounted the master:/usr/home directory as /usr/home.

Using ssh for remote shell execution:
The master node needs to be able to run programs on the processor nodes without hassling with login passwords and the like. To allow this, the nodes need to have some sort of remote shell capability set up. I use ssh for all my remote shell work, because it is encrypted and fairly secure. To allow remote logins without a password, I used ssh-keygen to generate a public/private key pair on the master node. Don't put in a key-phrase when generating this key. Copy the public key (usually .ssh/identity.pub) to your keyring (usually .ssh/authorized_keys) on the master node, which should automatically include it on the processor nodes because of the shared home directories. Test this by ssh'ing to another machine. The login should happen without asking for a password. One caveat: I changed my .ssh/authorized_keys entry to end with [email protected]. My key was generated using the public IP name of the machine, which the internal LAN had problems with, since it saw the connection coming from the master's internal NIC. If you get authorization errors, this is a good place to look. On the other hand, it might work just fine as it is, your mileage may vary.
Ssh adds encryption overhead to communications, and should actually not be used on the internal LAN if you're really worried about speed. Rsh should be configured to run on all the nodes within the cluster, which involves enabling it in the /etc/inetd.conf file, and authorizing machines in the /etc/hosts.equiv file. This comes disabled in FreeBSD because rsh is not terribly secure, but since the cluster LAN is not open to the public, it should not be a problem. If you're willing to take the processing hit, ssh will work just fine too. If you're clustering workstations on a regular network, ssh is the way to go, since using rsh on an open network is risky.

Installing MPICH:
Now that the nodes are synched and talking to each other in a trusting manner, it's time to actually install some message passing software. The first package I installed was MPICH. I uncompressed and untared the package, then ran the configure script with the prefix option to tell it where I wanted the package installed:
	./configure --prefix=/usr/local/mpich-1.2.4
The configure script does various things while building the makefile, including testing the ssh and rsh capabilities of the master node. This is why that must be running before installing MPICH, and also why you need to be able to ssh from the master to itself without passwords. After configure runs, it's time to run 'make' to actually build the package. Finally 'make install' (run as root) puts the package in its' final location. You then need to tell MPICH what machines are available to run processes on. This is accomplished by editing the machines.(os) file, in my case: /usr/local/mpich-1.2.4/share/machines.freebsd. MPICH puts five copies of the name of the master node in this file. Change it to a listing of all the nodes, one per line (in this case, master, alpha, and bravo).

Now it's time to test the cluster to see if the nodes can talk to each other via MPI. Run the tstmachines script in the sbin/ directory under the mpich directory to verify this. It will help to use the -v option to get more info. If this works, it's time to run a program on the cluster. Under the distribution tree for mpich you'll find an examples directory. Inside that, under the basic directory, you'll find the cpi program. This program calculates the value of Pi, and is a good tool for verifying the cluster is working properly. Run 'make cpi' in the basic directory to build the executable. Run the program using the mpirun command. Here's what I used to test mini-wulf:
	mpirun -np 3 -nolocal cpi
Note: I put /usr/local/mpich-1.2.4/bin in my path before doing this, so the machine could find mpirun. Also, the -nolocal flag was needed on my cluster to keep it from trying to run all the processes on the master node. I don't understand why this is, but it works for me. Update: the -nolocal flag is only needed when the 'outside' name of the node is included in the loopback line in /etc/hosts, which linux does by default. Change the loopback line to read '127.0.0.1 localhost.localdomain localhost' and MPICH won't require the flag. The -np flag tells mpirun how many processors to use to run the program, 3 in this case. Here's the output I got:
	% mpirun -np 3 -nolocal cpi
	Process 0 of 3 on (outside IP).rwic.und.edu
	pi is approximately 3.1415926535899121, Error is 0.0000000000001190
	wall clock time = 2.800307
	Process 1 of 3 on alpha.rwic.und.edu
	Process 2 of 3 on bravo.rwic.und.edu
Note: I changed cpi.c to add more loop cycles to the program to get a longer run time. This helped make the difference between using more nodes less influenced by communication lags and overhead. Looks like we're actually using all processors, but let's try some other configurations just to make sure:
	% mpirun -np 1 -nolocal cpi
	Process 0 of 1 on (outside IP).rwic.und.edu
	pi is approximately 3.1415926535897309, Error is 0.0000000000000622
	wall clock time = 8.395115
	% 
	% mpirun -np 2 -nolocal cpi
	Process 0 of 2 on (outside IP).rwic.und.edu
	pi is approximately 3.1415926535899850, Error is 0.0000000000001918
	wall clock time = 4.197473
	Process 1 of 2 on alpha.rwic.und.edu
Yep, looks like mpirun is calling on the specified number of CPUs to run the program, and the time savings using more CPUs is what you'd expect. Using two CPUs runs the program in 49.9% of the time it took one, and three runs it in 33.4%. This is a beautiful 1/N progression for the runtime vs. number of CPUs, but don't expect it to hold for more complex programs or huge numbers of cluster nodes.


Installing LAM/MPI:
Since MPICH has some issues with NFS, and the Klingon Bird of Prey cluster Mini-Wulf is based on runs it, I decided to install the LAM/MPI implementation of MPI.

Running
	./configure --prefix=/usr/local/lam_mpi --with-rsh=/usr/bin/ssh
revealed no problems, since LAM/MPI supports native FreeBSD. The INSTALL file did instruct me to run make with the '-i' option under FreeBSD, since 'BSDs version doesn't always handle script result codes the way they'd like. The usual 'make -i' and 'make -i install' followed. After adding /usr/local/lam_mpi/bin to my path, I also built the examples via 'make -i examples'.

While the mighty P90 CPU chewed on this task, I started another shell and edited the /usr/local/lam_mpi/etc/lam-bhost.def file, which contains a list of all the processor nodes in the cluster. This defaults to just one, the node that LAM is built on. I added the other two nodes in the cluster.

After the examples were built and the lam-bhost.def file adjusted, it was time to test! LAM runs a little differently than MPICH, in that it runs daemons on each node in the cluster to facilitate the message-passing. This means that the LAM executables must be on every node. I ran 'recon -v -a' to test the remote nodes, and got errors when they wouldn't run the LAM program 'tkill' (which is what recon uses to test the cluster). Since I hadn't shared out the /usr/local/lam_mpi directory on the master node, the slaves couldn't find it. I debated doing the NFS share, but for the moment just copied the directory to the remote nodes using scp. This keeps NFS traffic down, although it would make cluster maintainance more labor-intensive. (Note: I've since shared out the /usr/local/lam_mpi directory and NFS mounted it on the slaves. This makes things a lot easier for upgrades later) I also had to end up setting the LAMHOME environment variable to /usr/local/lam_mpi, since both recon and lamboot were having trouble finding executables (although putting the $prefix/bin directory in my PATH should have taken care of it. Oh well, whatever works). After that, I ran 'lamboot' and got the required output.

Now that LAM was actually running, it was time to actually run some parallel programs to test the cluster. I went to the examples/pi directory, and fired off my old friend, the cpi program. LAM syntax is a bit different to start:
	mpirun C cpi
The output was huge! This version of cpi was a bit different than the other I had tested, so I copied that one (from the MPICH distro) to the local directory, compiled it using mpicc, and ran that version using LAM. Here's the output:
	> mpirun C cpi
	Process 0 of 3 on (outside IP).rwic.und.edu
	Process 1 of 3 on alpha.rwic.und.edu
	Process 2 of 3 on bravo.rwic.und.edu
	pi is approximately 3.1415926535899121, Error is 0.0000000000001190
	wall clock time = 2.805890
So, it looks like the LAM version of MPI is running. It also compiles code written for MPICH with no modifications, and runs the resultant executable in a very similar elapsed time. Very satisfactory.

LAM requires one last step that MPICH doesn't: you have to shut down the LAM daemons on all the nodes. This is accomplished via the 'lamhalt' command. There seems to be no man page for this command, but you can do a man on the older 'wipe' command, which will give you more info.

Status: June 28, 2002
At this point the cluster is functional, and can be used as is. Most production clusters, however, need more than just the bare-bones of message- passing. Mini-wulf at this point would be fine for a single or small number of users running a small number of programs, but when you start adding lots of users and/or having more programs running, management of the cluster becomes a chore. Bigger clusters use tools for batch processing programs so that all programs get a fair share of the CPU cycles of the cluster. Also, using good old 'adduser' on each node to keep track of user accounts gets tedious.

Adding these tools to mini-wulf will be explored at a later date.

Status: July 1, 2002
Mini-wulf is currently off-line waiting for upgrades. I say, "waiting for upgrades." because it sounds better than "scavenged for parts." I had a need for a server at work, and since Mini was available, I concatenated together some of its parts to make the server. The slave nodes are basically intact, with one CPU downgraded from an AMD K6-2 400 to a Pentium 120, but I'll need to find a new master node. I've got a box in mind, I just have to make time to configure it.

Status: July 3, 2002
Mini-wulf is back in operation! After mucking about with a cranky 3Com 509 NIC, I got the new master node configured and functional. The Bravo node is still only running as a Pentium 100, even though the CPU was only downgraded to a P120. More investigation is needed. In any case, the overall performance of the cluster has suffered a bit. Here's the output from a cpi run in the new configuration:
	Process 0 of 3 on (outside ip).rwic.und.edu
	Process 1 of 3 on alpha.rwic.und.edu
	Process 2 of 3 on bravo.rwic.und.edu
	pi is approximately 3.1415926535899121, Error is 0.0000000000001190
	wall clock time = 3.301860
I thought that upgrading the weakest machine would help the overall performance, but as you can see, the clock time for the program is about one second longer. This result is consistant over several test runs. The new master node has only 32M of RAM, so there may be some disk swap going on before the program is passed out and run on the LAN. More testing will be conducted as I have time.

Status: July 5, 2002
Since the new master node was so pathetic, I just had to move stuff around again. I shuffled NICs and made the old alpha node the new master, since it now had the largest hard drive and most RAM. After fussing about with NFS mounts, rc.conf and hosts files, I finally got everything running properly. I shared out the /usr/local/lam_mpi directory to the slaves via NFS, since I had to end up rebuilding LAM due to damage I did during the move. I also added
set prompt = '%n@%m:%/%# '
to my
.cshrc
file, since doing the wrong commands on the wrong nodes is what messed me up in the first place. Hint: never try to scp a directory on to itself, it corrupts all the files and generally makes you unhappy. The cluster actually runs a bit faster now, interestingly enough:
	Process 0 of 3 on (outside ip).rwic.und.edu
	Process 1 of 3 on alpha.rwic.und.edu
	Process 2 of 3 on bravo.rwic.und.edu
	pi is approximately 3.1415926535899121, Error is 0.0000000000001190
	wall clock time = 2.526050
I also ran the lam test suite, just to verify that the package was properly built and installed. No problems at this point.

Status: July 16, 2002
I've been running the Pallas benchmark on miniwulf for about a week in various configurations. The Results are rather interesting. They indicate that the choice of MPI implementation and even hardware is dependent on your code and application of the cluster.

Status: September 3, 2002
Over the long weekend, I decided I wanted to try running some other distributed computer clients on the slave nodes ( www.distributed.net). Since these clients were designed to run on single computers attached to the internet, and miniwulf's slave nodes couldn't access the internet, I had some adjusting to do. I decided to setup NAT (network address translation) on the master node. This would allow the slave nodes with their unroutable IP numbers to pass packets to the master, which would strip off the old IPs and use its own, routable IP on the packets. When the packets return from the internet, natd uses its tables to figure out which slave the packet originated from, and puts the internal IP back on it. It's pretty slick, but under FreeBSD requires jumping through some hoops. I had to build a custom kernel with ipfw firewall capability, and write a simple firewall ruleset. It's ipfw that actually passes the packets off to natd. Also, the /etc/rc.conf file has to have a few adjustments as well, such as enabling forwarding and activating the firewall. Instructions for doing all this is available at www.freebsd.org.

The NAT routing makes the slave nodes think they're connected directly to the internet, but blocks any outside hosts from accessing the nodes. The cluster is thus still fairly secure, and can still run the MPI software without any problems.

Status: October 14, 2002
Mini-wulf continues to evolve. Since the distributed.net project was completed (at least the RC5-64 section that I was interested in), I removed the client programs from the nodes. I left the natd functionality intact, however, to allow easier upgrades and other maintainance of the cluster nodes.

Mini-wulf has finally been used for the purpose for which it was built: education. I enrolled in an online MPI programming course offered by the Ohio Supercomputer Center. Mini-wulf has been very handy for doing homework problems for this course. It's also interesting to note that programs run with more than three processors, which is all mini-wulf has, work just fine. This does cause more than one process to be run on each node, but for simple programs that don't require huge amounts of computing power, that's not a problem. Of course, a single computer with MPI installed could also be used to run simple MPI programs to teach and demonstrate message passing, but that would leave out all the fun of building the cluster :).

Status: January 22, 2003
"Charlie" node added. Mini is finally a four node cluster! When an old samba server underwent an upgrade (pronounced 'replacement'), I found myself with a fully fuctional Pentium 133 based FreeBSD box. After a quick re-read of this document, I made the necessary adjustments to the old box's network and NFS settings, plugged it into the Mini-wulf LAN, powered up, and away it went! Benchmarking with the pi calculation program, Mini's crunching abilities have increased 25%. Not the 33% I would have expected, but perhaps I'm basing my expectations on some dodgy math. It's still very gratifying that the cluster is so easy to upgrade.

I have room left on the hub and power strip for one more system to be added to Mini. However I don't know how likely it is that I'll do this. While Mini-wulf has been great fun and very educational to build, computationally it gets its butt whupped by our dual Pentium 3 Xeon system. I'm now starting the process of building our 'real' Beowulf cluster that will have some serious MFLOPs and storage. Mini taught me many of the things I needed to know to build the big cluster, but it will most likely be used for 'hot storage' of old hardware from now on. It's always possible students may want to use Mini for experimental purposes, but as a high-powered number cruncher it's just too limited to be useful for big problem solving.

Status: January 24, 2003
I modified the pi calculation program to include a crude MFLOP (million floating point operation) calculator, just so I could do some simple benchmarking. Since the pi program doesn't do any trig or other heavy math, the results should be used more as a relative guide rather than absolute. Here are the results using different numbers of nodes:

	Number of nodes		MFLOPS
        ---------------         ------
              1                  12.2
              2                  20.1
              3                  22.6
              4                  30.1

It should be remembered that the number three node is a pentium 100, while the others are 133s. Even so, it's a bit bizzare that the addition of the third node only increased the performance by about 12%.

The same code running on a dual 1.7GHz Pentium III Xeon system gave:

	Number of nodes		MFLOPS
	---------------		------
	      1                  235.6
              2                  471.3

As you can see, a modern dual-processor computer beats Mini's crunching capability by about an order of magnitude. I theorize the nice doubling of performance on the dually is because the interprocess communication is taking place on the bus, rather than across a network.

Status: March 25, 2003
Upgraded cluster OS to latest FreeBSD security branch.

Status: March 26, 2003
Installed ATLAS linear algebra math library.

Status: April 25, 2003
Ran Pi MFLOP benchmark again, this time for up to 20 processes (the cluster still has only 4 nodes).

	Number of processes	MFLOPS
	-------------------	------
		 1		 12.2
		 2		 23.7
		 3		 27.4
		 4		 36.6
		 5		 25.0
		 6		 30.1
		 7		 32.0
		 8		 36.6
		 9		 30.1
		10		 33.4
		11		 31.3
		12		 34.1
		13		 32.6
		14		 35.1
		15		 30.9
		16		 32.7
		17		 34.1
		18		 36.1
		19		 30.7
		20		 32.4


Graphical version.

Status: May 12, 2003
Mini is up and operational again. During the previous week a critical server failed, so I was forced to borrow the charlie node to fill in for it until a replacement could be built.
Over the weekend I built Deuce, a two-node cluster running Redhat Linux 9.0. Since Zeus will be running this OS, I wanted to get some clustering experience with it.

Status: May 23, 2003
Since Mini is now a four node cluster, I reran the Pallas benchmark on it. Here are the results. These are for MPICH running on a 10bT hub.

Status: August 5, 2003
Delta node added. Yet another pentium 133 was retired from active service and was added to Mini. This makes 5 nodes total, and fills the Addtron 10bT hub (and the power strip) to capacity. This is likely the last node I'll add to Mini. I do have an 8-port 10bT hub I could use for the LAN, but the counter where I have the cluster installed is running short on space. Since most of my energies as far as Beowulfs are concerned are being spent on Zeus, Mini is mostly a curiosity for me these days. I did run my Pi MFLOP benchmark on the new configuration, and found a ~24% increase in maximum MFLOPS over the 4-node configuration. Here are the results:

        Number of processes     MFLOPS
        -------------------     ------
                 1               12.1
                 2               24.2
                 3               22.6
                 4               29.6
                 5               37.6
                 6               36.2
                 7               42.4
                 8               30.1
                 9               33.8
                10               37.6
                11               41.4
                12               45.1
                13               32.6
                14               35.1
                15               37.6
                16               40.1
                17               42.6
                18               33.8
                19               35.7
                20               37.6

Graphical Version

It's interesting to note the changes in the maximum performance, which show up at 12 processes on the 5-node cluster, but at 8 processes on the 4-node. It should be noted that I've switched back to the MPICH implementation of MPI for this test, while the previous one was made using LAM/MPI. This could certainly have an effect on the response of the cluster to different processing loads.

Status: August 13, 2003
Clusters must be some sort of disease, or perhaps addictive. I just couldn't leave well enough alone. Another box became available, so I replaced the Bravo node with a K6/2 333. After a few abortive attempts, I replaced the 3Com ISA NIC with a PCI version, and got it working. Before the dust had settled, I renamed the old Bravo node to Echo, swapped out the 5-port hub for an 8-port, adjusted all the /etc/hosts files (strange things happen if all your nodes don't know about each other) and /usr/local/mpich-1.2.4/share/machines.freebsd, and ran the MFLOP benchmark for 20 processes again. This resulted in a peak performance jump of about 26% over the 5-node cluster configuration. Here's the output:

	Number of processes	MFLOPS
	-------------------	------
		 1		 12.2
		 2		 24.4
		 3		 36.4
		 4		 40.1
		 5		 50.1
		 6		 45.2
		 7		 42.6
		 8		 48.1
		 9		 54.1
		10		 50.1
		11		 55.1
		12		 45.2
		13		 48.9
		14		 52.7
		15		 56.4
		16		 53.4
		17		 56.8
		18		 45.2
		19		 47.6
		20		 50.2


Graphical Version

Changing the weak node from the second in line to the sixth has changed the shape of the repeating part of the performance graph. I haven't tested the benchmark code on the new K6/2 node individually yet, but it does look like it stacks up favorably to the Pentium 133s.

Looking back over this document I see that Mini is just over one year old, has doubled in size and almost doubled in computational power. It's come a long way since it began as an unsanctioned after-hours experiment with a few old computers I was going to surplus and some obsolete network gear. I remember searching the web for hours, trying to figure out what a Beowulf was and how it worked, and scratching for the little info available on FreeBSD 'wulfs amid the comparitive wealth on Linux clusters. When I first ran the Pi program on Mini, I was totally stoked. I started this web page and sent the URL to my Boss, who wasn't exactly bubbling over with enthusiasm ("Let's not waste too much time on this."). Since that time, however, he's become a cluster convert, and was willing to fund the building of Zeus. While neither of us think Beowulf clusters will send big-iron supercomputer builders packing, they do allow cash-strapped researchers some decent computational horsepower for certain algorithms that they are suited for.

Status: September 2, 2003
A sad day. Miniwulf has been given its marching orders. The counter where the cluster stood was needed for another project (a lab we were using to build stuff was needed again for teaching). Mini's master node was also used as a DNS and NTP server, so it had its secondary NIC removed, configuration adjusted, and was moved to a different machine room. The compute nodes were shut down and moved into a storage area. It is possible that another master node could be built from one of the compute nodes, and the cluster set up elsewhere. This will have to wait until I have sufficient free time and can find space and power/network resources to run the cluster. Given the lack of computational power Mini suffered from, the incentive to reassemble it is not terribly high.

So, 15 months. That's about how long Miniwulf was operational. It's been a fun and educational ride. I took a final snapshot of the critical configuration files that went into building Mini:





Status: September 18, 2003
I've pretty much accepted the fact that Miniwulf will most likely never be reassembled. I've raided the collection of compute nodes for replacement PCs and parts, and with the cost of much more powerful PCs as low as they are, it just doesn't make sense to rebuild a Beowulf that was based on Pentium 133s and 10bT ethernet. Mini's purpose was education: helping me learn how to build, program, and manage a cluster, and now that that purpose is fufilled, it's time to move on.

Links:


Other Clusters:
Tools for building clusters:
Other cluster stuff: