Gentoo Distcc Documentation

1. Introduction

What is distcc? 

Distcc is a program designed to distribute compiling tasks across a network to participating hosts. It is comprised of a server, distccd and a client program, distcc. Distcc can work transparently with ccache and Portage with a little work.

Using distcc to bootstrap 

If you are planning on using distcc to help you bootstrap a Gentoo installation, make sure you read the part on Using distcc to Bootstrap which is situated further down in this document.

2. Setup

Dependencies 

To start off with, to use Distcc all of the computers on your network need to have the same GCC versions. Mixing 3.3.x (where the x varies) is okay, but mixing 3.3.x and 3.2.x is not: your programs will have errors in it.

Installing Distcc 

There are a couple of options you should know about before you start installing distcc.

Distcc ships with a graphical monitor to monitor tasks that your computer is sending away for compilation. If you use Gnome then put 'gnome' in your USE flags. However, if you don't use Gnome and would still like to have the monitor then you should put 'gtk' in your USE flags.

# emerge distcc

Setting up Portage to use Distcc 

Setting up distcc is easy to do with Portage. Execute the following steps on each system that should participate in the distributed compiling:

# emerge distcc
# nano -w /etc/make.conf
(Set N to a suitable number for your particular setup)
(A common heuristic strategy is twice the number of total CPUs + 1)
MAKEOPTS="-jN"
(Add distcc to your FEATURES variable)
FEATURES="distcc"

Next you have to specify what hosts you want to use. Use the distcc-config command to set the list of hosts. Here is an example of some hosts that might be in your list:

192.168.0.1          192.168.0.2                       192.168.0.3
192.168.0.1/2        192.168.0.2                       192.168.0.3/10
192.168.0.1:4000/2   192.168.0.2/1                     192.168.0.3:3632/4
@192.168.0.1         @192.168.0.2:/usr/bin/distccd     192.168.0.3
(There are also several other methods of setting up hosts.  See the 
distcc manpage for more details.)

It may all look complicated, but in most cases a variant of line 1 or 2 will work.

• Line 1 is just a space-delimited list of hosts
• Line 2 is a list of hosts that specifies the maximum amount of jobs (by defining the /N setting) to send that host at any given time

Since most people won't be using lines 3 or 4, I'll point you to the distcc docs for more information.

For instance, to set the first line in the previous example:

# /usr/bin/distcc-config --set-hosts "192.168.0.1 192.168.0.2 192.168.0.3"

Now start the distcc daemon on all the participating computers:

(Add distccd to the default runlevel)
# rc-update add distccd default
(Start the distcc daemon)
# /etc/init.d/distccd start

Setting up Distcc to Work With Automake 

This is in some cases easier than the Portage setup. What you have to do is update your PATH variable to include /usr/lib/distcc/bin in front of the directory that contains gcc (/usr/bin). However, there is a caveat. If you use ccache you have to put distcc after the ccache part:

# export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"

Then, as you would normally type make, you would type make -jN (where N is an integer). The value of N depends on your network and the types of computers you are using to compile. For me, I have a dual processor P3 and a K6-2/400 that compiles everything for my desktop (a 1200MHz Athlon Thunderbird) and I use -j5. Test your own settings to find the number that yields the best performance.

3. Cross-Compiling

A Note on Cross-Compiling 

Cross-compiling is using one architecture to build programs for another architecture. This can be as simple as using an Athlon (i686) to build a program for a K6-2 (i586), or using a Sparc to build a program for a ppc.

An Introduction to Cross-compiling 

If you want to give Cross-compiling a try you can emerge the crossdev package. I can't support it since I didn't write it, but that's a good way to start cross-compiling.

4. Using Distcc to Bootstrap

Step 1: Configure Portage  

Boot your new box with a Gentoo Linux LiveCD and follow the installation instructions up until the bootstrapping part. Then configure Portage to use distcc:

# nano -w /etc/make.conf
(Add distcc to the FEATURES variable)
FEATURES="distcc"
(Modify MAKEOPTS to include -jN with N an appropriate integer)
MAKEOPTS="-jN"

# export PATH="/usr/lib/ccache/bin:/usr/lib/distcc/bin:${PATH}"

Step 2: Getting Distcc 

Before the installation of distcc, a user called distcc must be added to the /etc/passwd:

# echo "distcc:x:240:2:distccd:/dev/null:/bin/false" >> /etc/passwd

Install distcc:

# USE='-*' emerge --nodeps distcc

Step 3: Setting Up Distcc 

Run distcc-config to setup distcc:

(Substitute host1, host2, ... with the IP number(s) of the participating hosts)
# /usr/bin/distcc-config --set-hosts "localhost host1 host2 host3 ..."
An example: /usr/bin/distcc-config --set-hosts "localhost 192.168.0.4 192.168.0.6"

distcc is now set up to bootstrap! Continue the official installation instructions and don't forget to re-emerge distcc after emerge system. This is to make sure that all of the dependencies you want are installed as well.

5. Troubleshooting

Mozilla and Xfree 

As you emerge various packages, you'll notice that some of them aren't being distributed (and aren't being built in parallel). This is because the developers of the Mozilla and Xfree ebuilds intentionally disable parallel building because it is known to cause problems.

Sometimes distcc might cause a package to fail to compile. If this happens for you, please report it to us.

A Mixture of hardened-gcc and non-hardened-gcc Hosts Will Be Faulty 

With such a long title any explanation here is almost irrelevent. However, if you plan to use distcc across hosts that have the PaX/hardened-gcc and some that do not, you will run into problems.

The solution requires a little foresight on your part; you have to run hardened-gcc -R on the host that has PaX/hardened-gcc, or you have to enable PaX protections in your kernel and emerge hardened-gcc. Both of which are a good thing to do since for the most part the protections offered by both packages is a good thing and is transparent to the user.

Mixed GCC Versions 

If you have different GCC versions on your hosts, there will likely be very weird problems. The solution is to make certain all hosts have the same GCC version.

6. Distcc Extras

Distcc Monitors 

Distcc ships with two monitors. The text-based one is always built and is called distccmon-text. Running it for the first time can be a bit confusing, but it is really quite easy to use. If you run the program with no parameter it will run once. However, if you pass it a number it will update every N seconds, where N is the argument you passed.

The other monitor is only turned on if you enabled gtk or gnome in your USE flags. This one is GTK+ based, runs in an X environment and it is quite lovely.

# distccmon-text N
(Or)
# distccmon-gnome
To monitor Portage's distcc usage you can use:
# DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-text N
(Or)
# DISTCC_DIR=`portageq envvar DISTCC_DIR` distccmon-gnome