Package Building Procedures

1 Introduction

In order to provide pre-compiled binaries of third-party applications for FreeBSD, the Ports Collection is regularly built on one of the “Package Building Clusters.” Currently, the main cluster in use is at http://pointyhat.FreeBSD.org.

This article documents the internal workings of the cluster.

2 Build Client Management

The i386 clients co-located with pointyhat netboot from it (connected nodes); all other clients (disconnected nodes) are either self-hosted or netboot from some other pxe host. In all cases they set themselves up at boot-time to prepare to build packages.

The cluster master rsyncs the interesting data (ports and src trees, bindist tarballs, scripts, etc.) to disconnected nodes during the node-setup phase. Then, the disconnected portbuild directory is nullfs-mounted for chroot builds.

The ports-${arch} user can ssh(1) to the client nodes to monitor them. Use sudo and check the portbuild.hostname.conf for the user and access details.

The scripts/allgohans script can be used to run a command on all of the ${arch} clients.

The scripts/checkmachines script is used to monitor the load on all the nodes of the build cluster, and schedule which nodes build which ports. This script is not very robust, and has a tendency to die. It is best to start up this script on the build master (e.g. pointyhat) after boot time using a while(1) loop.

3 Chroot Build Environment Setup

Package builds are performed in a chroot populated by the portbuild script using the ${arch}/${branch}/builds/${buildid}/bindist.tar file.

The following command builds a world from the ${arch}/${branch}/builds/${buildid}/src/ tree and installs it into ${worlddir}. The tree will be updated first unless -nocvs is specified.

/var/portbuild# scripts/makeworld ${arch} ${branch} ${buildid} [-nocvs]

The bindist.tar tarball is created from the previously installed world by the mkbindist script. It should be run as root with the following command:

/var/portbuild# scripts/mkbindist ${arch} ${branch} ${buildid}

The per-machine tarballs are located in ${arch}/clients.

The bindist.tar file is extracted onto each client at client boot time, and at the start of each pass of the dopackages script.

4 Starting the Build

Several separate builds for each architecture - branch combination are supported. All data private to a build (ports tree, src tree, packages, distfiles, log files, bindist, Makefile, etc) are located under ${arch}/${branch}/builds/${buildid}. The last created build can be alternatively referenced under buildid latest, the one before is called previous.

New builds are cloned from the latest, which is fast since it uses ZFS.

dopackages.7 ${arch} ${buildid} [-options]

dopackages.wrapper ${arch} ${branch} ${buildid} [-options]

don't know how to make dns-all(continuing)

5 Anatomy of a Build

A full build without any -no options performs the following operations in the specified order:

1. An update of the current ports tree from the ZFS snapshot [*]

2. An update of the running branch's src tree from the ZFS snapshot [*]

3. Checks which ports do not have a SUBDIR entry in their respective category's Makefile [*]

4. Creates the duds file, which is a list of ports not to build [*] [+]

5. Generates a fresh INDEX file [*] [+]

6. Sets up the nodes that will be used in the build [*] [+]

7. Builds a list of restricted ports [*] [+]

8. Builds packages (phase 1) [++]

9. Performs another node setup [+]

10. Builds packages (phase 2) [++]

[*] Status of these steps can be found in ${arch}/${branch}/build.log as well as on stderr of the tty running the dopackages command.

[+] If any of these steps fail, the build will stop cold in its tracks.

[++] Status of these steps can be found in ${arch}/${branch}/make (old codebase) or ${arch}/${branch}/journal (new codebase). Individual ports will write their build logs to ${arch}/${branch}/logs and their error logs to ${arch}/${branch}/errors.

Formerly the docs tree was also checked out, however, it has been found to be unnecessary.

6 Build Maintenance

There are several cases where you will need to manually clean up a build:

1. You have manually interrupted it.

2. pointyhat has been rebooted while a build was running.

3. qmanager has crashed and has been restarted.

% /var/portbuild/scripts/build cleanup i386 8-exp 20080714120411 -full

ssh -l root gohan24 df

allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports"
allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/src"

7 Monitoring the Build

You can use qclient command to monitor the status of build nodes, and to list the currently scheduled jobs:

python /var/portbuild/evil/qmanager/qclient jobs

python /var/portbuild/evil/qmanager/qclient status

The scripts/stats ${branch} command shows the number of packages already built.

Running cat /var/portbuild/*/loads/* shows the client loads and number of concurrent builds in progress. The files that have been recently updated are the clients that are online; the others are the offline clients.

Running tail -f ${arch}/${branch}/build.log shows the overall build progress.

If a port build is failing, and it is not immediately obvious from the log as to why, you can preserve the WRKDIR for further analysis. To do this, touch a file called .keep in the port's directory. The next time the cluster tries to build this port, it will tar, compress, and copy the WRKDIR to ${arch}/${branch}/wrkdirs.

If you find that the system is looping trying to build the same package over and over again, you may be able to fix the problem by rebuilding the offending package by hand.

If all the builds start failing with complaints that they cannot load the dependent packages, check to see that httpd is still running, and restart it if not.

Keep an eye on df(1) output. If the /var/portbuild file system becomes full then Bad Things™ happen.

The status of all current builds is generated twice an hour and posted to http://pointyhat.FreeBSD.org/errorlogs/packagestats.html. For each buildenv, the following is displayed:

• cvs date is the contents of cvsdone. This is why we recommend that you update cvsdone for -exp runs (see below).

• date of latest log

• number of lines in INDEX

• the number of current build logs

• the number of completed packages

• the number of errors

• the number of duds (shown as skipped)

• missing shows the difference between INDEX and the other columns. If you have restarted a run after a cvs update, there will likely be duplicates in the packages and error columns, and this column will be meaningless. (The script is naive).

• running and completed are guesses based on a grep(1) of build.log.

8 Dealing With Build Errors

The easiest way to track build failures is to receive the emailed logs and sort them to a folder, so you can maintain a running list of current failures and detect new ones easily. To do this, add an email address to ${branch}/portbuild.conf. You can easily bounce the new ones to maintainers.

After a port appears broken on every build combination multiple times, it is time to mark it BROKEN. Two weeks' notification for the maintainers seems fair.

9 Release Builds

When building packages for a release, it may be necessary to manually update the ports and src trees to the release tag and use -nocvs and -noportscvs.

To build package sets intended for use on a CD-ROM, use the -cdrom option to dopackages.

If the disk space is not available on the cluster, use -nodistfiles to avoid collecting distfiles.

After the initial build completes, restart the build with -restart -fetch-original to collect updated distfiles as well. Then, once the build is post-processed, take an inventory of the list of files fetched:

% cd ${arch}/${branch}
% find distfiles > distfiles-${release}

This inventory file typically lives in i386/${branch} on the cluster master.

This is useful to aid in periodically cleaning out the distfiles from ftp-master. When space gets tight, distfiles from recent releases can be kept while others can be thrown away.

Once the distfiles have been uploaded (see below), the final release package set must be created. Just to be on the safe side, run the ${arch}/${branch}/cdrom.sh script by hand to make sure all the CD-ROM restricted packages and distfiles have been pruned. Then, copy the ${arch}/${branch}/packages directory to ${arch}/${branch}/packages-${release}. Once the packages are safely moved off, contact the Release Engineering Team <[email protected]> and inform them of the release package location.

Remember to coordinate with the Release Engineering Team <[email protected]> about the timing and status of the release builds.

10 Uploading Packages

Once a build has completed, packages and/or distfiles can be transferred to ftp-master for propagation to the FTP mirror network. If the build was run with -nofinish, then make sure to follow up with dopackages -finish to post-process the packages (removes RESTRICTED and NO_CDROM packages where appropriate, prunes packages not listed in INDEX, removes from INDEX references to packages not built, and generates a CHECKSUM.MD5 summary); and distfiles (moves them from the temporary distfiles/.pbtmp directory into distfiles/ and removes RESTRICTED and NO_CDROM distfiles).

It is usually a good idea to run the restricted.sh and/or cdrom.sh scripts by hand after dopackages finishes just to be safe. Run the restricted.sh script before uploading to ftp-master, then run cdrom.sh before preparing the final package set for a release.

The package subdirectories are named by whether they are for release, stable, or current. Examples:

• packages-7.2-release

• packages-7-stable

• packages-8-stable

• packages-9-current

If you are doing a completely new package set (e.g. for a new release), copy packages to the staging area on ftp-master with something like the following:

# cd /var/portbuild/${arch}/${branch}
# tar cfv - packages/ | ssh portmgr@ftp-master tar xfC - w/ports/${arch}/tmp/${subdir}

Then log into ftp-master, verify that the package set was transferred successfully, remove the package set that the new package set is to replace (in ~/w/ports/${arch}), and move the new set into place. (w/ is merely a shortcut.)

For incremental builds, packages should be uploaded using rsync so we do not put too much strain on the mirrors.

ALWAYS use -n first with rsync and check the output to make sure it is sane. If it looks good, re-run the rsync without the -n option.

Example rsync command for incremental package upload:

# rsync -n -r -v -l -t -p --delete packages/ portmgr@ftp-master:w/ports/${arch}/${subdir}/ | tee log

Distfiles should be transferred with the cpdistfiles script:

# /var/portbuild/scripts/cpdistfiles ${arch} ${branch}

Or you can do it by hand using rsync command:

# cd /var/portbuild/${arch}/${branch}
# rsync -n -r -v -l -p -c distfiles/ portmgr@ftp-master:w/ports/distfiles/ | tee log

Again, run the command without the -n option after you have checked it.

11 Experimental Patches Builds

Experimental patches builds are run from time to time to new features or bugfixes to the ports infrastructure (i.e. bsd.port.mk), or to test large sweeping upgrades. At any given time there may be several simultaneous experimental patches branches, such as 8-exp on the amd64 architecture.

In general, an experimental patches build is run the same way as any other build, except that you should first update the ports tree to the latest version and then apply your patches. To do the former, you can use the following:

% cvs -R update -dP > update.out
% date > cvsdone

You will need to edit update.out to look for lines beginning with ^M, ^C, or ^? and then deal with them.

It is always a good idea to save original copies of all changed files, as well as a list of what you are changing. You can then look back on this list when doing the final commit, to make sure you are committing exactly what you tested.

Since the machine is shared, someone else may delete your changes by mistake, so keep a copy of them in e.g. your home directory on freefall. Do not use tmp/; since pointyhat itself runs some version of -CURRENT, you can expect reboots (if nothing else, for updates).

In order to have a good control case with which to compare failures, you should first do a package build of the branch on which the experimental patches branch is based for the i386 architecture (currently this is 8). Then, when preparing for the experimental patches build, checkout a ports tree and a src tree with the same date as was used for the control build. This will ensure an apples-to-apples comparison later.

Once the build finishes, compare the control build failures to those of the experimental patches build. Use the following commands to facilitate this (this assumes the 8 branch is the control branch, and the 8-exp branch is the experimental patches branch):

% cd /var/portbuild/i386/8-exp/errors
% find . -name \*.log\* | sort > /tmp/8-exp-errs
% cd /var/portbuild/i386/8/errors
% find . -name \*.log\* | sort > /tmp/8-errs

% comm -3 /tmp/8-errs /tmp/8-exp-errs | less

This last command will produce a two-column report. The first column is ports that failed on the control build but not in the experimental patches build; the second column is vice versa. Reasons that the port might be in the first column include:

• Port was fixed since the control build was run, or was upgraded to a newer version that is also broken (thus the newer version should appear in the second column)

• Port is fixed by the patches in the experimental patches build

• Port did not build under the experimental patches build due to a dependency failure

Reasons for a port appearing in the second column include:

• Port was broken by the experimental patches [1]

• Port was upgraded since the control build and has become broken [2]

• Port was broken due to a transient error (e.g. FTP site down, package client error, etc.)

Both columns should be investigated and the reason for the errors understood before committing the experimental patches set. To differentiate between [1] and [2] above, you can do a rebuild of the affected packages under the control branch:

% cd /var/portbuild/i386/8/ports

The following command will set up the control branch for the partial build (old codebase):

% /var/portbuild/scripts/dopackages.8 -noportscvs -nobuild -nocvs -nofinish

The builds must be performed from the packages/All directory. This directory should initially be empty except for the Makefile symlink. If this symlink does not exist, it must be created:

% cd /var/portbuild/i386/8/packages/All
% ln -sf ../../Makefile .
% make -k -j<#> <list of packages to build>

This will build only those packages listed as well as all of their dependencies.

You can check the progress of this partial build the same way you would a regular build.

Once all the errors have been resolved, you can commit the package set. After committing, it is customary to send a HEADS UP email to [email protected] and copy [email protected] informing people of the changes. A summary of all changes should also be committed to /usr/ports/CHANGES.

12 How to configure a new package building node

Before following these steps, please coordinate with portmgr.

13 How to configure a new FreeBSD branch

When a new branch is created, some work needs to be done to specify that the previous branch is no longer equivalent to HEAD. The following instructions apply to the previous branch number:

• Create a new zfs filesystem for sources:

  zfs create a/snap/src-branch
  

  
  
  

• Checkout a src tree in the new filesystem:

  cvs -Rq -d /r/ncvs co -r RELENG-branch
  

  
  
  

• Edit the master copy of Tools/portbuild/portbuild.conf.

• For each arch, edit its copy of the above in /var/portbuild/arch/portbuild.conf.

• Edit /var/portbuild/scripts/buildenv.

• (Only necessary for old codebase): Add a link from /var/portbuild/scripts/dopackages to /var/portbuild/scripts/dopackages.branch.

• Modify HEAD_BRANCH and NON_HEAD_BRANCHES in /var/portbuild/scripts/updatesnap.

• (Only necessary for old codebase): Add the snap directory to /var/portbuild/scripts/zexpire.

• (Only necessary for old codebase): In the /var/portbuild/errorlogs/ directory, create links for the webserver:

  ln -s ../arch/branch/builds/latest/bak/errors arch-branch-full
  ln -s ../arch/branch/builds/latest/bak/logs arch-branch-full-logs
  ln -s ../arch/branch/builds/latest/errors arch-branch-latest
  ln -s ../arch/branch/builds/latest/logs arch-branch-latest-logs
  ln -s ../arch/branch/builds/latest/bak/packages arch-branch-packages-full
  ln -s ../arch/branch/builds/latest/packages arch-branch-packages-latest
  

  
  
  

• Kick-start the build for the branch with

  build create arch branch
  

  
  
  

• Create bindist.tar .

14 How to configure a new architecture

• Create a new ports-arch user and group.

• mkdir /var/portbuild/arch; cd /var/portbuild/arch
  

• Create a new zfs filesystem:

  zfs create -o mountpoint=/a/portbuild/arch a/portbuild/arch
  

  
  
  

• Create a directory for buildlogs and errorlogs:

  mkdir /dumpster/pointyhat/arch/archive
  

  
  
  

  Note: It is possible that /dumpster/pointyhat will not have enough space. In that case, create the archive directory as /dumpster/pointyhat/arch/archive and symlink to that. (This needs to be sorted out.)

• Create a link to the above for the webserver:

  ln -s /dumpster/pointyhat/arch/archive archive
  

  
  
  

• In the /var/portbuild/arch directory:

  mkdir clients
  

  
  
  

• Populate clients as usual.

• mkdir loads
  

  
  
  

• mkdir lockfiles
  

  
  
  

• Create a local make.conf. In the most trivial case, you can

  ln ../make.conf ./make.conf
  

  
  
  

• Create an empty mlist file.

• Create pnohang.arch. (The easiest way may be to do the following on a client, and then copy it back):

  cc pnohang.c -o pnohang-arch
  

  
  
  

• Create a fresh portbuild.conf file from one of the ones for another architecture.

• Create customized portbuild.machinename.conf files as appropriate.

• cd .ssh && ssh-keygen
  

  
  
  

• Edit the .ssh/config file for convenience in using ssh.

• Make the private configuration directory:

  mkdir /var/portbuild/conf/arch
  

  
  
  

• In that directory: create any dotunnel.* scripts needed.

• Tell qmanager about the arch:

  python /var/portbuild/evil/qmanager/qclient add_acl name=ports-arch uidlist=ports-arch gidlist=portmgr sense=1
  

  
  
  

• Edit /var/portbuild/scripts/buildenv.

• Add the arch directory to /var/portbuild/scripts/zbackup and /var/portbuild/scripts/zexpire.

• (Only necessary for old codebase): As with the procedure for creating a new branch: in the /var/portbuild/errorlogs/ directory, create links for the webserver:

  ln -s ../arch/branch/builds/latest/bak/errors arch-branch-full
  ln -s ../arch/branch/builds/latest/bak/logs arch-branch-full-logs
  ln -s ../arch/branch/builds/latest/errors arch-branch-latest
  ln -s ../arch/branch/builds/latest/logs arch-branch-latest-logs
  ln -s ../arch/branch/builds/latest/bak/packages arch-branch-packages-full
  ln -s ../arch/branch/builds/latest/packages arch-branch-packages-latest
  

  
  
  

• (Only necessary for old codebase): In that directory, create two more links for the webserver:

  ln -s ../arch/archive/buildlogs arch-buildlogs
  ln -s ../arch/archive/errorlogs arch-errorlogs
  

  
  
  

• For each branch that will be supported, do the following:

  • Kick-start the build for the branch with

    build create arch branch
    

    
    
    

  • Create bindist.tar.

• Only after the first time a dopackages has been run for the arch: add the arch to /var/portbuild/scripts/dopackagestats.

15 How to configure a new head node (pointyhat instance)

This section is in progress.

16 Procedures for dealing with disk failures

When a machine has a disk failure (e.g. panics due to read errors, etc), then we should do the following steps:

• Note the time and failure mode (e.g. paste in the relevant console output) in /var/portbuild/${arch}/reboots

• For i386 gohan clients, scrub the disk by touching /SCRUB in the nfsroot (e.g. /a/nfs/8.dir1/SCRUB) and rebooting. This will dd if=/dev/zero of=/dev/ad0 and force the drive to remap any bad sectors it finds, if it has enough spares left. This is a temporary measure to extend the lifetime of a drive that is on the way out.

  Note: For the i386 blade systems another signal of a failing disk seems to be that the blade will completely hang and be unresponsive to either console break, or even NMI.

  For other build systems that don't newfs their disk at boot (e.g. amd64 systems) this step has to be skipped.

• If the problem recurs, then the disk is probably toast. Take the machine out of mlist and (for ata disks) run smartctl on the drive:

  smartctl -t long /dev/ad0
  

  It will take about 1/2 hour:

  gohan51# smartctl -t long /dev/ad0
  smartctl version 5.38 [i386-portbld-freebsd8.0] Copyright (C) 2002-8
  Bruce Allen
  Home page is http://smartmontools.sourceforge.net/
  
  === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
  Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
  Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
  Testing has begun.
  Please wait 31 minutes for test to complete.
  Test will complete after Fri Jul  4 03:59:56 2008
  
  Use smartctl -X to abort test.
  

  Then smartctl -a /dev/ad0 shows the status after it finishes:

  # SMART Self-test log structure revision number 1
  # Num  Test_Description    Status                  Remaining 
  LifeTime(hours)  LBA_of_first_error
  #   1  Extended offline    Completed: read failure       80%     15252    319286
  

  It will also display other data including a log of previous drive errors. It is possible for the drive to show previous DMA errors without failing the self-test though (because of sector remapping).

When a disk has failed, please inform the cluster administrators so we can try to get it replaced.