Table of Contents
For a number of reasons you may want to build binary packages for a large selected set of packages in pkgsrc or even for all pkgsrc packages. For instance, when you have multiple machines that should run the same software, it is wasted time if they all build their packages themselves from source. Or you may want to build a list of packages you want and check them before deploying onto production system. There is a way of getting a set of binary packages: The bulk build system, or pbulk ("p" stands for "parallel). This chapter describes how to set it up so that the packages are most likely to be usable later.
First of all, you have to decide whether you build all packages or a limited set of them. Full bulk builds usually consume a lot more resources, both space and time, than builds for some practical sets of packages. There exists a number of particularly heavy packages that are not actually interesting to a wide audience. For a limited bulk builds you need to make a list of packages you want to build. Note, that all their dependencies will be built, so you don't need to track them manually.
During bulk builds various packages are installed and deinstalled
in /usr/pkg (or whatever LOCALBASE is),
so make sure that you don't need any package during the builds.
Essentially, you should provide fresh system, either a chroot environment
or something even more restrictive, depending on what the operating system provides,
or dedicate the whole physical machine.
As a useful side effect this makes sure that bulk builds cannot
break anything in your system. There have been numerous cases where
certain packages tried to install files outside the
LOCALBASE or wanted to edit some files in
/etc.
Running a pbulk-style bulk build works roughly as follows:
First, build the pbulk infrastructure in a fresh pkgsrc location.
Then, build each of the packages from a clean installation directory using the infrastructure.
To simplify configuration we provide helper script mk/pbulk/pbulk.sh.
In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). Configure network access to fetch distribution files. Create user with name "pbulk".
Fetch and extract pkgsrc. Use a command like one of these:
#(cd /usr && ftp -o - http://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)#(cd /usr && fetch -o - http://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)#(cd /usr && cvs -Q -z3 -d [email protected]:/cvsroot get -P pkgsrc)
Or any other way that fits (e.g., curl, wget).
Deploy and configure pbulk tools, e.g.:
#sh pbulk.sh -n # native (NetBSD)#sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment#sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build
mk.conf.frag is a fragment of
mk.conf that contains settings you want to
apply to packages you build. For instance,
PKG_DEVELOPER= yes # perform more checks X11_TYPE= modular # use pkgsrc X11 SKIP_LICENSE_CHECK= yes # accept all licences (useful when building all packages)
If configured for limited list, replace the list in /usr/pbulk/etc/pbulk.list
with your list of packages one per line without empty lines or comments. E.g.:
www/firefox mail/thunderbird misc/libreoffice4
At this point you can also review configuration in /usr/pbulk/etc
and make final amendments, if wanted.
Start it:
#/usr/pbulk/bin/bulkbuild
After it finishes, you'll have /mnt filled with distribution files, binary packages, and reports,
plain text summary in /mnt/bulklog/meta/report.txt
The pbulk.sh script does not cover all possible use cases.
While being ready to run, it serves as a good starting point to understand and build more complex setups.
The script is kept small enough for better understanding.
The pbulk.sh script supports running
unprivileged bulk build and helps configuring distributed bulk builds.
A complete bulk build requires lots of disk space. Some of the disk space can be read-only, some other must be writable. Some can be on remote filesystems (such as NFS) and some should be local. Some can be temporary filesystems, others must survive a sudden reboot.
40 GB for the distfiles (read-write, remote, temporary)
30 GB for the binary packages (read-write, remote, permanent)
1 GB for the pkgsrc tree (read-only, remote, permanent)
5 GB for LOCALBASE (read-write, local, temporary)
10 GB for the log files (read-write, remote, permanent)
5 GB for temporary files (read-write, local, temporary)
After your pkgsrc bulk-build has completed, you may wish to
create a CD-ROM set of the resulting binary packages to assist
in installing packages on other machines. The
pkgtools/cdpack package provides
a simple tool for creating the ISO 9660 images.
cdpack arranges the packages on the CD-ROMs in a
way that keeps all the dependencies for a given package on the same
CD as that package.
Complete documentation for cdpack is found in the cdpack(1)
man page. The following short example assumes that the binary
packages are left in
/usr/pkgsrc/packages/All and that
sufficient disk space exists in /u2 to
hold the ISO 9660 images.
#mkdir /u2/images#pkg_add /usr/pkgsrc/packages/All/cdpack#cdpack /usr/pkgsrc/packages/All /u2/images
If you wish to include a common set of files
(COPYRIGHT, README,
etc.) on each CD in the collection, then you need to create a
directory which contains these files. e.g.
#mkdir /tmp/common#echo "This is a README" > /tmp/common/README#echo "Another file" > /tmp/common/COPYING#mkdir /tmp/common/bin#echo "#!/bin/sh" > /tmp/common/bin/myscript#echo "echo Hello world" >> /tmp/common/bin/myscript#chmod 755 /tmp/common/bin/myscript
Now create the images:
#cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images
Each image will contain README,
COPYING, and bin/myscript
in their root directories.