Ok, now you know the requirements and how to offer the services, but not how to get it. :-) This section explains how to actually mirror the various parts of FreeBSD, what tools to use, and where to mirror from.
The FTP area is the largest amount of data that needs to be mirrored. It includes the distribution sets required for network installation, the branches which are actually snapshots of checked-out source trees, the ISO Images to write CD-ROMs with the installation distribution, a live file system, and a snapshot of the ports tree. All of course for various FreeBSD versions, and various architectures.
The best way to mirror the FTP area is rsync. You can install the port net/rsync and then use rsync to sync with your upstream host. rsync is already mentioned in Section 2.4.2, “Rsync (optional for FTP fileset)”. Since rsync access is not required, your preferred upstream site may not allow it. You may need to hunt around a little bit to find a site that allows rsync access.
Since the number of rsync clients will have a significant impact on the server machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from about their policy, and maybe an exception for your host (since you are a mirror).
A command line to mirror FreeBSD might look like:
%rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/
Consult the documentation for rsync, which is also available at http://rsync.samba.org/, about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via cron(8).
The FreeBSD website should only be mirrored via rsync.
A command line to mirror the FreeBSD web site might look like:
%rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/
Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages. For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the pkg(8) process. Alternatively specific packages and their dependencies can be fetched by running something like the following:
%pkg fetch -d -o/usr/local/mirrorvim
Once those packages have been fetched, the repository metadata must be generated by running:
%pkg repo/usr/local/mirror
Once the packages have been fetched and the metadata for the repository has been generated, serve the packages up to the client machines via HTTP. For additional information see the man pages for pkg(8), specifically the pkg-repo(8) page.
Every mirror should be updated at a minimum of once per day. Certainly a script with locking to prevent multiple runs happening at the same time will be needed to run from cron(8). Since nearly every admin does this in their own way, specific instructions cannot be provided. It could work something like this:
Put the command to run your mirroring application
in a script. Use of a plain /bin/sh
script is recommended.
Add some output redirections so diagnostic messages are logged to a file.
Test if your script works. Check the logs.
Use crontab(1) to add the script to the appropriate user's crontab(5). This should be a different user than what your FTP daemon runs as so that if file permissions inside your FTP area are not world-readable those files can not be accessed by anonymous FTP. This is used to “stage” releases — making sure all of the official mirror sites have all of the necessary release files on release day.
Here are some recommended schedules:
FTP fileset: daily
WWW pages: daily
All FreeBSD documents are available for download at http://ftp.FreeBSD.org/pub/FreeBSD/doc/
Questions that are not answered by the
documentation may be
sent to <[email protected]>.
Send questions about this document to <[email protected]>.