Disk space is one of the most important requirements. Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind that official mirrors are probably required to be complete. The CVS repository and the web pages should always be mirrored completely. Also note that the numbers stated here are reflecting the current state (at 7.3-RELEASE/8.1-RELEASE). Further development and releases will only increase the required amount. Also make sure to keep some (ca. 10-20%) extra space around just to be sure. Here are some approximate figures:
Full FTP Distribution: 899 GB
CVS repository: 5.4 GB
CTM deltas: 3.2 GB
Web pages: 463 MB
The current disk usage of FTP Distribution can be found at ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes.
Of course, you need to be connected to the Internet. The required bandwidth depends on your intended use of the mirror. If you just want to mirror some parts of FreeBSD for local use at your site/intranet, the demand may be much smaller than if you want to make the files publicly available. If you intend to become an official mirror, the bandwidth required will be even higher. We can only give rough estimates here:
Local site, no public access: basically no minimum, but < 2 Mbps could make syncing too slow.
Unofficial public site: 34 Mbps is probably a good start.
Official site: > 100 Mbps is recommended, and your host should be connected as close as possible to your border router.
One thing this depends on the expected number of clients, which is determined by the server's policy. It is also affected by the types of services you want to offer. Plain FTP or HTTP services may not require a huge amount of resources. Watch out if you provide CVSup, rsync or even AnonCVS. This can have a huge impact on CPU and memory requirements. Especially rsync is considered a memory hog, and CVSup does indeed consume some CPU. For AnonCVS it might be a nice idea to set up a memory resident file system (MFS) of at least 300 MB, so you need to take this into account for your memory requirements. The following are just examples to give you a very rough hint.
For a moderately visited site that offers Rsync, you might consider a current CPU with around 800MHz - 1 GHz, and at least 512MB RAM. This is probably the minimum you want for an official site.
For a frequently used site you definitely need more RAM (consider 2GB as a good start) and possibly more CPU, which could also mean that you need to go for a SMP system.
You also want to consider a fast disk subsystem. Operations on the CVS repository require a fast disk subsystem (RAID is highly advised). A SCSI controller that has a cache of its own can also speed up things since most of these services incur a large number of small modifications to the disk.
Every mirror site is required to have a set of core services available. In addition to these required services, there are a number of optional services that server administrators may choose to offer. This section explains which services you can provide and how to go about implementing them.
This is one of the most basic services, and it is required for each mirror offering public FTP distributions. FTP access must be anonymous, and no upload/download ratios are allowed (a ridiculous thing anyway). Upload capability is not required (and must never be allowed for the FreeBSD file space). Also the FreeBSD archive should be available under the path /pub/FreeBSD.
There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order).
/usr/libexec/ftpd: FreeBSD's own ftpd can be used. Be sure to read ftpd(8).
ftp/ncftpd: A commercial package, free for educational use.
ftp/oftpd: An ftpd designed with security as a main focus.
ftp/proftpd: A modular and very flexible ftpd.
ftp/pure-ftpd: Another ftpd developed with security in mind.
ftp/twoftpd: As above.
ftp/vsftpd: The “very secure” ftpd.
ftp/wu-ftpd: The ftpd from Washington University. It has become infamous, because of the huge amount of security issues that have been found in it. If you do choose to use this software be sure to keep it up to date.
FreeBSD's ftpd, proftpd, wu-ftpd and maybe ncftpd are among the most commonly used FTPds. The others do not have a large userbase among mirror sites. One thing to consider is that you may need flexibility in limiting how many simultaneous connections are allowed, thus limiting how much network bandwidth and system resources are consumed.
Rsync is often offered for access to the contents of the FTP area of FreeBSD, so other mirror sites can use your system as their source. The protocol is different from FTP in many ways. It is much more bandwidth friendly, as only differences between files are transferred instead of whole files when they change. Rsync does require a significant amount of memory for each instance. The size depends on the size of the synced module in terms of the number of directories and files. Rsync can use rsh and ssh (now default) as a transport, or use its own protocol for stand-alone access (this is the preferred method for public rsync servers). Authentication, connection limits, and other restrictions may be applied. There is just one software package available:
If you want to offer the FreeBSD web pages, you will need to install a web server. You may optionally offer the FTP fileset via HTTP. The choice of web server software is left up to the mirror administrator. Some of the most popular choices are:
www/apache13: Apache is the most widely deployed web server on the Internet. It is used extensively by the FreeBSD Project. You may also wish to use the next generation of the Apache web server, available in the ports collection as www/apache22.
www/thttpd: If you are going to be serving a large amount of static content you may find that using an application such as thttpd is more efficient than Apache. It is optimized for excellent performance on FreeBSD.
www/boa: Boa is another alternative to thttpd and Apache. It should provide considerably better performance than Apache for purely static content. It does not, at the time of this writing, contain the same set of optimizations for FreeBSD that are found in thttpd.
CVSup is a very efficient way of distributing files. It
works similar to rsync, but was specially designed for use
with CVS repositories. If you want to offer the FreeBSD CVS repository, you really want
to consider offering it via CVSup. It is possible to offer the
CVS repository via AnonCVS, FTP, rsync or HTTP, but people would benefit much more from CVSup access. CVSup was developed by
John Polstra <[email protected]>. It is a bit tricky to
install on non-FreeBSD platforms, since it is written in Modula-3 and therefore requires
a Modula-3 environment. John Polstra has built a stripped down version of M3 that is
sufficient to run CVSup, and can be installed much easier. See
Ezm3 for
details. Related ports are:
net/cvsup: The native CVSup port (client and server) which requires lang/ezm3 now.
net/cvsup-mirror: The CVSup mirror kit, which requires net/cvsup-without-gui, and configures it mirror-ready. Some site administrators may want a different setup though.
There are a few more like net/cvsup-without-gui you might want to have a look at. If you prefer a static binary package, take a look here. This page still refers to the S1G bug that was present in CVSup. Maybe John will set up a generic download-site to get static binaries for various platforms.
It is possible to use CVSup to offer any kind of fileset, not just CVS repositories, but configuration can be complex. CVSup is known to eat some CPU on both the server and the client, since it needs to compare lots of files.
If you have the CVS repository, you may want to offer anonymous CVS access. A short warning first: There is not much demand for it, it requires some experience, and you need to know what you are doing.
Generally there are two ways to access a CVS repository remotely: via pserver or via ssh (we do not consider rsh). For anonymous access, pserver is very well suited, but some still offer ssh access as well. There is a custom crafted wrapper in the CVS repository, to be used as a login-shell for the anonymous ssh account. It does a chroot, and therefore requires the CVS repository to be available under the anonymous user's home-directory. This may not be possible for all sites. If you just offer pserver this restriction does not apply, but you may run with more security risks. You do not need to install any special software, since cvs(1) comes with FreeBSD. You need to enable access via inetd, so add an entry into your /etc/inetd.conf like this:
cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
See the manpage for details of the options. Also see the CVS info page about additional ways to make sure access is read-only. It is advised that you create an unprivileged account, preferably called anoncvs. Also you need to create a file passwd in your /home/ncvs/CVSROOT and assign a CVS password (empty or anoncvs) to that user. The directory /anoncvstmp is a special purpose memory based file system. It is not required but advised since cvs(1) creates a shadow directory structure in your /tmp which is not used after the operation but slows things dramatically if real disk operations are required. Here is an excerpt from /etc/fstab, how to set up such a MFS:
/dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid 0 0
This is (of course) tuned a lot, and was suggested by John Polstra <[email protected]>.