Table of Contents
Subversion is an open-source software project developed under an Apache-style software license. The project is financially backed by CollabNet, Inc., a California-based software development company. The community that has formed around the development of Subversion always welcomes new members who can donate their time and attention to the project. Volunteers are encouraged to assist in any way they can, whether that means finding and diagnosing bugs, refining existing source code, or fleshing out whole new features.
This chapter is for those who wish to assist in the continued evolution of Subversion by actually getting their hands dirty with the source code. We will cover some of the software's more intimate details, the kind of technical nitty-gritty that those developing Subversion itself—or writing entirely new tools based on the Subversion libraries—should be aware of. If you don't foresee yourself participating with the software at such a level, feel free to skip this chapter with confidence that your experience as a Subversion user will not be affected.
Subversion has a modular design, implemented as a collection of C libraries. Each library has a well-defined purpose and interface, and most modules are said to exist in one of three main layers—the Repository Layer, the Repository Access (RA) Layer, or the Client Layer. We will examine these layers shortly, but first, see our brief inventory of Subversion's libraries in Table 8.1, “A Brief Inventory of the Subversion Libraries”. For the sake of consistency, we will refer to the libraries by their extensionless Unix library names (e.g.: libsvn_fs, libsvn_wc, mod_dav_svn).
Table 8.1. A Brief Inventory of the Subversion Libraries
| Library | Description |
|---|---|
| libsvn_client | Primary interface for client programs |
| libsvn_delta | Tree and byte-stream differencing routines |
| libsvn_diff | Contextual differencing and merging routines |
| libsvn_fs | Filesystem commons and module loader |
| libsvn_fs_base | The Berkeley DB filesystem back-end |
| libsvn_fs_fs | The native filesystem (FSFS) back-end |
| libsvn_ra | Repository Access commons and module loader |
| libsvn_ra_dav | The WebDAV Repository Access module |
| libsvn_ra_local | The local Repository Access module |
| libsvn_ra_svn | The custom protocol Repository Access module |
| libsvn_repos | Repository interface |
| libsvn_subr | Miscellaneous helpful subroutines |
| libsvn_wc | The working copy management library |
| mod_authz_svn | Apache authorization module for Subversion repositories access via WebDAV |
| mod_dav_svn | Apache module for mapping WebDAV operations to Subversion ones |
The fact that the word “miscellaneous” only appears once in Table 8.1, “A Brief Inventory of the Subversion Libraries” is a good sign. The Subversion development team is serious about making sure that functionality lives in the right layer and libraries. Perhaps the greatest advantage of the modular design is its lack of complexity from a developer's point of view. As a developer, you can quickly formulate that kind of “big picture” that allows you to pinpoint the location of certain pieces of functionality with relative ease.
Another benefit of modularity is the ability to replace a given module with a whole new library that implements the same API without affecting the rest of the code base. In some sense, this happens within Subversion already. The libsvn_ra_dav, libsvn_ra_local, and libsvn_ra_svn all implement the same interface. And all three communicate with the Repository Layer—libsvn_ra_dav and libsvn_ra_svn do so across a network, and libsvn_ra_local connects to it directly. The libsvn_fs_base and libsvn_fs_fs libraries are another example of this.
The client itself also highlights modularity in the Subversion design. While Subversion itself comes with only a command-line client program, there are several third party programs which provide various forms of client GUI. These GUIs use the same APIs that the stock command-line client does. Subversion's libsvn_client library is the one-stop shop for most of the functionality necessary for designing a working Subversion client (see the section called “Client Layer”).
When referring to Subversion's Repository Layer, we're generally talking about two libraries—the repository library, and the filesystem library. These libraries provide the storage and reporting mechanisms for the various revisions of your version-controlled data. This layer is connected to the Client Layer via the Repository Access Layer, and is, from the perspective of the Subversion user, the stuff at the “other end of the line.”
The Subversion Filesystem is accessed via the libsvn_fs API, and is not a kernel-level filesystem that one would install in an operating system (like the Linux ext2 or NTFS), but a virtual filesystem. Rather than storing “files” and “directories” as real files and directories (as in, the kind you can navigate through using your favorite shell program), it uses one of two available abstract storage backends—either a Berkeley DB database environment, or a flat-file representation. (To learn more about the two repository back-ends, see the section called “Repository Data Stores”.) However, there has been considerable interest by the development community in giving future releases of Subversion the ability to use other back-end database systems, perhaps through a mechanism such as Open Database Connectivity (ODBC).
The filesystem API exported by libsvn_fs contains the kinds of functionality you would expect from any other filesystem API: you can create and remove files and directories, copy and move them around, modify file contents, and so on. It also has features that are not quite as common, such as the ability to add, modify, and remove metadata (“properties”) on each file or directory. Furthermore, the Subversion Filesystem is a versioning filesystem, which means that as you make changes to your directory tree, Subversion remembers what your tree looked like before those changes. And before the previous changes. And the previous ones. And so on, all the way back through versioning time to (and just beyond) the moment you first started adding things to the filesystem.
All the modifications you make to your tree are done within the context of a Subversion transaction. The following is a simplified general routine for modifying your filesystem:
Begin a Subversion transaction.
Make your changes (adds, deletes, property modifications, etc.).
Commit your transaction.
Once you have committed your transaction, your filesystem modifications are permanently stored as historical artifacts. Each of these cycles generates a single new revision of your tree, and each revision is forever accessible as an immutable snapshot of “the way things were.”
Most of the functionality provided by the filesystem
interface comes as an action that occurs on a filesystem path.
That is, from outside of the filesystem, the primary mechanism
for describing and accessing the individual revisions of files
and directories comes through the use of path strings like
/foo/bar, just as if you were addressing
files and directories through your favorite shell program.
You add new files and directories by passing their paths-to-be
to the right API functions. You query for information about
them by the same mechanism.
Unlike most filesystems, though, a path alone is not enough information to identify a file or directory in Subversion. Think of a directory tree as a two-dimensional system, where a node's siblings represent a sort of left-and-right motion, and descending into subdirectories a downward motion. Figure 8.1, “Files and directories in two dimensions” shows a typical representation of a tree as exactly that.
Of course, the Subversion filesystem has a nifty third
dimension that most filesystems do not have—Time!
[42]
In the filesystem interface, nearly every function that has a
path argument also expects a
root argument. This
svn_fs_root_t argument describes
either a revision or a Subversion transaction (which is
usually just a revision-to-be), and provides that
third-dimensional context needed to understand the difference
between /foo/bar in revision 32, and the
same path as it exists in revision 98. Figure 8.2, “Versioning time—the third dimension!” shows revision history as an
added dimension to the Subversion filesystem universe.
As we mentioned earlier, the libsvn_fs API looks and feels like any other filesystem, except that it has this wonderful versioning capability. It was designed to be usable by any program interested in a versioning filesystem. Not coincidentally, Subversion itself is interested in that functionality. But while the filesystem API should be sufficient for basic file and directory versioning support, Subversion wants more—and that is where libsvn_repos comes in.
The Subversion repository library (libsvn_repos) is basically a wrapper library around the filesystem functionality. This library is responsible for creating the repository layout, making sure that the underlying filesystem is initialized, and so on. Libsvn_repos also implements a set of hooks—scripts that are executed by the repository code when certain actions take place. These scripts are useful for notification, authorization, or whatever purposes the repository administrator desires. This type of functionality, and other utilities provided by the repository library, are not strictly related to implementing a versioning filesystem, which is why it was placed into its own library.
Developers who wish to use the libsvn_repos API will find that it is not a complete wrapper around the filesystem interface. That is, only certain major events in the general cycle of filesystem activity are wrapped by the repository interface. Some of these include the creation and commit of Subversion transactions, and the modification of revision properties. These particular events are wrapped by the repository layer because they have hooks associated with them. In the future, other events may be wrapped by the repository API. All of the remaining filesystem interaction will continue to occur directly via the libsvn_fs API, though.
For example, here is a code segment that illustrates the
use of both the repository and filesystem interfaces to create
a new revision of the filesystem in which a directory is
added. Note that in this example (and all others throughout
this book), the SVN_ERR() macro simply
checks for a non-successful error return from the function it
wraps, and returns that error if it exists.
Example 8.1. Using the Repository Layer
/* Create a new directory at the path NEW_DIRECTORY in the Subversion
repository located at REPOS_PATH. Perform all memory allocation in
POOL. This function will create a new revision for the addition of
NEW_DIRECTORY. */
static svn_error_t *
make_new_directory (const char *repos_path,
const char *new_directory,
apr_pool_t *pool)
{
svn_error_t *err;
svn_repos_t *repos;
svn_fs_t *fs;
svn_revnum_t youngest_rev;
svn_fs_txn_t *txn;
svn_fs_root_t *txn_root;
const char *conflict_str;
/* Open the repository located at REPOS_PATH. */
SVN_ERR (svn_repos_open (&repos, repos_path, pool));
/* Get a pointer to the filesystem object that is stored in
REPOS. */
fs = svn_repos_fs (repos);
/* Ask the filesystem to tell us the youngest revision that
currently exists. */
SVN_ERR (svn_fs_youngest_rev (&youngest_rev, fs, pool));
/* Begin a new transaction that is based on YOUNGEST_REV. We are
less likely to have our later commit rejected as conflicting if we
always try to make our changes against a copy of the latest snapshot
of the filesystem tree. */
SVN_ERR (svn_fs_begin_txn (&txn, fs, youngest_rev, pool));
/* Now that we have started a new Subversion transaction, get a root
object that represents that transaction. */
SVN_ERR (svn_fs_txn_root (&txn_root, txn, pool));
/* Create our new directory under the transaction root, at the path
NEW_DIRECTORY. */
SVN_ERR (svn_fs_make_dir (txn_root, new_directory, pool));
/* Commit the transaction, creating a new revision of the filesystem
which includes our added directory path. */
err = svn_repos_fs_commit_txn (&conflict_str, repos,
&youngest_rev, txn, pool);
if (! err)
{
/* No error? Excellent! Print a brief report of our success. */
printf ("Directory '%s' was successfully added as new revision "
"'%ld'.\n", new_directory, youngest_rev);
}
else if (err->apr_err == SVN_ERR_FS_CONFLICT)
{
/* Uh-oh. Our commit failed as the result of a conflict
(someone else seems to have made changes to the same area
of the filesystem that we tried to modify). Print an error
message. */
printf ("A conflict occurred at path '%s' while attempting "
"to add directory '%s' to the repository at '%s'.\n",
conflict_str, new_directory, repos_path);
}
else
{
/* Some other error has occurred. Print an error message. */
printf ("An error occurred while attempting to add directory '%s' "
"to the repository at '%s'.\n",
new_directory, repos_path);
}
/* Return the result of the attempted commit to our caller. */
return err;
}
In the previous code segment, calls were made to both the
repository and filesystem interfaces. We could just as easily
have committed the transaction using
svn_fs_commit_txn(). But the filesystem
API knows nothing about the repository library's hook
mechanism. If you want your Subversion repository to
automatically perform some set of non-Subversion tasks every
time you commit a transaction (like, for example, sending an
email that describes all the changes made in that transaction
to your developer mailing list), you need to use the
libsvn_repos-wrapped version of that
function—svn_repos_fs_commit_txn().
This function will actually first run the
pre-commit hook script if one exists, then
commit the transaction, and finally will run a
post-commit hook script. The hooks provide
a special kind of reporting mechanism that does not really
belong in the core filesystem library itself. (For more
information regarding Subversion's repository hooks, see the section called “Hook Scripts”.)
The hook mechanism requirement is but one of the reasons for the abstraction of a separate repository library from the rest of the filesystem code. The libsvn_repos API provides several other important utilities to Subversion. These include the abilities to:
create, open, destroy, and perform recovery steps on a Subversion repository and the filesystem included in that repository.
describe the differences between two filesystem trees.
query for the commit log messages associated with all (or some) of the revisions in which a set of files was modified in the filesystem.
generate a human-readable “dump” of the filesystem, a complete representation of the revisions in the filesystem.
parse that dump format, loading the dumped revisions into a different Subversion repository.
As Subversion continues to evolve, the repository library will grow with the filesystem library to offer increased functionality and configurable option support.
If the Subversion Repository Layer is at “the other end of the line”, the Repository Access Layer is the line itself. Charged with marshalling data between the client libraries and the repository, this layer includes the libsvn_ra module loader library, the RA modules themselves (which currently includes libsvn_ra_dav, libsvn_ra_local, and libsvn_ra_svn), and any additional libraries needed by one or more of those RA modules, such as the mod_dav_svn Apache module with which libsvn_ra_dav communicates or libsvn_ra_svn's server, svnserve.
Since Subversion uses URLs to identify its repository
resources, the protocol portion of the URL schema (usually
file:, http:,
https:, or svn:) is used
to determine which RA module will handle the communications.
Each module registers a list of the protocols it knows how to
“speak” so that the RA loader can, at runtime,
determine which module to use for the task at hand. You can
determine which RA modules are available to the Subversion
command-line client, and what protocols they claim to support,
by running svn --version:
$ svn --version svn, version 1.2.3 (r15833) compiled Sep 13 2005, 22:45:22 Copyright (C) 2000-2005 CollabNet. Subversion is open source software, see http://subversion.tigris.org/ This product includes software developed by CollabNet (http://www.Collab.Net/). The following repository access (RA) modules are available: * ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol. - handles 'http' scheme - handles 'https' scheme * ra_svn : Module for accessing a repository using the svn network protocol. - handles 'svn' scheme * ra_local : Module for accessing a repository on local disk. - handles 'file' scheme
The libsvn_ra_dav library is designed for use by clients
that are being run on different machines than the servers
with which they communicating, specifically servers reached
using URLs that contain the http: or
https: protocol portions. To understand
how this module works, we should first mention a couple of
other key components in this particular configuration of the
Repository Access Layer—the powerful Apache HTTP
Server, and the Neon HTTP/WebDAV client library.
Subversion's primary network server is the Apache HTTP Server. Apache is a time-tested, extensible open-source server process that is ready for serious use. It can sustain a high network load and runs on many platforms. The Apache server supports a number of different standard authentication protocols, and can be extended through the use of modules to support many others. It also supports optimizations like network pipelining and caching. By using Apache as a server, Subversion gets all of these features for free. And since most firewalls already allow HTTP traffic to pass through, system administrators typically don't even have to change their firewall configurations to allow Subversion to work.
Subversion uses HTTP and WebDAV (with DeltaV) to communicate with an Apache server. You can read more about this in the WebDAV section of this chapter, but in short, WebDAV and DeltaV are extensions to the standard HTTP 1.1 protocol that enable sharing and versioning of files over the web. Apache 2.0 and later versions come with mod_dav, an Apache module that understands the DAV extensions to HTTP. Subversion itself supplies mod_dav_svn, though, which is another Apache module that works in conjunction with (really, as a back-end to) mod_dav to provide Subversion's specific implementations of WebDAV and DeltaV.
When communicating with a repository over HTTP, the RA
loader library chooses libsvn_ra_dav as the proper access
module. The Subversion client makes calls into the generic
RA interface, and libsvn_ra_dav maps those calls (which
embody rather large-scale Subversion actions) to a set of
HTTP/WebDAV requests. Using the Neon library, libsvn_ra_dav
transmits those requests to the Apache server. Apache
receives these requests (exactly as it does generic HTTP
requests that your web browser might make), notices that the
requests are directed at a URL that is configured as a DAV
location (using the <Location>
directive in httpd.conf), and hands the
request off to its own mod_dav module. When properly
configured, mod_dav knows to use Subversion's mod_dav_svn for
any filesystem-related needs, as opposed to the generic
mod_dav_fs that comes with Apache. So ultimately, the client
is communicating with mod_dav_svn, which binds directly to the
Subversion Repository Layer.
That was a simplified description of the actual exchanges taking place, though. For example, the Subversion repository might be protected by Apache's authorization directives. This could result in initial attempts to communicate with the repository being rejected by Apache on authorization grounds. At this point, libsvn_ra_dav gets back the notice from Apache that insufficient identification was supplied, and calls back into the Client Layer to get some updated authentication data. If the data is supplied correctly, and the user has the permissions that Apache seeks, libsvn_ra_dav's next automatic attempt at performing the original operation will be granted, and all will be well. If sufficient authentication information cannot be supplied, the request will ultimately fail, and the client will report the failure to the user.
By using Neon and Apache, Subversion gets free functionality in several other complex areas, too. For example, if Neon finds the OpenSSL libraries, it allows the Subversion client to attempt to use SSL-encrypted communications with the Apache server (whose own mod_ssl can “speak the language”). Also, both Neon itself and Apache's mod_deflate can understand the “deflate” algorithm (the same one used by the PKZIP and gzip programs), so requests can be sent in smaller, compressed chunks across the wire. Other complex features that Subversion hopes to support in the future include the ability to automatically handle server-specified redirects (for example, when a repository has been moved to a new canonical URL) and taking advantage of HTTP pipelining.
In addition to the standard HTTP/WebDAV protocol,
Subversion also provides an RA implementation that uses a
custom protocol. The libsvn_ra_svn module implements
its own network socket connectivity, and communicates with a
stand-alone server—the svnserve
program—on the machine that hosts the
repository. Clients access the repository using the
svn:// schema.
This RA implementation lacks most of the advantages of
Apache mentioned in the previous section; however, it may be
appealing to some system administrators nonetheless. It is
dramatically easier to configure and run; setting up an
svnserve process is nearly
instantaneous. It is also much smaller (in terms of lines
of code) than Apache, making it much easier to audit, for
security reasons or otherwise. Furthermore, some system
administrators may already have an SSH security
infrastructure in place, and want Subversion to use it.
Clients using ra_svn can easily tunnel the protocol over
SSH.
Not all communications with a Subversion repository
require a powerhouse server process and a network layer.
For users who simply wish to access the repositories on
their local disk, they may do so using
file: URLs and the functionality provided
by libsvn_ra_local. This RA module binds directly with the
repository and filesystem libraries, so no network
communication is required at all.
Subversion requires that the server name included as part
of the file: URL be either
localhost or empty, and that there be no
port specification. In other words, your URLs should look
like either
file://localhost/path/to/repos or
file:///path/to/repos.
Also, be aware that Subversion's
file: URLs cannot be used in a regular
web browser the way typical file: URLs
can. When you attempt to view a file:
URL in a regular web browser, it reads and displays the
contents of the file at that location by examining the
filesystem directly. However, Subversion's resources exist
in a virtual filesystem (see the section called “Repository Layer”), and your browser will not
understand how to read that filesystem.
For those who wish to access a Subversion repository using still another protocol, that is precisely why the Repository Access Layer is modularized! Developers can simply write a new library that implements the RA interface on one side and communicates with the repository on the other. Your new library can use existing network protocols, or you can invent your own. You could use inter-process communication (IPC) calls, or—let's get crazy, shall we?—you could even implement an email-based protocol. Subversion supplies the APIs; you supply the creativity.
On the client side, the Subversion working copy is where all the action takes place. The bulk of functionality implemented by the client-side libraries exists for the sole purpose of managing working copies—directories full of files and other subdirectories which serve as a sort of local, editable “reflection” of one or more repository locations—and propagating changes to and from the Repository Access layer.
Subversion's working copy library, libsvn_wc, is directly
responsible for managing the data in the working copies. To
accomplish this, the library stores administrative information
about each working copy directory within a special
subdirectory. This subdirectory, named
.svn, is present in each working copy
directory and contains various other files and directories
which record state and provide a private workspace for
administrative action. For those familiar with CVS, this
.svn subdirectory is similar in purpose
to the CVS administrative directories
found in CVS working copies. For more information about the
.svn administrative area, see the section called “Inside the Working Copy Administration Area”in this chapter.
The Subversion client library, libsvn_client, has the
broadest responsibility; its job is to mingle the
functionality of the working copy library with that of the
Repository Access Layer, and then to provide the highest-level
API to any application that wishes to perform general revision
control actions. For example, the function
svn_client_checkout() takes a URL as an
argument. It passes this URL to the RA layer and opens an
authenticated session with a particular repository. It then
asks the repository for a certain tree, and sends this tree
into the working copy library, which then writes a full
working copy to disk (.svn directories
and all).
The client library is designed to be used by any application. While the Subversion source code includes a standard command-line client, it should be very easy to write any number of GUI clients on top of the client library. New GUIs (or any new client, really) for Subversion need not be clunky wrappers around the included command-line client—they have full access via the libsvn_client API to same functionality, data, and callback mechanisms that the command-line client uses.
[42] We understand that this may come as a shock to sci-fi fans who have long been under the impression that Time was actually the fourth dimension, and we apologize for any emotional trauma induced by our assertion of a different theory.