Pluggable On-Disk Back-end APIs

The internal REST API used between the proxy server and the account, container and object server is almost identical to public Swift REST API, but with a few internal extentsions (for example, update an account with a new container).

The pluggable back-end APIs for the three REST API servers (account, container, object) abstracts the needs for servicing the various REST APIs from the details of how data is laid out and stored on-disk.

The APIs are documented in the reference implementations for all three servers. For historical reasons, the object server backend reference implementation module is named diskfile, while the account and container server backend reference implementation modules are named appropriately.

This API is still under development and not yet finalized.

Back-end API for Account Server REST APIs

Pluggable Back-end for Account Server

class swift.account.backend.AccountBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=None, stale_reads_ok=False)

Encapsulates working with an account database.

AccountBroker.create_account_stat_table(conn, put_timestamp)

Create account_stat table which is specific to the account DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:

• conn – DB connection object
• put_timestamp – put timestamp

AccountBroker.create_container_table(conn)

Create container table which is specific to the account DB.

Parameters:conn – DB connection object

AccountBroker.empty()

Check if the account DB is empty.

Returns:True if the database has no active containers.

AccountBroker.get_info()

Get global data for the account.

Returns:dict with keys: account, created_at, put_timestamp, delete_timestamp, container_count, object_count, bytes_used, hash, id

AccountBroker.is_deleted()

Check if the account DB is considered to be deleted.

Returns:True if the account DB is considered to be deleted, False otherwise

AccountBroker.is_status_deleted()

Only returns true if the status field is set to DELETED.

AccountBroker.list_containers_iter(limit, marker, end_marker, prefix, delimiter)

Get a list of containers sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:

• limit – maximum number of entries to get
• marker – marker query
• end_marker – end marker query
• prefix – prefix query
• delimiter – delimiter for query

Returns:

list of tuples of (name, object_count, bytes_used, 0)

AccountBroker.merge_items(item_list, source=None)

Merge items into the container table.

Parameters:

• item_list – list of dictionaries of {‘name’, ‘put_timestamp’, ‘delete_timestamp’, ‘object_count’, ‘bytes_used’, ‘deleted’}
• source – if defined, update incoming_sync with the source

AccountBroker.put_container(name, put_timestamp, delete_timestamp, object_count, bytes_used)

Create a container with the given attributes.

Parameters:

• name – name of the container to create
• put_timestamp – put_timestamp of the container to create
• delete_timestamp – delete_timestamp of the container to create
• object_count – number of objects in the container
• bytes_used – number of bytes used by the container

Back-end API for Container Server REST APIs

Pluggable Back-ends for Container Server

class swift.container.backend.ContainerBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=None, stale_reads_ok=False)

Encapsulates working with a container database.

ContainerBroker.create_container_stat_table(conn, put_timestamp=None)

Create the container_stat table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:

• conn – DB connection object
• put_timestamp – put timestamp

ContainerBroker.create_object_table(conn)

Create the object table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:conn – DB connection object

ContainerBroker.delete_object(name, timestamp)

Mark an object deleted.

Parameters:

• name – object name to be deleted
• timestamp – timestamp when the object was marked as deleted

ContainerBroker.empty()

Check if container DB is empty.

Returns:True if the database has no active objects, False otherwise

ContainerBroker.get_info()

Get global data for the container.

Returns:dict with keys: account, container, created_at, put_timestamp, delete_timestamp, object_count, bytes_used, reported_put_timestamp, reported_delete_timestamp, reported_object_count, reported_bytes_used, hash, id, x_container_sync_point1, and x_container_sync_point2.

ContainerBroker.is_deleted(timestamp=None)

Check if the DB is considered to be deleted.

Returns:True if the DB is considered to be deleted, False otherwise

ContainerBroker.list_objects_iter(limit, marker, end_marker, prefix, delimiter, path=None)

Get a list of objects sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:

• limit – maximum number of entries to get
• marker – marker query
• end_marker – end marker query
• prefix – prefix query
• delimiter – delimiter for query
• path – if defined, will set the prefix and delimter based on the path

Returns:

list of tuples of (name, created_at, size, content_type, etag)

ContainerBroker.merge_items(item_list, source=None)

Merge items into the object table.

Parameters:

• item_list – list of dictionaries of {‘name’, ‘created_at’, ‘size’, ‘content_type’, ‘etag’, ‘deleted’}
• source – if defined, update incoming_sync with the source

ContainerBroker.put_object(name, timestamp, size, content_type, etag, deleted=0)

Creates an object in the DB with its metadata.

Parameters:

• name – object name to be created
• timestamp – timestamp of when the object was created
• size – object size
• content_type – object content-type
• etag – object etag
• deleted – if True, marks the object as deleted and sets the deteleted_at timestamp to timestamp

ContainerBroker.reported(put_timestamp, delete_timestamp, object_count, bytes_used)

Update reported stats, available with container’s get_info.

Parameters:

• put_timestamp – put_timestamp to update
• delete_timestamp – delete_timestamp to update
• object_count – object_count to update
• bytes_used – bytes_used to update

Back-end API for Object Server REST APIs

Disk File Interface for the Swift Object Server

The DiskFile, DiskFileWriter and DiskFileReader classes combined define the on-disk abstraction layer for supporting the object server REST API interfaces (excluding REPLICATE). Other implementations wishing to provide an alternative backend for the object server must implement the three classes. An example alternative implementation can be found in the mem_server.py and mem_diskfile.py modules along size this one.

The DiskFileManager is a reference implemenation specific class and is not part of the backend API.

The remaining methods in this module are considered implementation specific and are also not considered part of the backend API.

class swift.obj.diskfile.AuditLocation(path, device, partition)

Represents an object location to be audited.

Other than being a bucket of data, the only useful thing this does is stringify to a filesystem path so the auditor’s logs look okay.

class swift.obj.diskfile.DiskFile(mgr, device_path, threadpool, partition, account=None, container=None, obj=None, _datadir=None)

Manage object files.

This specific implementation manages object files on a disk formatted with a POSIX-compliant file system that supports extended attributes as metadata on a file or directory.

Note

The arguments to the constructor are considered implementation specific. The API does not define the constructor arguments.

Parameters:

• mgr – associated DiskFileManager instance
• device_path – path to the target device or drive
• threadpool – thread pool to use for blocking operations
• partition – partition on the device in which the object lives
• account – account name for the object
• container – container name for the object
• obj – object name for the object

DiskFile.create(*args, **kwds)

Context manager to create a file. We create a temporary file first, and then return a DiskFileWriter object to encapsulate the state.

Note

An implementation is not required to perform on-disk preallocations even if the parameter is specified. But if it does and it fails, it must raise a DiskFileNoSpace exception.

Parameters:size – optional initial size of file to explicitly allocate on disk Raises DiskFileNoSpace:  if a size is specified and allocation fails

DiskFile.delete(timestamp)

Delete the object.

This implementation creates a tombstone file using the given timestamp, and removes any older versions of the object file. Any file that has an older timestamp than timestamp will be deleted.

Note

An implementation is free to use or ignore the timestamp parameter.

Parameters:timestamp – timestamp to compare with each file Raises DiskFileError:  this implementation will raise the same errors as the create() method.

DiskFile.get_metadata()

Provide the metadata for a previously opened object as a dictionary.

Returns:object’s metadata dictionary Raises DiskFileNotOpen:  if the swift.obj.diskfile.DiskFile.open() method was not previously invoked

DiskFile.open()

Open the object.

This implementation opens the data file representing the object, reads the associated metadata in the extended attributes, additionally combining metadata from fast-POST .meta files.

Note

An implementation is allowed to raise any of the following exceptions, but is only required to raise DiskFileNotExist when the object representation does not exist.

Raises:

• DiskFileCollision – on name mis-match with metadata
• DiskFileNotExist – if the object does not exist
• DiskFileDeleted – if the object was previously deleted
• DiskFileQuarantined – if while reading metadata of the file some data did pass cross checks

Returns:

itself for use as a context manager

DiskFile.read_metadata()

Return the metadata for an object without requiring the caller to open the object first.

Returns:metadata dictionary for an object Raises DiskFileError:  this implementation will raise the same errors as the open() method.

DiskFile.reader(keep_cache=False, _quarantine_hook=<function <lambda> at 0x5e539b0>)

Return a swift.common.swob.Response class compatible “app_iter” object as defined by swift.obj.diskfile.DiskFileReader.

For this implementation, the responsibility of closing the open file is passed to the swift.obj.diskfile.DiskFileReader object.

Parameters:

• keep_cache – caller’s preference for keeping data read in the OS buffer cache
• _quarantine_hook – 1-arg callable called when obj quarantined; the arg is the reason for quarantine. Default is to ignore it. Not needed by the REST layer.

Returns:

a swift.obj.diskfile.DiskFileReader object

DiskFile.write_metadata(metadata)

Write a block of metadata to an object without requiring the caller to create the object first. Supports fast-POST behavior semantics.

Parameters:metadata – dictionary of metadata to be associated with the object Raises DiskFileError:  this implementation will raise the same errors as the create() method.

class swift.obj.diskfile.DiskFileManager(conf, logger)

Management class for devices, providing common place for shared parameters and methods not provided by the DiskFile class (which primarily services the object server REST API layer).

The get_diskfile() method is how this implementation creates a DiskFile object.

Note

This class is reference implementation specific and not part of the pluggable on-disk backend API.

Note

TODO(portante): Not sure what the right name to recommend here, as “manager” seemed generic enough, though suggestions are welcome.

Parameters:

• conf – caller provided configuration object
• logger – caller provided logger

DiskFileManager.construct_dev_path(device)

Construct the path to a device without checking if it is mounted.

Parameters:device – name of target device Returns:full path to the device

DiskFileManager.get_dev_path(device, mount_check=None)

Return the path to a device, checking to see that it is a proper mount point based on a configuration parameter.

Parameters:

• device – name of target device
• mount_check – whether or not to check mountedness of device. Defaults to bool(self.mount_check).

Returns:

full path to the device, None if the path to the device is not a proper mount point.

DiskFileManager.get_diskfile_from_hash(device, partition, object_hash, **kwargs)

Returns a DiskFile instance for an object at the given object_hash. Just in case someone thinks of refactoring, be sure DiskFileDeleted is not raised, but the DiskFile instance representing the tombstoned object is returned instead.

Raises DiskFileNotExist:  if the object does not exist

DiskFileManager.replication_lock(*args, **kwds)

A context manager that will lock on the device given, if configured to do so.

Raises ReplicationLockTimeout:  If the lock on the device cannot be granted within the configured timeout.

DiskFileManager.yield_hashes(device, partition, suffixes=None)

Yields tuples of (full_path, hash_only, timestamp) for object information stored for the given device, partition, and (optionally) suffixes. If suffixes is None, all stored suffixes will be searched for object hashes. Note that if suffixes is not None but empty, such as [], then nothing will be yielded.

DiskFileManager.yield_suffixes(device, partition)

Yields tuples of (full_path, suffix_only) for suffixes stored on the given device and partition.

class swift.obj.diskfile.DiskFileReader(fp, data_file, obj_size, etag, threadpool, disk_chunk_size, keep_cache_size, device_path, logger, quarantine_hook, keep_cache=False)

Encapsulation of the WSGI read context for servicing GET REST API requests. Serves as the context manager object for the swift.obj.diskfile.DiskFile class’s swift.obj.diskfile.DiskFile.reader() method.

Note

The quarantining behavior of this method is considered implementation specific, and is not required of the API.

Note

The arguments to the constructor are considered implementation specific. The API does not define the constructor arguments.

Parameters:

• fp – open file object pointer reference
• data_file – on-disk data file name for the object
• obj_size – verified on-disk size of the object
• etag – expected metadata etag value for entire file
• threadpool – thread pool to use for read operations
• disk_chunk_size – size of reads from disk in bytes
• keep_cache_size – maximum object size that will be kept in cache
• device_path – on-disk device path, used when quarantining an obj
• logger – logger caller wants this object to use
• quarantine_hook – 1-arg callable called w/reason when quarantined
• keep_cache – should resulting reads be kept in the buffer cache

DiskFileReader.app_iter_range(start, stop)

Returns an iterator over the data file for range (start, stop)

DiskFileReader.app_iter_ranges(ranges, content_type, boundary, size)

Returns an iterator over the data file for a set of ranges

DiskFileReader.close()

Close the open file handle if present.

For this specific implementation, this method will handle quarantining the file if necessary.

class swift.obj.diskfile.DiskFileWriter(name, datadir, fd, tmppath, bytes_per_sync, threadpool)

Encapsulation of the write context for servicing PUT REST API requests. Serves as the context manager object for the swift.obj.diskfile.DiskFile class’s swift.obj.diskfile.DiskFile.create() method.

Note

It is the responsibility of the swift.obj.diskfile.DiskFile.create() method context manager to close the open file descriptor.

Note

The arguments to the constructor are considered implementation specific. The API does not define the constructor arguments.

Parameters:

• name – name of object from REST API
• datadir – on-disk directory object will end up in on swift.obj.diskfile.DiskFileWriter.put()
• fd – open file descriptor of temporary file to receive data
• tmppath – full path name of the opened file descriptor
• bytes_per_sync – number bytes written between sync calls
• threadpool – internal thread pool to use for disk operations

DiskFileWriter.put(metadata)

Finalize writing the file on disk.

For this implementation, this method is responsible for renaming the temporary file to the final name and directory location. This method should be called after the final call to swift.obj.diskfile.DiskFileWriter.write().

Parameters:metadata – dictionary of metadata to be associated with the object

DiskFileWriter.write(chunk)

Write a chunk of data to disk. All invocations of this method must come before invoking the :func:

For this implementation, the data is written into a temporary file.

Parameters:chunk – the chunk of data to write as a string object Returns:the total number of bytes written to an object

swift.obj.diskfile.get_hashes(partition_dir, recalculate=None, do_listdir=False, reclaim_age=604800)

Get a list of hashes for the suffix dir. do_listdir causes it to mistrust the hash cache for suffix existence at the (unexpectedly high) cost of a listdir. reclaim_age is just passed on to hash_suffix.

Parameters:

• partition_dir – absolute path of partition to get hashes for
• recalculate – list of suffixes which should be recalculated when got
• do_listdir – force existence check for all hashes in the partition
• reclaim_age – age at which to remove tombstones

Returns:

tuple of (number of suffix dirs hashed, dictionary of hashes)

swift.obj.diskfile.get_ondisk_files(files, datadir)

Given a simple list of files names, determine the files to use.

Params files:simple set of files as a python list Params datadir:directory name files are from for convenience Returns:a tuple of data, meta and ts (tombstone) files, in one of two states:

• ts_file is not None, data_file is None, meta_file is None

  object is considered deleted

• data_file is not None, ts_file is None

  object exists, and optionally has fast-POST metadata

swift.obj.diskfile.hash_cleanup_listdir(hsh_path, reclaim_age=604800)

List contents of a hash directory and clean up any old files.

Parameters:

• hsh_path – object hash path
• reclaim_age – age in seconds at which to remove tombstones

Returns:

list of files remaining in the directory, reverse sorted

swift.obj.diskfile.hash_suffix(path, reclaim_age)

Performs reclamation and returns an md5 of all (remaining) files.

Parameters:

reclaim_age – age in seconds at which to remove tombstones

Raises:

• PathNotDir – if given path is not a valid directory
• OSError – for non-ENOTDIR errors

swift.obj.diskfile.invalidate_hash(suffix_dir)

Invalidates the hash for a suffix_dir in the partition’s hashes file.

Parameters:suffix_dir – absolute path to suffix dir whose hash needs invalidating

swift.obj.diskfile.object_audit_location_generator(devices, mount_check=True, logger=None, device_dirs=None)

Given a devices path (e.g. “/srv/node”), yield an AuditLocation for all objects stored under that directory if device_dirs isn’t set. If device_dirs is set, only yield AuditLocation for the objects under the entries in device_dirs. The AuditLocation only knows the path to the hash directory, not to the .data file therein (if any). This is to avoid a double listdir(hash_dir); the DiskFile object will always do one, so we don’t.

Parameters:

• devices – parent directory of the devices to be audited
• mount_check – flag to check if a mount check should be performed on devices
• logger – a logger object

Device_dirs :

a list of directories under devices to traverse

swift.obj.diskfile.quarantine_renamer(device_path, corrupted_file_path)

In the case that a file is corrupted, move it to a quarantined area to allow replication to fix it.

Params device_path:  The path to the device the corrupted file is on. Params corrupted_file_path:  The path to the file you want quarantined. Returns:path (str) of directory the file was moved to Raises OSError:re-raises non errno.EEXIST / errno.ENOTEMPTY exceptions from rename

swift.obj.diskfile.read_metadata(fd)

Helper function to read the pickled metadata from an object file.

Parameters:fd – file descriptor or filename to load the metadata from Returns:dictionary of metadata

swift.obj.diskfile.write_metadata(fd, metadata)

Helper function to write pickled metadata for an object file.

Parameters:

• fd – file descriptor or filename to write the metadata
• metadata – metadata to write