Container

Container Auditor

class swift.container.auditor.ContainerAuditor(conf)

Bases: swift.common.daemon.Daemon

Audit containers.

container_audit(path)

Audits the given container path

Parameters:path – the path to a container db

run_forever(*args, **kwargs)

Run the container audit until stopped.

run_once(*args, **kwargs)

Run the container audit once.

swift.container.auditor.random() → x in the interval [0, 1).

Container Backend

Pluggable Back-ends for Container Server

class swift.container.backend.ContainerBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=None, stale_reads_ok=False)

Bases: swift.common.db.DatabaseBroker

Encapsulates working with a container database.

create_container_stat_table(conn, put_timestamp=None)

Create the container_stat table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:

• conn – DB connection object
• put_timestamp – put timestamp

create_object_table(conn)

Create the object table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:conn – DB connection object

db_contains_type = 'object'

db_reclaim_timestamp = 'created_at'

db_type = 'container'

delete_object(name, timestamp)

Mark an object deleted.

Parameters:

• name – object name to be deleted
• timestamp – timestamp when the object was marked as deleted

empty()

Check if container DB is empty.

Returns:True if the database has no active objects, False otherwise

get_db_version(conn)

get_info()

Get global data for the container.

Returns:dict with keys: account, container, created_at, put_timestamp, delete_timestamp, object_count, bytes_used, reported_put_timestamp, reported_delete_timestamp, reported_object_count, reported_bytes_used, hash, id, x_container_sync_point1, and x_container_sync_point2.

is_deleted(timestamp=None)

Check if the DB is considered to be deleted.

Returns:True if the DB is considered to be deleted, False otherwise

list_objects_iter(limit, marker, end_marker, prefix, delimiter, path=None)

Get a list of objects sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:

• limit – maximum number of entries to get
• marker – marker query
• end_marker – end marker query
• prefix – prefix query
• delimiter – delimiter for query
• path – if defined, will set the prefix and delimter based on the path

Returns:

list of tuples of (name, created_at, size, content_type, etag)

merge_items(item_list, source=None)

Merge items into the object table.

Parameters:

• item_list – list of dictionaries of {‘name’, ‘created_at’, ‘size’, ‘content_type’, ‘etag’, ‘deleted’}
• source – if defined, update incoming_sync with the source

put_object(name, timestamp, size, content_type, etag, deleted=0)

Creates an object in the DB with its metadata.

Parameters:

• name – object name to be created
• timestamp – timestamp of when the object was created
• size – object size
• content_type – object content-type
• etag – object etag
• deleted – if True, marks the object as deleted and sets the deteleted_at timestamp to timestamp

reported(put_timestamp, delete_timestamp, object_count, bytes_used)

Update reported stats, available with container’s get_info.

Parameters:

• put_timestamp – put_timestamp to update
• delete_timestamp – delete_timestamp to update
• object_count – object_count to update
• bytes_used – bytes_used to update

set_x_container_sync_points(sync_point1, sync_point2)

Container Server

class swift.container.server.ContainerController(conf, logger=None)

Bases: object

WSGI Controller for the container server.

DELETE(*a, **kw)

Handle HTTP DELETE request.

GET(*a, **kw)

Handle HTTP GET request.

HEAD(*a, **kw)

Handle HTTP HEAD request.

POST(*a, **kw)

Handle HTTP POST request.

PUT(*a, **kw)

Handle HTTP PUT request.

REPLICATE(*a, **kw)

Handle HTTP REPLICATE request (json-encoded RPC calls for replication.)

account_update(req, account, container, broker)

Update the account server(s) with latest container info.

Parameters:

• req – swob.Request object
• account – account name
• container – container name
• broker – container DB broker object

Returns:

if all the account requests return a 404 error code, HTTPNotFound response object, if the account cannot be updated due to a malformed header, an HTTPBadRequest response object, otherwise None.

allowed_sync_hosts = None

The list of hosts we’re allowed to send syncs to. This can be overridden by data in self.realms_conf

realms_conf = None

ContainerSyncCluster instance for validating sync-to values.

save_headers = ['x-container-read', 'x-container-write', 'x-container-sync-key', 'x-container-sync-to']

update_data_record(record)

Perform any mutations to container listing records that are common to all serialization formats, and returns it as a dict.

Converts created time to iso timestamp. Replaces size with ‘swift_bytes’ content type parameter.

Params record:object entry record Returns:modified record

swift.container.server.app_factory(global_conf, **local_conf)

paste.deploy app factory for creating WSGI container server apps

Container Sync

class swift.container.sync.ContainerSync(conf, container_ring=None, object_ring=None)

Bases: swift.common.daemon.Daemon

Daemon to sync syncable containers.

This is done by scanning the local devices for container databases and checking for x-container-sync-to and x-container-sync-key metadata values. If they exist, newer rows since the last sync will trigger PUTs or DELETEs to the other container.

Note

Container sync will sync object POSTs only if the proxy server is set to use “object_post_as_copy = true” which is the default. So-called fast object posts, “object_post_as_copy = false” do not update the container listings and therefore can’t be detected for synchronization.

The actual syncing is slightly more complicated to make use of the three (or number-of-replicas) main nodes for a container without each trying to do the exact same work but also without missing work if one node happens to be down.

Two sync points are kept per container database. All rows between the two sync points trigger updates. Any rows newer than both sync points cause updates depending on the node’s position for the container (primary nodes do one third, etc. depending on the replica count of course). After a sync run, the first sync point is set to the newest ROWID known and the second sync point is set to newest ROWID for which all updates have been sent.

An example may help. Assume replica count is 3 and perfectly matching ROWIDs starting at 1.

First sync run, database has 6 rows:

• SyncPoint1 starts as -1.
• SyncPoint2 starts as -1.
• No rows between points, so no “all updates” rows.
• Six rows newer than SyncPoint1, so a third of the rows are sent by node 1, another third by node 2, remaining third by node 3.
• SyncPoint1 is set as 6 (the newest ROWID known).
• SyncPoint2 is left as -1 since no “all updates” rows were synced.

Next sync run, database has 12 rows:

• SyncPoint1 starts as 6.
• SyncPoint2 starts as -1.
• The rows between -1 and 6 all trigger updates (most of which should short-circuit on the remote end as having already been done).
• Six more rows newer than SyncPoint1, so a third of the rows are sent by node 1, another third by node 2, remaining third by node 3.
• SyncPoint1 is set as 12 (the newest ROWID known).
• SyncPoint2 is set as 6 (the newest “all updates” ROWID).

In this way, under normal circumstances each node sends its share of updates each run and just sends a batch of older updates to ensure nothing was missed.

Parameters:

• conf – The dict of configuration values from the [container-sync] section of the container-server.conf
• container_ring – If None, the <swift_dir>/container.ring.gz will be loaded. This is overridden by unit tests.
• object_ring – If None, the <swift_dir>/object.ring.gz will be loaded. This is overridden by unit tests.

allowed_sync_hosts = None

The list of hosts we’re allowed to send syncs to. This can be overridden by data in self.realms_conf

conf = None

The dict of configuration values from the [container-sync] section of the container-server.conf.

container_deletes = None

Number of successful DELETEs triggered.

container_failures = None

Number of containers that had a failure of some type.

container_puts = None

Number of successful PUTs triggered.

container_ring = None

swift.common.ring.Ring for locating containers.

container_skips = None

Number of containers that didn’t have sync turned on.

container_sync(path)

Checks the given path for a container database, determines if syncing is turned on for that database and, if so, sends any updates to the other container.

Parameters:path – the path to a container db

container_sync_row(row, sync_to, user_key, broker, info, realm, realm_key)

Sends the update the row indicates to the sync_to container.

Parameters:

• row – The updated row in the local database triggering the sync update.
• sync_to – The URL to the remote container.
• user_key – The X-Container-Sync-Key to use when sending requests to the other container.
• broker – The local container database broker.
• info – The get_info result from the local container database broker.
• realm – The realm from self.realms_conf, if there is one. If None, fallback to using the older allowed_sync_hosts way of syncing.
• realm_key – The realm key from self.realms_conf, if there is one. If None, fallback to using the older allowed_sync_hosts way of syncing.

Returns:

True on success

container_syncs = None

Number of containers with sync turned on that were successfully synced.

container_time = None

Maximum amount of time to spend syncing a container before moving on to the next one. If a conatiner sync hasn’t finished in this time, it’ll just be resumed next scan.

devices = None

Path to the local device mount points.

interval = None

Minimum time between full scans. This is to keep the daemon from running wild on near empty systems.

logger = None

Logger to use for container-sync log lines.

mount_check = None

Indicates whether mount points should be verified as actual mount points (normally true, false for tests and SAIO).

object_ring = None

swift.common.ring.Ring for locating objects.

realms_conf = None

ContainerSyncCluster instance for validating sync-to values.

report()

Writes a report of the stats to the logger and resets the stats for the next report.

reported = None

Time of last stats report.

run_forever(*args, **kwargs)

Runs container sync scans until stopped.

run_once(*args, **kwargs)

Runs a single container sync scan.

select_http_proxy()

swift.container.sync.random() → x in the interval [0, 1).

Container Updater

class swift.container.updater.ContainerUpdater(conf)

Bases: swift.common.daemon.Daemon

Update container information in account listings.

container_report(node, part, container, put_timestamp, delete_timestamp, count, bytes)

Report container info to an account server.

Parameters:

• node – node dictionary from the account ring
• part – partition the account is on
• container – container name
• put_timestamp – put timestamp
• delete_timestamp – delete timestamp
• count – object count in the container
• bytes – bytes used in the container

container_sweep(path)

Walk the path looking for container DBs and process them.

Parameters:path – path to walk

get_account_ring()

Get the account ring. Load it if it hasn’t been yet.

get_paths()

Get paths to all of the partitions on each drive to be processed.

Returns:a list of paths

process_container(dbfile)

Process a container, and update the information in the account.

Parameters:dbfile – container DB to process

run_forever(*args, **kwargs)

Run the updator continuously.

run_once(*args, **kwargs)

Run the updater once.

swift.container.updater.random() → x in the interval [0, 1).