Misc¶

ACLs¶

swift.common.middleware.acl.acls_from_account_info(info)¶

Extract the account ACLs from the given account_info, and return the ACLs.

Raises ValueError:
Parameters:	info – a dict of the form returned by get_account_info
Returns:	None (no ACL system metadata is set), or a dict of the form:: {‘admin’: [...], ‘read-write’: [...], ‘read-only’: [...]}
	on a syntactically invalid header

swift.common.middleware.acl.clean_acl(name, value)¶

Returns a cleaned ACL header value, validating that it meets the formatting requirements for standard Swift ACL strings.

The ACL format is:

[item[,item...]]

Each item can be a group name to give access to or a referrer designation to grant or deny based on the HTTP Referer header.

The referrer designation format is:

.r:[-]value

The .r can also be .ref, .referer, or .referrer; though it will be shortened to just .r for decreased character count usage.

The value can be * to specify any referrer host is allowed access, a specific host name like www.example.com, or if it has a leading period . or leading *. it is a domain name specification, like .example.com or *.example.com. The leading minus sign - indicates referrer hosts that should be denied access.

Referrer access is applied in the order they are specified. For example, .r:.example.com,.r:-thief.example.com would allow all hosts ending with .example.com except for the specific host thief.example.com.

Example valid ACLs:

.r:*
.r:*,.r:-.thief.com
.r:*,.r:.example.com,.r:-thief.example.com
.r:*,.r:-.thief.com,bobs_account,sues_account:sue
bobs_account,sues_account:sue

Example invalid ACLs:

.r:
.r:-

By default, allowing read access via .r will not allow listing objects in the container – just retrieving objects from the container. To turn on listings, use the .rlistings directive.

Also, .r designations aren’t allowed in headers whose names include the word ‘write’.

ACLs that are “messy” will be cleaned up. Examples:

Original	Cleaned
`bob, sue`	`bob,sue`
`bob , sue`	`bob,sue`
`bob,,,sue`	`bob,sue`
`.referrer : *`	`.r:*`
`.ref:*.example.com`	`.r:.example.com`
`.r:*, .rlistings`	`.r:*,.rlistings`

Raises ValueError:
Parameters:	name – The name of the header being cleaned, such as X-Container-Read or X-Container-Write. value – The value of the header being cleaned.
Returns:	The value, cleaned of extraneous formatting.
	If the value does not meet the ACL formatting requirements; the error message will indicate why.

swift.common.middleware.acl.format_acl(version=1, **kwargs)¶

Compatibility wrapper to help migrate ACL syntax from version 1 to 2. Delegates to the appropriate version-specific format_acl method, defaulting to version 1 for backward compatibility.

Parameters:	kwargs – keyword args appropriate for the selected ACL syntax version (see `format_acl_v1()` or `format_acl_v2()`)

swift.common.middleware.acl.format_acl_v1(groups=None, referrers=None, header_name=None)¶

Returns a standard Swift ACL string for the given inputs.

Caller is responsible for ensuring that :referrers: parameter is only given if the ACL is being generated for X-Container-Read. (X-Container-Write and the account ACL headers don’t support referrers.)

Parameters:	groups – a list of groups (and/or members in most auth systems) to grant access referrers – a list of referrer designations (without the leading .r:) header_name – (optional) header name of the ACL we’re preparing, for clean_acl; if None, returned ACL won’t be cleaned
Returns:	a Swift ACL string for use in X-Container-{Read,Write}, X-Account-Access-Control, etc.

swift.common.middleware.acl.format_acl_v2(acl_dict)¶

Returns a version-2 Swift ACL JSON string.

HTTP headers for Version 2 ACLs have the following form:: Header-Name: {“arbitrary”:”json”,”encoded”:”string”}

JSON will be forced ASCII (containing six-char uNNNN sequences rather than UTF-8; UTF-8 is valid JSON but clients vary in their support for UTF-8 headers), and without extraneous whitespace.

Advantages over V1: forward compatibility (new keys don’t cause parsing exceptions); Unicode support; no reserved words (you can have a user named .rlistings if you want).

Parameters:	acl_dict – dict of arbitrary data to put in the ACL; see specific auth systems such as tempauth for supported values
Returns:	a JSON string which encodes the ACL

swift.common.middleware.acl.parse_acl(*args, **kwargs)¶

Compatibility wrapper to help migrate ACL syntax from version 1 to 2. Delegates to the appropriate version-specific parse_acl method, attempting to determine the version from the types of args/kwargs.

Parameters:	args – positional args for the selected ACL syntax version kwargs – keyword args for the selected ACL syntax version (see `parse_acl_v1()` or `parse_acl_v2()`)
Returns:	the return value of `parse_acl_v1()` or `parse_acl_v2()`

swift.common.middleware.acl.parse_acl_v1(acl_string)¶

Parses a standard Swift ACL string into a referrers list and groups list.

See clean_acl() for documentation of the standard Swift ACL format.

Parameters:	acl_string – The standard Swift ACL string to parse.
Returns:	A tuple of (referrers, groups) where referrers is a list of referrer designations (without the leading .r:) and groups is a list of groups to allow access.

swift.common.middleware.acl.parse_acl_v2(data)¶

Parses a version-2 Swift ACL string and returns a dict of ACL info.

Parameters:	data – string containing the ACL data in JSON format
Returns:	A dict (possibly empty) containing ACL info, e.g.: {“groups”: [...], “referrers”: [...]}
Returns:	None if data is None, is not valid JSON or does not parse as a dict
Returns:	empty dictionary if data is an empty string

swift.common.middleware.acl.referrer_allowed(referrer, referrer_acl)¶

Returns True if the referrer should be allowed based on the referrer_acl list (as returned by parse_acl()).

See clean_acl() for documentation of the standard Swift ACL format.

Parameters:	referrer – The value of the HTTP Referer header. referrer_acl – The list of referrer designations as returned by `parse_acl()`.
Returns:	True if the referrer should be allowed; False if not.

Buffered HTTP¶

Monkey Patch httplib.HTTPResponse to buffer reads of headers. This can improve performance when making large numbers of small HTTP requests. This module also provides helper functions to make HTTP connections using BufferedHTTPResponse.

Warning

If you use this, be sure that the libraries you are using do not access the socket directly (xmlrpclib, I’m looking at you :/), and instead make all calls through httplib.

class swift.common.bufferedhttp.BufferedHTTPConnection(host, port=None, strict=None, timeout=<object object at 0x7f95ac78b0c0>, source_address=None)¶

Bases: httplib.HTTPConnection

HTTPConnection class that uses BufferedHTTPResponse

response_class¶: alias of BufferedHTTPResponse

class swift.common.bufferedhttp.BufferedHTTPResponse(sock, debuglevel=0, strict=0, method=None)¶

Bases: httplib.HTTPResponse

HTTPResponse class that buffers reading of headers

nuke_from_orbit()¶

Terminate the socket with extreme prejudice.

Closes the underlying socket regardless of whether or not anyone else has references to it. Use this when you are certain that nobody else you care about has a reference to this socket.

swift.common.bufferedhttp.http_connect(ipaddr, port, device, partition, method, path, headers=None, query_string=None, ssl=False)¶

Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.

Parameters:

ipaddr – IPv4 address to connect to
port – port to connect to
device – device of the node to query
partition – partition on the device
method – HTTP method to request (‘GET’, ‘PUT’, ‘POST’, etc.)
path – request path
headers – dictionary of headers
query_string – request query string
ssl – set True if SSL should be used (default: False)

Returns:

HTTPConnection object

swift.common.bufferedhttp.http_connect_raw(ipaddr, port, method, path, headers=None, query_string=None, ssl=False)¶

Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.

Parameters:	ipaddr – IPv4 address to connect to port – port to connect to method – HTTP method to request (‘GET’, ‘PUT’, ‘POST’, etc.) path – request path headers – dictionary of headers query_string – request query string ssl – set True if SSL should be used (default: False)
Returns:	HTTPConnection object

Constraints¶

swift.common.constraints.FORMAT2CONTENT_TYPE = {'xml': 'application/xml', 'plain': 'text/plain', 'json': 'application/json'}¶: Query string format= values to their corresponding content-type values

swift.common.constraints.check_copy_from_header(req)¶

Validate that the value from x-copy-from header is well formatted. We assume the caller ensures that x-copy-from header is present in req.headers.

Parameters:	req – HTTP request object
Returns:	A tuple with container name and object name
Raise :	HTTPPreconditionFailed if x-copy-from value is not well formatted.

swift.common.constraints.check_float(string)¶

Helper function for checking if a string can be converted to a float.

Parameters:	string – string to be verified as a float
Returns:	True if the string can be converted to a float, False otherwise

swift.common.constraints.check_metadata(req, target_type)¶

Check metadata sent in the request headers.

Parameters:	req – request object target_type – str: one of: object, container, or account: indicates which type the target storage for the metadata is
Returns:	HTTPBadRequest with bad metadata otherwise None

swift.common.constraints.check_mount(root, drive)¶

Verify that the path to the device is a mount point and mounted. This allows us to fast fail on drives that have been unmounted because of issues, and also prevents us for accidentally filling up the root partition.

Parameters:	root – base path where the devices are mounted drive – drive name to be checked
Returns:	True if it is a valid mounted device, False otherwise

swift.common.constraints.check_object_creation(req, object_name)¶

Check to ensure that everything is alright about an object to be created.

Returns HTTPRequestEntityTooLarge:
Parameters:	req – HTTP request object object_name – name of object to be created
	the object is too large
Returns HTTPLengthRequired:
	missing content-length header and not a chunked request
Returns HTTPBadRequest:
	missing or bad content-type header, or bad metadata

swift.common.constraints.check_utf8(string)¶

Validate if a string is valid UTF-8 str or unicode and that it does not contain any null character.

Parameters:	string – string to be validated
Returns:	True if the string is valid utf-8 str or unicode and contains no null characters, False otherwise

swift.common.constraints.reload_constraints()¶: Parse SWIFT_CONF_FILE and reset module level global contraint attrs, populating OVERRIDE_CONSTRAINTS AND EFFECTIVE_CONSTRAINTS along the way.

Container Sync Realms¶

class swift.common.container_sync_realms.ContainerSyncRealms(conf_path, logger)¶

Bases: object

Loads and parses the container-sync-realms.conf, occasionally checking the file’s mtime to see if it needs to be reloaded.

clusters(realm)¶: Returns a list of clusters for the realm.

endpoint(realm, cluster)¶: Returns the endpoint for the cluster in the realm.

get_sig(request_method, path, x_timestamp, nonce, realm_key, user_key)¶

Returns the hexdigest string of the HMAC-SHA1 (RFC 2104) for the information given.

Parameters:	request_method – HTTP method of the request. path – The path to the resource. x_timestamp – The X-Timestamp header value for the request. nonce – A unique value for the request. realm_key – Shared secret at the cluster operator level. user_key – Shared secret at the user’s container level.
Returns:	hexdigest str of the HMAC-SHA1 for the request.

key(realm)¶: Returns the key for the realm.

key2(realm)¶: Returns the key2 for the realm.

realms()¶: Returns a list of realms.

reload()¶: Forces a reload of the conf file.

Direct Client¶

Internal client library for making calls directly to the servers rather than through the proxy.

swift.common.direct_client.direct_delete_container(node, part, account, container, conn_timeout=5, response_timeout=15, headers=None)¶

swift.common.direct_client.direct_delete_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶

Delete object directly from the object server.

Parameters:	node – node dictionary from the ring part – partition the container is on account – account name container – container name obj – object name conn_timeout – timeout in seconds for establishing the connection response_timeout – timeout in seconds for getting the response
Returns:	response from server

swift.common.direct_client.direct_get_account(node, part, account, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15)¶

Get listings directly from the account server.

Parameters:	node – node dictionary from the ring part – partition the account is on account – account name marker – marker query limit – query limit prefix – prefix query delimeter – delimeter for the query conn_timeout – timeout in seconds for establishing the connection response_timeout – timeout in seconds for getting the response
Returns:	a tuple of (response headers, a list of containers) The response headers will be a dict and all header names will be lowercase.

swift.common.direct_client.direct_get_container(node, part, account, container, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15)¶

Get container listings directly from the container server.

Parameters:

node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
marker – marker query
limit – query limit
prefix – prefix query
delimeter – delimeter for the query
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response

Returns:

a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase.

swift.common.direct_client.direct_get_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, resp_chunk_size=None, headers=None)¶

Get object directly from the object server.

Parameters:

node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
obj – object name
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
resp_chunk_size – if defined, chunk size of data to read.
headers – dict to be passed into HTTPConnection headers

Returns:

a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase.

swift.common.direct_client.direct_head_container(node, part, account, container, conn_timeout=5, response_timeout=15)¶

Request container information directly from the container server.

Parameters:	node – node dictionary from the ring part – partition the container is on account – account name container – container name conn_timeout – timeout in seconds for establishing the connection response_timeout – timeout in seconds for getting the response
Returns:	a dict containing the response’s headers (all header names will be lowercase)

swift.common.direct_client.direct_head_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15)¶

Request object information directly from the object server.

Parameters:	node – node dictionary from the ring part – partition the container is on account – account name container – container name obj – object name conn_timeout – timeout in seconds for establishing the connection response_timeout – timeout in seconds for getting the response
Returns:	a dict containing the response’s headers (all header names will be lowercase)

swift.common.direct_client.direct_post_object(node, part, account, container, name, headers, conn_timeout=5, response_timeout=15)¶

Direct update to object metadata on object server.

Raises ClientException:
Parameters:	node – node dictionary from the ring part – partition the container is on account – account name container – container name name – object name headers – headers to store as metadata conn_timeout – timeout in seconds for establishing the connection response_timeout – timeout in seconds for getting the response
	HTTP POST request failed

swift.common.direct_client.direct_put_object(node, part, account, container, name, contents, content_length=None, etag=None, content_type=None, headers=None, conn_timeout=5, response_timeout=15, chunk_size=65535)¶

Put object directly from the object server.

Parameters:

node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
name – object name
contents – an iterable or string to read object data from
content_length – value to send as content-length header
etag – etag of contents
content_type – value to send as content-type header
headers – additional headers to include in the request
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
chunk_size – if defined, chunk size of data to send.

Returns:

etag from the server response

swift.common.direct_client.gen_headers(hdrs_in=None, add_ts=False)¶

swift.common.direct_client.retry(func, *args, **kwargs)¶

Helper function to retry a given function a number of times.

Parameters:	func – callable to be called retries – number of retries error_log – logger for errors args – arguments to send to func kwargs – keyward arguments to send to func (if retries or error_log are sent, they will be deleted from kwargs before sending on to func)
Returns:	restult of func

Exceptions¶

exception swift.common.exceptions.ChunkReadTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.ChunkWriteTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.ClientException(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=0, http_reason='', http_device='', http_response_content='')¶: Bases: exceptions.Exception

exception swift.common.exceptions.ConnectionTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.DiskFileCollision¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileDeleted(metadata=None)¶: Bases: swift.common.exceptions.DiskFileNotExist

exception swift.common.exceptions.DiskFileDeviceUnavailable¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.DiskFileExpired(metadata=None)¶: Bases: swift.common.exceptions.DiskFileDeleted

exception swift.common.exceptions.DiskFileNoSpace¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileNotExist¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileNotOpen¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileQuarantined¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DriveNotMounted¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.DuplicateDeviceError¶: Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.EmptyRingError¶: Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.ListingIterError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ListingIterNotAuthorized(aresp)¶: Bases: swift.common.exceptions.ListingIterError

exception swift.common.exceptions.ListingIterNotFound¶: Bases: swift.common.exceptions.ListingIterError

exception swift.common.exceptions.LockTimeout(seconds=None, msg=None)¶: Bases: swift.common.exceptions.MessageTimeout

exception swift.common.exceptions.MessageTimeout(seconds=None, msg=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.PathNotDir¶: Bases: exceptions.OSError

exception swift.common.exceptions.ReplicationException¶: Bases: exceptions.Exception

exception swift.common.exceptions.ReplicationLockTimeout(seconds=None, msg=None)¶: Bases: swift.common.exceptions.LockTimeout

exception swift.common.exceptions.RingBuilderError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.RingValidationError¶: Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.SegmentError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.SwiftException¶: Bases: exceptions.Exception

Internal Client¶

class swift.common.internal_client.CompressingFileReader(file_obj, compresslevel=9, chunk_size=4096)¶

Bases: object

Wrapper for file object to compress object while reading.

Can be used to wrap file objects passed to InternalClient.upload_object().

Used in testing of InternalClient.

Parameters:	file_obj – File object to wrap. compresslevel – Compression level, defaults to 9. chunk_size – Size of chunks read when iterating using object, defaults to 4096.

next()¶

read(*a, **kw)¶

Reads a chunk from the file object.

Params are passed directly to the underlying file object’s read().

Returns:	Compressed chunk from file object.

seek(offset, whence=0)¶

set_initial_state()¶: Sets the object to the state needed for the first read.

class swift.common.internal_client.InternalClient(conf_path, user_agent, request_tries, allow_modify_pipeline=False)¶

Bases: object

An internal client that uses a swift proxy app to make requests to Swift.

This client will exponentially slow down for retries.

Parameters:	conf_path – Full path to proxy config. user_agent – User agent to be sent to requests to Swift. request_tries – Number of tries before InternalClient.make_request() gives up.

container_exists(account, container)¶

Checks to see if a container exists.

Parameters:	account – The container’s account. container – Container to check.
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

:returns : True if container exists, false otherwise.

create_container(account, container, headers=None, acceptable_statuses=(2, ))¶

Creates container.

Parameters:	account – The container’s account. container – Container to create. headers – Defaults to empty dict. acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

delete_container(account, container, acceptable_statuses=(2, 404))¶

Deletes a container.

Parameters:	account – The container’s account. container – Container to delete. acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

delete_object(account, container, obj, acceptable_statuses=(2, 404))¶

Deletes an object.

Parameters:	account – The object’s account. container – The object’s container. obj – The object. acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

get_account_info(account, acceptable_statuses=(2, 404))¶

Returns (container_count, object_count) for an account.

Parameters:	account – Account on which to get the information. acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

get_account_metadata(account, metadata_prefix='', acceptable_statuses=(2, ))¶

Gets account metadata.

Parameters:	account – Account on which to get the metadata. metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’. acceptable_statuses – List of status for valid responses, defaults to (2,).

:returns : Returns dict of account metadata. Keys will be lowercase.

Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

get_container_metadata(account, container, metadata_prefix='', acceptable_statuses=(2, ))¶

Gets container metadata.

Parameters:	account – The container’s account. container – Container to get metadata on. metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’. acceptable_statuses – List of status for valid responses, defaults to (2,).

:returns : Returns dict of container metadata. Keys will be lowercase.

Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

get_object_metadata(account, container, obj, metadata_prefix='', acceptable_statuses=(2, ))¶

Gets object metadata.

Parameters:	account – The object’s account. container – The object’s container. obj – The object. metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’. acceptable_statuses – List of status for valid responses, defaults to (2,).

:returns : Dict of object metadata.

Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

iter_containers(account, marker='', end_marker='', acceptable_statuses=(2, 404))¶

Returns an iterator of containers dicts from an account.

Parameters:	account – Account on which to do the container listing. marker – Prefix of first desired item, defaults to ‘’. end_marker – Last item returned will be ‘less’ than this, defaults to ‘’. acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

iter_object_lines(account, container, obj, headers=None, acceptable_statuses=(2, ))¶

Returns an iterator of object lines from an uncompressed or compressed text object.

Uncompress object as it is read if the object’s name ends with ‘.gz’.

Parameters:	account – The object’s account. container – The object’s container. obj – The object. acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

iter_objects(account, container, marker='', end_marker='', acceptable_statuses=(2, 404))¶

Returns an iterator of object dicts from a container.

Parameters:	account – The container’s account. container – Container to iterate objects on. marker – Prefix of first desired item, defaults to ‘’. end_marker – Last item returned will be ‘less’ than this, defaults to ‘’. acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

make_path(account, container=None, obj=None)¶

Returns a swift path for a request quoting and utf-8 encoding the path parts as need be.

Raises ValueError:
Parameters:	account – swift account container – container, defaults to None obj – object, defaults to None
	Is raised if obj is specified and container is not.

make_request(method, path, headers, acceptable_statuses, body_file=None)¶

Makes a request to Swift with retries.

Parameters:	method – HTTP method of request. path – Path of request. headers – Headers to be sent with request. acceptable_statuses – List of acceptable statuses for request. body_file – Body file to be passed along with request, defaults to None.

:returns : Response object on success.

Raises:	UnexpectedResponse – Exception raised when make_request() fails to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

set_account_metadata(account, metadata, metadata_prefix='', acceptable_statuses=(2, ))¶

Sets account metadata. A call to this will add to the account metadata and not overwrite all of it with values in the metadata dict. To clear an account metadata value, pass an empty string as the value for the key in the metadata dict.

Parameters:	account – Account on which to get the metadata. metadata – Dict of metadata to set. metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’. acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

set_container_metadata(account, container, metadata, metadata_prefix='', acceptable_statuses=(2, ))¶

Sets container metadata. A call to this will add to the container metadata and not overwrite all of it with values in the metadata dict. To clear a container metadata value, pass an empty string as the value for the key in the metadata dict.

Parameters:	account – The container’s account. container – Container to set metadata on. metadata – Dict of metadata to set. metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’. acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

set_object_metadata(account, container, obj, metadata, metadata_prefix='', acceptable_statuses=(2, ))¶

Sets an object’s metadata. The object’s metadata will be overwritten by the values in the metadata dict.

Parameters:	account – The object’s account. container – The object’s container. obj – The object. metadata – Dict of metadata to set. metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’. acceptable_statuses – List of status for valid responses, defaults to (2,).
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

upload_object(fobj, account, container, obj, headers=None)¶

Parameters:	fobj – File object to read object’s content from. account – The object’s account. container – The object’s container. obj – The object. headers – Headers to send with request, defaults ot empty dict.
Raises:	UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status Exception – Exception is raised when code fails in an unexpected way.

class swift.common.internal_client.SimpleClient(url=None, token=None, starting_backoff=1, max_backoff=5, retries=5)¶

Bases: object

Simple client that is used in bin/swift-dispersion-* and container sync

base_request(method, container=None, name=None, prefix=None, headers={}, proxy=None, contents=None, full_listing=None)¶

get_account(*args, **kwargs)¶

get_container(container, **kwargs)¶

put_container(container, **kwargs)¶

put_object(container, name, contents, **kwargs)¶

retry_request(method, **kwargs)¶

exception swift.common.internal_client.UnexpectedResponse(message, resp)¶

Bases: exceptions.Exception

Exception raised on invalid responses to InternalClient.make_request().

Parameters:	message – Exception message. resp – The unexpected response.

swift.common.internal_client.delete_object(url, **kwargs)¶: For usage with container sync

swift.common.internal_client.get_auth(url, user, key, auth_version='1.0', **kwargs)¶

swift.common.internal_client.put_object(url, **kwargs)¶: For usage with container sync

Manager¶

class swift.common.manager.Manager(servers, run_dir='/var/run/swift')¶

Bases: object

Main class for performing commands on groups of servers.

Parameters:	servers – list of server names as strings

force_reload(*a, **kw)¶: alias for reload

get_command(cmd)¶

Find and return the decorated method named like cmd

Parameters:	cmd – the command to get, a string, if not found raises UnknownCommandError

kill(*a, **kw)¶: stop a server (no error if not running)

classmethod list_commands()¶

Get all publicly accessible commands

Returns:	a list of string tuples (cmd, help), the method names who are decorated as commands

no_daemon(*a, **kw)¶: start a server interactively

no_wait(*a, **kw)¶: spawn server and return immediately

once(*a, **kw)¶: start server and run one pass on supporting daemons

reload(*a, **kw)¶: graceful shutdown then restart on supporting servers

restart(*a, **kw)¶: stops then restarts server

run_command(cmd, **kwargs)¶

Find the named command and run it

Parameters:	cmd – the command name to run

shutdown(*a, **kw)¶: allow current requests to finish on supporting servers

start(*a, **kw)¶: starts a server

status(*a, **kw)¶: display status of tracked pids for server

stop(*a, **kw)¶: stops a server

class swift.common.manager.Server(server, run_dir='/var/run/swift')¶

Bases: object

Manage operations on a server or group of servers of similar type

Parameters:	server – name of server

conf_files(**kwargs)¶

Get conf files for this server

Param :	number, if supplied will only lookup the nth server
Returns:	list of conf files

get_conf_file_name(pid_file)¶

Translate pid_file to a corresponding conf_file

Parameters:	pid_file – a pid_file for this server, a string
Returns:	the conf_file for this pid_file

get_pid_file_name(conf_file)¶

Translate conf_file to a corresponding pid_file

Parameters:	conf_file – an conf_file for this server, a string
Returns:	the pid_file for this conf_file

get_running_pids(**kwargs)¶

Get running pids

Returns:	a dict mapping pids (ints) to pid_files (paths)

interact(**kwargs)¶: wait on spawned procs to terminate

iter_pid_files(**kwargs)¶: Generator, yields (pid_file, pids)

kill_running_pids(**kwargs)¶

Kill running pids

Parameters:	graceful – if True, attempt SIGHUP on supporting servers
Returns:	a dict mapping pids (ints) to pid_files (paths)

launch(**kwargs)¶: Collect conf files and attempt to spawn the processes for this server

pid_files(**kwargs)¶

Get pid files for this server

Param :	number, if supplied will only lookup the nth server
Returns:	list of pid files

signal_pids(sig, **kwargs)¶

Send a signal to pids for this server

Parameters:	sig – signal to send
Returns:	a dict mapping pids (ints) to pid_files (paths)

spawn(conf_file, once=False, wait=True, daemon=True, **kwargs)¶

Launch a subprocess for this server.

Parameters:	conf_file – path to conf_file to use as first arg once – boolean, add once argument to command wait – boolean, if true capture stdout with a pipe daemon – boolean, if false ask server to log to console

:returns : the pid of the spawned process

status(pids=None, **kwargs)¶

Display status of server

Param :	pids, if not supplied pids will be populated automatically
Param :	number, if supplied will only lookup the nth server
Returns:	1 if server is not running, 0 otherwise

stop(**kwargs)¶

Send stop signals to pids for this server

Returns:	a dict mapping pids (ints) to pid_files (paths)

wait(**kwargs)¶: wait on spawned procs to start

swift.common.manager.command(func)¶

Decorator to declare which methods are accessible as commands, commands always return 1 or 0, where 0 should indicate success.

Parameters:	func – function to make public

swift.common.manager.setup_env()¶: Try to increase resource limits of the OS. Move PYTHON_EGG_CACHE to /tmp

swift.common.manager.watch_server_pids(server_pids, interval=1, **kwargs)¶

Monitor a collection of server pids yielding back those pids that aren’t responding to signals.

Parameters:	server_pids – a dict, lists of pids [int,...] keyed on Server objects

MemCacheD¶

Why our own memcache client? By Michael Barton

python-memcached doesn’t use consistent hashing, so adding or removing a memcache server from the pool invalidates a huge percentage of cached items.

If you keep a pool of python-memcached client objects, each client object has its own connection to every memcached server, only one of which is ever in use. So you wind up with n * m open sockets and almost all of them idle. This client effectively has a pool for each server, so the number of backend connections is hopefully greatly reduced.

python-memcache uses pickle to store things, and there was already a huge stink about Swift using pickles in memcache (http://osvdb.org/show/osvdb/86581). That seemed sort of unfair, since nova and keystone and everyone else use pickles for memcache too, but it’s hidden behind a “standard” library. But changing would be a security regression at this point.

Also, pylibmc wouldn’t work for us because it needs to use python sockets in order to play nice with eventlet.

Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:

http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt

class swift.common.memcached.MemcacheConnPool(server, size, connect_timeout)¶

Bases: eventlet.pools.Pool

Connection pool for Memcache Connections

class swift.common.memcached.MemcacheRing(servers, connect_timeout=0.3, io_timeout=2.0, pool_timeout=1.0, tries=3, allow_pickle=False, allow_unpickle=False, max_conns=2)¶

Bases: object

Simple, consistent-hashed memcache client.

decr(key, delta=1, time=0, timeout=0)¶

Decrements a key which has a numeric value by delta. Calls incr with -delta.

Raises MemcacheConnectionError:
Parameters:	key – key delta – amount to subtract to the value of key (or set the value to 0 if the key is not found) will be cast to an int time – the time to live. This parameter depcates parameter timeout. The addition of this parameter is to make the interface consistent with set and set_multi methods timeout – ttl in memcache, deprecated, will be removed in future OpenStack releases
Returns:	result of decrementing

delete(key)¶

Deletes a key/value pair from memcache.

Parameters:	key – key to be deleted

get(key)¶

Gets the object specified by key. It will also unserialize the object before returning if it is serialized in memcache with JSON, or if it is pickled and unpickling is allowed.

Parameters:	key – key
Returns:	value of the key in memcache

get_multi(keys, server_key)¶

Gets multiple values from memcache for the given keys.

Parameters:	keys – keys for values to be retrieved from memcache servery_key – key to use in determining which server in the ring is used
Returns:	list of values

incr(key, delta=1, time=0, timeout=0)¶

Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta or 0 if delta < 0. If passed a negative number, will use memcached’s decr. Returns the int stored in memcached Note: The data memcached stores as the result of incr/decr is an unsigned int. decr’s that result in a number below 0 are stored as 0.

Raises MemcacheConnectionError:
Parameters:	key – key delta – amount to add to the value of key (or set as the value if the key is not found) will be cast to an int time – the time to live. This parameter deprecates parameter timeout. The addition of this parameter is to make the interface consistent with set and set_multi methods timeout – ttl in memcache, deprecated, will be removed in future OpenStack releases
Returns:	result of incrementing

set(key, value, serialize=True, timeout=0, time=0, min_compress_len=0)¶

Set a key/value pair in memcache

Min_compress_len :
Parameters:	key – key value – value serialize – if True, value is serialized with JSON before sending to memcache, or with pickle if configured to use pickle instead of JSON (to avoid cache poisoning) timeout – ttl in memcache, this parameter is now deprecated. It will be removed in next release of OpenStack, use time parameter instead in the future
Time :	equivalent to timeout, this parameter is added to keep the signature compatible with python-memcached interface. This implementation will take this value and sign it to the parameter timeout
	minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it.

set_multi(mapping, server_key, serialize=True, timeout=0, time=0, min_compress_len=0)¶

Sets multiple key/value pairs in memcache.

Min_compress_len :
Parameters:	mapping – dictonary of keys and values to be set in memcache servery_key – key to use in determining which server in the ring is used serialize – if True, value is serialized with JSON before sending to memcache, or with pickle if configured to use pickle instead of JSON (to avoid cache poisoning) timeout – ttl for memcache. This parameter is now deprecated, it will be removed in next release of OpenStack, use time parameter instead in the future
Time :	equalvent to timeout, this parameter is added to keep the signature compatible with python-memcached interface. This implementation will take this value and sign it to parameter timeout
	minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it

swift.common.memcached.sanitize_timeout(timeout)¶: Sanitize a timeout value to use an absolute expiration time if the delta is greater than 30 days (in seconds). Note that the memcached server translates negative values to mean a delta of 30 days in seconds (and 1 additional second), client beware.

Request Helpers¶

Miscellaneous utility functions for use in generating responses.

Why not swift.common.utils, you ask? Because this way we can import things from swob in here without creating circular imports.

class swift.common.request_helpers.SegmentedIterable(req, app, listing_iter, max_get_time, logger, ua_suffix, swift_source, name='<not specified>', response=None)¶

Bases: object

Iterable that returns the object contents for a large object.

Parameters:

req – original request object
app – WSGI application from which segments will come
listing_iter – iterable yielding the object segments to fetch, along with the byte subranges to fetch, in the form of a tuple (object-path, first-byte, last-byte) or (object-path, None, None) to fetch the whole thing.
max_get_time – maximum permitted duration of a GET request (seconds)
logger – logger object
swift_source – value of swift.source in subrequest environ (just for logging)
ua_suffix – string to append to user-agent.
name – name of manifest (used in logging only)
response – optional response object for the response being sent to the client.

app_iter_range(*a, **kw)¶

swob.Response will only respond with a 206 status in certain cases; one of those is if the body iterator responds to .app_iter_range().

However, this object (or really, its listing iter) is smart enough to handle the range stuff internally, so we just no-op this out for swob.

swift.common.request_helpers.close_if_possible(maybe_closable)¶

swift.common.request_helpers.closing_if_possible(*args, **kwds)¶

Like contextlib.closing(), but doesn’t crash if the object lacks a close() method.

PEP 333 (WSGI) says: “If the iterable returned by the application has a close() method, the server or gateway must call that method upon completion of the current request[.]” This function makes that easier.

swift.common.request_helpers.get_listing_content_type(req)¶

Determine the content type to use for an account or container listing response.

Parameters:	req – request object
Returns:	content type as a string (e.g. text/plain, application/json)
Raises :	HTTPNotAcceptable if the requested content type is not acceptable
Raises :	HTTPBadRequest if the ‘format’ query param is provided and not valid UTF-8

swift.common.request_helpers.get_param(req, name, default=None)¶

Get parameters from an HTTP request ensuring proper handling UTF-8 encoding.

Parameters:	req – request object name – parameter name default – result to return if the parameter is not found
Returns:	HTTP request parameter value (as UTF-8 encoded str, not unicode object)
Raises :	HTTPBadRequest if param not valid UTF-8 byte sequence

swift.common.request_helpers.get_sys_meta_prefix(server_type)¶

Returns the prefix for system metadata headers for given server type.

This prefix defines the namespace for headers that will be persisted by backend servers.

Parameters:	server_type – type of backend server i.e. [account\|container\|object]
Returns:	prefix string for server type’s system metadata headers

swift.common.request_helpers.get_user_meta_prefix(server_type)¶

Returns the prefix for user metadata headers for given server type.

This prefix defines the namespace for headers that will be persisted by backend servers.

Parameters:	server_type – type of backend server i.e. [account\|container\|object]
Returns:	prefix string for server type’s user metadata headers

swift.common.request_helpers.is_sys_meta(server_type, key)¶

Tests if a header key starts with and is longer than the system metadata prefix for given server type.

Parameters:	server_type – type of backend server i.e. [account\|container\|object] key – header key
Returns:	True if the key satisfies the test, False otherwise

swift.common.request_helpers.is_sys_or_user_meta(server_type, key)¶

Tests if a header key starts with and is longer than the user or system metadata prefix for given server type.

Parameters:	server_type – type of backend server i.e. [account\|container\|object] key – header key
Returns:	True if the key satisfies the test, False otherwise

swift.common.request_helpers.is_user_meta(server_type, key)¶

Tests if a header key starts with and is longer than the user metadata prefix for given server type.

Parameters:	server_type – type of backend server i.e. [account\|container\|object] key – header key
Returns:	True if the key satisfies the test, False otherwise

swift.common.request_helpers.remove_items(headers, condition)¶

Removes items from a dict whose keys satisfy the given condition.

Parameters:	headers – a dict of headers condition – a function that will be passed the header key as a single argument and should return True if the header is to be removed.
Returns:	a dict, possibly empty, of headers that have been removed

swift.common.request_helpers.split_and_validate_path(request, minsegs=1, maxsegs=None, rest_with_last=False)¶

Utility function to split and validate the request path.

Returns:	result of split_path if everything’s okay
Raises :	HTTPBadRequest if something’s not okay

swift.common.request_helpers.strip_sys_meta_prefix(server_type, key)¶

Removes the system metadata prefix for a given server type from the start of a header key.

Parameters:	server_type – type of backend server i.e. [account\|container\|object] key – header key
Returns:	stripped header key

swift.common.request_helpers.strip_user_meta_prefix(server_type, key)¶

Removes the user metadata prefix for a given server type from the start of a header key.

Parameters:	server_type – type of backend server i.e. [account\|container\|object] key – header key
Returns:	stripped header key

Swob¶

Implementation of WSGI Request and Response objects.

This library has a very similar API to Webob. It wraps WSGI request environments and response values into objects that are more friendly to interact with.

Why Swob and not just use WebOb? By Michael Barton

We used webob for years. The main problem was that the interface wasn’t stable. For a while, each of our several test suites required a slightly different version of webob to run, and none of them worked with the then-current version. It was a huge headache, so we just scrapped it.

This is kind of a ton of code, but it’s also been a huge relief to not have to scramble to add a bunch of code branches all over the place to keep Swift working every time webob decides some interface needs to change.

class swift.common.swob.Accept(headerval)¶

Bases: object

Wraps a Request’s Accept header as a friendly object.

Parameters:	headerval – value of the header as a str

__weakref__¶: list of weak references to the object (if defined)

best_match(options)¶

Returns the item from “options” that best matches the accept header. Returns None if no available options are acceptable to the client.

Parameters:	options – a list of content-types the server can respond with

class swift.common.swob.HeaderEnvironProxy(environ)¶

Bases: UserDict.DictMixin

A dict-like object that proxies requests to a wsgi environ, rewriting header keys to environ keys.

For example, headers[‘Content-Range’] sets and gets the value of headers.environ[‘HTTP_CONTENT_RANGE’]

class swift.common.swob.HeaderKeyDict(base_headers=None, **kwargs)¶

Bases: dict

A dict that title-cases all keys on the way in, so as to be case-insensitive.

__weakref__¶: list of weak references to the object (if defined)

class swift.common.swob.Match(headerval)¶

Bases: object

Wraps a Request’s If-[None-]Match header as a friendly object.

Parameters:	headerval – value of the header as a str

__weakref__¶: list of weak references to the object (if defined)

class swift.common.swob.Range(headerval)¶

Bases: object

Wraps a Request’s Range header as a friendly object. After initialization, “range.ranges” is populated with a list of (start, end) tuples denoting the requested ranges.

If there were any syntactically-invalid byte-range-spec values, “range.ranges” will be an empty list, per the relevant RFC:

“The recipient of a byte-range-set that includes one or more syntactically invalid byte-range-spec values MUST ignore the header field that includes that byte-range-set.”

According to the RFC 2616 specification, the following cases will be all considered as syntactically invalid, thus, a ValueError is thrown so that the range header will be ignored. If the range value contains at least one of the following cases, the entire range is considered invalid, ValueError will be thrown so that the header will be ignored.

value not starts with bytes=
range value start is greater than the end, eg. bytes=5-3
range does not have start or end, eg. bytes=-
range does not have hyphen, eg. bytes=45
range value is non numeric
any combination of the above

Every syntactically valid range will be added into the ranges list even when some of the ranges may not be satisfied by underlying content.

Parameters:	headerval – value of the header as a str

__weakref__¶: list of weak references to the object (if defined)

ranges_for_length(length)¶

This method is used to return multiple ranges for a given length which should represent the length of the underlying content. The constructor method __init__ made sure that any range in ranges list is syntactically valid. So if length is None or size of the ranges is zero, then the Range header should be ignored which will eventually make the response to be 200.

If an empty list is returned by this method, it indicates that there are unsatisfiable ranges found in the Range header, 416 will be returned.

if a returned list has at least one element, the list indicates that there is at least one range valid and the server should serve the request with a 206 status code.

The start value of each range represents the starting position in the content, the end value represents the ending position. This method purposely adds 1 to the end number because the spec defines the Range to be inclusive.

The Range spec can be found at the following link: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1

Parameters:	length – length of the underlying content

class swift.common.swob.Request(environ)¶

Bases: object

WSGI Request object.

__weakref__¶: list of weak references to the object (if defined)

accept¶: Retrieve and set the accept property in the WSGI environ, as a Accept object

acl¶: Get and set the swob.ACL property in the WSGI environment

classmethod blank(path, environ=None, headers=None, body=None, **kwargs)¶

Create a new request object with the given parameters, and an environment otherwise filled in with non-surprising default values.

Parameters:	path – encoded, parsed, and unquoted into PATH_INFO environ – WSGI environ dictionary headers – HTTP headers body – stuffed in a StringIO and hung on wsgi.input kwargs – any environ key with an property setter

body¶: Get and set the request body str

body_file¶: Get and set the wsgi.input property in the WSGI environment

call_application(application)¶

Calls the application with this request’s environment. Returns the status, headers, and app_iter for the response as a tuple.

Parameters:	application – the WSGI application to call

content_length¶: Retrieve and set the content-length header as an int

copy_get()¶: Makes a copy of the request, converting it to a GET.

get_response(application)¶

Calls the application with this request’s environment. Returns a Response object that wraps up the application’s result.

Parameters:	application – the WSGI application to call

host¶: Get and set the HTTP_HOST property in the WSGI environment

host_url¶: Get url for request/response up to path

if_match¶: Retrieve and set the if-match property in the WSGI environ, as a Match object

if_modified_since¶: Retrieve and set the if-modified-since header as a datetime, set it with a datetime, int, or str

if_none_match¶: Retrieve and set the if-none-match property in the WSGI environ, as a Match object

if_unmodified_since¶: Retrieve and set the if-unmodified-since header as a datetime, set it with a datetime, int, or str

message_length()¶

Properly determine the message length for this request. It will return an integer if the headers explicitly contain the message length, or None if the headers don’t contain a length. The ValueError exception will be raised if the headers are invalid.

Raises:	ValueError – if either transfer-encoding or content-length headers have bad values AttributeError – if the last value of the transfer-encoding header is not “chunked”

method¶: Get and set the REQUEST_METHOD property in the WSGI environment

params¶: Provides QUERY_STRING parameters as a dictionary

path¶: Provides the full path of the request, excluding the QUERY_STRING

path_info¶: Get and set the PATH_INFO property in the WSGI environment

path_info_pop()¶: Takes one path portion (delineated by slashes) from the path_info, and appends it to the script_name. Returns the path segment.

path_qs¶: The path of the request, without host but with query string.

query_string¶: Get and set the QUERY_STRING property in the WSGI environment

range¶: Retrieve and set the range property in the WSGI environ, as a Range object

referer¶: Get and set the HTTP_REFERER property in the WSGI environment

referrer¶: Get and set the HTTP_REFERER property in the WSGI environment

remote_addr¶: Get and set the REMOTE_ADDR property in the WSGI environment

remote_user¶: Get and set the REMOTE_USER property in the WSGI environment

script_name¶: Get and set the SCRIPT_NAME property in the WSGI environment

split_path(minsegs=1, maxsegs=None, rest_with_last=False)¶

Validate and split the Request’s path.

Examples:

['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)

Parameters:	minsegs – Minimum number of segments to be extracted maxsegs – Maximum number of segments to be extracted rest_with_last – If True, trailing data will be returned as part of last segment. If False, and there is trailing data, raises ValueError.
Returns:	list of segments with a length of maxsegs (non-existent segments will return as None)
Raises :	ValueError if given an invalid path

str_params¶: Provides QUERY_STRING parameters as a dictionary

swift_entity_path¶

Provides the account/container/object path, sans API version.

This can be useful when constructing a path to send to a backend server, as that path will need everything after the “/v1”.

url¶: Provides the full url of the request

user_agent¶: Get and set the HTTP_USER_AGENT property in the WSGI environment

class swift.common.swob.Response(body=None, status=200, headers=None, app_iter=None, request=None, conditional_response=False, **kw)¶

Bases: object

WSGI Response object.

__call__(env, start_response)¶: Respond to the WSGI request.

Warning

This will translate any relative Location header value to an absolute URL using the WSGI environment’s HOST_URL as a prefix, as RFC 2616 specifies.

However, it is quite common to use relative redirects, especially when it is difficult to know the exact HOST_URL the browser would have used when behind several CNAMEs, CDN services, etc. All modern browsers support relative redirects.

To skip over RFC enforcement of the Location header value, you may set env['swift.leave_relative_location'] = True in the WSGI environment.

__weakref__¶: list of weak references to the object (if defined)

absolute_location()¶: Attempt to construct an absolute location.

accept_ranges¶: Retrieve and set the accept-ranges header

app_iter¶: Retrieve and set the response app_iter

body¶: Retrieve and set the Response body str

charset¶: Retrieve and set the response charset

content_length¶: Retrieve and set the content-length header as an int

content_range¶: Retrieve and set the content-range header

content_type¶: Retrieve and set the response Content-Type header

etag¶: Retrieve and set the response Etag header

host_url¶: Get url for request/response up to path

last_modified¶: Retrieve and set the last-modified header as a datetime, set it with a datetime, int, or str

location¶: Retrieve and set the location header

status¶: Retrieve and set the Response status, e.g. ‘200 OK’

www_authenticate()¶

Construct a suitable value for WWW-Authenticate response header

If we have a request and a valid-looking path, the realm is the account; otherwise we set it to ‘unknown’.

class swift.common.swob.StatusMap¶

Bases: object

A dict-like object that returns HTTPException subclasses/factory functions where the given key is the status code.

__weakref__¶: list of weak references to the object (if defined)

swift.common.swob.wsgify(func)¶: A decorator for translating functions which take a swob Request object and return a Response object into WSGI callables. Also catches any raised HTTPExceptions and treats them as a returned Response.

Utils¶

Miscellaneous utility functions for use with Swift.

class swift.common.utils.CloseableChain(*iterables)¶

Bases: object

Like itertools.chain, but with a close method that will attempt to invoke its sub-iterators’ close methods, if any.

class swift.common.utils.ContextPool(size=1000)¶

Bases: eventlet.greenpool.GreenPool

GreenPool subclassed to kill its coros when it gets gc’ed

class swift.common.utils.GreenAsyncPile(size)¶

Bases: object

Runs jobs in a pool of green threads, and the results can be retrieved by using this object as an iterator.

This is very similar in principle to eventlet.GreenPile, except it returns results as they become available rather than in the order they were launched.

Correlating results with jobs (if necessary) is left to the caller.

spawn(func, *args, **kwargs)¶: Spawn a job in a green thread on the pile.

waitall(timeout)¶

Wait timeout seconds for any results to come in.

Parameters:	timeout – seconds to wait for results
Returns:	list of results accrued in that time

class swift.common.utils.GreenthreadSafeIterator(unsafe_iterable)¶

Bases: object

Wrap an iterator to ensure that only one greenthread is inside its next() method at a time.

This is useful if an iterator’s next() method may perform network IO, as that may trigger a greenthread context switch (aka trampoline), which can give another greenthread a chance to call next(). At that point, you get an error like “ValueError: generator already executing”. By wrapping calls to next() with a mutex, we avoid that error.

class swift.common.utils.InputProxy(wsgi_input)¶

Bases: object

File-like object that counts bytes read. To be swapped in for wsgi.input for accounting purposes.

read(*args, **kwargs)¶: Pass read request to the underlying file-like object and add bytes read to total.

readline(*args, **kwargs)¶: Pass readline request to the underlying file-like object and add bytes read to total.

class swift.common.utils.LogAdapter(logger, server)¶

Bases: logging.LoggerAdapter, object

A Logger like object which performs some reformatting on calls to exception(). Can be used to store a threadlocal transaction id and client ip.

notice(msg, *args, **kwargs)¶: Convenience function for syslog priority LOG_NOTICE. The python logging lvl is set to 25, just above info. SysLogHandler is monkey patched to map this log lvl to the LOG_NOTICE syslog priority.

process(msg, kwargs)¶: Add extra info to message

set_statsd_prefix(prefix)¶: The StatsD client prefix defaults to the “name” of the logger. This method may override that default with a specific value. Currently used in the proxy-server to differentiate the Account, Container, and Object controllers.

statsd_delegate(statsd_func_name)¶

Factory to create methods which delegate to methods on self.logger.statsd_client (an instance of StatsdClient). The created methods conditionally delegate to a method whose name is given in ‘statsd_func_name’. The created delegate methods are a no-op when StatsD logging is not configured.

Parameters:	statsd_func_name – the name of a method on StatsdClient.

class swift.common.utils.LoggingHandlerWeakRef¶

Bases: weakref

Like a weak reference, but passes through a couple methods that logging handlers need.

class swift.common.utils.ModifiedParseResult¶

Bases: urlparse.ParseResult

Parse results class for urlparse.

class swift.common.utils.NullLogger¶

Bases: object

A no-op logger for eventlet wsgi.

class swift.common.utils.RateLimitedIterator(iterable, elements_per_second, limit_after=0)¶

Bases: object

Wrap an iterator to only yield elements at a rate of N per second.

Parameters:	iterable – iterable to wrap elements_per_second – the rate at which to yield elements limit_after – rate limiting kicks in only after yielding this many elements; default is 0 (rate limit immediately)

class swift.common.utils.SwiftLogFormatter(fmt=None, datefmt=None)¶

Bases: logging.Formatter

Custom logging.Formatter will append txn_id to a log message if the record has one and the message does not.

swift.common.utils.affinity_key_function(affinity_str)¶

Turns an affinity config value into a function suitable for passing to sort(). After doing so, the array will be sorted with respect to the given ordering.

For example, if affinity_str is “r1=1, r2z7=2, r2z8=2”, then the array will be sorted with all nodes from region 1 (r1=1) first, then all the nodes from region 2 zones 7 and 8 (r2z7=2 and r2z8=2), then everything else.

Note that the order of the pieces of affinity_str is irrelevant; the priority values are what comes after the equals sign.

If affinity_str is empty or all whitespace, then the resulting function will not alter the ordering of the nodes.

Parameters:	affinity_str – affinity config value, e.g. “r1z2=3” or “r1=1, r2z1=2, r2z2=2”
Returns:	single-argument function
Raises :	ValueError if argument invalid

swift.common.utils.affinity_locality_predicate(write_affinity_str)¶

Turns a write-affinity config value into a predicate function for nodes. The returned value will be a 1-arg function that takes a node dictionary and returns a true value if it is “local” and a false value otherwise. The definition of “local” comes from the affinity_str argument passed in here.

For example, if affinity_str is “r1, r2z2”, then only nodes where region=1 or where (region=2 and zone=2) are considered local.

If affinity_str is empty or all whitespace, then the resulting function will consider everything local

Parameters:	affinity_str – affinity config value, e.g. “r1z2” or “r1, r2z1, r2z2”
Returns:	single-argument function, or None if affinity_str is empty
Raises :	ValueError if argument invalid

swift.common.utils.audit_location_generator(devices, datadir, suffix='', mount_check=True, logger=None)¶

Given a devices path and a data directory, yield (path, device, partition) for all files in that directory

Parameters:	devices – parent directory of the devices to be audited datadir – a directory located under self.devices. This should be one of the DATADIR constants defined in the account, container, and object servers. suffix – path name suffix required for all names returned mount_check – Flag to check if a mount check should be performed on devices logger – a logger object

swift.common.utils.backward(f, blocksize=4096)¶

A generator returning lines from a file starting with the last line, then the second last line, etc. i.e., it reads lines backwards. Stops when the first line (if any) is read. This is useful when searching for recent activity in very large files.

Parameters:	f – file object to read blocksize – no of characters to go backwards at each block

swift.common.utils.cache_from_env(env, allow_none=False)¶

Get memcache connection pool from the environment (which had been previously set by the memcache middleware

Parameters:	env – wsgi environment dict
Returns:	swift.common.memcached.MemcacheRing from environment

swift.common.utils.capture_stdio(logger, **kwargs)¶

Log unhandled exceptions, close stdio, capture stdout and stderr.

param logger: Logger object to use

swift.common.utils.compute_eta(start_time, current_value, final_value)¶

Compute an ETA. Now only if we could also have a progress bar...

Parameters:	start_time – Unix timestamp when the operation began current_value – Current value final_value – Final value
Returns:	ETA as a tuple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)

swift.common.utils.config_auto_int_value(value, default)¶: Returns default if value is None or ‘auto’. Returns value as an int or raises ValueError otherwise.

swift.common.utils.config_true_value(value)¶: Returns True if the value is either True or a string in TRUE_VALUES. Returns False otherwise.

swift.common.utils.csv_append(csv_string, item)¶

Appends an item to a comma-separated string.

If the comma-separated string is empty/None, just returns item.

swift.common.utils.drop_buffer_cache(fd, offset, length)¶

Drop ‘buffer’ cache for the given range of the given file.

Parameters:	fd – file descriptor offset – start offset length – length

swift.common.utils.drop_privileges(user)¶

Sets the userid/groupid of the current process, get session leader, etc.

Parameters:	user – User name to change privileges to

swift.common.utils.dump_recon_cache(cache_dict, cache_file, logger, lock_timeout=2)¶

Update recon cache values

Parameters:	cache_dict – Dictionary of cache key/value pairs to write out cache_file – cache file to update logger – the logger to use to log an encountered error lock_timeout – timeout (in seconds)

swift.common.utils.fallocate(fd, size)¶

Pre-allocate disk space for a file.

Parameters:	fd – file descriptor size – size to allocate (in bytes)

swift.common.utils.fdatasync(fd)¶

Sync modified file data to disk.

Parameters:	fd – file descriptor

swift.common.utils.fsync(fd)¶

Sync modified file data and metadata to disk.

Parameters:	fd – file descriptor

swift.common.utils.get_hmac(request_method, path, expires, key)¶

Returns the hexdigest string of the HMAC-SHA1 (RFC 2104) for the request.

Parameters:	request_method – Request method to allow. path – The path to the resource to allow access to. expires – Unix timestamp as an int for when the URL expires. key – HMAC shared secret.
Returns:	hexdigest str of the HMAC-SHA1 for the request.

swift.common.utils.get_hub()¶

Checks whether poll is available and falls back on select if it isn’t.

Note about epoll:

Review: https://review.openstack.org/#/c/18806/

There was a problem where once out of every 30 quadrillion connections, a coroutine wouldn’t wake up when the client closed its end. Epoll was not reporting the event or it was getting swallowed somewhere. Then when that file descriptor was re-used, eventlet would freak right out because it still thought it was waiting for activity from it in some other coro.

swift.common.utils.get_log_line(req, res, trans_time, additional_info)¶

Make a line for logging that matches the documented log line format for backend servers.

Parameters:	req – the request. res – the response. trans_time – the time the request took to complete, a float. additional_info – a string to log at the end of the line
Returns:	a properly formated line for logging.

swift.common.utils.get_logger(conf, name=None, log_to_console=False, log_route=None, fmt='%(server)s: %(message)s')¶

Get the current system logger using config settings.

Log config and defaults:

log_facility = LOG_LOCAL0
log_level = INFO
log_name = swift
log_udp_host = (disabled)
log_udp_port = logging.handlers.SYSLOG_UDP_PORT
log_address = /dev/log
log_statsd_host = (disabled)
log_statsd_port = 8125
log_statsd_default_sample_rate = 1.0
log_statsd_sample_rate_factor = 1.0
log_statsd_metric_prefix = (empty-string)

Parameters:	conf – Configuration dict to read settings from name – Name of the logger log_to_console – Add handler which writes to console on stderr log_route – Route for the logging, not emitted to the log, just used to separate logging configurations fmt – Override log format

swift.common.utils.get_swift_info(admin=False, disallowed_sections=None)¶

Returns information about the swift cluster that has been previously registered with the register_swift_info call.

Parameters:	admin – boolean value, if True will additionally return an ‘admin’ section with information previously registered as admin info. disallowed_sections – list of section names to be withheld from the information returned.
Returns:	dictionary of information about the swift cluster.

swift.common.utils.get_time_units(time_amount)¶

Get a nomralized length of time in the largest unit of time (hours, minutes, or seconds.)

Parameters:	time_amount – length of time in seconds
Returns:	A touple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)

swift.common.utils.get_valid_utf8_str(str_or_unicode)¶

Get valid parts of utf-8 str from str, unicode and even invalid utf-8 str

Parameters:	str_or_unicode – a string or an unicode which can be invalid utf-8

swift.common.utils.hash_path(account, container=None, object=None, raw_digest=False)¶

Get the canonical hash for an account/container/object

Parameters:	account – Account container – Container object – Object raw_digest – If True, return the raw version rather than a hex digest
Returns:	hash string

swift.common.utils.human_readable(value)¶: Returns the number in a human readable format; for example 1048576 = “1Mi”.

swift.common.utils.ismount(path)¶: Test whether a path is a mount point. This will catch any exceptions and translate them into a False return value Use ismount_raw to have the exceptions raised instead.

swift.common.utils.ismount_raw(path)¶

Test whether a path is a mount point. Whereas ismount will catch any exceptions and just return False, this raw version will not catch exceptions.

This is code hijacked from C Python 2.6.8, adapted to remove the extra lstat() system call.

swift.common.utils.item_from_env(env, item_name, allow_none=False)¶

Get a value from the wsgi environment

Parameters:	env – wsgi environment dict item_name – name of item to get
Returns:	the value from the environment

swift.common.utils.list_from_csv(comma_separated_str)¶: Splits the str given and returns a properly stripped list of the comma separated values.

swift.common.utils.load_libc_function(func_name, log_error=True)¶

Attempt to find the function in libc, otherwise return a no-op func.

Parameters:	func_name – name of the function to pull from libc.

swift.common.utils.lock_file(*args, **kwds)¶

Context manager that acquires a lock on a file. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

Parameters:	filename – file to be locked timeout – timeout (in seconds) append – True if file should be opened in append mode unlink – True if the file should be unlinked at the end

swift.common.utils.lock_parent_directory(filename, timeout=10)¶

Context manager that acquires a lock on the parent directory of the given file path. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

Parameters:	filename – file path of the parent directory to be locked timeout – timeout (in seconds)

swift.common.utils.lock_path(*args, **kwds)¶

Context manager that acquires a lock on a directory. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

For locking exclusively, file or directory has to be opened in Write mode. Python doesn’t allow directories to be opened in Write Mode. So we workaround by locking a hidden file in the directory.

Parameters:	directory – directory to be locked timeout – timeout (in seconds) timeout_class – The class of the exception to raise if the lock cannot be granted within the timeout. Will be constructed as timeout_class(timeout, lockpath). Default: LockTimeout

swift.common.utils.mkdirs(path)¶

Ensures the path is a directory or makes it if not. Errors if the path exists but is a file or on permissions failure.

Parameters:	path – path to create

swift.common.utils.normalize_delete_at_timestamp(timestamp)¶

Format a timestamp (string or numeric) into a standardized xxxxxxxxxx (10) format.

Note that timestamps less than 0000000000 are raised to 0000000000 and values greater than November 20th, 2286 at 17:46:39 UTC will be capped at that date and time, resulting in no return value exceeding 9999999999.

This cap is because the expirer is already working through a sorted list of strings that were all a length of 10. Adding another digit would mess up the sort and cause the expirer to break from processing early. By 2286, this problem will need to be fixed, probably by creating an additional .expiring_objects account to work from with 11 (or more) digit container names.

Parameters:	timestamp – unix timestamp
Returns:	normalized timestamp as a string

swift.common.utils.normalize_timestamp(timestamp)¶

Format a timestamp (string or numeric) into a standardized xxxxxxxxxx.xxxxx (10.5) format.

Note that timestamps using values greater than or equal to November 20th, 2286 at 17:46 UTC will use 11 digits to represent the number of seconds.

Parameters:	timestamp – unix timestamp
Returns:	normalized timestamp as a string

swift.common.utils.override_bytes_from_content_type(listing_dict, logger=None)¶: Takes a dict from a container listing and overrides the content_type, bytes fields if swift_bytes is set.

swift.common.utils.parse_content_type(content_type)¶

Parse a content-type and its parameters into values. RFC 2616 sec 14.17 and 3.7 are pertinent.

Examples:

'text/plain; charset=UTF-8' -> ('text/plain', [('charset, 'UTF-8')])
'text/plain; charset=UTF-8; level=1' ->
    ('text/plain', [('charset, 'UTF-8'), ('level', '1')])

Parameters:	content_type – content_type to parse
Returns:	a typle containing (content type, list of k, v parameter tuples)

swift.common.utils.parse_options(parser=None, once=False, test_args=None)¶

Parse standard swift server/daemon options with optparse.OptionParser.

Parameters:	parser – OptionParser to use. If not sent one will be created. once – Boolean indicating the “once” option is available test_args – Override sys.argv; used in testing

:returns : Tuple of (config, options); config is an absolute path to the: config file, options is the parser options as a dictionary.

Raises SystemExit:
	First arg (CONFIG) is required, file must exist

swift.common.utils.public(func)¶

Decorator to declare which methods are publicly accessible as HTTP requests

Parameters:	func – function to make public

swift.common.utils.put_recon_cache_entry(cache_entry, key, item)¶: Function that will check if item is a dict, and if so put it under cache_entry[key]. We use nested recon cache entries when the object auditor runs in ‘once’ mode with a specified subset of devices.

swift.common.utils.quorum_size(n)¶: Number of successful backend requests needed for the proxy to consider the client request successful.

swift.common.utils.quote(value, safe='/')¶: Patched version of urllib.quote that encodes utf-8 strings before quoting

swift.common.utils.random() → x in the interval [0, 1).¶

swift.common.utils.ratelimit_sleep(running_time, max_rate, incr_by=1, rate_buffer=5)¶

Will eventlet.sleep() for the appropriate time so that the max_rate is never exceeded. If max_rate is 0, will not ratelimit. The maximum recommended rate should not exceed (1000 * incr_by) a second as eventlet.sleep() does involve some overhead. Returns running_time that should be used for subsequent calls.

Parameters:

running_time – the running time in milliseconds of the next allowable request. Best to start at zero.
max_rate – The maximum rate per second allowed for the process.
incr_by – How much to increment the counter. Useful if you want to ratelimit 1024 bytes/sec and have differing sizes of requests. Must be > 0 to engage rate-limiting behavior.
rate_buffer – Number of seconds the rate counter can drop and be allowed to catch up (at a faster than listed rate). A larger number will result in larger spikes in rate but better average accuracy. Must be > 0 to engage rate-limiting behavior.

swift.common.utils.readconf(conf_path, section_name=None, log_name=None, defaults=None, raw=False)¶

Read config file(s) and return config items as a dict

Parameters:	conf_path – path to config file/directory, or a file-like object (hasattr readline) section_name – config section to read (will return all sections if not defined) log_name – name to be used with logging (will use section_name if not defined) defaults – dict of default values to pre-populate the config with
Returns:	dict of config items

swift.common.utils.register_swift_info(name='swift', admin=False, **kwargs)¶

Registers information about the swift cluster to be retrieved with calls to get_swift_info.

Parameters:	name – string, the section name to place the information under. admin – boolean, if True, information will be registered to an admin section which can optionally be withheld when requesting the information. kwargs – key value arguments representing the information to be added.

swift.common.utils.reiterate(iterable)¶

Consume the first item from an iterator, then re-chain it to the rest of the iterator. This is useful when you want to make sure the prologue to downstream generators have been executed before continuing.

Parameters:	iterable – an iterable object

swift.common.utils.remove_file(path)¶

Quiet wrapper for os.unlink, OSErrors are suppressed

Parameters:	path – first and only argument passed to os.unlink

swift.common.utils.renamer(old, new)¶

Attempt to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.

Parameters:	old – old path to be renamed new – new path to be renamed to

swift.common.utils.replication(func)¶

Decorator to declare which methods are accessible for different type of servers:

If option replication_server is None then this decorator doesn’t matter.
If option replication_server is True then ONLY decorated with this decorator methods will be started.
If option replication_server is False then decorated with this decorator methods will NOT be started.

Parameters:	func – function to mark accessible for replication

swift.common.utils.rsync_ip(ip)¶

Transform ip string to an rsync-compatible form

Will return ipv4 addresses unchanged, but will nest ipv6 addresses inside square brackets.

Parameters:	ip – an ip string (ipv4 or ipv6)
Returns:	a string ip address

swift.common.utils.search_tree(root, glob_match, ext='', dir_ext=None)¶

Look in root, for any files/dirs matching glob, recursively traversing any found directories looking for files ending with ext

Parameters:	root – start of search path glob_match – glob to match in root, matching dirs are traversed with os.walk ext – only files that end in ext will be returned dir_ext – if present directories that end with dir_ext will not be traversed and instead will be returned as a matched path
Returns:	list of full paths to matching files, sorted

swift.common.utils.split_path(path, minsegs=1, maxsegs=None, rest_with_last=False)¶

Validate and split the given HTTP request path.

Examples:

['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)

Parameters:	path – HTTP Request path to be split minsegs – Minimum number of segments to be extracted maxsegs – Maximum number of segments to be extracted rest_with_last – If True, trailing data will be returned as part of last segment. If False, and there is trailing data, raises ValueError.
Returns:	list of segments with a length of maxsegs (non-existent segments will return as None)
Raises :	ValueError if given an invalid path

swift.common.utils.storage_directory(datadir, partition, name_hash)¶

Get the storage directory

Parameters:	datadir – Base data directory partition – Partition name_hash – Account, container or object name hash
Returns:	Storage directory

swift.common.utils.streq_const_time(s1, s2)¶

Constant-time string comparison.

Params s1:	the first string
Params s2:	the second string
Returns:	True if the strings are equal.

This function takes two strings and compares them. It is intended to be used when doing a comparison for authentication purposes to help guard against timing attacks.

swift.common.utils.timing_stats(**dec_kwargs)¶: Returns a decorator that logs timing events or errors for public methods in swift’s wsgi server controllers, based on response code.

swift.common.utils.tpool_reraise(func, *args, **kwargs)¶: Hack to work around Eventlet’s tpool not catching and reraising Timeouts.

swift.common.utils.unlink_older_than(path, mtime)¶

Remove any file in a given path that that was last modified before mtime.

Parameters:	path – path to remove file from
Mtime :	timestamp of oldest file to keep

swift.common.utils.urlparse(url)¶

urlparse augmentation. This is necessary because urlparse can’t handle RFC 2732 URLs.

Parameters:	url – URL to parse.

swift.common.utils.validate_device_partition(device, partition)¶

Validate that a device and a partition are valid and won’t lead to directory traversal when used.

Parameters:	device – device to validate partition – partition to validate
Raises :	ValueError if given an invalid device or partition

swift.common.utils.validate_sync_to(value, allowed_sync_hosts, realms_conf)¶

Validates an X-Container-Sync-To header value, returning the validated endpoint, realm, and realm_key, or an error string.

Parameters:	value – The X-Container-Sync-To header value to validate. allowed_sync_hosts – A list of allowed hosts in endpoints, if realms_conf does not apply. realms_conf – A instance of swift.common.container_sync_realms.ContainerSyncRealms to validate against.
Returns:	A tuple of (error_string, validated_endpoint, realm, realm_key). The error_string will None if the rest of the values have been validated. The validated_endpoint will be the validated endpoint to sync to. The realm and realm_key will be set if validation was done through realms_conf.

swift.common.utils.whataremyips()¶

Get the machine’s ip addresses

Returns:	list of Strings of ip addresses

swift.common.utils.write_file(path, contents)¶

Write contents to file at path

Parameters:	path – any path, subdirs will be created as needed contents – data to write to file, will be converted to string

swift.common.utils.write_pickle(obj, dest, tmp=None, pickle_protocol=0)¶

Ensure that a pickle file gets written to disk. The file is first written to a tmp location, ensure it is synced to disk, then perform a move to its final location

Parameters:	obj – python object to be pickled dest – path of final destination file tmp – path to tmp to use, defaults to None pickle_protocol – protocol to pickle the obj with, defaults to 0

WSGI¶

WSGI tools for use with swift.

class swift.common.wsgi.ConfigDirLoader(conf_dir)¶

Bases: swift.common.wsgi.NamedConfigLoader

Read configuration from multiple files under the given path.

class swift.common.wsgi.NamedConfigLoader(filename)¶

Bases: paste.deploy.loadwsgi.ConfigLoader

Patch paste.deploy’s ConfigLoader so each context object will know what config section it came from.

class swift.common.wsgi.PipelineWrapper(context)¶

Bases: object

This class provides a number of utility methods for modifying the composition of a wsgi pipeline.

create_filter(entry_point_name)¶

Creates a context for a filter that can subsequently be added to a pipeline context.

Parameters:	entry_point_name – entry point of the middleware (Swift only)
Returns:	a filter context

index(entry_point_name)¶

Returns the first index of the given entry point name in the pipeline.

Raises ValueError if the given module is not in the pipeline.

insert_filter(ctx, index=0)¶

Inserts a filter module into the pipeline context.

Parameters:	ctx – the context to be inserted index – (optional) index at which filter should be inserted in the list of pipeline filters. Default is 0, which means the start of the pipeline.

startswith(entry_point_name)¶

Tests if the pipeline starts with the given entry point name.

Parameters:	entry_point_name – entry point of middleware or app (Swift only)
Returns:	True if entry_point_name is first in pipeline, False otherwise

class swift.common.wsgi.RestrictedGreenPool(size=1024)¶

Bases: eventlet.greenpool.GreenPool

Works the same as GreenPool, but if the size is specified as one, then the spawn_n() method will invoke waitall() before returning to prevent the caller from doing any other work (like calling accept()).

class swift.common.wsgi.WSGIContext(wsgi_app)¶

Bases: object

This class provides a means to provide context (scope) for a middleware filter to have access to the wsgi start_response results like the request status and headers.

swift.common.wsgi.get_socket(conf, default_port=8080)¶

Bind socket to bind ip:port in conf

Parameters:	conf – Configuration dict to read settings from default_port – port to use if not specified in conf

:returns : a socket object as returned from socket.listen or: ssl.wrap_socket if conf specifies cert_file

swift.common.wsgi.init_request_processor(conf_path, app_section, *args, **kwargs)¶

Loads common settings from conf Sets the logger Loads the request processor

Raises ConfigFileError:
Parameters:	conf_path – Path to paste.deploy style configuration file/directory app_section – App name from conf file to load config from
Returns:	the loaded application entry point
	Exception is raised for config file error

swift.common.wsgi.loadapp(conf_file, global_conf=None, allow_modify_pipeline=True)¶: Loads a context from a config file, and if the context is a pipeline then presents the app with the opportunity to modify the pipeline.

swift.common.wsgi.make_env(env, method=None, path=None, agent='Swift', query_string=None, swift_source=None)¶

Returns a new fresh WSGI environment.

Parameters:

env – The WSGI environment to base the new environment on.
method – The new REQUEST_METHOD or None to use the original.
path – The new path_info or none to use the original. path should NOT be quoted. When building a url, a Webob Request (in accordance with wsgi spec) will quote env[‘PATH_INFO’]. url += quote(environ[‘PATH_INFO’])
query_string – The new query_string or none to use the original. When building a url, a Webob Request will append the query string directly to the url. url += ‘?’ + env[‘QUERY_STRING’]
agent – The HTTP user agent to use; default ‘Swift’. You can put %(orig)s in the agent to have it replaced with the original env’s HTTP_USER_AGENT, such as ‘%(orig)s StaticWeb’. You also set agent to None to use the original env’s HTTP_USER_AGENT or ‘’ to have no HTTP_USER_AGENT.
swift_source – Used to mark the request as originating out of middleware. Will be logged in proxy logs.

Returns:

Fresh WSGI environment.

swift.common.wsgi.make_pre_authed_env(env, method=None, path=None, agent='Swift', query_string=None, swift_source=None)¶: Same as make_env() but with preauthorization.

swift.common.wsgi.make_pre_authed_request(env, method=None, path=None, body=None, headers=None, agent='Swift', swift_source=None)¶: Same as make_subrequest() but with preauthorization.

swift.common.wsgi.make_subrequest(env, method=None, path=None, body=None, headers=None, agent='Swift', swift_source=None, make_env=<function make_env at 0x4a71500>)¶

Makes a new swob.Request based on the current env but with the parameters specified.

Parameters:

env – The WSGI environment to base the new request on.
method – HTTP method of new request; default is from the original env.
path – HTTP path of new request; default is from the original env. path should be compatible with what you would send to Request.blank. path should be quoted and it can include a query string. for example: ‘/a%20space?unicode_str%E8%AA%9E=y%20es’
body – HTTP body of new request; empty by default.
headers – Extra HTTP headers of new request; None by default.
agent – The HTTP user agent to use; default ‘Swift’. You can put %(orig)s in the agent to have it replaced with the original env’s HTTP_USER_AGENT, such as ‘%(orig)s StaticWeb’. You also set agent to None to use the original env’s HTTP_USER_AGENT or ‘’ to have no HTTP_USER_AGENT.
swift_source – Used to mark the request as originating out of middleware. Will be logged in proxy logs.
make_env – make_subrequest calls this make_env to help build the swob.Request.

Returns:

Fresh swob.Request object.

swift.common.wsgi.monkey_patch_mimetools()¶: mimetools.Message defaults content-type to “text/plain” This changes it to default to None, so we can detect missing headers.

swift.common.wsgi.run_wsgi(conf_path, app_section, *args, **kwargs)¶

Runs the server using the specified number of workers.

Parameters:	conf_path – Path to paste.deploy style configuration file/directory app_section – App name from conf file to load config from
Returns:	0 if successful, nonzero otherwise

swift.common.wsgi.wrap_conf_type(f)¶: Wrap a function whos first argument is a paste.deploy style config uri, such that you can pass it an un-adorned raw filesystem path and the config directive (either config: or config_dir:) will be added automatically based on the type of filesystem entity at the given path (either a file or directory) before passing it through to the paste.deploy function.

Table Of Contents

Previous topic

Next topic

This Page

Misc¶

ACLs¶

Buffered HTTP¶

Constraints¶

Container Sync Realms¶

Direct Client¶

Exceptions¶

Internal Client¶

Manager¶

MemCacheD¶

Request Helpers¶

Swob¶

Utils¶

WSGI¶

Navigation

Table Of Contents

Previous topic

Next topic

This Page

Quick search

Misc¶

ACLs¶

Buffered HTTP¶

Constraints¶

Container Sync Realms¶

Direct Client¶

Exceptions¶

Internal Client¶

Manager¶

MemCacheD¶

Request Helpers¶

Swob¶

Utils¶

WSGI¶

Navigation