account_quotas is a middleware which blocks write requests (PUT, POST) if a given account quota (in bytes) is exceeded while DELETE requests are still allowed.
account_quotas uses the x-account-meta-quota-bytes metadata entry to store the quota. Write requests to this metadata entry are only permitted for resellers. There is no quota limit if x-account-meta-quota-bytes is not set.
The account_quotas middleware should be added to the pipeline in your /etc/swift/proxy-server.conf file just after any auth middleware. For example:
[pipeline:main]
pipeline = catch_errors cache tempauth account_quotas proxy-server
[filter:account_quotas]
use = egg:swift#account_quotas
To set the quota on an account:
swift -A http://127.0.0.1:8080/auth/v1.0 -U account:reseller -K secret post -m quota-bytes:10000
Remove the quota:
swift -A http://127.0.0.1:8080/auth/v1.0 -U account:reseller -K secret post -m quota-bytes:
The same limitations apply for the account quotas as for the container quotas.
For example, when uploading an object without a content-length header the proxy server doesn’t know the final size of the currently uploaded object and the upload will be allowed if the current account size is within the quota. Due to the eventual consistency further uploads might be possible until the account size has been updated.
Bases: object
Account quota middleware
See above for a full description.
Returns a WSGI filter app for use with paste.deploy.
Bases: object
Middleware that will do many operations on a single request.
Extract Archive:
Expand tar files into a swift account. Request must be a PUT with the query parameter ?extract-archive=format specifying the format of archive file. Accepted formats are tar, tar.gz, and tar.bz2.
For a PUT to the following url:
/v1/AUTH_Account/$UPLOAD_PATH?extract-archive=tar.gz
UPLOAD_PATH is where the files will be expanded to. UPLOAD_PATH can be a container, a pseudo-directory within a container, or an empty string. The destination of a file in the archive will be built as follows:
/v1/AUTH_Account/$UPLOAD_PATH/$FILE_PATH
Where FILE_PATH is the file name from the listing in the tar file.
If the UPLOAD_PATH is an empty string, containers will be auto created accordingly and files in the tar that would not map to any container (files in the base directory) will be ignored.
Only regular files will be uploaded. Empty directories, symlinks, etc will not be uploaded.
The response from bulk operations functions differently from other swift responses. This is because a short request body sent from the client could result in many operations on the proxy server and precautions need to be made to prevent the request from timing out due to lack of activity. To this end, the client will always receive a 200 OK response, regardless of the actual success of the call. The body of the response must be parsed to determine the actual success of the operation. In addition to this the client may receive zero or more whitespace characters prepended to the actual response body while the proxy server is completing the request.
The format of the response body defaults to text/plain but can be either json or xml depending on the Accept header. Acceptable formats are text/plain, application/json, application/xml, and text/xml. An example body is as follows:
If all valid files were uploaded successfully the Response Status will be 201 Created. If any files failed to be created the response code corresponds to the subrequest’s error. Possible codes are 400, 401, 502 (on server errors), etc. In both cases the response body will specify the number of files successfully uploaded and a list of the files that failed.
There are proxy logs created for each file (which becomes a subrequest) in the tar. The subrequest’s proxy log will have a swift.source set to “EA” the log’s content length will reflect the unzipped size of the file. If double proxy-logging is used the leftmost logger will not have a swift.source set and the content length will reflect the size of the payload sent to the proxy (the unexpanded size of the tar.gz).
Bulk Delete:
Will delete multiple objects or containers from their account with a single request. Responds to POST requests with query parameter ?bulk-delete set. The request url is your storage url. The Content-Type should be set to text/plain. The body of the POST request will be a newline separated list of url encoded objects to delete. You can delete 10,000 (configurable) objects per request. The objects specified in the POST request body must be URL encoded and in the form:
/container_name/obj_name
or for a container (which must be empty at time of delete)
/container_name
The response is similar to extract archive as in every response will be a 200 OK and you must parse the response body for actual results. An example response is:
If all items were successfully deleted (or did not exist), the Response Status will be 200 OK. If any failed to delete, the response code corresponds to the subrequest’s error. Possible codes are 400, 401, 502 (on server errors), etc. In all cases the response body will specify the number of items successfully deleted, not found, and a list of those that failed. The return body will be formatted in the way specified in the request’s Accept header. Acceptable formats are text/plain, application/json, application/xml, and text/xml.
There are proxy logs created for each object or container (which becomes a subrequest) that is deleted. The subrequest’s proxy log will have a swift.source set to “BD” the log’s content length of 0. If double proxy-logging is used the leftmost logger will not have a swift.source set and the content length will reflect the size of the payload sent to the proxy (the list of objects/containers to be deleted).
Checks if the container exists and if not try to create it. :params container_path: an unquoted path to a container to be created :returns: True if created container, False if container exists :raises: CreateContainerError when unable to create container
Will populate objs_to_delete with data from request input. :params req: a Swob request :returns: a list of the contents of req.body when separated by newline. :raises: HTTPException on failures
A generator that can be assigned to a swob Response’s app_iter which, when iterated over, will delete the objects specified in request body. Will occasionally yield whitespace while request is being processed. When the request is completed will yield a response body that can be parsed to determine success. See above documentation for details.
| Params req: | a swob Request |
|---|---|
| Params objs_to_delete: | |
| a list of dictionaries that specifies the objects to be deleted. If None, uses self.get_objs_to_delete to query request. | |
A generator that can be assigned to a swob Response’s app_iter which, when iterated over, will extract and PUT the objects pulled from the request body. Will occasionally yield whitespace while request is being processed. When the request is completed will yield a response body that can be parsed to determine success. See above documentation for details.
| Params req: | a swob Request |
|---|---|
| Params compress_type: | |
| specifying the compression type of the tar. Accepts ‘’, ‘gz’, or ‘bz2’ | |
Returns a properly formatted response body according to format. Handles json and xml, otherwise will return text/plain. Note: xml response does not include xml declaration. :params data_format: resulting format :params data_dict: generated data about results. :params error_list: list of quoted filenames that failed
Bases: object
Middleware that provides high-level error handling and ensures that a transaction id will be set for every request.
CNAME Lookup Middleware
Middleware that translates an unknown domain in the host header to something that ends with the configured storage_domain by looking up the given domain’s CNAME record in DNS.
This middleware will continue to follow a CNAME chain in DNS until it finds a record ending in the configured storage domain or it reaches the configured maximum lookup depth. If a match is found, the environment’s Host header is rewritten and the request is passed further down the WSGI chain.
Bases: object
CNAME Lookup Middleware
See above for a full description.
| Parameters: |
|
|---|
Given a domain, returns its DNS CNAME mapping and DNS ttl.
| Parameters: | domain – domain to query on |
|---|---|
| Returns: | (ttl, result) |
The container_quotas middleware implements simple quotas that can be imposed on swift containers by a user with the ability to set container metadata, most likely the account administrator. This can be useful for limiting the scope of containers that are delegated to non-admin users, exposed to formpost uploads, or just as a self-imposed sanity check.
Any object PUT operations that exceed these quotas return a 413 response (request entity too large) with a descriptive body.
Quotas are subject to several limitations: eventual consistency, the timeliness of the cached container_info (60 second ttl by default), and it’s unable to reject chunked transfer uploads that exceed the quota (though once the quota is exceeded, new chunked transfers will be refused).
Quotas are set by adding meta values to the container, and are validated when set:
| Metadata | Use |
|---|---|
| X-Container-Meta-Quota-Bytes | Maximum size of the container, in bytes. |
| X-Container-Meta-Quota-Count | Maximum object count of the container. |
Bases: object
WSGI middleware that validates an incoming container sync request using the container-sync-realms.conf style of container sync.
Bases: object
Cross domain middleware used to respond to requests for cross domain policy information.
If the path is /crossdomain.xml it will respond with an xml cross domain policy document. This allows web pages hosted elsewhere to use client side technologies such as Flash, Java and Silverlight to interact with the Swift API.
To enable this middleware, add it to the pipeline in your proxy-server.conf file. It should be added before any authentication (e.g., tempauth or keystone) middleware. In this example ellipsis (...) indicate other middleware you may have chosen to use:
[pipeline:main]
pipeline = ... crossdomain ... authtoken ... proxy-server
And add a filter section, such as:
[filter:crossdomain]
use = egg:swift#crossdomain
cross_domain_policy = <allow-access-from domain="*.example.com" />
<allow-access-from domain="www.example.com" secure="false" />
For continuation lines, put some whitespace before the continuation text. Ensure you put a completely blank line to terminate the cross_domain_policy value.
The cross_domain_policy name/value is optional. If omitted, the policy defaults as if you had specified:
cross_domain_policy = <allow-access-from domain="*" secure="false" />
Returns a 200 response with cross domain policy information
Swift will by default provide clients with an interface providing details about the installation. Unless disabled (i.e expose_info=false in Proxy Server Configuration), a GET request to /info will return configuration data in JSON format. An example response:
{"swift": {"version": "1.11.0"}, "staticweb": {}, "tempurl": {}}
This would signify to the client that swift version 1.11.0 is running and that staticweb and tempurl are available in this installation.
There may be administrator-only information available via /info. To retrieve it, one must use an HMAC-signed request, similar to TempURL. The signature may be produced like so:
swift-temp-url GET 3600 /info secret 2>/dev/null | sed s/temp_url/swiftinfo/g
Domain Remap Middleware
Middleware that translates container and account parts of a domain to path parameters that the proxy server understands.
container.account.storageurl/object gets translated to container.account.storageurl/path_root/account/container/object
account.storageurl/path_root/container/object gets translated to account.storageurl/path_root/account/container/object
Browsers can convert a host header to lowercase, so check that reseller prefix on the account is the correct case. This is done by comparing the items in the reseller_prefixes config option to the found prefix. If they match except for case, the item from reseller_prefixes will be used instead of the found reseller prefix. The reseller_prefixes list is exclusive. If defined, any request with an account prefix not in that list will be ignored by this middleware. reseller_prefixes defaults to ‘AUTH’.
Note that this middleware requires that container names and account names (except as described above) must be DNS-compatible. This means that the account name created in the system and the containers created by users cannot exceed 63 characters or have UTF-8 characters. These are restrictions over and above what swift requires and are not explicitly checked. Simply put, the this middleware will do a best-effort attempt to derive account and container names from elements in the domain name and put those derived values into the URL path (leaving the Host header unchanged).
Also note that using container sync with remapped domain names is not advised. With container sync, you should use the true storage end points as sync destinations.
Bases: object
Domain Remap Middleware
See above for a full description.
| Parameters: |
|
|---|
FormPost Middleware
Translates a browser form post into a regular Swift object PUT.
The format of the form is:
<form action="<swift-url>" method="POST"
enctype="multipart/form-data">
<input type="hidden" name="redirect" value="<redirect-url>" />
<input type="hidden" name="max_file_size" value="<bytes>" />
<input type="hidden" name="max_file_count" value="<count>" />
<input type="hidden" name="expires" value="<unix-timestamp>" />
<input type="hidden" name="signature" value="<hmac>" />
<input type="file" name="file1" /><br />
<input type="submit" />
</form>
The <swift-url> is the URL to the Swift desination, such as:
https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix
The name of each file uploaded will be appended to the <swift-url> given. So, you can upload directly to the root of container with a url like:
https://swift-cluster.example.com/v1/AUTH_account/container/
Optionally, you can include an object prefix to better separate different users’ uploads, such as:
https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix
Note the form method must be POST and the enctype must be set as “multipart/form-data”.
The redirect attribute is the URL to redirect the browser to after the upload completes. This is an optional parameter. If you are uploading the form via an XMLHttpRequest the redirect should not be included. The URL will have status and message query parameters added to it, indicating the HTTP status code for the upload (2xx is success) and a possible message for further information if there was an error (such as “max_file_size exceeded”).
The max_file_size attribute must be included and indicates the largest single file upload that can be done, in bytes.
The max_file_count attribute must be included and indicates the maximum number of files that can be uploaded with the form. Include additional <input type="file" name="filexx" /> attributes if desired.
The expires attribute is the Unix timestamp before which the form must be submitted before it is invalidated.
The signature attribute is the HMAC-SHA1 signature of the form. Here is sample code for computing the signature:
import hmac
from hashlib import sha1
from time import time
path = '/v1/account/container/object_prefix'
redirect = 'https://srv.com/some-page' # set to '' if redirect not in form
max_file_size = 104857600
max_file_count = 10
expires = int(time() + 600)
key = 'mykey'
hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect,
max_file_size, max_file_count, expires)
signature = hmac.new(key, hmac_body, sha1).hexdigest()
The key is the value of the X-Account-Meta-Temp-URL-Key header on the account.
Be certain to use the full path, from the /v1/ onward.
The command line tool swift-form-signature may be used (mostly just when testing) to compute expires and signature.
Also note that the file attributes must be after the other attributes in order to be processed correctly. If attributes come after the file, they won’t be sent with the subrequest (there is no way to parse all the attributes on the server-side without reading the whole thing into memory – to service many requests, some with large files, there just isn’t enough memory on the server, so attributes following the file are simply ignored).
Bases: object
FormPost Middleware
See above for a full description.
The proxy logs created for any subrequests made will have swift.source set to “FP”.
| Parameters: |
|
|---|
The next WSGI application/filter in the paste.deploy pipeline.
The filter configuration dict.
Returns the WSGI filter for use with paste.deploy.
The size of data to read from the form at any given time.
The maximum size of any attribute’s value. Any additional data will be truncated.
The gatekeeper middleware imposes restrictions on the headers that may be included with requests and responses. Request headers are filtered to remove headers that should never be generated by a client. Similarly, response headers are filtered to remove private headers that should never be passed to a client.
The gatekeeper middleware must always be present in the proxy server wsgi pipeline. It should be configured close to the start of the pipeline specified in /etc/swift/proxy-server.conf, immediately after catch_errors and before any other middleware. It is essential that it is configured ahead of all middlewares using system metadata in order that they function correctly.
If gatekeeper middleware is not configured in the pipeline then it will be automatically inserted close to the start of the pipeline by the proxy server.
A list of python regular expressions that will be used to match against inbound request headers. Matching headers will be removed from the request.
A list of python regular expressions that will be used to match against outbound response headers. Matching headers will be removed from the response.
Bases: object
Healthcheck middleware used for monitoring.
If the path is /healthcheck, it will respond 200 with “OK” as the body.
If the optional config parameter “disable_path” is set, and a file is present at that path, it will respond 503 with “DISABLED BY FILE” as the body.
Returns a 503 response with “DISABLED BY FILE” in the body.
Returns a 200 response with “OK” in the body.
Bases: object
Swift middleware to Keystone authorization system.
In Swift’s proxy-server.conf add this middleware to your pipeline:
[pipeline:main]
pipeline = catch_errors cache authtoken keystoneauth proxy-server
Make sure you have the authtoken middleware before the keystoneauth middleware.
The authtoken middleware will take care of validating the user and keystoneauth will authorize access.
The authtoken middleware is shipped directly with keystone it does not have any other dependences than itself so you can either install it by copying the file directly in your python path or by installing keystone.
If support is required for unvalidated users (as with anonymous access) or for formpost/staticweb/tempurl middleware, authtoken will need to be configured with delay_auth_decision set to true. See the Keystone documentation for more detail on how to configure the authtoken middleware.
In proxy-server.conf you will need to have the setting account auto creation to true:
[app:proxy-server]
account_autocreate = true
And add a swift authorization filter section, such as:
[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = admin, swiftoperator
This maps tenants to account in Swift.
The user whose able to give ACL / create Containers permissions will be the one that are inside the operator_roles setting which by default includes the admin and the swiftoperator roles.
If you need to have a different reseller_prefix to be able to mix different auth servers you can configure the option reseller_prefix in your keystoneauth entry like this:
reseller_prefix = NEWAUTH
| Parameters: |
|
|---|
Authorize an anonymous request.
| Returns: | None if authorization is granted, an error page otherwise. |
|---|
Deny WSGI Response.
Returns a standard WSGI response callable with the status of 403 or 401 depending on whether the REMOTE_USER is set or not.
Returns a WSGI filter app for use with paste.deploy.
List endpoints for an object, account or container.
This middleware makes it possible to integrate swift with software that relies on data locality information to avoid network overhead, such as Hadoop.
Answers requests of the form:
/endpoints/{account}/{container}/{object}
/endpoints/{account}/{container}
/endpoints/{account}
with a JSON-encoded list of endpoints of the form:
http://{server}:{port}/{dev}/{part}/{acc}/{cont}/{obj}
http://{server}:{port}/{dev}/{part}/{acc}/{cont}
http://{server}:{port}/{dev}/{part}/{acc}
correspondingly, e.g.:
http://10.1.1.1:6000/sda1/2/a/c2/o1
http://10.1.1.1:6000/sda1/2/a/c2
http://10.1.1.1:6000/sda1/2/a
The ‘/endpoints/’ path is customizable (‘list_endpoints_path’ configuration parameter).
Intended for consumption by third-party services living inside the cluster (as the endpoints make sense only inside the cluster behind the firewall); potentially written in a different language.
This is why it’s provided as a REST API and not just a Python API: to avoid requiring clients to write their own ring parsers in their languages, and to avoid the necessity to distribute the ring file to clients and keep it up-to-date.
Note that the call is not authenticated, which means that a proxy with this middleware enabled should not be open to an untrusted environment (everyone can query the locality data using this middleware).
Bases: object
List endpoints for an object, account or container.
See above for a full description.
Uses configuration parameter swift_dir (default /etc/swift).
| Parameters: |
|
|---|
Bases: object
Caching middleware that manages caching in swift.
Created on February 27, 2012
A filter that disallows any paths that contain defined forbidden characters or that exceed a defined length.
Place early in the proxy-server pipeline after the left-most occurrence of the proxy-logging middleware (if present) and before the final proxy-logging middleware (if present) or the proxy-serer app itself, e.g.:
[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging name_check cache ratelimit tempauth sos proxy-logging proxy-server
[filter:name_check]
use = egg:swift#name_check
forbidden_chars = '"`<>
maximum_length = 255
There are default settings for forbidden_chars (FORBIDDEN_CHARS) and maximum_length (MAX_LENGTH)
The filter returns HTTPBadRequest if path is invalid.
@author: eamonn-otoole
Logging middleware for the Swift proxy.
This serves as both the default logging implementation and an example of how to plug in your own logging format/method.
The logging format implemented below is as follows:
These values are space-separated, and each is url-encoded, so that they can be separated with a simple .split()
The proxy-logging can be used twice in the proxy server’s pipeline when there is middleware installed that can return custom responses that don’t follow the standard pipeline to the proxy server.
For example, with staticweb, the middleware might intercept a request to /v1/AUTH_acc/cont/, make a subrequest to the proxy to retrieve /v1/AUTH_acc/cont/index.html and, in effect, respond to the client’s original request using the 2nd request’s body. In this instance the subrequest will be logged by the rightmost middleware (with a swift.source set) and the outgoing request (with body overridden) will be logged by leftmost middleware.
Requests that follow the normal pipeline (use the same wsgi environment throughout) will not be double logged because an environment variable (swift.proxy_access_log_made) is checked/set when a log is made.
All middleware making subrequests should take care to set swift.source when needed. With the doubled proxy logs, any consumer/processor of swift’s proxy logs should look at the swift.source field, the rightmost log value, to decide if this is a middleware subrequest or not. A log processor calculating bandwidth usage will want to only sum up logs with no swift.source.
Bases: object
Middleware that logs Swift proxy requests in the swift log format.
Log a request.
| Parameters: |
|
|---|
Bases: object
Rate limiting middleware
Rate limits requests on both an Account and Container level. Limits are configurable.
Returns a list of key (used in memcache), ratelimit tuples. Keys should be checked in order.
| Parameters: |
|
|---|
Performs rate limiting and account white/black listing. Sleeps if necessary. If self.memcache_client is not set, immediately returns None.
| Parameters: |
|
|---|
paste.deploy app factory for creating WSGI proxy apps.
Returns number of requests allowed per second for given size.
Bases: object
Recon middleware used for monitoring.
/recon/load|mem|async... will return various system metrics.
Needs to be added to the pipeline and requires a filter declaration in the object-server.conf:
[filter:recon] use = egg:swift#recon recon_cache_path = /var/cache/swift
get # of async pendings
get auditor info
get devices
get disk utilization statistics
get expirer info
get info from /proc/loadavg
get info from /proc/meminfo
get ALL mounted fs from /proc/mounts
get obj/container/account quarantine counts
get replication info
get all ring md5sum’s
get info from /proc/net/sockstat and sockstat6
Note: The mem value is actually kernel pages, but we return bytes allocated based on the systems page size.
get md5 of swift.conf
list unmounted (failed?) devices
get updater info
get swift version
Middleware that will provide Static Large Object (SLO) support.
This feature is very similar to Dynamic Large Object (DLO) support in that it allows the user to upload many objects concurrently and afterwards download them as a single object. It is different in that it does not rely on eventually consistent container listings to do so. Instead, a user defined manifest of the object segments is used.
After the user has uploaded the objects to be concatenated a manifest is uploaded. The request must be a PUT with the query parameter:
?multipart-manifest=put
The body of this request will be an ordered list of files in json data format. The data to be supplied for each segment is:
path: the path to the segment (not including account)
/container/object_name
etag: the etag given back when the segment was PUT
size_bytes: the size of the segment in bytes
The format of the list will be:
json:
[{"path": "/cont/object",
"etag": "etagoftheobjectsegment",
"size_bytes": 1048576}, ...]
The number of object segments is limited to a configurable amount, default 1000. Each segment, except for the final one, must be at least 1 megabyte (configurable). On upload, the middleware will head every segment passed in and verify the size and etag of each. If any of the objects do not match (not found, size/etag mismatch, below minimum size) then the user will receive a 4xx error response. If everything does match, the user will receive a 2xx response and the SLO object is ready for downloading.
Behind the scenes, on success, a json manifest generated from the user input is sent to object servers with an extra “X-Static-Large-Object: True” header and a modified Content-Type. The parameter: swift_bytes=$total_size will be appended to the existing Content-Type, where total_size is the sum of all the included segments’ size_bytes. This extra parameter will be hidden from the user.
Manifest files can reference objects in separate containers, which will improve concurrent upload speed. Objects can be referenced by multiple manifests. The segments of a SLO manifest can even be other SLO manifests. Treat them as any other object i.e., use the Etag and Content-Length given on the PUT of the sub-SLO in the manifest to the parent SLO.
A GET request to the manifest object will return the concatenation of the objects from the manifest much like DLO. If any of the segments from the manifest are not found or their Etag/Content Length no longer match the connection will drop. In this case a 409 Conflict will be logged in the proxy logs and the user will receive incomplete results.
The headers from this GET or HEAD request will return the metadata attached to the manifest object itself with some exceptions:
Content-Length: the total size of the SLO (the sum of the sizes of
the segments in the manifest)
X-Static-Large-Object: True
Etag: the etag of the SLO (generated the same way as DLO)
A GET request with the query parameter:
?multipart-manifest=get
Will return the actual manifest file itself. This is generated json and does not match the data sent from the original multipart-manifest=put. This call’s main purpose is for debugging.
When the manifest object is uploaded you are more or less guaranteed that every segment in the manifest exists and matched the specifications. However, there is nothing that prevents the user from breaking the SLO download by deleting/replacing a segment referenced in the manifest. It is left to the user to use caution in handling the segments.
A DELETE request will just delete the manifest object itself.
A DELETE with a query parameter:
?multipart-manifest=delete
will delete all the segments referenced in the manifest and then the manifest itself. The failure response will be similar to the bulk delete middleware.
PUTs / POSTs will work as expected, PUTs will just overwrite the manifest object for example.
In a container listing the size listed for SLO manifest objects will be the total_size of the concatenated segments in the manifest. The overall X-Container-Bytes-Used for the container (and subsequently for the account) will not reflect total_size of the manifest but the actual size of the json data stored. The reason for this somewhat confusing discrepancy is we want the container listing to reflect the size of the manifest object when it is downloaded. We do not, however, want to count the bytes-used twice (for both the manifest and the segments it’s referring to) in the container and account metadata which can be used for stats purposes.
Bases: object
StaticLargeObject Middleware
See above for a full description.
The proxy logs created for any subrequests made will have swift.source set to “SLO”.
| Parameters: |
|
|---|
A generator function to be used to delete all the segments and sub-segments referenced in a manifest.
| Params req: | a swob.Request with an SLO manifest in path |
|---|---|
| Raises: |
|
Performs a swob.Request and returns the SLO manifest’s segments.
| Raises: |
|
|---|---|
| Returns: | SLO manifest’s segments |
Will delete all the segments in the SLO manifest and then, if successful, will delete the manifest file.
| Params req: | a swob.Request with an obj in path |
|---|---|
| Returns: | swob.Response whose app_iter set to Bulk.handle_delete_iter |
Handles the GET or HEAD of a SLO manifest.
The response body (only on GET, of course) will consist of the concatenation of the segments.
| Params req: | a swob.Request with a path referencing an object |
|---|---|
| Raises : | HttpException on errors |
Will handle the PUT of a SLO manifest. Heads every object in manifest to check if is valid and if so will save a manifest generated from the user input. Uses WSGIContext to call self and start_response and returns a WSGI iterator.
| Params req: | a swob.Request with an obj in path |
|---|---|
| Raises : | HttpException on errors |
Given a request will parse the body and return a list of dictionaries :raises: HTTPException on parse errors :returns: a list of dictionaries on success
This StaticWeb WSGI middleware will serve container data as a static web site with index file and error file resolution and optional file listings. This mode is normally only active for anonymous requests. When using keystone for authentication set delay_auth_decision = true in the authtoken middleware configuration in your /etc/swift/proxy-server.conf file. If you want to use it with authenticated requests, set the X-Web-Mode: true header on the request.
The staticweb filter should be added to the pipeline in your /etc/swift/proxy-server.conf file just after any auth middleware. Also, the configuration section for the staticweb middleware itself needs to be added. For example:
[DEFAULT]
...
[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging cache ratelimit tempauth
staticweb proxy-logging proxy-server
...
[filter:staticweb]
use = egg:swift#staticweb
Any publicly readable containers (for example, X-Container-Read: .r:*, see ACLs for more information on this) will be checked for X-Container-Meta-Web-Index and X-Container-Meta-Web-Error header values:
X-Container-Meta-Web-Index <index.name>
X-Container-Meta-Web-Error <error.name.suffix>
If X-Container-Meta-Web-Index is set, any <index.name> files will be served without having to specify the <index.name> part. For instance, setting X-Container-Meta-Web-Index: index.html will be able to serve the object .../pseudo/path/index.html with just .../pseudo/path or .../pseudo/path/
If X-Container-Meta-Web-Error is set, any errors (currently just 401 Unauthorized and 404 Not Found) will instead serve the .../<status.code><error.name.suffix> object. For instance, setting X-Container-Meta-Web-Error: error.html will serve .../404error.html for requests for paths not found.
For pseudo paths that have no <index.name>, this middleware can serve HTML file listings if you set the X-Container-Meta-Web-Listings: true metadata item on the container.
If listings are enabled, the listings can have a custom style sheet by setting the X-Container-Meta-Web-Listings-CSS header. For instance, setting X-Container-Meta-Web-Listings-CSS: listing.css will make listings link to the .../listing.css style sheet. If you “view source” in your browser on a listing page, you will see the well defined document structure that can be styled.
The content-type of directory marker objects can be modified by setting the X-Container-Meta-Web-Directory-Type header. If the header is not set, application/directory is used by default. Directory marker objects are 0-byte objects that represent directories to create a simulated hierarchical structure.
Example usage of this middleware via swift:
Make the container publicly readable:
swift post -r '.r:*' containerYou should be able to get objects directly, but no index.html resolution or listings.
Set an index file directive:
swift post -m 'web-index:index.html' containerYou should be able to hit paths that have an index.html without needing to type the index.html part.
Turn on listings:
swift post -m 'web-listings: true' containerNow you should see object listings for paths and pseudo paths that have no index.html.
Enable a custom listings style sheet:
swift post -m 'web-listings-css:listings.css' containerSet an error file:
swift post -m 'web-error:error.html' containerNow 401’s should load 401error.html, 404’s should load 404error.html, etc.
Set Content-Type of directory marker object:
swift post -m 'web-directory-type:text/directory' containerNow 0-byte objects with a content-type of text/directory will be treated as directories rather than objects.
Bases: object
The Static Web WSGI middleware filter; serves container data as a static web site. See staticweb for an overview.
The proxy logs created for any subrequests made will have swift.source set to “SW”.
| Parameters: |
|
|---|
The next WSGI application/filter in the paste.deploy pipeline.
The filter configuration dict.
Returns a Static Web WSGI filter for use with paste.deploy.
Bases: object
Test authentication and authorization system.
Add to your pipeline in proxy-server.conf, such as:
[pipeline:main]
pipeline = catch_errors cache tempauth proxy-server
Set account auto creation to true in proxy-server.conf:
[app:proxy-server]
account_autocreate = true
And add a tempauth filter section, such as:
[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3
# To allow accounts/users with underscores you can base64 encode them.
# Here is the account "under_score" and username "a_b" (note the lack
# of padding equal signs):
user64_dW5kZXJfc2NvcmU_YV9i = testing4
See the proxy-server.conf-sample for more information.
If a swift_owner issues a POST or PUT to the account, with the X-Account-Access-Control header set in the request, then this may allow certain types of access for additional users.
To generate headers for setting an account ACL:
from swift.common.middleware.acl import format_acl
acl_data = { 'admin': ['alice'], 'read-write': ['bob', 'carol'] }
header_value = format_acl(version=2, acl_dict=acl_data)
To generate a curl command line from the above:
token=...
storage_url=...
python -c '
from swift.common.middleware.acl import format_acl
acl_data = { 'admin': ['alice'], 'read-write': ['bob', 'carol'] }
headers = {'X-Account-Access-Control':
format_acl(version=2, acl_dict=acl_data)}
header_str = ' '.join(["-H '%s: %s'" % (k, v)
for k, v in headers.items()])
print ('curl -D- -X POST -H "x-auth-token: $token" %s '
'$storage_url' % header_str)
'
| Parameters: |
|
|---|
Return a dict of ACL data from the account server via get_account_info.
Auth systems may define their own format, serialization, structure, and capabilities implemented in the ACL headers and persisted in the sysmeta data. However, auth systems are strongly encouraged to be interoperable with Tempauth.
Returns None if the request is authorized to continue or a standard WSGI response callable if not.
Returns a standard WSGI response callable with the status of 403 or 401 depending on whether the REMOTE_USER is set or not.
Return a user-readable string indicating the errors in the input ACL, or None if there are no errors.
Get groups for the given token.
| Parameters: |
|
|---|---|
| Returns: | None if the token is invalid or a string containing a comma separated list of groups the authenticated user is a member of. The first group in the list is also considered a unique identifier for that user. |
WSGI entry point for auth requests (ones that match the self.auth_prefix). Wraps env in swob.Request object and passes it down.
| Parameters: |
|
|---|
Handles the various request for token and service end point(s) calls. There are various formats to support the various auth servers in the past. Examples:
GET <auth-prefix>/v1/<act>/auth
X-Auth-User: <act>:<usr> or X-Storage-User: <usr>
X-Auth-Key: <key> or X-Storage-Pass: <key>
GET <auth-prefix>/auth
X-Auth-User: <act>:<usr> or X-Storage-User: <act>:<usr>
X-Auth-Key: <key> or X-Storage-Pass: <key>
GET <auth-prefix>/v1.0
X-Auth-User: <act>:<usr> or X-Storage-User: <act>:<usr>
X-Auth-Key: <key> or X-Storage-Pass: <key>
On successful authentication, the response will have X-Auth-Token and X-Storage-Token set to the token to use with Swift and X-Storage-URL set to the URL to the default Swift cluster to use.
| Parameters: | req – The swob.Request to process. |
|---|---|
| Returns: | swob.Response, 2xx on success with data set as explained above. |
Entry point for auth requests (ones that match the self.auth_prefix). Should return a WSGI-style callable (such as swob.Response).
| Parameters: | req – swob.Request object |
|---|
Returns a WSGI filter app for use with paste.deploy.
TempURL Middleware
Allows the creation of URLs to provide temporary access to objects.
For example, a website may wish to provide a link to download a large object in Swift, but the Swift account has no public access. The website can generate a URL that will provide GET access for a limited time to the resource. When the web browser user clicks on the link, the browser will download the object directly from Swift, obviating the need for the website to act as a proxy for the request.
If the user were to share the link with all his friends, or accidentally post it on a forum, etc. the direct access would be limited to the expiration time set when the website created the link.
To create such temporary URLs, first an X-Account-Meta-Temp-URL-Key header must be set on the Swift account. Then, an HMAC-SHA1 (RFC 2104) signature is generated using the HTTP method to allow (GET or PUT), the Unix timestamp the access should be allowed until, the full path to the object, and the key set on the account.
For example, here is code generating the signature for a GET for 60 seconds on /v1/AUTH_account/container/object:
import hmac
from hashlib import sha1
from time import time
method = 'GET'
expires = int(time() + 60)
path = '/v1/AUTH_account/container/object'
key = 'mykey'
hmac_body = '%s\n%s\n%s' % (method, expires, path)
sig = hmac.new(key, hmac_body, sha1).hexdigest()
Be certain to use the full path, from the /v1/ onward.
Let’s say the sig ends up equaling da39a3ee5e6b4b0d3255bfef95601890afd80709 and expires ends up 1323479485. Then, for example, the website could provide a link to:
https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485
Any alteration of the resource path or query arguments would result in 401 Unauthorized. Similary, a PUT where GET was the allowed method would 401. HEAD is allowed if GET or PUT is allowed.
Using this in combination with browser form post translation middleware could also allow direct-from-browser uploads to specific locations in Swift.
TempURL supports up to two keys, specified by X-Account-Meta-Temp-URL-Key and X-Account-Meta-Temp-URL-Key-2. Signatures are checked against both keys, if present. This is to allow for key rotation without invalidating all existing temporary URLs.
With GET TempURLs, a Content-Disposition header will be set on the response so that browsers will interpret this as a file attachment to be saved. The filename chosen is based on the object name, but you can override this with a filename query parameter. Modifying the above example:
https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485&filename=My+Test+File.pdf
If you do not want the object to be downloaded, you can cause “Content-Disposition: inline” to be set on the response by adding the “inline” parameter to the query string, like so:
https://swift-cluster.example.com/v1/AUTH_account/container/object?
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
temp_url_expires=1323479485&inline
Bases: object
WSGI Middleware to grant temporary URLs specific access to Swift resources. See the overview for more information.
This middleware understands the following configuration settings:
incoming_remove_headers
The headers to remove from incoming requests. Simply a
whitespace delimited list of header names and names can
optionally end with '*' to indicate a prefix match.
incoming_allow_headers is a list of exceptions to these
removals.
Default: x-timestamp
incoming_allow_headers
The headers allowed as exceptions to
incoming_remove_headers. Simply a whitespace delimited
list of header names and names can optionally end with
'*' to indicate a prefix match.
Default: None
outgoing_remove_headers
The headers to remove from outgoing responses. Simply a
whitespace delimited list of header names and names can
optionally end with '*' to indicate a prefix match.
outgoing_allow_headers is a list of exceptions to these
removals.
Default: x-object-meta-*
outgoing_allow_headers
The headers allowed as exceptions to
outgoing_remove_headers. Simply a whitespace delimited
list of header names and names can optionally end with
'*' to indicate a prefix match.
Default: x-object-meta-public-*
The proxy logs created for any subrequests made will have swift.source set to “FP”.
| Parameters: |
|
|---|
HTTP user agent to use for subrequests.
The next WSGI application/filter in the paste.deploy pipeline.
The filter configuration dict.
Headers to allow in incoming requests. Uppercase WSGI env style, like HTTP_X_MATCHES_REMOVE_PREFIX_BUT_OKAY.
Header with match prefixes to allow in incoming requests. Uppercase WSGI env style, like HTTP_X_MATCHES_REMOVE_PREFIX_BUT_OKAY_*.
Headers to remove from incoming requests. Uppercase WSGI env style, like HTTP_X_PRIVATE.
Header with match prefixes to remove from incoming requests. Uppercase WSGI env style, like HTTP_X_SENSITIVE_*.
The methods allowed with Temp URLs.
Headers to allow in outgoing responses. Lowercase, like x-matches-remove-prefix-but-okay.
Header with match prefixes to allow in outgoing responses. Lowercase, like x-matches-remove-prefix-but-okay-*.
Headers to remove from outgoing responses. Lowercase, like x-account-meta-temp-url-key.
Header with match prefixes to remove from outgoing responses. Lowercase, like x-account-meta-private-*.
Returns the WSGI filter for use with paste.deploy.
Default headers to remove from incoming requests. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match. DEFAULT_INCOMING_ALLOW_HEADERS is a list of exceptions to these removals.
Default headers as exceptions to DEFAULT_INCOMING_REMOVE_HEADERS. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match.
Default headers to remove from outgoing responses. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match. DEFAULT_OUTGOING_ALLOW_HEADERS is a list of exceptions to these removals.
Default headers as exceptions to DEFAULT_OUTGOING_REMOVE_HEADERS. Simply a whitespace delimited list of header names and names can optionally end with ‘*’ to indicate a prefix match.