Proxy backend that mirrors writes to several internal backends. More...

Inheritance diagram for FileBackendMultiWrite:

Collaboration diagram for FileBackendMultiWrite:

Public Member Functions
	__construct (array $config)
	Construct a proxy backend that consists of several internal backends.
	accessibilityCheck (array $paths)
	Check that a set of file paths are usable across all internal backends.
	clearCache (array $paths=null)
	Invalidate any in-process file stat and property cache.
	concatenate (array $params)
	Concatenate a list of storage files into a single file system file.
	consistencyCheck (array $paths)
	Check that a set of files are consistent across all internal backends.
	directoryExists (array $params)
	Check if a directory exists at a given storage path.
	fileExists (array $params)
	Check if a file exists at a storage path in the backend.
	getDirectoryList (array $params)
	Get an iterator to list all directories under a storage directory.
	getFeatures ()
	Get the a bitfield of extra features supported by the backend medium.
	getFileContentsMulti (array $params)
	Like getFileContents() except it takes an array of storage paths and returns a map of storage paths to strings (or null on failure).
	getFileHttpUrl (array $params)
	Return an HTTP URL to a given file that requires no authentication to use.
	getFileList (array $params)
	Get an iterator to list all stored files under a storage directory.
	getFileProps (array $params)
	Get the properties of the file at a storage path in the backend.
	getFileSha1Base36 (array $params)
	Get a SHA-1 hash of the file at a storage path in the backend.
	getFileSize (array $params)
	Get the size (bytes) of a file at a storage path in the backend.
	getFileStat (array $params)
	Get quick information about a file at a storage path in the backend.
	getFileTimestamp (array $params)
	Get the last-modified timestamp of the file at a storage path.
	getFileXAttributes (array $params)
	Get metadata about a file at a storage path in the backend.
	getLocalCopyMulti (array $params)
	Like getLocalCopy() except it takes an array of storage paths and returns a map of storage paths to TempFSFile objects (or null on failure).
	getLocalReferenceMulti (array $params)
	Like getLocalReference() except it takes an array of storage paths and returns a map of storage paths to FSFile objects (or null on failure).
	getScopedLocksForOps (array $ops, Status $status)
	Get an array of scoped locks needed for a batch of file operations.
	preloadCache (array $paths)
	Preload persistent file stat cache and property cache into in-process cache.
	preloadFileStat (array $params)
	Preload file stat information (concurrently if possible) into in-process cache.
	resyncFiles (array $paths)
	Check that a set of files are consistent across all internal backends and re-synchronize those files againt the "multi master" if needed.
	streamFile (array $params)
	Stream the file at a storage path in the backend.
Public Attributes
const	CHECK_SHA1 = 4
const	CHECK_SIZE = 1
const	CHECK_TIME = 2
Protected Member Functions
	doClean (array $params)
	doOperationsInternal (array $ops, array $opts)
	doPrepare (array $params)
	doPublish (array $params)
	doQuickOperationsInternal (array $ops)
	doSecure (array $params)
	fileStoragePathsForOps (array $ops)
	Get a list of file storage paths to read or write for a list of operations.
	replicateContainerDirChanges ($path)
	substOpBatchPaths (array $ops, FileBackendStore $backend)
	Substitute the backend name in storage path parameters for a set of operations with that of a given internal backend.
	substOpPaths (array $ops, FileBackendStore $backend)
	Same as substOpBatchPaths() but for a single operation.
	substPaths ($paths, FileBackendStore $backend)
	Substitute the backend of storage paths with an internal backend's name.
	unsubstPaths ($paths)
	Substitute the backend of internal storage paths with the proxy backend's name.
Protected Attributes
string bool	$autoResync = false
	*
array	$backends = array()
	Prioritized list of FileBackendStore objects.
int	$masterIndex = -1
	Index of master backend *.
array	$noPushDirConts = array()
	*
bool	$noPushQuickOps = false
	*
int	$syncChecks = 0
	Bitfield *.

Detailed Description

Proxy backend that mirrors writes to several internal backends.

This class defines a multi-write backend. Multiple backends can be registered to this proxy backend and it will act as a single backend. Use this when all access to those backends is through this proxy backend. At least one of the backends must be declared the "master" backend.

Only use this class when transitioning from one storage system to another.

Read operations are only done on the 'master' backend for consistency. Write operations are performed on all backends, in the order defined. If an operation fails on one backend it will be rolled back from the others.

Since:: 1.19

Definition at line 42 of file FileBackendMultiWrite.php.

Constructor & Destructor Documentation

FileBackendMultiWrite::__construct ( array $ config )

Construct a proxy backend that consists of several internal backends.

Locking, journaling, and read-only checks are handled by the proxy backend.

Additional $config params include:

backends : Array of backend config and multi-backend settings. Each value is the config used in the constructor of a FileBackendStore class, but with these additional settings:
- class : The name of the backend class
- isMultiMaster : This must be set for one backend.
- template: : If given a backend name, this will use the config of that backend as a template. Values specified here take precedence.
syncChecks : Integer bitfield of internal backend sync checks to perform. Possible bits include the FileBackendMultiWrite::CHECK_* constants. There are constants for SIZE, TIME, and SHA1. The checks are done before allowing any file operations.
autoResync : Automatically resync the clone backends to the master backend when pre-operation sync checks fail. This should only be used if the master backend is stable and not missing any files. Use "conservative" to limit resyncing to copying newer master backend files over older (or non-existing) clone backend files. Cases that cannot be handled will result in operation abortion.
noPushQuickOps : (hack) Only apply doQuickOperations() to the master backend.
noPushDirConts : (hack) Only apply directory functions to the master backend.

Parameters:

array $config

Exceptions:

FileBackendError

Reimplemented from FileBackend.

Definition at line 91 of file FileBackendMultiWrite.php.

Member Function Documentation

FileBackendMultiWrite::accessibilityCheck ( array $ paths )

Check that a set of file paths are usable across all internal backends.

Parameters:

array $paths List of storage paths

Returns:: Status

Definition at line 276 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::clearCache ( array $ paths = null )

Invalidate any in-process file stat and property cache.

If $paths is given, then only the cache for those files will be cleared.

See also:: FileBackend::getFileStat()

Parameters:

array $paths Storage paths (optional)

Reimplemented from FileBackend.

Definition at line 652 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::concatenate ( array $ params )

Concatenate a list of storage files into a single file system file.

The target path should refer to a file that is already locked or otherwise safe from modification from other processes. Normally, the file will be a new temp file, which should be adequate.

Parameters:

array

$params

Operation parameters, include:

srcs : ordered source storage paths (e.g. chunk1, chunk2, ...)
dst : file system path to 0-byte temp file
parallelize : try to do operations in parallel when possible

Returns:: Status

Reimplemented from FileBackend.

Definition at line 533 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::consistencyCheck ( array $ paths )

Check that a set of files are consistent across all internal backends.

Parameters:

array $paths List of storage paths

Returns:: Status

Definition at line 210 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::directoryExists ( array $ params )

Check if a directory exists at a given storage path.

Backends using key/value stores will check if the path is a virtual directory, meaning there are files under the given directory.

Storage backends with eventual consistency might return stale data.

Parameters:

array

$params

Parameters include:

dir : storage directory

Returns:: bool|null Returns null on failure

Since:: 1.20

Reimplemented from FileBackend.

Definition at line 630 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::doClean ( array $ params ) [protected]

See also:: FileBackend::clean()

Reimplemented from FileBackend.

Definition at line 520 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::doOperationsInternal	(	array $	ops,
		array $	opts
	)		`[final, protected]`

See also:: FileBackend::doOperations()

Reimplemented from FileBackend.

Definition at line 143 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::doPrepare ( array $ params ) [protected]

See also:: FileBackend::prepare()

Reimplemented from FileBackend.

Definition at line 481 of file FileBackendMultiWrite.php.

References list, and FileBackend\splitStoragePath().

FileBackendMultiWrite::doPublish ( array $ params ) [protected]

See also:: FileBackend::publish()

Reimplemented from FileBackend.

Definition at line 507 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::doQuickOperationsInternal ( array $ ops ) [protected]

See also:: FileBackend::doQuickOperations()

Since:: 1.20

Reimplemented from FileBackend.

Definition at line 446 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::doSecure ( array $ params ) [protected]

See also:: FileBackend::secure()

Reimplemented from FileBackend.

Definition at line 494 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::fileExists ( array $ params )

Check if a file exists at a storage path in the backend.

This returns false if only a directory exists at the path.

Parameters:

array

$params

Parameters include:

src : source storage path
latest : use the latest available data

Returns:: bool|null Returns null on failure

Reimplemented from FileBackend.

Definition at line 540 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::fileStoragePathsForOps ( array $ ops ) [protected]

Get a list of file storage paths to read or write for a list of operations.

Parameters:

array $ops Same format as doOperations()

Returns:: array List of storage paths to files (does not include directories)

Definition at line 358 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::getDirectoryList ( array $ params )

Get an iterator to list *all* directories under a storage directory.

If the directory is of the form "mwstore://backend/container", then all directories in the container will be listed. If the directory is of form "mwstore://backend/container/dir", then all directories directly under that directory will be listed. Results will be storage directories relative to the given directory.

Storage backends with eventual consistency might return stale data.

Failures during iteration can result in FileBackendError exceptions (since 1.22).

Parameters:

array

$params

Parameters include:

dir : storage directory
topOnly : only return direct child dirs of the directory

Returns:: Traversable|array|null Returns null on failure

Since:: 1.20

Reimplemented from FileBackend.

Definition at line 636 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getFeatures ( )

Get the a bitfield of extra features supported by the backend medium.

Returns:: int Bitfield of FileBackend::ATTR_* flags

Since:: 1.23

Reimplemented from FileBackend.

Definition at line 648 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getFileContentsMulti ( array $ params )

Like getFileContents() except it takes an array of storage paths and returns a map of storage paths to strings (or null on failure).

The map keys (paths) are in the same order as the provided list of paths.

See also:: FileBackend::getFileContents()

Parameters:

array

$params

Parameters include:

srcs : list of source storage paths
latest : use the latest available data
parallelize : try to do operations in parallel when possible

Returns:: array Map of (path name => string or false on failure)

Since:: 1.20

Reimplemented from FileBackend.

Definition at line 570 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getFileHttpUrl ( array $ params )

Return an HTTP URL to a given file that requires no authentication to use.

The URL may be pre-authenticated (via some token in the URL) and temporary. This will return null if the backend cannot make an HTTP URL for the file.

This is useful for key/value stores when using scripts that seek around large files and those scripts (and the backend) support HTTP Range headers. Otherwise, one would need to use getLocalReference(), which involves loading the entire file on to local disk.

Parameters:

array

$params

Parameters include:

src : source storage path
ttl : lifetime (seconds) if pre-authenticated; default is 1 day

Returns:: string|null

Since:: 1.21

Reimplemented from FileBackend.

Definition at line 624 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::getFileList ( array $ params )

Get an iterator to list *all* stored files under a storage directory.

If the directory is of the form "mwstore://backend/container", then all files in the container will be listed. If the directory is of form "mwstore://backend/container/dir", then all files under that directory will be listed. Results will be storage paths relative to the given directory.

Storage backends with eventual consistency might return stale data.

Failures during iteration can result in FileBackendError exceptions (since 1.22).

Parameters:

array

$params

Parameters include:

dir : storage directory
topOnly : only return direct child files of the directory (since 1.20)
adviseStat : set to true if stat requests will be made on the files (since 1.22)

Returns:: Traversable|array|null Returns null on failure

Reimplemented from FileBackend.

Definition at line 642 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getFileProps ( array $ params )

Get the properties of the file at a storage path in the backend.

This gives the result of FSFile::getProps() on a local copy of the file.

Parameters:

array

$params

Parameters include:

src : source storage path
latest : use the latest available data

Returns:: array Returns FSFile::placeholderProps() on failure

Reimplemented from FileBackend.

Definition at line 588 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getFileSha1Base36 ( array $ params )

Get a SHA-1 hash of the file at a storage path in the backend.

Parameters:

array

$params

Parameters include:

src : source storage path
latest : use the latest available data

Returns:: string|bool Hash string or false on failure

Reimplemented from FileBackend.

Definition at line 582 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::getFileSize ( array $ params )

Get the size (bytes) of a file at a storage path in the backend.

Parameters:

array

$params

Parameters include:

src : source storage path
latest : use the latest available data

Returns:: int|bool Returns false on failure

Reimplemented from FileBackend.

Definition at line 552 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getFileStat ( array $ params )

Get quick information about a file at a storage path in the backend.

If the file does not exist, then this returns false. Otherwise, the result is an associative array that includes:

mtime : the last-modified timestamp (TS_MW)
size : the file size (bytes) Additional values may be included for internal use only.

Parameters:

array

$params

Parameters include:

src : source storage path
latest : use the latest available data

Returns:: array|bool|null Returns null on failure

Reimplemented from FileBackend.

Definition at line 558 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getFileTimestamp ( array $ params )

Get the last-modified timestamp of the file at a storage path.

Parameters:

array

$params

Parameters include:

src : source storage path
latest : use the latest available data

Returns:: string|bool TS_MW timestamp or false on failure

Reimplemented from FileBackend.

Definition at line 546 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getFileXAttributes ( array $ params )

Get metadata about a file at a storage path in the backend.

If the file does not exist, then this returns false. Otherwise, the result is an associative array that includes:

headers : map of HTTP headers used for GET/HEAD requests (name => value)
metadata : map of file metadata (name => value) Metadata keys and headers names will be returned in all lower-case. Additional values may be included for internal use only.

Use FileBackend::hasFeatures() to check how well this is supported.

Parameters:

array

$params

$params include:

src : source storage path
latest : use the latest available data

Returns:: array|bool Returns false on failure

Since:: 1.23

Reimplemented from FileBackend.

Definition at line 564 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getLocalCopyMulti ( array $ params )

Like getLocalCopy() except it takes an array of storage paths and returns a map of storage paths to TempFSFile objects (or null on failure).

The map keys (paths) are in the same order as the provided list of paths.

See also:: FileBackend::getLocalCopy()

Parameters:

array

$params

Parameters include:

srcs : list of source storage paths
latest : use the latest available data
parallelize : try to do operations in parallel when possible

Returns:: array Map of (path name => TempFSFile or null on failure)

Since:: 1.20

Reimplemented from FileBackend.

Definition at line 612 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::getLocalReferenceMulti ( array $ params )

Like getLocalReference() except it takes an array of storage paths and returns a map of storage paths to FSFile objects (or null on failure).

The map keys (paths) are in the same order as the provided list of paths.

See also:: FileBackend::getLocalReference()

Parameters:

array

$params

Parameters include:

srcs : list of source storage paths
latest : use the latest available data
parallelize : try to do operations in parallel when possible

Returns:: array Map of (path name => FSFile or null on failure)

Since:: 1.20

Reimplemented from FileBackend.

Definition at line 600 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::getScopedLocksForOps	(	array $	ops,
		Status $	status
	)

Get an array of scoped locks needed for a batch of file operations.

Normally, FileBackend::doOperations() handles locking, unless the 'nonLocking' param is passed in. This function is useful if you want the files to be locked for a broader scope than just when the files are changing. For example, if you need to update DB metadata, you may want to keep the files locked until finished.

See also:: FileBackend::doOperations()

Parameters:

array	$ops	List of file operations to FileBackend::doOperations()
Status	$status	Status to update on lock/unlock

Returns:: array List of ScopedFileLocks or null values

Since:: 1.20

Reimplemented from FileBackend.

Definition at line 669 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::preloadCache ( array $ paths )

Preload persistent file stat cache and property cache into in-process cache.

This should be used when stat calls will be made on a known list of a many files.

See also:: FileBackend::getFileStat()

Parameters:

array $paths Storage paths

Reimplemented from FileBackend.

Definition at line 659 of file FileBackendMultiWrite.php.

References substPaths().

FileBackendMultiWrite::preloadFileStat ( array $ params )

Preload file stat information (concurrently if possible) into in-process cache.

This should be used when stat calls will be made on a known list of a many files.

See also:: FileBackend::getFileStat()

Parameters:

array

$params

Parameters include:

srcs : list of source storage paths
latest : use the latest available data

Returns:: bool All requests proceeded without I/O errors (since 1.24)

Since:: 1.23

Reimplemented from FileBackend.

Definition at line 664 of file FileBackendMultiWrite.php.

References substPaths().

FileBackendMultiWrite::replicateContainerDirChanges ( $ path ) [protected]

Parameters:

string $path Storage path

Returns:: bool Path container should have dir changes pushed to all backends

Definition at line 475 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::resyncFiles ( array $ paths )

Check that a set of files are consistent across all internal backends and re-synchronize those files againt the "multi master" if needed.

Parameters:

array $paths List of storage paths

Returns:: Status

Definition at line 301 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::streamFile ( array $ params )

Stream the file at a storage path in the backend.

If the file does not exists, an HTTP 404 error will be given. Appropriate HTTP headers (Status, Content-Type, Content-Length) will be sent if streaming began, while none will be sent otherwise. Implementations should flush the output buffer before sending data.

Parameters:

array

$params

Parameters include:

src : source storage path
headers : list of additional HTTP headers to send on success
latest : use the latest available data

Returns:: Status

Reimplemented from FileBackend.

Definition at line 594 of file FileBackendMultiWrite.php.

References substOpPaths().

FileBackendMultiWrite::substOpBatchPaths	(	array $	ops,
		FileBackendStore $	backend
	)		`[protected]`

Substitute the backend name in storage path parameters for a set of operations with that of a given internal backend.

Parameters:

array	$ops	List of file operation arrays
FileBackendStore	$backend

Returns:: array

Definition at line 389 of file FileBackendMultiWrite.php.

FileBackendMultiWrite::substOpPaths	(	array $	ops,
		FileBackendStore $	backend
	)		`[protected]`

Same as substOpBatchPaths() but for a single operation.

Parameters:

array	$ops	File operation array
FileBackendStore	$backend

Returns:: array

Definition at line 411 of file FileBackendMultiWrite.php.

Referenced by directoryExists(), getDirectoryList(), getFeatures(), getFileContentsMulti(), getFileList(), getFileProps(), getFileSize(), getFileStat(), getFileTimestamp(), getFileXAttributes(), getLocalReferenceMulti(), getScopedLocksForOps(), and streamFile().

FileBackendMultiWrite::substPaths	(	$	paths,
		FileBackendStore $	backend
	)		`[protected]`

Substitute the backend of storage paths with an internal backend's name.

Parameters:

array \| string	$paths	List of paths or single string path
FileBackendStore	$backend

Returns:: array|string

Definition at line 424 of file FileBackendMultiWrite.php.

Referenced by preloadCache(), and preloadFileStat().

FileBackendMultiWrite::unsubstPaths ( $ paths ) [protected]

Substitute the backend of internal storage paths with the proxy backend's name.

Parameters:

array | string $paths List of paths or single string path

Returns:: array|string

Definition at line 438 of file FileBackendMultiWrite.php.

Member Data Documentation

string bool FileBackendMultiWrite::$autoResync = false [protected]

*

Definition at line 51 of file FileBackendMultiWrite.php.

array FileBackendMultiWrite::$backends = array() [protected]

Prioritized list of FileBackendStore objects.

array of (backend index => backends)

Definition at line 45 of file FileBackendMultiWrite.php.

int FileBackendMultiWrite::$masterIndex = -1 [protected]

Index of master backend *.

Definition at line 47 of file FileBackendMultiWrite.php.

array FileBackendMultiWrite::$noPushDirConts = array() [protected]

*

Definition at line 53 of file FileBackendMultiWrite.php.

bool FileBackendMultiWrite::$noPushQuickOps = false [protected]

*

Definition at line 55 of file FileBackendMultiWrite.php.

int FileBackendMultiWrite::$syncChecks = 0 [protected]

Bitfield *.

Definition at line 49 of file FileBackendMultiWrite.php.

const FileBackendMultiWrite::CHECK_SHA1 = 4

Definition at line 60 of file FileBackendMultiWrite.php.

const FileBackendMultiWrite::CHECK_SIZE = 1

Definition at line 58 of file FileBackendMultiWrite.php.

const FileBackendMultiWrite::CHECK_TIME = 2

Definition at line 59 of file FileBackendMultiWrite.php.

The documentation for this class was generated from the following file:

includes/filebackend/FileBackendMultiWrite.php

Public Member Functions

Public Attributes

Protected Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation