Class to handle enqueueing and running of background jobs for federated queues. More...

Inheritance diagram for JobQueueFederated:

Collaboration diagram for JobQueueFederated:

Public Member Functions
	getAllDelayedJobs ()
	Get an iterator to traverse over all delayed jobs in this queue.
	getAllQueuedJobs ()
	Get an iterator to traverse over all available jobs in this queue.
	getCoalesceLocationInternal ()
	Do not use this function outside of JobQueue/JobQueueGroup.
	setTestingPrefix ($key)
	Namespace the queue with a key to isolate it for testing.
Public Attributes
const	CACHE_TTL_LONG = 300
const	CACHE_TTL_SHORT = 30
Protected Member Functions
	__construct (array $params)
	doAck (Job $job)
	doBatchPush (array $jobs, $flags)
	doDeduplicateRootJob (Job $job)
	doDelete ()
	doFlushCaches ()
	doGetAbandonedCount ()
	doGetAcquiredCount ()
	doGetDelayedCount ()
	doGetPeriodicTasks ()
	doGetSiblingQueueSizes (array $types)
	doGetSiblingQueuesWithJobs (array $types)
	doGetSize ()
	doIsEmpty ()
	doIsRootJobOldDuplicate (Job $job)
	doPop ()
	doWaitForBackups ()
	getCrossPartitionSum ($type, $method)
	optimalOrder ()
	Get the default queue order to use if configuration does not specify one.
	supportedOrders ()
	Get the allowed queue orders for configuration validation.
	supportsDelayedJobs ()
	Find out if delayed jobs are supported for configuration validation.
	throwErrorIfAllPartitionsDown ($down)
	Throw an error if no partitions available.
	tryJobInsertions (array $jobs, HashRing &$partitionRing, $flags)
Protected Attributes
BagOStuff	$cache
	*
int	$maxPartitionsTry
	Maximum number of partitions to try *.
HashRing	$partitionPushRing
	*
array	$partitionQueues = array()
	(partition name => JobQueue) reverse sorted by weight *
HashRing	$partitionRing
	*
Private Member Functions
	getCacheKey ($property)

Detailed Description

Class to handle enqueueing and running of background jobs for federated queues.

This class allows for queues to be partitioned into smaller queues. A partition is defined by the configuration for a JobQueue instance. For example, one can set $wgJobTypeConf['refreshLinks'] to point to a JobQueueFederated instance, which itself would consist of three JobQueueRedis instances, each using their own redis server. This would allow for the jobs to be split (evenly or based on weights) accross multiple servers if a single server becomes impractical or expensive. Different JobQueue classes can be mixed.

The basic queue configuration (e.g. "order", "claimTTL") of a federated queue is inherited by the partition queues. Additional configuration defines what section each wiki is in, what partition queues each section uses (and their weight), and the JobQueue configuration for each partition. Some sections might only need a single queue partition, like the sections for groups of small wikis.

If used for performance, then $wgMainCacheType should be set to memcached/redis. Note that "fifo" cannot be used for the ordering, since the data is distributed. One can still use "timestamp" instead, as in "roughly timestamp ordered". Also, queue classes used by this should ignore down servers (with TTL) to avoid slowness.

Since:: 1.22

Definition at line 49 of file JobQueueFederated.php.

Constructor & Destructor Documentation

JobQueueFederated::__construct ( array $ params ) [protected]

Parameters:

array

$params

Possible keys:

sectionsByWiki : A map of wiki IDs to section names. Wikis will default to using the section "default".
partitionsBySection : Map of section names to maps of (partition name => weight). A section called 'default' must be defined if not all wikis have explicitly defined sections.
configByPartition : Map of queue partition names to configuration arrays. These configuration arrays are passed to JobQueue::factory(). The options set here are overriden by those passed to this the federated queue itself (e.g. 'order' and 'claimTTL').
partitionsNoPush : List of partition names that can handle pop() but not push(). This can be used to migrate away from a certain partition.
maxPartitionsTry : Maximum number of times to attempt job insertion using different partition queues. This improves availability during failure, at the cost of added latency and somewhat less reliable job de-duplication mechanisms.

Exceptions:

MWException

Reimplemented from JobQueue.

Definition at line 80 of file JobQueueFederated.php.

Member Function Documentation

JobQueueFederated::doAck ( Job $ job ) [protected]

See also:: JobQueue::ack()

Parameters:

Job $job

Reimplemented from JobQueue.

Definition at line 348 of file JobQueueFederated.php.

JobQueueFederated::doBatchPush	(	array $	jobs,
		$	flags
	)		`[protected]`

See also:: JobQueue::batchPush()

Parameters:

array	$jobs
int	$flags

Reimplemented from JobQueue.

Definition at line 212 of file JobQueueFederated.php.

JobQueueFederated::doDeduplicateRootJob ( Job $ job ) [protected]

See also:: JobQueue::deduplicateRootJob()

Parameters:

Job $job

Exceptions:

MWException

Returns:: bool

Reimplemented from JobQueue.

Definition at line 372 of file JobQueueFederated.php.

JobQueueFederated::doDelete ( ) [protected]

See also:: JobQueue::delete()

Exceptions:

MWException

Reimplemented from JobQueue.

Definition at line 388 of file JobQueueFederated.php.

JobQueueFederated::doFlushCaches ( ) [protected]

See also:: JobQueue::flushCaches()

Returns:: void

Reimplemented from JobQueue.

Definition at line 429 of file JobQueueFederated.php.

JobQueueFederated::doGetAbandonedCount ( ) [protected]

See also:: JobQueue::getAbandonedCount()

Returns:: int

Reimplemented from JobQueue.

Definition at line 179 of file JobQueueFederated.php.

References getCrossPartitionSum().

JobQueueFederated::doGetAcquiredCount ( ) [protected]

See also:: JobQueue::getAcquiredCount()

Returns:: int

Reimplemented from JobQueue.

Definition at line 171 of file JobQueueFederated.php.

References getCrossPartitionSum().

JobQueueFederated::doGetDelayedCount ( ) [protected]

See also:: JobQueue::getDelayedCount()

Returns:: int

Reimplemented from JobQueue.

Definition at line 175 of file JobQueueFederated.php.

References getCrossPartitionSum().

JobQueueFederated::doGetPeriodicTasks ( ) [protected]

See also:: JobQueue::getPeriodicTasks()

Returns:: array

Reimplemented from JobQueue.

Definition at line 417 of file JobQueueFederated.php.

JobQueueFederated::doGetSiblingQueueSizes ( array $ types ) [protected]

See also:: JobQueue::getSiblingQueuesSize()

Parameters:

array $types List of queues types

Returns:: array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 501 of file JobQueueFederated.php.

JobQueueFederated::doGetSiblingQueuesWithJobs ( array $ types ) [protected]

See also:: JobQueue::getSiblingQueuesWithJobs()

Parameters:

array $types List of queues types

Returns:: array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 475 of file JobQueueFederated.php.

JobQueueFederated::doGetSize ( ) [protected]

See also:: JobQueue::getSize()

Returns:: int

Reimplemented from JobQueue.

Definition at line 167 of file JobQueueFederated.php.

JobQueueFederated::doIsEmpty ( ) [protected]

See also:: JobQueue::isEmpty()

Returns:: bool

Reimplemented from JobQueue.

Definition at line 141 of file JobQueueFederated.php.

JobQueueFederated::doIsRootJobOldDuplicate ( Job $ job ) [protected]

See also:: JobQueue::isRootJobOldDuplicate()

Parameters:

Job $job

Returns:: bool

Reimplemented from JobQueue.

Definition at line 356 of file JobQueueFederated.php.

JobQueueFederated::doPop ( ) [protected]

See also:: JobQueue::pop()

Returns:: Job

Reimplemented from JobQueue.

Definition at line 313 of file JobQueueFederated.php.

JobQueueFederated::doWaitForBackups ( ) [protected]

See also:: JobQueue::waitForBackups()

Returns:: void

Reimplemented from JobQueue.

Definition at line 403 of file JobQueueFederated.php.

JobQueueFederated::getAllDelayedJobs ( )

Get an iterator to traverse over all delayed jobs in this queue.

Note: results may be stale if the queue is concurrently modified.

Returns:: Iterator

Exceptions:

JobQueueError

Since:: 1.22

Reimplemented from JobQueue.

Definition at line 459 of file JobQueueFederated.php.

JobQueueFederated::getAllQueuedJobs ( )

Get an iterator to traverse over all available jobs in this queue.

This does not include jobs that are currently acquired or delayed. Note: results may be stale if the queue is concurrently modified.

Returns:: Iterator

Exceptions:

JobQueueError

Reimplemented from JobQueue.

Definition at line 448 of file JobQueueFederated.php.

JobQueueFederated::getCacheKey ( $ property ) [private]

Parameters:

string $property

Returns:: string

Definition at line 549 of file JobQueueFederated.php.

References $property, list, type, wfForeignMemcKey(), and wfSplitWikiID().

JobQueueFederated::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Returns:: string

Since:: 1.22

Reimplemented from JobQueue.

Definition at line 470 of file JobQueueFederated.php.

JobQueueFederated::getCrossPartitionSum	(	$	type,
		$	method
	)		`[protected]`

Parameters:

string	$type
string	$method

Returns:: int

Definition at line 188 of file JobQueueFederated.php.

Referenced by doGetAbandonedCount(), doGetAcquiredCount(), and doGetDelayedCount().

JobQueueFederated::optimalOrder ( ) [protected]

Get the default queue order to use if configuration does not specify one.

Returns:: string One of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 133 of file JobQueueFederated.php.

References array().

JobQueueFederated::setTestingPrefix ( $ key )

Namespace the queue with a key to isolate it for testing.

Parameters:

string $key

Returns:: void

Exceptions:

MWException

Reimplemented from JobQueue.

Definition at line 538 of file JobQueueFederated.php.

JobQueueFederated::supportedOrders ( ) [protected]

Get the allowed queue orders for configuration validation.

Returns:: array Subset of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 128 of file JobQueueFederated.php.

JobQueueFederated::supportsDelayedJobs ( ) [protected]

Find out if delayed jobs are supported for configuration validation.

Returns:: bool Whether delayed jobs are supported

Reimplemented from JobQueue.

Definition at line 137 of file JobQueueFederated.php.

JobQueueFederated::throwErrorIfAllPartitionsDown ( $ down ) [protected]

Throw an error if no partitions available.

Parameters:

int $down The number of up partitions down

Returns:: void

Exceptions:

JobQueueError

Definition at line 532 of file JobQueueFederated.php.

JobQueueFederated::tryJobInsertions	(	array $	jobs,
		HashRing &$	partitionRing,
		$	flags
	)		`[protected]`

Parameters:

array	$jobs
HashRing	$partitionRing
int	$flags

Exceptions:

JobQueueError

Returns:: array List of Job object that could not be inserted

Definition at line 241 of file JobQueueFederated.php.

Member Data Documentation

BagOStuff JobQueueFederated::$cache [protected]

*

Definition at line 54 of file JobQueueFederated.php.

int JobQueueFederated::$maxPartitionsTry [protected]

Maximum number of partitions to try *.

Definition at line 56 of file JobQueueFederated.php.

HashRing JobQueueFederated::$partitionPushRing [protected]

*

Definition at line 51 of file JobQueueFederated.php.

array JobQueueFederated::$partitionQueues = array() [protected]

(partition name => JobQueue) reverse sorted by weight *

Definition at line 52 of file JobQueueFederated.php.

HashRing JobQueueFederated::$partitionRing [protected]

*

Definition at line 50 of file JobQueueFederated.php.

const JobQueueFederated::CACHE_TTL_LONG = 300

Definition at line 59 of file JobQueueFederated.php.

const JobQueueFederated::CACHE_TTL_SHORT = 30

Definition at line 58 of file JobQueueFederated.php.

The documentation for this class was generated from the following file:

includes/jobqueue/JobQueueFederated.php

Public Member Functions

Public Attributes

Protected Member Functions

Protected Attributes

Private Member Functions

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation