Class to handle enqueueing and running of background jobs. More...

Inheritance diagram for JobQueue:

Collaboration diagram for JobQueue:

Public Member Functions
	ack (Job $job)
	Acknowledge that a job was completed.
	batchPush (array $jobs, $flags=0)
	Push a batch of jobs into the queue.
	deduplicateRootJob (Job $job)
	Register the "root job" of a given job into the queue for de-duplication.
	delayedJobsEnabled ()
	delete ()
	Deleted all unclaimed and delayed jobs from the queue.
	flushCaches ()
	Clear any process and persistent caches.
	getAbandonedCount ()
	Get the number of acquired jobs that can no longer be attempted.
	getAcquiredCount ()
	Get the number of acquired jobs (these are temporarily out of the queue).
	getAllDelayedJobs ()
	Get an iterator to traverse over all delayed jobs in this queue.
	getAllQueuedJobs ()
	Get an iterator to traverse over all available jobs in this queue.
	getCoalesceLocationInternal ()
	Do not use this function outside of JobQueue/JobQueueGroup.
	getDelayedCount ()
	Get the number of delayed jobs (these are temporarily out of the queue).
	getOrder ()
	getPeriodicTasks ()
	Return a map of task names to task definition maps.
	getSiblingQueueSizes (array $types)
	Check the size of each of the given queues.
	getSiblingQueuesWithJobs (array $types)
	Check whether each of the given queues are empty.
	getSize ()
	Get the number of available (unacquired, non-delayed) jobs in the queue.
	getType ()
	getWiki ()
	isEmpty ()
	Quickly check if the queue has no available (unacquired, non-delayed) jobs.
	pop ()
	Pop a job off of the queue.
	push ($jobs, $flags=0)
	Push a single jobs into the queue.
	setTestingPrefix ($key)
	Namespace the queue with a key to isolate it for testing.
	waitForBackups ()
	Wait for any slaves or backup servers to catch up.
Static Public Member Functions
static	factory (array $params)
	Get a job queue object of the specified type.
static	incrStats ($key, $type, $delta=1)
	Call wfIncrStats() for the queue overall and for the queue type.
Public Attributes
const	QOS_ATOMIC = 1
const	ROOTJOB_TTL = 2419200
Protected Member Functions
	__construct (array $params)
	doAck (Job $job)
	doBatchPush (array $jobs, $flags)
	doDeduplicateRootJob (Job $job)
	doDelete ()
	doFlushCaches ()
	doGetAbandonedCount ()
	doGetAcquiredCount ()
	doGetDelayedCount ()
	doGetPeriodicTasks ()
	doGetSiblingQueueSizes (array $types)
	doGetSiblingQueuesWithJobs (array $types)
	doGetSize ()
	doIsEmpty ()
	doIsRootJobOldDuplicate (Job $job)
	doPop ()
	doWaitForBackups ()
	getRootJobCacheKey ($signature)
	isRootJobOldDuplicate (Job $job)
	Check if the "root" job of a given job has been superseded by a newer one.
	optimalOrder ()
	Get the default queue order to use if configuration does not specify one.
	supportedOrders ()
	Get the allowed queue orders for configuration validation.
	supportsDelayedJobs ()
	Find out if delayed jobs are supported for configuration validation.
Protected Attributes
	$checkDelay
	$claimTTL
BagOStuff	$dupCache
	*
	$maxTries
	$order
	$type
	$wiki

Detailed Description

Class to handle enqueueing and running of background jobs.

Since:: 1.21

Definition at line 31 of file JobQueue.php.

Constructor & Destructor Documentation

JobQueue::__construct ( array $ params ) [protected]

Parameters:

$params array

Reimplemented in JobQueueRedis, JobQueueFederated, and JobQueueDB.

Definition at line 48 of file JobQueue.php.

Member Function Documentation

JobQueue::ack ( Job $ job ) [final]

Acknowledge that a job was completed.

This does nothing for certain queue classes or if "claimTTL" is not set. Outside callers should use JobQueueGroup::ack() instead of this function.

Parameters:

$job Job

Returns:: bool

Exceptions:

JobQueueError

Definition at line 381 of file JobQueue.php.

JobQueue::batchPush	(	array $	jobs,
		$	flags = `0`
	)		`[final]`

Push a batch of jobs into the queue.

This does not require $wgJobClasses to be set for the given job type. Outside callers should use JobQueueGroup::push() instead of this function.

Parameters:

array	$jobs	List of Jobs
	$flags	integer Bitfield (supports JobQueue::QOS_ATOMIC)

Returns:: bool Returns false on failure

Exceptions:

JobQueueError

Definition at line 305 of file JobQueue.php.

JobQueue::deduplicateRootJob ( Job $ job ) [final]

Register the "root job" of a given job into the queue for de-duplication.

This should only be called right *after* all the new jobs have been inserted. This is used to turn older, duplicate, job entries into no-ops. The root job information will remain in the registry until it simply falls out of cache.

This requires that $job has two special fields in the "params" array:

rootJobSignature : hash (e.g. SHA1) that identifies the task
rootJobTimestamp : TS_MW timestamp of this instance of the task

A "root job" is a conceptual job that consist of potentially many smaller jobs that are actually inserted into the queue. For example, "refreshLinks" jobs are spawned when a template is edited. One can think of the task as "update links of pages that use template X" and an instance of that task as a "root job". However, what actually goes into the queue are potentially many refreshLinks2 jobs. Since these jobs include things like page ID ranges and DB master positions, and morph into smaller refreshLinks2 jobs recursively, simple duplicate detection (like job_sha1) for individual jobs being identical is not useful.

In the case of "refreshLinks", if these jobs are still in the queue when the template is edited again, we want all of these old refreshLinks jobs for that template to become no-ops. This can greatly reduce server load, since refreshLinks jobs involves parsing. Essentially, the new batch of jobs belong to a new "root job" and the older ones to a previous "root job" for the same task of "update links of pages that use template X".

This does nothing for certain queue classes.

Parameters:

$job Job

Returns:: bool

Exceptions:

JobQueueError

Definition at line 428 of file JobQueue.php.

JobQueue::delayedJobsEnabled ( ) [final]

Returns:: bool Whether delayed jobs are enabled

Since:: 1.22

Definition at line 136 of file JobQueue.php.

JobQueue::delete ( ) [final]

Deleted all unclaimed and delayed jobs from the queue.

Returns:: bool Success

Exceptions:

JobQueueError

Since:: 1.22

Definition at line 516 of file JobQueue.php.

JobQueue::doAck ( Job $ job ) [abstract, protected]

See also:: JobQueue::ack()

Returns:: bool

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

JobQueue::doBatchPush	(	array $	jobs,
		$	flags
	)		`[abstract, protected]`

See also:: JobQueue::batchPush()

Returns:: bool

Reimplemented in JobQueueDB, JobQueueFederated, and JobQueueRedis.

JobQueue::doDeduplicateRootJob ( Job $ job ) [protected]

See also:: JobQueue::deduplicateRootJob()

Parameters:

$job Job

Returns:: bool

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 443 of file JobQueue.php.

JobQueue::doDelete ( ) [protected]

See also:: JobQueue::delete()

Returns:: bool Success

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 527 of file JobQueue.php.

JobQueue::doFlushCaches ( ) [protected]

See also:: JobQueue::flushCaches()

Returns:: void

Reimplemented in JobQueueDB, and JobQueueFederated.

Definition at line 594 of file JobQueue.php.

JobQueue::doGetAbandonedCount ( ) [protected]

See also:: JobQueue::getAbandonedCount()

Returns:: integer

Reimplemented in JobQueueRedis, JobQueueFederated, and JobQueueDB.

Definition at line 277 of file JobQueue.php.

JobQueue::doGetAcquiredCount ( ) [abstract, protected]

See also:: JobQueue::getAcquiredCount()

Returns:: integer

Reimplemented in JobQueueFederated, JobQueueRedis, and JobQueueDB.

JobQueue::doGetDelayedCount ( ) [protected]

See also:: JobQueue::getDelayedCount()

Returns:: integer

Reimplemented in JobQueueFederated, and JobQueueRedis.

Definition at line 253 of file JobQueue.php.

JobQueue::doGetPeriodicTasks ( ) [protected]

See also:: JobQueue::getPeriodicTasks()

Returns:: Array

Reimplemented in JobQueueRedis, JobQueueDB, and JobQueueFederated.

Definition at line 575 of file JobQueue.php.

JobQueue::doGetSiblingQueueSizes ( array $ types ) [protected]

See also:: JobQueue::getSiblingQueuesSize()

Parameters:

array $types List of queues types

Returns:: array|null (list of queue types) or null if unsupported

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 671 of file JobQueue.php.

JobQueue::doGetSiblingQueuesWithJobs ( array $ types ) [protected]

See also:: JobQueue::getSiblingQueuesWithJobs()

Parameters:

array $types List of queues types

Returns:: array|null (list of queue types) or null if unsupported

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 647 of file JobQueue.php.

JobQueue::doGetSize ( ) [abstract, protected]

See also:: JobQueue::getSize()

Returns:: integer

Reimplemented in JobQueueFederated, JobQueueRedis, and JobQueueDB.

JobQueue::doIsEmpty ( ) [abstract, protected]

See also:: JobQueue::isEmpty()

Returns:: bool

Reimplemented in JobQueueFederated, JobQueueRedis, and JobQueueDB.

JobQueue::doIsRootJobOldDuplicate ( Job $ job ) [protected]

See also:: JobQueue::isRootJobOldDuplicate()

Parameters:

Job $job

Returns:: bool

Reimplemented in JobQueueRedis, and JobQueueFederated.

Definition at line 486 of file JobQueue.php.

JobQueue::doPop ( ) [abstract, protected]

See also:: JobQueue::pop()

Returns:: Job

Reimplemented in JobQueueRedis, JobQueueFederated, and JobQueueDB.

JobQueue::doWaitForBackups ( ) [protected]

See also:: JobQueue::waitForBackups()

Returns:: void

Reimplemented in JobQueueDB, and JobQueueFederated.

Definition at line 549 of file JobQueue.php.

static JobQueue::factory ( array $ params ) [static, final]

Get a job queue object of the specified type.

$params includes:

class : What job class to use (determines job type)
wiki : wiki ID of the wiki the jobs are for (defaults to current wiki)
type : The name of the job types this queue handles
order : Order that pop() selects jobs, one of "fifo", "timestamp" or "random". If "fifo" is used, the queue will effectively be FIFO. Note that job completion will not appear to be exactly FIFO if there are multiple job runners since jobs can take different times to finish once popped. If "timestamp" is used, the queue will at least be loosely ordered by timestamp, allowing for some jobs to be popped off out of order. If "random" is used, pop() will pick jobs in random order. Note that it may only be weakly random (e.g. a lottery of the oldest X). If "any" is choosen, the queue will use whatever order is the fastest. This might be useful for improving concurrency for job acquisition.
claimTTL : If supported, the queue will recycle jobs that have been popped but not acknowledged as completed after this many seconds. Recycling of jobs simple means re-inserting them into the queue. Jobs can be attempted up to three times before being discarded.
checkDelay : If supported, respect Job::getReleaseTimestamp() in the push functions. This lets delayed jobs wait in a staging area until a given timestamp is reached, at which point they will enter the queue. If this is not enabled or not supported, an exception will be thrown on delayed job insertion.

Queue classes should throw an exception if they do not support the options given.

Parameters:

$params array

Returns:: JobQueue

Exceptions:

MWException

Definition at line 99 of file JobQueue.php.

Referenced by CopyJobQueue\execute(), and JobQueueTest\setUp().

JobQueue::flushCaches ( ) [final]

Clear any process and persistent caches.

Returns:: void

Definition at line 584 of file JobQueue.php.

JobQueue::getAbandonedCount ( ) [final]

Get the number of acquired jobs that can no longer be attempted.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns:: integer

Exceptions:

JobQueueError

Definition at line 266 of file JobQueue.php.

JobQueue::getAcquiredCount ( ) [final]

Get the number of acquired jobs (these are temporarily out of the queue).

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns:: integer

Exceptions:

JobQueueError

Definition at line 219 of file JobQueue.php.

JobQueue::getAllDelayedJobs ( )

Get an iterator to traverse over all delayed jobs in this queue.

Note: results may be stale if the queue is concurrently modified.

Returns:: Iterator

Exceptions:

JobQueueError

Since:: 1.22

Reimplemented in JobQueueRedis, and JobQueueFederated.

Definition at line 614 of file JobQueue.php.

JobQueue::getAllQueuedJobs ( ) [abstract]

Get an iterator to traverse over all available jobs in this queue.

This does not include jobs that are currently acquired or delayed. Note: results may be stale if the queue is concurrently modified.

Returns:: Iterator

Exceptions:

JobQueueError

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

JobQueue::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Returns:: string

Since:: 1.22

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 624 of file JobQueue.php.

JobQueue::getDelayedCount ( ) [final]

Get the number of delayed jobs (these are temporarily out of the queue).

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns:: integer

Exceptions:

JobQueueError

Since:: 1.22

Definition at line 242 of file JobQueue.php.

JobQueue::getOrder ( ) [final]

Returns:: string One of (random, timestamp, fifo, undefined)

Definition at line 128 of file JobQueue.php.

JobQueue::getPeriodicTasks ( ) [final]

Return a map of task names to task definition maps.

A "task" is a fast periodic queue maintenance action. Mutually exclusive tasks must implement their own locking in the callback.

Each task value is an associative array with:

name : the name of the task
callback : a PHP callable that performs the task
period : the period in seconds corresponding to the task frequency

Returns:: Array

Definition at line 563 of file JobQueue.php.

JobQueue::getRootJobCacheKey ( $ signature ) [protected]

Parameters:

string $signature Hash identifier of the root job

Returns:: string

Definition at line 504 of file JobQueue.php.

JobQueue::getSiblingQueueSizes ( array $ types ) [final]

Check the size of each of the given queues.

For queues not served by the same store as this one, 0 is returned. This is used for batching checks for queues stored at the same place.

Parameters:

array $types List of queues types

Returns:: array|null (job type => whether queue is empty) or null if unsupported

Exceptions:

MWException

Since:: 1.22

Definition at line 661 of file JobQueue.php.

JobQueue::getSiblingQueuesWithJobs ( array $ types ) [final]

Check whether each of the given queues are empty.

This is used for batching checks for queues stored at the same place.

Parameters:

array $types List of queues types

Returns:: array|null (list of non-empty queue types) or null if unsupported

Exceptions:

MWException

Since:: 1.22

Definition at line 637 of file JobQueue.php.

JobQueue::getSize ( ) [final]

Get the number of available (unacquired, non-delayed) jobs in the queue.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns:: integer

Exceptions:

JobQueueError

Definition at line 197 of file JobQueue.php.

JobQueue::getType ( ) [final]

Returns:: string Job type that this queue handles

Definition at line 121 of file JobQueue.php.

JobQueue::getWiki ( ) [final]

Returns:: string Wiki ID

Definition at line 114 of file JobQueue.php.

static JobQueue::incrStats	(	$	key,
		$	type,
		$	delta = `1`
	)		`[static]`

Call wfIncrStats() for the queue overall and for the queue type.

Parameters:

string	$key	Event type
string	$type	Job type
integer	$delta

Since:: 1.22

Definition at line 683 of file JobQueue.php.

JobQueue::isEmpty ( ) [final]

Quickly check if the queue has no available (unacquired, non-delayed) jobs.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this might return false when there are actually no jobs. If pop() is called and returns false then it should correct the cache. Also, calling flushCaches() first prevents this. However, this affect is typically not distinguishable from the race condition between isEmpty() and pop().