Class to handle enqueueing and running of background jobs. More...

Inheritance diagram for JobQueue:

Collaboration diagram for JobQueue:

Public Member Functions
	ack (Job $job)
	Acknowledge that a job was completed.
	batchPush (array $jobs, $flags=0)
	Push a batch of jobs into the queue.
	deduplicateRootJob (Job $job)
	Register the "root job" of a given job into the queue for de-duplication.
	delayedJobsEnabled ()
	delete ()
	Deleted all unclaimed and delayed jobs from the queue.
	flushCaches ()
	Clear any process and persistent caches.
	getAbandonedCount ()
	Get the number of acquired jobs that can no longer be attempted.
	getAcquiredCount ()
	Get the number of acquired jobs (these are temporarily out of the queue).
	getAllDelayedJobs ()
	Get an iterator to traverse over all delayed jobs in this queue.
	getAllQueuedJobs ()
	Get an iterator to traverse over all available jobs in this queue.
	getCoalesceLocationInternal ()
	Do not use this function outside of JobQueue/JobQueueGroup.
	getDelayedCount ()
	Get the number of delayed jobs (these are temporarily out of the queue).
	getOrder ()
	getPeriodicTasks ()
	Return a map of task names to task definition maps.
	getSiblingQueueSizes (array $types)
	Check the size of each of the given queues.
	getSiblingQueuesWithJobs (array $types)
	Check whether each of the given queues are empty.
	getSize ()
	Get the number of available (unacquired, non-delayed) jobs in the queue.
	getType ()
	getWiki ()
	isEmpty ()
	Quickly check if the queue has no available (unacquired, non-delayed) jobs.
	pop ()
	Pop a job off of the queue.
	push ($jobs, $flags=0)
	Push one or more jobs into the queue.
	setTestingPrefix ($key)
	Namespace the queue with a key to isolate it for testing.
	waitForBackups ()
	Wait for any slaves or backup servers to catch up.
Static Public Member Functions
static	factory (array $params)
	Get a job queue object of the specified type.
static	incrStats ($key, $type, $delta=1, $wiki=null)
	Call wfIncrStats() for the queue overall and for the queue type.
Public Attributes
const	QOS_ATOMIC = 1
const	ROOTJOB_TTL = 2419200
Protected Member Functions
	__construct (array $params)
	doAck (Job $job)
	doBatchPush (array $jobs, $flags)
	doDeduplicateRootJob (Job $job)
	doDelete ()
	doFlushCaches ()
	doGetAbandonedCount ()
	doGetAcquiredCount ()
	doGetDelayedCount ()
	doGetPeriodicTasks ()
	doGetSiblingQueueSizes (array $types)
	doGetSiblingQueuesWithJobs (array $types)
	doGetSize ()
	doIsEmpty ()
	doIsRootJobOldDuplicate (Job $job)
	doPop ()
	doWaitForBackups ()
	getRootJobCacheKey ($signature)
	isRootJobOldDuplicate (Job $job)
	Check if the "root" job of a given job has been superseded by a newer one.
	optimalOrder ()
	Get the default queue order to use if configuration does not specify one.
	supportedOrders ()
	Get the allowed queue orders for configuration validation.
	supportsDelayedJobs ()
	Find out if delayed jobs are supported for configuration validation.
Protected Attributes
bool	$checkDelay
	Allow delayed jobs *.
int	$claimTTL
	Time to live in seconds *.
BagOStuff	$dupCache
	*
int	$maxTries
	Maximum number of times to try a job *.
string	$order
	Job priority for pop() *.
string	$type
	Job type *.
string	$wiki
	Wiki ID *.

Detailed Description

Class to handle enqueueing and running of background jobs.

Since:: 1.21

Definition at line 31 of file JobQueue.php.

Constructor & Destructor Documentation

JobQueue::__construct ( array $ params ) [protected]

Parameters:

array $params

Exceptions:

MWException

Reimplemented in JobQueueRedis, JobQueueFederated, and JobQueueDB.

Definition at line 54 of file JobQueue.php.

References empty, optimalOrder(), order, supportedOrders(), supportsDelayedJobs(), type, wfGetCache(), and wiki.

Member Function Documentation

JobQueue::ack ( Job $ job ) [final]

Acknowledge that a job was completed.

This does nothing for certain queue classes or if "claimTTL" is not set. Outside callers should use JobQueueGroup::ack() instead of this function.

Parameters:

Job $job

Returns:: void

Exceptions:

MWException

Definition at line 395 of file JobQueue.php.

JobQueue::batchPush	(	array $	jobs,
		$	flags = `0`
	)		`[final]`

Push a batch of jobs into the queue.

This does not require $wgJobClasses to be set for the given job type. Outside callers should use JobQueueGroup::push() instead of this function.

Parameters:

array	$jobs	List of Jobs
int	$flags	Bitfield (supports JobQueue::QOS_ATOMIC)

Returns:: void

Exceptions:

MWException

Definition at line 317 of file JobQueue.php.

JobQueue::deduplicateRootJob ( Job $ job ) [final]

Register the "root job" of a given job into the queue for de-duplication.

This should only be called right *after* all the new jobs have been inserted. This is used to turn older, duplicate, job entries into no-ops. The root job information will remain in the registry until it simply falls out of cache.

This requires that $job has two special fields in the "params" array:

rootJobSignature : hash (e.g. SHA1) that identifies the task
rootJobTimestamp : TS_MW timestamp of this instance of the task

A "root job" is a conceptual job that consist of potentially many smaller jobs that are actually inserted into the queue. For example, "refreshLinks" jobs are spawned when a template is edited. One can think of the task as "update links of pages that use template X" and an instance of that task as a "root job". However, what actually goes into the queue are range and leaf job subtypes. Since these jobs include things like page ID ranges and DB master positions, and can morph into smaller jobs recursively, simple duplicate detection for individual jobs being identical (like that of job_sha1) is not useful.

In the case of "refreshLinks", if these jobs are still in the queue when the template is edited again, we want all of these old refreshLinks jobs for that template to become no-ops. This can greatly reduce server load, since refreshLinks jobs involves parsing. Essentially, the new batch of jobs belong to a new "root job" and the older ones to a previous "root job" for the same task of "update links of pages that use template X".

This does nothing for certain queue classes.

Parameters:

Job $job

Exceptions:

MWException

Returns:: bool

Definition at line 441 of file JobQueue.php.

JobQueue::delayedJobsEnabled ( ) [final]

Returns:: bool Whether delayed jobs are enabled

Since:: 1.22

Definition at line 143 of file JobQueue.php.

JobQueue::delete ( ) [final]

Deleted all unclaimed and delayed jobs from the queue.

Exceptions:

JobQueueError

Since:: 1.22

Returns:: void

Definition at line 533 of file JobQueue.php.

JobQueue::doAck ( Job $ job ) [abstract, protected]

See also:: JobQueue::ack()

Parameters:

Job $job

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

JobQueue::doBatchPush	(	array $	jobs,
		$	flags
	)		`[abstract, protected]`

See also:: JobQueue::batchPush()

Parameters:

array	$jobs
int	$flags

Reimplemented in JobQueueFederated, JobQueueDB, and JobQueueRedis.

JobQueue::doDeduplicateRootJob ( Job $ job ) [protected]

See also:: JobQueue::deduplicateRootJob()

Parameters:

Job $job

Exceptions:

MWException

Returns:: bool

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 458 of file JobQueue.php.

References $key, $timestamp, getRootJobCacheKey(), Job\getRootJobParams(), Job\hasRootJobParams(), and ROOTJOB_TTL.

JobQueue::doDelete ( ) [protected]

See also:: JobQueue::delete()

Exceptions:

MWException

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 543 of file JobQueue.php.

JobQueue::doFlushCaches ( ) [protected]

See also:: JobQueue::flushCaches()

Returns:: void

Reimplemented in JobQueueDB, and JobQueueFederated.

Definition at line 612 of file JobQueue.php.

JobQueue::doGetAbandonedCount ( ) [protected]

See also:: JobQueue::getAbandonedCount()

Returns:: int

Reimplemented in JobQueueFederated, JobQueueRedis, and JobQueueDB.

Definition at line 289 of file JobQueue.php.

JobQueue::doGetAcquiredCount ( ) [abstract, protected]

See also:: JobQueue::getAcquiredCount()

Returns:: int

Reimplemented in JobQueueFederated, JobQueueRedis, and JobQueueDB.

JobQueue::doGetDelayedCount ( ) [protected]

See also:: JobQueue::getDelayedCount()

Returns:: int

Reimplemented in JobQueueFederated, and JobQueueRedis.

Definition at line 264 of file JobQueue.php.

JobQueue::doGetPeriodicTasks ( ) [protected]

See also:: JobQueue::getPeriodicTasks()

Returns:: array

Reimplemented in JobQueueRedis, JobQueueDB, and JobQueueFederated.

Definition at line 593 of file JobQueue.php.

JobQueue::doGetSiblingQueueSizes ( array $ types ) [protected]

See also:: JobQueue::getSiblingQueuesSize()

Parameters:

array $types List of queues types

Returns:: array|null (list of queue types) or null if unsupported

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 692 of file JobQueue.php.

JobQueue::doGetSiblingQueuesWithJobs ( array $ types ) [protected]

See also:: JobQueue::getSiblingQueuesWithJobs()

Parameters:

array $types List of queues types

Returns:: array|null (list of queue types) or null if unsupported

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 667 of file JobQueue.php.

JobQueue::doGetSize ( ) [abstract, protected]

See also:: JobQueue::getSize()

Returns:: int

Reimplemented in JobQueueFederated, JobQueueRedis, and JobQueueDB.

JobQueue::doIsEmpty ( ) [abstract, protected]

See also:: JobQueue::isEmpty()

Returns:: bool

Reimplemented in JobQueueFederated, JobQueueRedis, and JobQueueDB.

JobQueue::doIsRootJobOldDuplicate ( Job $ job ) [protected]

See also:: JobQueue::isRootJobOldDuplicate()

Parameters:

Job $job

Returns:: bool

Reimplemented in JobQueueRedis, and JobQueueFederated.

Definition at line 502 of file JobQueue.php.

JobQueue::doPop ( ) [abstract, protected]

See also:: JobQueue::pop()

Returns:: Job

Reimplemented in JobQueueFederated, JobQueueRedis, and JobQueueDB.

JobQueue::doWaitForBackups ( ) [protected]

See also:: JobQueue::waitForBackups()

Returns:: void

Reimplemented in JobQueueDB, and JobQueueFederated.

Definition at line 565 of file JobQueue.php.

static JobQueue::factory ( array $ params ) [static, final]

Get a job queue object of the specified type.

$params includes:

class : What job class to use (determines job type)
wiki : wiki ID of the wiki the jobs are for (defaults to current wiki)
type : The name of the job types this queue handles
order : Order that pop() selects jobs, one of "fifo", "timestamp" or "random". If "fifo" is used, the queue will effectively be FIFO. Note that job completion will not appear to be exactly FIFO if there are multiple job runners since jobs can take different times to finish once popped. If "timestamp" is used, the queue will at least be loosely ordered by timestamp, allowing for some jobs to be popped off out of order. If "random" is used, pop() will pick jobs in random order. Note that it may only be weakly random (e.g. a lottery of the oldest X). If "any" is choosen, the queue will use whatever order is the fastest. This might be useful for improving concurrency for job acquisition.
claimTTL : If supported, the queue will recycle jobs that have been popped but not acknowledged as completed after this many seconds. Recycling of jobs simple means re-inserting them into the queue. Jobs can be attempted up to three times before being discarded.
checkDelay : If supported, respect Job::getReleaseTimestamp() in the push functions. This lets delayed jobs wait in a staging area until a given timestamp is reached, at which point they will enter the queue. If this is not enabled or not supported, an exception will be thrown on delayed job insertion.

Queue classes should throw an exception if they do not support the options given.

Parameters:

array $params

Returns:: JobQueue

Exceptions:

MWException

Definition at line 105 of file JobQueue.php.

Referenced by CopyJobQueue\execute(), JobQueueRedis\getJobFromFields(), and JobQueueTest\setUp().

JobQueue::flushCaches ( ) [final]

Clear any process and persistent caches.

Returns:: void

Definition at line 602 of file JobQueue.php.

JobQueue::getAbandonedCount ( ) [final]

Get the number of acquired jobs that can no longer be attempted.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns:: int

Exceptions:

JobQueueError

Definition at line 277 of file JobQueue.php.

JobQueue::getAcquiredCount ( ) [final]

Get the number of acquired jobs (these are temporarily out of the queue).

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns:: int

Exceptions:

JobQueueError

Definition at line 228 of file JobQueue.php.

JobQueue::getAllDelayedJobs ( )

Get an iterator to traverse over all delayed jobs in this queue.

Note: results may be stale if the queue is concurrently modified.

Returns:: Iterator

Exceptions:

JobQueueError

Since:: 1.22

Reimplemented in JobQueueRedis, and JobQueueFederated.

Definition at line 633 of file JobQueue.php.

JobQueue::getAllQueuedJobs ( ) [abstract]

Get an iterator to traverse over all available jobs in this queue.

This does not include jobs that are currently acquired or delayed. Note: results may be stale if the queue is concurrently modified.

Returns:: Iterator

Exceptions:

JobQueueError

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

JobQueue::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Returns:: string

Since:: 1.22

Reimplemented in JobQueueDB, JobQueueRedis, and JobQueueFederated.

Definition at line 643 of file JobQueue.php.

JobQueue::getDelayedCount ( ) [final]

Get the number of delayed jobs (these are temporarily out of the queue).

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns:: int

Exceptions:

JobQueueError

Since:: 1.22

Definition at line 252 of file JobQueue.php.

JobQueue::getOrder ( ) [final]

Returns:: string One of (random, timestamp, fifo, undefined)

Definition at line 135 of file JobQueue.php.

References type.

JobQueue::getPeriodicTasks ( ) [final]

Return a map of task names to task definition maps.

A "task" is a fast periodic queue maintenance action. Mutually exclusive tasks must implement their own locking in the callback.

Each task value is an associative array with:

name : the name of the task
callback : a PHP callable that performs the task
period : the period in seconds corresponding to the task frequency

Returns:: array

Definition at line 580 of file JobQueue.php.

JobQueue::getRootJobCacheKey ( $ signature ) [protected]

Parameters:

string $signature Hash identifier of the root job

Returns:: string

Definition at line 520 of file JobQueue.php.

Referenced by doDeduplicateRootJob().

JobQueue::getSiblingQueueSizes ( array $ types ) [final]

Check the size of each of the given queues.

For queues not served by the same store as this one, 0 is returned. This is used for batching checks for queues stored at the same place.

Parameters:

array $types List of queues types

Returns:: array|null (job type => whether queue is empty) or null if unsupported

Exceptions:

MWException

Since:: 1.22

Definition at line 681 of file JobQueue.php.

JobQueue::getSiblingQueuesWithJobs ( array $ types ) [final]

Check whether each of the given queues are empty.

This is used for batching checks for queues stored at the same place.

Parameters:

array $types List of queues types

Returns:: array|null (list of non-empty queue types) or null if unsupported

Exceptions:

MWException

Since:: 1.22

Definition at line 656 of file JobQueue.php.

JobQueue::getSize ( ) [final]

Get the number of available (unacquired, non-delayed) jobs in the queue.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns:: int

Exceptions:

JobQueueError

Definition at line 205 of file JobQueue.php.

JobQueue::getType ( ) [final]

Returns:: string Job type that this queue handles

Definition at line 128 of file JobQueue.php.

References wiki.

JobQueue::getWiki ( ) [final]

Returns:: string Wiki ID

Definition at line 121 of file JobQueue.php.

static JobQueue::incrStats	(	$	key,
		$	type,
		$	delta = `1`,
		$	wiki = `null`
	)		`[static]`

Call wfIncrStats() for the queue overall and for the queue type.

Parameters:

string	$key	Event type
string	$type	Job type
int	$delta
string	$wiki	Wiki ID (added in 1.23)

Since:: 1.22

Definition at line 705 of file JobQueue.php.

JobQueue::isEmpty ( ) [final]

Quickly check if the queue has no available (unacquired, non-delayed) jobs.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this might return false when there are actually no jobs. If pop() is called and returns false then it should correct the cache. Also, calling flushCaches() first prevents this. However, this affect is typically not distinguishable from the race condition between isEmpty() and pop().