Class to handle job queues stored in Redis. More...

Inheritance diagram for JobQueueRedis:

Collaboration diagram for JobQueueRedis:

Public Member Functions
	__construct (array $params)
	getAllDelayedJobs ()
	getAllQueuedJobs ()
	getCoalesceLocationInternal ()
	Do not use this function outside of JobQueue/JobQueueGroup.
	getJobFromUidInternal ($uid, RedisConnRef $conn)
	This function should not be called outside JobQueueRedis.
	recyclePruneAndUndelayJobs ()
	Recycle or destroy any jobs that have been claimed for too long and release any ready delayed jobs into the queue.
	setTestingPrefix ($key)
Public Attributes
const	MAX_AGE_PRUNE = 604800
Protected Member Functions
	doAck (Job $job)
	doBatchPush (array $jobs, $flags)
	doDeduplicateRootJob (Job $job)
	doDelete ()
	doGetAbandonedCount ()
	doGetAcquiredCount ()
	doGetDelayedCount ()
	doGetPeriodicTasks ()
	doGetSiblingQueueSizes (array $types)
	doGetSiblingQueuesWithJobs (array $types)
	doGetSize ()
	doIsEmpty ()
	doIsRootJobOldDuplicate (Job $job)
	doPop ()
	getConnection ()
	Get a connection to the server that handles all sub-queues for this queue.
	getJobFromFields (array $fields)
	getNewJobFields (IJobSpecification $job)
	optimalOrder ()
	Get the default queue order to use if configuration does not specify one.
	popAndAcquireBlob (RedisConnRef $conn)
	popAndDeleteBlob (RedisConnRef $conn)
	pushBlobs (RedisConnRef $conn, array $items)
	serialize (array $fields)
	supportedOrders ()
	Get the allowed queue orders for configuration validation.
	supportsDelayedJobs ()
	Find out if delayed jobs are supported for configuration validation.
	throwRedisException (RedisConnRef $conn, $e)
	unserialize ($blob)
Protected Attributes
string	$compression
	Compression method to use *.
bool	$daemonized
	*
string	$key
	Key to prefix the queue keys with (used for testing) *.
RedisConnectionPool	$redisPool
	*
string	$server
	Server address *.
Private Member Functions
	getQueueKey ($prop, $type=null)

Detailed Description

Class to handle job queues stored in Redis.

This is faster, less resource intensive, queue that JobQueueDB. All data for a queue using this class is placed into one redis server.

There are eight main redis keys used to track jobs:

l-unclaimed : A list of job IDs used for ready unclaimed jobs
z-claimed : A sorted set of (job ID, UNIX timestamp as score) used for job retries
z-abandoned : A sorted set of (job ID, UNIX timestamp as score) used for broken jobs
z-delayed : A sorted set of (job ID, UNIX timestamp as score) used for delayed jobs
h-idBySha1 : A hash of (SHA1 => job ID) for unclaimed jobs used for de-duplication
h-sha1ById : A hash of (job ID => SHA1) for unclaimed jobs used for de-duplication
h-attempts : A hash of (job ID => attempt count) used for job claiming/retries
h-data : A hash of (job ID => serialized blobs) for job storage A job ID can be in only one of z-delayed, l-unclaimed, z-claimed, and z-abandoned. If an ID appears in any of those lists, it should have a h-data entry for its ID. If a job has a SHA1 de-duplication value and its ID is in l-unclaimed or z-delayed, then there should be no other such jobs with that SHA1. Every h-idBySha1 entry has an h-sha1ById entry and every h-sha1ById must refer to an ID that is l-unclaimed. If a job has its ID in z-claimed or z-abandoned, then it must also have an h-attempts entry for its ID.

Additionally, "rootjob:* keys track "root jobs" used for additional de-duplication. Aside from root job keys, all keys have no expiry, and are only removed when jobs are run. All the keys are prefixed with the relevant wiki ID information.

This class requires Redis 2.6 as it makes use Lua scripts for fast atomic operations. Additionally, it should be noted that redis has different persistence modes, such as rdb snapshots, journaling, and no persistent. Appropriate configuration should be made on the servers based on what queues are using it and what tolerance they have.

Since:: 1.22

Definition at line 59 of file JobQueueRedis.php.

Constructor & Destructor Documentation

JobQueueRedis::__construct ( array $ params )

Parameters:

array

$params

Possible keys:

redisConfig : An array of parameters to RedisConnectionPool::__construct(). Note that the serializer option is ignored as "none" is always used.
redisServer : A hostname/port combination or the absolute path of a UNIX socket. If a hostname is specified but no port, the standard port number 6379 will be used. Required.
compression : The type of compression to use; one of (none,gzip).
daemonized : Set to true if the redisJobRunnerService runs in the background. This will disable job recycling/undelaying from the MediaWiki side to avoid redundance and out-of-sync configuration.

Reimplemented from JobQueue.

Definition at line 82 of file JobQueueRedis.php.

Member Function Documentation

JobQueueRedis::doAck ( Job $ job ) [protected]

See also:: JobQueue::doAck()

Parameters:

Job $job

Returns:: Job|bool

Exceptions:

MWException|JobQueueError

Reimplemented from JobQueue.

Definition at line 410 of file JobQueueRedis.php.

JobQueueRedis::doBatchPush	(	array $	jobs,
		$	flags
	)		`[protected]`

See also:: JobQueue::doBatchPush()

Parameters:

array	$jobs
int	$flags

Returns:: void

Exceptions:

JobQueueError

Reimplemented from JobQueue.

Definition at line 188 of file JobQueueRedis.php.

JobQueueRedis::doDeduplicateRootJob ( Job $ job ) [protected]

See also:: JobQueue::doDeduplicateRootJob()

Parameters:

Job $job

Returns:: bool

Exceptions:

MWException|JobQueueError

Reimplemented from JobQueue.

Definition at line 455 of file JobQueueRedis.php.

JobQueueRedis::doDelete ( ) [protected]

See also:: JobQueue::doDelete()

Returns:: bool

Exceptions:

JobQueueError

Reimplemented from JobQueue.

Definition at line 506 of file JobQueueRedis.php.

JobQueueRedis::doGetAbandonedCount ( ) [protected]

See also:: JobQueue::doGetAbandonedCount()

Returns:: int

Exceptions:

JobQueueError

Reimplemented from JobQueue.

Definition at line 169 of file JobQueueRedis.php.

JobQueueRedis::doGetAcquiredCount ( ) [protected]

See also:: JobQueue::doGetAcquiredCount()

Returns:: int

Exceptions:

JobQueueError

Reimplemented from JobQueue.

Definition at line 131 of file JobQueueRedis.php.

JobQueueRedis::doGetDelayedCount ( ) [protected]

See also:: JobQueue::doGetDelayedCount()

Returns:: int

Exceptions:

JobQueueError

Reimplemented from JobQueue.

Definition at line 152 of file JobQueueRedis.php.

JobQueueRedis::doGetPeriodicTasks ( ) [protected]

Returns:: array

Reimplemented from JobQueue.

Definition at line 717 of file JobQueueRedis.php.

JobQueueRedis::doGetSiblingQueueSizes ( array $ types ) [protected]

See also:: JobQueue::getSiblingQueuesSize()

Parameters:

array $types List of queues types

Returns:: array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 577 of file JobQueueRedis.php.

JobQueueRedis::doGetSiblingQueuesWithJobs ( array $ types ) [protected]

See also:: JobQueue::getSiblingQueuesWithJobs()

Parameters:

array $types List of queues types

Returns:: array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 573 of file JobQueueRedis.php.

References server.

JobQueueRedis::doGetSize ( ) [protected]

See also:: JobQueue::doGetSize()

Returns:: int

Exceptions:

MWException

Reimplemented from JobQueue.

Definition at line 117 of file JobQueueRedis.php.

JobQueueRedis::doIsEmpty ( ) [protected]

See also:: JobQueue::doIsEmpty()

Returns:: bool

Exceptions:

MWException

Reimplemented from JobQueue.

Definition at line 108 of file JobQueueRedis.php.

JobQueueRedis::doIsRootJobOldDuplicate ( Job $ job ) [protected]

See also:: JobQueue::doIsRootJobOldDuplicate()

Parameters:

Job $job

Returns:: bool

Exceptions:

JobQueueError

Reimplemented from JobQueue.

Definition at line 483 of file JobQueueRedis.php.

JobQueueRedis::doPop ( ) [protected]

See also:: JobQueue::doPop()

Returns:: Job|bool

Exceptions:

JobQueueError

Reimplemented from JobQueue.

Definition at line 294 of file JobQueueRedis.php.

JobQueueRedis::getAllDelayedJobs ( )

See also:: JobQueue::getAllQueuedJobs()

Returns:: Iterator

Reimplemented from JobQueue.

Definition at line 550 of file JobQueueRedis.php.

References array(), and getConnection().

JobQueueRedis::getAllQueuedJobs ( )

See also:: JobQueue::getAllQueuedJobs()

Returns:: Iterator

Reimplemented from JobQueue.

Definition at line 527 of file JobQueueRedis.php.

References array(), getConnection(), and throwRedisException().

JobQueueRedis::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Returns:: string

Since:: 1.22

Reimplemented from JobQueue.

Definition at line 569 of file JobQueueRedis.php.

JobQueueRedis::getConnection ( ) [protected]

Get a connection to the server that handles all sub-queues for this queue.

Returns:: RedisConnRef

Exceptions:

JobQueueConnectionError

Definition at line 819 of file JobQueueRedis.php.

Referenced by getAllDelayedJobs(), and getAllQueuedJobs().

JobQueueRedis::getJobFromFields ( array $ fields ) [protected]

Parameters:

array $fields

Returns:: Job|bool

Definition at line 765 of file JobQueueRedis.php.

References $job, $title, JobQueue\factory(), and Title\makeTitleSafe().

JobQueueRedis::getJobFromUidInternal	(	$	uid,
		RedisConnRef $	conn
	)

This function should not be called outside JobQueueRedis.

Parameters:

string	$uid
RedisConnRef	$conn

Returns:: Job|bool Returns false if the job does not exist

Exceptions:

MWException|JobQueueError

Definition at line 607 of file JobQueueRedis.php.

JobQueueRedis::getNewJobFields ( IJobSpecification $ job ) [protected]

Parameters:

IJobSpecification $job

Returns:: array

Definition at line 743 of file JobQueueRedis.php.

References array(), IJobSpecification\getDeduplicationInfo(), IJobSpecification\getParams(), IJobSpecification\getReleaseTimestamp(), IJobSpecification\getTitle(), IJobSpecification\getType(), IJobSpecification\ignoreDuplicates(), UIDGenerator\newRawUUIDv4(), UIDGenerator\QUICK_RAND, serialize(), and wfBaseConvert().

JobQueueRedis::getQueueKey	(	$	prop,
		$	type = `null`
	)		`[private]`

Parameters:

string	$prop
string \| null	$type

Returns:: string

Definition at line 843 of file JobQueueRedis.php.

JobQueueRedis::optimalOrder ( ) [protected]

Get the default queue order to use if configuration does not specify one.

Returns:: string One of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 95 of file JobQueueRedis.php.

References array().

JobQueueRedis::popAndAcquireBlob ( RedisConnRef $ conn ) [protected]

Parameters:

RedisConnRef $conn

Returns:: array Serialized string or false

Exceptions:

RedisException

Definition at line 374 of file JobQueueRedis.php.

JobQueueRedis::popAndDeleteBlob ( RedisConnRef $ conn ) [protected]

Parameters:

RedisConnRef $conn

Returns:: array Serialized string or false

Exceptions:

RedisException

Definition at line 341 of file JobQueueRedis.php.

JobQueueRedis::pushBlobs	(	RedisConnRef $	conn,
		array $	items
	)		`[protected]`

Parameters:

RedisConnRef	$conn
array	$items	List of results from JobQueueRedis::getNewJobFields()

Returns:: int Number of jobs inserted (duplicates are ignored)

Exceptions:

RedisException

Definition at line 241 of file JobQueueRedis.php.

JobQueueRedis::recyclePruneAndUndelayJobs ( )

Recycle or destroy any jobs that have been claimed for too long and release any ready delayed jobs into the queue.

Returns:: int Number of jobs recycled/deleted/undelayed

Exceptions:

MWException|JobQueueError

Definition at line 634 of file JobQueueRedis.php.

JobQueueRedis::serialize ( array $ fields ) [protected]

Parameters:

array $fields

Returns:: string Serialized and possibly compressed version of $fields

Definition at line 781 of file JobQueueRedis.php.

References $blob, array(), and object.

Referenced by getNewJobFields().

JobQueueRedis::setTestingPrefix ( $ key )

Parameters:

string $key

Returns:: void

Reimplemented from JobQueue.

Definition at line 857 of file JobQueueRedis.php.

References $key, and key.

JobQueueRedis::supportedOrders ( ) [protected]

Get the allowed queue orders for configuration validation.

Returns:: array Subset of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 91 of file JobQueueRedis.php.

JobQueueRedis::supportsDelayedJobs ( ) [protected]

Find out if delayed jobs are supported for configuration validation.

Returns:: bool Whether delayed jobs are supported

Reimplemented from JobQueue.

Definition at line 99 of file JobQueueRedis.php.

JobQueueRedis::throwRedisException	(	RedisConnRef $	conn,
		$	e
	)		`[protected]`

Parameters:

RedisConnRef	$conn
RedisException	$e

Exceptions:

JobQueueError

Definition at line 833 of file JobQueueRedis.php.

Referenced by getAllQueuedJobs().

JobQueueRedis::unserialize ( $ blob ) [protected]

Parameters:

string $blob

Returns:: array|bool Unserialized version of $blob or false

Definition at line 800 of file JobQueueRedis.php.

References $blob.

Member Data Documentation

string JobQueueRedis::$compression [protected]

Compression method to use *.

Definition at line 63 of file JobQueueRedis.php.

bool JobQueueRedis::$daemonized [protected]

*

Definition at line 64 of file JobQueueRedis.php.

string JobQueueRedis::$key [protected]

Key to prefix the queue keys with (used for testing) *.

Definition at line 68 of file JobQueueRedis.php.

Referenced by setTestingPrefix().

RedisConnectionPool JobQueueRedis::$redisPool [protected]

*

Definition at line 60 of file JobQueueRedis.php.

string JobQueueRedis::$server [protected]

Server address *.

Definition at line 62 of file JobQueueRedis.php.

const JobQueueRedis::MAX_AGE_PRUNE = 604800

Definition at line 66 of file JobQueueRedis.php.

The documentation for this class was generated from the following file:

includes/jobqueue/JobQueueRedis.php

Public Member Functions

Public Attributes

Protected Member Functions

Protected Attributes

Private Member Functions

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation