Chapter 2. Using the AMQ Message Store

The default message store used by FUSE Message Broker is a light-weight transactional store that is fast and reliable. It is a hybrid system that couples a transactional journal for message storage and a reference store for quick retrieval. The AMQ message store is highly configurable.

Overview

By default, FUSE Message Broker uses the AMQ Message Store to persist message data. The AMQ message store is an embeddable, transactional message store that is extremely fast and reliable. It is an evolution of the Kaha system used by Active MQ 4.x. It uses a transactional journal to store message data and a Kaha-based index to store message locations for quick retrieval.

Figure 2.1 shows a high-level view of the AMQ message store.

Figure 2.1. Overview of the AMQ Message Store

the AMQ message store has disk-based data logs that support an indexed in-memory cache

Messages are stored in file-based data logs. When all of the messages in a data log have been successfully consumed, the data log is marked as ready to be deleted. At a predetermined clean-up interval, logs marked as deletable are removed from the system.

	Note
	Message logs can also be archived.

An index of message locations is cached in memory to facilitate quick retrieval of message data. At configurable checkpoint intervals, the references are inserted into the persistent reference store.

Data structure

The AMQ message store is a file-based message store and uses a layered directory structure to store its data. Figure 2.2 shows the layout of the AMQ message store's files.

Figure 2.2. AMQ Message Store Directory Layout

image of the multi-layer message store layout

The top-level directory of a broker's message store is identified by the name of the broker. For example a broker configured with the name JoeFred would have a message store folder named JoeFred. Beneath the message store's top-level folder are four folders:

archive

The archive folder stores archived message logs.

	Note
	This folder only exists when log archiving is activated.

	Tip
	You can change the name of this folder by setting the AMQ persistence adapter's `directoryArchive` attribute. See Configuration.

journal

The journal folder stores the active message logs.

kr-store

The kr-store folder is used by the Kaha reference store when it saves message references to disk. It has two sub-folders:

data: The data folder stores the indexes for referencing the logged messages.
state: The state folder maintains state information regarding the message store. The information contained her includes the name of durable subscribers and information about transactions.

	Note
	This folder is only used if the AMQ persistence adapter's `persistentIndex` attribute is set to `true`. See Configuration.

tmp-storage

The tmp-storage folder stores transient messages that are cached to free up memory. For example non-persistent messages may be stored here while awaiting consumption by an active, but slow consumer.

Configuration

FUSE Message Broker comes preconfigured to use the AMQ message store. However, you can modify how the message store behaves by explicitly defining its persistence adapter using the amqPersistenceAdapter element as shown in Example 2.1.

Example 2.1. Configuring the AMQ Message Store

<broker brokerName="broker" persistent="true" useShutdownHook="false">
  ...
  <persistenceAdapter>
    <amqPersistenceAdapter directory="activemq-data" maxFileLength="32mb"/>
  </persistenceAdapter>
</broker>

Table 2.1 describes all of the attributes that can be used to configure the AMQ message store.

Table 2.1. Configuration Attributes for the AMQ Message Store

Attribute	Default Value	Description
`directory`	`activemq-data`	Specifies the path to the top-level folder used to hold the message store's data files.
`useNIO`	`true`	Specifies whether or not NIO is used to write messages to the data logs.
`syncOnWrite`	`false`	Specifies whether or not to sync every write to a data log to the disk.
`maxFileLength`	`32mb`	Sets the maximum size of the logs used to store message data. Any string value provided, in kilobytes, megabytes, or gigabytes, is converted to a long value in bytes at runtime. The conversion process ignores whitespace, is case-insensitive, and accepts one- or two-letter acronyms as units of measurement. Values written without units are treated as bytes. For example, the following values are all valid: `1024kb`, `256 MB`, `512m`, `1G`, `10000000`.
`persistentIndex`	`true`	Specifies whether or not to write the reference index to a persistent store. If set this attribute is set to `false` the reference index will be maintained in non-persistent memory.
`checkpointInterval`	`6000`	Specifies how often, in milliseconds, that data from the journal is synchronized with the indexes.
`maxCheckpointMessageAddSize`	`4096`	Specifies the maximum number of messages to keep in a transaction before automatically commiting them to the message store.
`cleanupInterval`	`30000`	Specifies the interval, in milliseconds, between clean-up sweeps of the message store.
`indexBinSize`	`1024`	Specifies the default number of bins used by the reference index. Increasing the number of bins in use increases the relative performance of the index.
`indexKeySize`	`96`	Specifies the size of the index key to use. The index key is the message id.
`indexPageSize`	`16kb`	Specifies the size of the page file to use for the reference index. Increasing the page file size increases the index's write performance.
`archiveDataLogs`	`false`	Specifies whether or not old message logs are archived or deleted. Setting this attribute to `true` specifies that old message logs are copied to the message store's archive folder instead of being deleted.
`directoryArchive`	`archive`	Specifies the name of the folder into which archived message logs are stored.

Failure recovery

If the message broker does not shutdown properly, the reference store indexes are cleaned and the message data files are replayed to rebuild the message store's state information. It is possible to force automatic recovery by deleting the kr-store/state/index-store-state file.