Skip Headers
Oracle GlassFish Server Message Queue Administration Guide
Release 4.5.2
Part Number E24943-01
Go to Documentation Home
Home		 Go to Book List
Book List		 Go to Table of Contents
Contents		 Go to Feedback page
Contact Us
Go to previous page
Previous		 Go to next page
Next
View PDF

15 Troubleshooting

This chapter explains how to understand and resolve the following problems:

When problems occur, it is useful to check the version number of the installed Message Queue software. Use the version number to ensure that you are using documentation whose version matches the software version. You also need the version number to report a problem to Oracle. To check the version number, issue the following command:

imqcmd -v

A Client Cannot Establish a Connection

Symptoms:

Possible causes:

Client applications are not closing connections, causing the number of connections to exceed resource limitations.

To confirm this cause of the problem: List all connections to a broker:

imqcmd list cxn

The output will list all connections and the host from which each connection has been made, revealing an unusual number of open connections for specific clients.

To resolve the problem: Rewrite the offending clients to close unused connections.

Broker is not running or there is a network connectivity problem.

To confirm this cause of the problem:

To resolve the problem:

Connection service is inactive or paused.

To confirm this cause of the problem: Check the status of all connection services:

imqcmd list svc

If the status of a connection service is shown as unknown or paused, clients will not be able to establish a connection using that service.

To resolve the problem:

Too few threads available for the number of connections required.

To confirm this cause of the problem: Check for the following entry in the broker log:

WARNING [B3004]: No threads are available to process a new connection on service ...Closing the new connection.

Also check the number of connections on the connection service and the number of threads currently in use, using one of the following formats:

imqcmd query svc -n serviceNameimqcmd metrics svc -n serviceName-m cxn

Each connection requires two threads: one for incoming messages and one for outgoing messages (see Thread Pool Management).

To resolve the problem:

Too few file descriptors for the number of connections required on the Solaris or Linux platform.

For more information about this issue, see Setting the File Descriptor Limit.

To confirm this cause of the problem: Check for an entry in the broker log similar to the following:

Too many open files

To resolve the problem: Increase the file descriptor limit, as described in the man page for the ulimit command.

TCP backlog limits the number of simultaneous new connection requests that can be established.

The TCP backlog places a limit on the number of simultaneous connection requests that can be stored in the system backlog (imq.portmapper.backlog) before the Port Mapper rejects additional requests. (On the Windows platform there is a hard-coded backlog limit of 5 for Windows desktops and 200 for Windows servers.)

The rejection of requests because of backlog limits is usually a transient phenomenon, due to an unusually high number of simultaneous connection requests.

To confirm this cause of the problem: Examine the broker log. First, check to see whether the broker is accepting some connections during the same time period that it is rejecting others. Next, check for messages that explain rejected connections. If you find such messages, the TCP backlog is probably not the problem, because the broker does not log connection rejections due to the TCP backlog. If some successful connections are logged, and no connection rejections are logged, the TCP backlog is probably the problem.

To resolve the problem:

Operating system limits the number of concurrent connections.

The Windows operating system license places limits on the number of concurrent remote connections that are supported.

To confirm this cause of the problem: Check that there are plenty of threads available for connections (using imqcmd query svc) and check the terms of your Windows license agreement. If you can make connections from a local client, but not from a remote client, operating system limitations might be the cause of the problem.

To resolve the problem:

Authentication or authorization of the user is failing.

The authentication may be failing for any of the following reasons:

To confirm this cause of the problem: Check entries in the broker log for the Forbidden error message. This will indicate an authentication error, but will not indicate the reason for it.

To resolve the problem:

Connection Throughput Is Too Slow

Symptoms:

Possible causes:

Network connection or WAN is too slow.

To confirm this cause of the problem:

To resolve the problem: Upgrade the network link.

Connection service protocol is inherently slow compared to TCP.

For example, SSL-based or HTTP-based protocols are slower than TCP (see Transport Protocols).

To confirm this cause of the problem: If you are using SSL-based or HTTP-based protocols, try using TCP and compare the delivery times.

To resolve the problem: Application requirements usually dictate the protocols being used, so there is little you can do other than attempt to tune the protocol as described in Tuning Transport Protocols.

Connection service protocol is not optimally tuned.

To confirm this cause of the problem: Try tuning the protocol to see whether it makes a difference.

To resolve the problem: Try tuning the protocol, as described in Tuning Transport Protocols.

Messages are so large that they consume too much bandwidth.

To confirm this cause of the problem: Try running your benchmark with smaller-sized messages.

To resolve the problem:

What appears to be slow connection throughput is actually a bottleneck in some other step of the message delivery process.

To confirm this cause of the problem: If what appears to be slow connection throughput cannot be explained by any of the causes above, see Factors Affecting Performance for other possible bottlenecks and check for symptoms associated with the following problems:

To resolve the problem: Follow the problem resolution guidelines provided in the troubleshooting sections listed above.

A Client Cannot Create a Message Producer

Symptom:

Possible causes:

A physical destination has been configured to allow only a limited number of producers.

One of the ways of avoiding the accumulation of messages on a physical destination is to limit the number of producers (maxNumProducers) that it supports.

To confirm this cause of the problem: Check the physical destination:

imqcmd query dst

(see Viewing Physical Destination Information). The output will show the current number of producers and the value of maxNumProducers. If the two values are the same, the number of producers has reached its configured limit. When a new producer is rejected by the broker, the broker returns the exception

ResourceAllocationException [C4088]: A JMS destination limit was reached

and makes the following entry in the broker log:

[B4183]: Producer can not be added to destination

To resolve the problem: Increase the value of the maxNumProducers property (see Updating Physical Destination Properties).

The user is not authorized to create a message producer due to settings in the access control file.

To confirm this cause of the problem: When a new producer is rejected by the broker, the broker returns the exception

JMSSecurityException [C4076]: Client does not have permission to create producer on destination

and makes the following entries in the broker log:

[B2041]: Producer on destination denied[B4051]: Forbidden guest.

To resolve the problem: Change the access control properties to allow the user to produce messages (see Authorization Rules for Physical Destinations).

Message Production Is Delayed or Slowed

Symptoms:

Possible causes:

The broker is backlogged and has responded by slowing message producers.

A backlogged broker accumulates messages in broker memory. When the number of messages or message bytes in physical destination memory reaches configured limits, the broker attempts to conserve memory resources in accordance with the specified limit behavior. The following limit behaviors slow down message producers:

Similarly, when the number of messages or message bytes in brokerwide memory (for all physical destinations) reaches configured limits, the broker will attempt to conserve memory resources by rejecting the newest messages. Also, when system memory limits are reached because physical destination or brokerwide limits have not been set properly, the broker takes increasingly serious action to prevent memory overload. These actions include throttling back message producers.

To confirm this cause of the problem: When a message is rejected by the broker because of configured message limits, the broker returns the exception

JMSException [C4036]: A server error occurred

and makes the following entry in the broker log:

[B2011]: Storing of JMS message from IMQconn failed

This message is followed by another indicating the limit that has been reached:

[B4120]: Cannot store message on destination destNamebecause capacity of maxNumMsgswould be exceeded.

if the exceeded message limit is on a physical destination, or

[B4024]: The maximum number of messages currrently in the system has been exceeded, rejecting message.

if the limit is brokerwide.

More generally, you can check for message limit conditions before the rejections occur as follows:

To resolve the problem:

The broker cannot save a persistent message to the data store.

If the broker cannot access a data store or write a persistent message to it, the producing client is blocked. This condition can also occur if destination or brokerwide message limits are reached, as described above.

To confirm this cause of the problem: If the broker is unable to write to the data store, it makes one of the following entries in the broker log:

[B2011]: Storing of JMS message from connectionIDfailed[B4004]: Failed to persist message messageID

To resolve the problem:

Broker acknowledgment timeout is too short.

Because of slow connections or a lethargic broker (caused by high CPU utilization or scarce memory resources), a broker may require more time to acknowledge receipt of a persistent message than allowed by the value of the connection factory's imqAckTimeout attribute.

To confirm this cause of the problem: If the imqAckTimeout value is exceeded, the broker returns the exception

JMSException [C4000]: Packet acknowledge failed

To resolve the problem: Change the value of the imqAckTimeout connection factory attribute (see Reliability And Flow Control).

A producing client is encountering JVM limitations.

To confirm this cause of the problem:

To resolve the problem: Adjust the JVM (see Java Virtual Machine Adjustments).

Messages Are Backlogged

Symptoms:

To see whether messages are accumulating, check how the number of messages or message bytes in the broker changes over time and compare to configured limits. First check the configured limits:

imqcmd query bkr

Note:

The imqcmd metrics bkr subcommand does not display this information.

Then check for message accumulation in each destination:

imqcmd list dst

To see whether messages have exceeded configured destination or brokerwide limits, check the broker log for the entry

[B2011]: Storing of JMS message from …failed.

This entry will be followed by another identifying the limit that has been exceeded.

Possible causes:

There are inactive durable subscriptions on a topic destination.

If a durable subscription is inactive, messages are stored in a destination until the corresponding consumer becomes active and can consume the messages.

To confirm this cause of the problem: Check the state of durable subscriptions on each topic destination:

imqcmd list dur -d destName

To resolve the problem:

Too few consumers are available to consume messages in a multiple-consumer queue.

If there are too few active consumers to which messages can be delivered, a queue destination can become backlogged as messages accumulate. This condition can occur for any of the following reasons:

To confirm this cause of the problem: To help determine the reason for unavailable consumers, check the number of active consumers on a destination:

imqcmd metrics dst -n destName-t q -m con

To resolve the problem: Depending on the reason for unavailable consumers,

Message consumers are processing too slowly to keep up with message producers.

In this case, topic subscribers or queue receivers are consuming messages more slowly than the producers are sending messages. One or more destinations are getting backlogged with messages because of this imbalance.

To confirm this cause of the problem: Check for the rate of flow of messages into and out of the broker:

imqcmd metrics bkr -m rts

Then check flow rates for each of the individual destinations:

imqcmd metrics bkr -t destType-n destName-m rts

To resolve the problem:

Client acknowledgment processing is slowing down message consumption.

Two factors affect the processing of client acknowledgments:

To confirm this cause of the problem:

To resolve the problem:

The broker cannot keep up with produced messages.

In this case, messages are flowing into the broker faster than the broker can route and dispatch them to consumers. The sluggishness of the broker can be due to limitations in any or all of the following:

To confirm this cause of the problem: Check that none of the other possible causes of this problem are responsible.

To resolve the problem:

Client code defects; consumers are not acknowledging messages.

Messages are held in a destination until they have been acknowledged by all consumers to which they have been sent. If a client is not acknowledging consumed messages, the messages accumulate in the destination without being deleted.

For example, client code might have the following defects:

To confirm this cause of the problem: First check all other possible causes listed in this section. Next, list the destination with the following command:

imqcmd list dst

Notice whether the number of messages listed under the UnAcked header is the same as the number of messages in the destination. Messages under this header were sent to consumers but not acknowledged. If this number is the same as the total number of messages, then the broker has sent all the messages and is waiting for acknowledgment.

To resolve the problem: Request the help of application developers in debugging this problem.

Broker Throughput Is Sporadic

Symptom:

Possible causes:

The broker is very low on memory resources.

Because destination and broker limits were not properly set, the broker takes increasingly serious action to prevent memory overload; this can cause the broker to become sluggish until the message backlog is cleared.

To confirm this cause of the problem: Check the broker log for a low memory condition

[B1089]: In low memory condition, broker is attempting to free up resources

followed by an entry describing the new memory state and the amount of total memory being used. Also check the free memory available in the JVM heap:

imqcmd metrics bkr -m cxn

Free memory is low when the value of total JVM memory is close to the maximum JVM memory value.

To resolve the problem:

JVM memory reclamation (garbage collection) is taking place.

Memory reclamation periodically sweeps through the system to free up memory. When this occurs, all threads are blocked. The larger the amount of memory to be freed up and the larger the JVM heap size, the longer the delay due to memory reclamation.

To confirm this cause of the problem: Monitor CPU usage on your computer. CPU usage drops when memory reclamation is taking place.

Also start your broker using the following command line options:

-vmargs-verbose:gc

Standard output indicates the time when memory reclamation takes place.

To resolve the problem: In computers with multiple CPUs, set the memory reclamation to take place in parallel:

-XX:+UseParallelGC=true

The JVM is using the just-in-time compiler to speed up performance.

To confirm this cause of the problem: Check that none of the other possible causes of this problem are responsible.

To resolve the problem: Let the system run for awhile; performance should improve.

Messages Are Not Reaching Consumers

Symptom:

Possible causes:

Limit behaviors are causing messages to be deleted on the broker.

When the number of messages or message bytes in destination memory reach configured limits, the broker attempts to conserve memory resources. Three of the configurable behaviors adopted by the broker when these limits are reached will cause messages to be lost:

To confirm this cause of the problem: Use the QBrowser demo application to inspect the contents of the dead message queue (see To Inspect the Dead Message Queue).

Check whether the JMS_SUN_DMQ_UNDELIVERED_REASON property of messages in the queue has the value REMOVE_OLDEST or REMOVE_LOW_PRIORITY.

To resolve the problem: Increase the destination limits. For example:

imqcmd update dst -n MyDest-o maxNumMsgs=1000

Message timeout value is expiring.

The broker deletes messages whose timeout value has expired. If a destination gets sufficiently backlogged with messages, messages whose time-to-live value is too short might be deleted.

To confirm this cause of the problem: Use the QBrowser demo application to inspect the contents of the dead message queue (see To Inspect the Dead Message Queue).

Check whether the JMS_SUN_DMQ_UNDELIVERED_REASON property of messages in the queue has the value EXPIRED.

To resolve the problem: Contact the application developers and have them increase the time-to-live value.

The broker clock and producer clock are not synchronized.

If clocks are not synchronized, broker calculations of message lifetimes can be wrong, causing messages to exceed their expiration times and be deleted.

To confirm this cause of the problem: Use the QBrowser demo application to inspect the contents of the dead message queue (see To Inspect the Dead Message Queue).

Check whether the JMS_SUN_DMQ_UNDELIVERED_REASON property of messages in the queue has the value EXPIRED.

In the broker log file, look for any of the following messages: B2102, B2103, B2104. These messages all report that possible clock skew was detected.

To resolve the problem: Check that you are running a time synchronization program, as described in Preparing System Resources.

Consuming client failed to start message delivery on a connection.

Messages cannot be delivered until client code establishes a connection and starts message delivery on the connection.

To confirm this cause of the problem: Check that client code establishes a connection and starts message delivery.

To resolve the problem: Rewrite the client code to establish a connection and start message delivery.

Dead Message Queue Contains Messages

Symptom:

Listing all the destinations on the broker specified by:
---------------------------------
Host         Primary Port
---------------------------------
localhost    7676
----------------------------------------------------------------------
   Name     Type    State   Producers  Consumers  Msgs
                                         Total    Count  UnAck  Avg Size
------------------------------------------------- ----------------------
MyDest      Queue  RUNNING       0          0        5      0    1177.0
mq.sys.dmq  Queue  RUNNING       0          0       35      0    1422.0
Successfully listed destinations.

In this example, the dead message queue, mq.sys.dmq, contains 35 messages.

Possible causes:

The number of messages, or their sizes, exceed destination limits.

To confirm this cause of the problem: Use the QBrowser demo application to inspect the contents of the dead message queue (see To Inspect the Dead Message Queue).

Check the values for the following message properties:

Under JMS Headers, scroll down to the value for JMSDestination to determine the destination whose messages are becoming dead.

To resolve the problem: Increase the destination limits. For example:

imqcmd update dst -n MyDest -o maxNumMsgs=1000

The broker clock and producer clock are not synchronized.

If clocks are not synchronized, broker calculations of message lifetimes can be wrong, causing messages to exceed their expiration times and be deleted.

To confirm this cause of the problem: Use the QBrowser demo application to inspect the contents of the dead message queue (see To Inspect the Dead Message Queue).

Check whether the JMS_SUN_DMQ_UNDELIVERED_REASON property of messages in the queue has the value EXPIRED.

In the broker log file, look for any of the following messages: B2102, B2103, B2104. These messages all report that possible clock skew was detected.

To resolve the problem: Check that you are running a time synchronization program, as described in Preparing System Resources.

An unexpected broker error has occurred.

To confirm this cause of the problem: Use the QBrowser demo application to inspect the contents of the dead message queue (see To Inspect the Dead Message Queue).

Check whether the JMS_SUN_DMQ_UNDELIVERED_REASON property of messages in the queue has the value ERROR.

To resolve the problem:

Consumers are not consuming messages before they time out.

To confirm this cause of the problem: Use the QBrowser demo application to inspect the contents of the dead message queue (see To Inspect the Dead Message Queue).

Check whether the JMS_SUN_DMQ_UNDELIVERED_REASON property of messages in the queue has the value EXPIRED.

Check to see if there any consumers on the destination and the value for the Current Number of Active Consumers. For example:

imqcmd query dst -t q -n MyDest

If there are active consumers, then there might be any number of possible reasons why messages are timing out before being consumed. One is that the message timeout is too short for the speed at which the consumer executes. In that case, request that application developers increase message time-to-live values. Otherwise, investigate the following possible causes for messages to time out before being consumed:

There are too many producers for the number of consumers.

To confirm this cause of the problem: Use the QBrowser demo application to inspect the contents of the dead message queue (see To Inspect the Dead Message Queue).

Check whether the JMS_SUN_DMQ_UNDELIVERED_REASON property of messages in the queue has the value REMOVE_OLDEST or REMOVE_LOW_PRIORITY. If so, use the imqcmd query dst command to check the number of producers and consumers on the destination. If the number of producers exceeds the number of consumers, the production rate might be overwhelming the consumption rate.

To resolve the problem: Add more consumer clients or set the destination's limit behavior to FLOW_CONTROL (which uses consumption rate to control production rate), using a command such as the following:

imqcmd update dst -n myDst -t q -o limitBehavior=FLOW_CONTROL

Producers are faster than consumers.

To confirm this cause of the problem: To determine whether slow consumers are causing producers to slow down, set the destination's limit behavior to FLOW_CONTROL (which uses consumption rate to control production rate), using a command such as the following:

imqcmd update dst -n myDst -t q -o limitBehavior=FLOW_CONTROL

Use metrics to examine the destination's input and output, using a command such as the following:

imqcmd metrics dst -n myDst -t q -m rts

In the metrics output, examine the following values:

Because flow control aligns production to consumption, note whether production slows or stops. If so, there is a discrepancy between the processing speeds of producers and consumers. You can also check the number of unacknowledged (UnAcked) messages sent, by using the imqcmd list dst command. If the number of unacknowledged messages is less than the size of the destination, the destination has additional capacity and is being held back by client flow control.

To resolve the problem: If production rate is consistently faster than consumption rate, consider using flow control regularly, to keep the system aligned. In addition, consider and attempt to resolve each of the following possible causes, which are subsequently described in more detail:

A consumer is too slow.

To confirm this cause of the problem: Use imqcmd metrics to determine the rate of production and consumption, as described above under Producers are faster than consumers..

To resolve the problem:

Clients are not committing transactions.

To confirm this cause of the problem: Check with application developers to find out whether the application uses transactions. If so, list the active transactions as follows:

imqcmd list txn

Here is an example of the command output:

----------------------------------------------------------------------
Transaction ID       State    User name  # Msgs/# Acks   Creation time
----------------------------------------------------------------------
6800151593984248832  STARTED  guest           3/2     7/19/04 11:03:08 AM

Note the numbers of messages and number of acknowledgments. If the number of messages is high, producers may be sending individual messages but failing to commit transactions. Until the broker receives a commit, it cannot route and deliver the messages for that transaction. If the number of acknowledgments is high, consumers may be sending acknowledgments for individual messages but failing to commit transactions. Until the broker receives a commit, it cannot remove the acknowledgments for that transaction.

To resolve the problem: Contact application developers to fix the coding error.

Consumers are failing to acknowledge messages.

To confirm this cause of the problem: Contact application developers to determine whether the application uses system-based acknowledgment (AUTO_ACKNOWLEDGE or DUPES_ONLY) or client-based acknowledgment (CLIENT_ACKNOWLEDGE). If the application uses system-based acknowledgment , skip this section; if it uses client-based acknowledgment), first decrease the number of messages stored on the client, using a command like the following:

imqcmd update dst -n myDst -t q -o consumerFlowLimit=1

Next, you will determine whether the broker is buffering messages because a consumer is slow, or whether the consumer processes messages quickly but does not acknowledge them. List the destination, using the following command:

imqcmd list dst

After you supply a user name and password, output like the following appears:

Listing all the destinations on the broker specified by:
---------------------------------
Host         Primary Port
---------------------------------
localhost    7676
----------------------------------------------------------------------
   Name     Type    State   Producers  Consumers  Msgs
                                         Total    Count  UnAck  Avg Size
------------------------------------------------ -----------------------
MyDest      Queue  RUNNING       0          0        5    200    1177.0
mq.sys.dmq  Queue  RUNNING       0          0       35      0    1422.0
Successfully listed destinations.

The UnAck number represents messages that the broker has sent and for which it is waiting for acknowledgment. If this number is high or increasing, you know that the broker is sending messages, so it is not waiting for a slow consumer. You also know that the consumer is not acknowledging the messages.

To resolve the problem: Contact application developers to fix the coding error.

Durable subscribers are inactive.

To confirm this cause of the problem: Look at the topic's durable subscribers, using the following command format:

imqcmd list dur -d topicName

To resolve the problem:

To Inspect the Dead Message Queue

A number of troubleshooting procedures involve an inspection of the dead message queue (mq.sys.dmq). The following procedure explains how to carry out such an inspection by using the QBrowser demo application.

  1. Locate the QBrowser demo application in IMQ_HOME/examples/applications/qbrowser.

  2. Run the QBrowser application:

    java QBrowser

    The QBrowser main window appears.

  3. Select the queue name mq.sys.dmq and click Browse.

    A list like the following appears:

    QBrowser screenshot of dead message queue
    Description of the illustration queuebrowsermain.png

  4. Double-click any message to display details about that message:

    The display should resemble the following:

    QBrowser screenshot of dead message details
    Description of the illustration queuebrowsermsgdetails.png

    You can inspect the Message Properties pane to determine the reason why the message was placed in the dead message queue.

Go to previous page
Previous		 Go to next page
Next
Go to Documentation Home
Home		 Go to Book List
Book List		 Go to Table of Contents
Contents		 Go to Feedback page
Contact Us