Extended Monitoring Guide
ZenJMX is a ZenPack that allows Zenoss to communicate with remote Java Management Extensions (JMX) agents. The ZenJMX ZenPack defines a data source named JMX that allows you to query any single or complex-value attribute, or invoke an MBean operation. It also comes with a built-in template named Java that contains MBean information for a few beans built into the JVM.
Note
ZenJMX also includes a built-in template named ZenJMX. This template should only be used on the device running the zenjmx daemon. To monitor other Java servers use the included Java template.
When the zenjmx daemon is started it communicates with ZenHub and retrieves a list of devices that possess JMX data sources. It also spawns a Java process. ZenJMX asynchronously issues queries for each of those devices to the Java process via XML-RPC. The Java process then collects the data from the Java application and returns the results to ZenJMX. Any collection or configuration errors are sent as events to Zenoss and will appear in the event console.
Lastly, ZenJMX heartbeats after each collect to ZenHub to let Zenoss know that ZenJMX is still alive and well.
The JMX technology is used throughout the Java Virtual Machine to provide performance and management information to clients. Using a combination of JConsole (Sun Microsystems' JMX client that is shipped with the JDK) and JMX, a system operator can examine the number of threads that are active in the JVM or change the log level. There are numerous other performance metrics that can be gleaned from the JVM, as well as several management interfaces that can be invoked that change the behavior of the JVM.
In Java 5, Sun introduced the Remote API for Java Management Extensions. This enhancement defines an RMI wrapper around a JMX agent and allows for independent client development. ZenJMX accesses remote JMX agents via the "Remote API for Java Management Extensions." It currently does not support local connections (provided via the temporary directory) to JMX Agents.
ZenJMX is a full-featured JMX client that works "out of the box" with JMX agents that have their remote APIs enabled. It supports authenticated and unauthenticated connections, and it can retrieve single-value attributes, complex-value attributes, and the results of invoking an operation. Operations with parameters are also supported so long as the parameters are primitive types (Strings, booleans, numbers), as well as the object version of primitives (such as java.lang.Integer and java.lang.Float). Multi-value responses from operations (Maps and Lists) are supported, as are primitive responses from operations.
The JMX data source installed by ZenJMX allows you to define the connection, authentication, and retrieval information you want to use to retrieve performance information. The IP address is extracted from the parent device, but the port number of the JMX Agent is configurable in each data source. This allows you to operate multiple JMX Agents on a single device and retrieve performance information for each agent separately. This is commonly used on production servers that run multiple applications.
Authentication information is also associated with each JMX data source. This offers the most flexibility for site administrators because they can run some JMX agents in an open, unauthenticated fashion and others in a hardened and authenticated fashion. SSL-wrapped connections are supported by the underlying JMX Remote subsystem built into the JDK, but were not tested in the Zenoss labs. As a result, your success with SSL encrypted access to JMX Agents may vary.
The data source allows you to define the type of performance information you want to achieve: single-value attribute, complex-value attribute, or operation invocation. To specify the type of retrieval, you must specify an attribute name (and one or more data points) or provide operation information.
Any numerical value returned by a JMX agent can be retrieved by Zenoss and graphed and checked against thresholds. Non-numerical values (Strings and complex types) cannot be retrieved and stored by Zenoss.
When setting up data points, make sure you understand the semantics of the attribute name and choose the correct Zenoss data point type. Many JMX Agent implementations use inconsistent nomenclature when describing attributes. In some cases the term "Count" refers to an ever-increasing number (a "Counter" data point type). In other cases the term "Count" refers to a snapshot number (a "Gauge" data point type).
The following primitive data types are allowed in JMX calls:
java.lang.Integerjava.lang.Longjava.lang.Doublejava.lang.Floatjava.lang.Stringjava.lang.Booleanintlongdoublefloatboolean
This is the most basic usage scenario. If you are interested in retrieving a single value from an MBean in a JMX Agent, and the attribute returns simple numeric data, you fall into the "single value attribute" category. To define a single-value attribute call simply provide the fully qualified name of your MBean and then provide the name of the attribute in the Attribute Name field of the data source. Lastly, you must define a data point.
Some examples of this include the commonly referenced JDK Threading information:
MBean Name: java.lang:type=Threading
Attribute Name: ThreadCount
Data Points:
ThreadCount (type: gauge)
Java uses lots of file descriptors during normal operation. The number of open file descriptors the JVM is working with can be measured using the following information:
MBean Name: java.lang:type=OperatingSystem
Attribute Name: OpenFileDescriptorCount
Data Points:
OpenFileDescriptorCount (type: gauge)
There are several other single-value attributes that can be retrieved from the JDK. We recommend using JConsole to interactively navigate through the MBean hierarchy to determine which MBeans contain useful information to you. See Section 5, “Using JConsole to Query a JMX Agent” for additional information on how to inspect the MBeans deployed in an JMX Agent.
If your MBean attribute defines multiple sub-attributes (via CompositeData or Tabular) that you are interested in capturing, then you fall into the category of a "complex-value attribute" call. The JDK contains a few complex-value attributes you might be interested in capturing, including garbage collection statistics that were captured during the copy and mark-sweep compact collection cycles.
To extract data from a complex-value attribute, you must define one or more data points in the data source. The names of the data points are used as keys into the complex-value data structure returned from the MBean attribute. For JMX CompositeData attributes, the data point names are used as a key to map the results. For JMX TabularData, the data point names are used as indexes into the structure to map the result.
The JDK also provides heap memory information via a complex-value attribute. The amount of committed, used, and maximum heap memory can be viewed by setting up a complex-value attribute in Zenoss with the following information:
MBean Name: java.lang:type=Memory
Attribute Name: HeapMemoryUsage
Data Points:
committed (type: gauge)
used (type: gauge)
max (type: gauge)
Some management values need to be computed. These situations frequently arise when custom MBeans are deployed alongside an enterprise application. An MBean named "Accounting" might be deployed within an enterprise application that defines operations intended for operators or support staff. These operations might include methods such as "getBankBalance()" or "countTotalDeposits()".
ZenJMX has the ability to invoke operations, but there are some subtleties in how ZenJMX sends parameters to the JMX Agent and interprets the response.
In the most basic usage scenario no arguments are passed to the operation and a single value is returned. This usage scenario is very similar to a single-value attribute call, except we're invoking an operation to retrieve the value rather than accessing an attribute. The configuration for this hypothetical usage scenario follows:
MBean Name: Application:Name=Accounting,Type=Accounting
Operation Name: getBankBalance()
Data Points:
balance (type: gauge)
In this scenario no parameters are passed to an operation, but multiple response values are provided in a List. The values returned are expressed in a List<Object>, but they are coerced (but not casted) to doubles prior to being stored in Zenoss. This means that returning a numeric value as "1234" will work, but "1,234" will not work. The litmus test is to evaluate if Double.valueOf(object.toString()) will successfully evaluate.
ZenJMX can be configured to read multiple values from an operation's results by defining multiple data points. You must define a data point for each value returned from the operation, and if there is a mismatch between the number of data points you define and the size of the List<Object> returned an exception will be generated. The configuration for ZenJMX follows:
MBean Name: Application:Name=Accounting,Type=Accounting
Operation Name: getBalanceSummary()
Data Points:
dailyBalance (type: gauge)
annualBalance (type: gauge)
In this scenario no parameters are passed to an operation, but multiple response values are provided in a Map<String, Object>. The keyset of the Map contains the names of data points that can be defined, and the values are the values of said data points. When a Map<String, Object> is returned you need not capture all of the returned values as data points, and you can instead pick the exact values you are interested in. To choose the values to capture you simply define data points with the same names as Strings in the keyset.
The following configuration demonstrates how to extract specific data points from an operation that returns a Map<String, Object>. The key item to note in this configuration is that "dailyBalance" and "annualBalance" must be present as keys in the returned Map<String, Object> and their values must be coercible via the Double.valueOf(object.toString()) idiom.
MBean Name: Application:Name=Accounting,Type=Accounting
Operation Name: getBalances()
Data Points:
dailyBalance (type: gauge)
annualBalance (type: gauge)
MBeans are implemented as Java classes and Java permits parameterized polymorphic behavior. This means that multiple methods can be defined with the same name so long as their parameter signatures differ. You can safely define "getBalance(String)" and "getBalance()" and the two exist as separate methods.
In order to properly resolve methods with the same name the caller must provide a Class[] that lists the types of parameters that exist in the method's signature. This resolves the candidate methods to an individual method which can then be invoked by passing an Object[].
ZenJMX allows you to resolve methods of the same name and asks you to provide the fully qualified class names of each parameter in comma delimited format when you set up the data source. Note that primitive types (String, Boolean, Integer, Float) are supported but complex types are not supported, and that you must include the class' package name when providing the information (java.lang.String).
The Object[] of parameter values must line up with Class[] of parameter types, and if there is a mismatch in the number of types and values that are provided an exception will be generated.
The marshaling of values from String to Boolean, Integer, and Float types is provided via the .valueOf() static method on each of those types. That is, if you define an attribute of type java.lang.Integer you must provide a String that can be successfully passed to java.lang.Integer.fromValue(). If you fail to do so an exception is generated.
This example illustrates how to pass a single parameter to a polymorphic operation:
MBean Name: Application:Name=Accounting,Type=Accounting
Operation Name: getBalances()
Paramater Types: java.lang.Integer
Parameter Values: 1234
Data Points:
balance (type: gauge)
Here is another example where we've changed the type of the parameter passed to the method to be a String. Semantically it represents a different type of Account in our example:
MBean Name: Application:Name=Accounting,Type=Accounting
Operation Name: getBalances()
Paramater Types: java.lang.String
Parameter Values: sbb552349999
Data Points:
balance (type: gauge)
The above example describes how polymorphic behavior in Java functions and how method resolution can be provided by identifying the Class[] that represents the parameters passed to a method. The situation where multiple parameters are passed to a polymorphic operation is no different then the situation where a single parameter is passed to a polymorphic operation, except that the length of the Class[] and Object[] is greater than one.
When multiple parameters are required to invoke an operation you must provide the fully qualified class names of each parameter's type in comma delimited format, as well as the object values for each type (also in comma delimited format).
The following example demonstrates a configuration that passes two parameters to an MBean operation. The second parameter passed is a default value to return if no account can be located matching the first parameter.
MBean Name: Application:Name=Accounting,Type=Accounting
Operation Name: getBalances()
Parameter Types: java.lang.String, java.lang.Integer
Parameter Values: sbb552349999, 0
Data Points:
balance (type: gauge)
There are additional combinations that are possible with polymorphic methods and the values they return, and those combinations are left as an exercise for the reader to explore. The logic for extracting results from multi-value operation invocations follows the same rules as the logic for extracting results from a multi-value attribute read. For additional information on the rules of that logic see the section above on multi-value attributes.