Domain and Cluster Management in JOnAS

The content of this guide is the following:

Target Audience and Rationale

This guide is intended for JOnAS administrators responsible for the configuration and administration of JOnAS servers running within a management domain.

Domain and Cluster Management Functions

A JOnAS management domain is composed of a set of JOnAS servers that are running under the same management authority. All the servers in the domain must have a distinct server name and a common domain name.

The servers in a domain can be administered by a management application running on a server playing the role of administrator or master. The managed servers play the role of slaves. Note that it is possible to have several masters in a domain. Also note that default configuration corresponds to a slave server running without the discovery service (the role of this service is described below and its configuration here).

Typically, when deploying the JonasAdmin application on a slave, the administrator can manage only the server on which the application is running. When deploying JonasAdmin on a master server, the administrator can manage all the known servers in the domain:

the administrator can see the state of the servers belonging to the domain;
can apply management operations to one or more servers in the domain;
switch management from the master to any other running server in the domain, then switch back to the master.
Starting with JOnAS 4.8, administration is enriched with domain and cluster monitoring features, and remote control.

Cluster management facilities was introduced in JOnAS 4.7. A cluster is a group of servers having common properties within a domain. A cluster may be the target of a domain level management operation - currently applications and J2EE modules deployment only.

The domain topology (servers and clusters composing the domain) can be defined using a new domain.xml configuration file. Also, servers and clusters may be dynamically added and removed to the domain via JonasAdmin.

Starting with JOnAS 4.8, domain and cluster management is enhanced with monitoring functions available in JonasAdmin. As in the previous version, clusters may be created by an administrator in a static (via domain.xml) or dynamic (via JonasAdmin) way. These clusters are now called logical cluster. A new class of clusters is supported in the current version, the physical clusters that are detected automatically by the management infrastructure. Several types of physical clusters are supported: Mod JK clusters, Tomcat clusters, CMI clusters, etc. All the members of a type of physical cluster share specific properties which depend on the cluster type, e.g., all the members of a CMI cluster have the same JGroups configuration. Note that a given JOnAS instance may belong to several physical and logical clusters.

An important domain level administration operation introduced in JOnAS 4.8, is the possiblity to start/stop the managed servers via a so called cluster daemon. Cluster daemons can be defined and associated to servers using the domain.xml configuration file. A cluster daemon has to be collocalized (located on the same machine) with the servers it controls.

Mechanisms used by Domain Management

Basically, domain management in JOnAS relies on JSR 160 specification. When a JOnAS server is started, it creates at least one JSR 160 connector server as explained in JSR 160 support in JOnAS.

A connector server makes it possible for a JMX remote API client to access and manage the MBeans exposed through the MBean server running in a remote JVM. In order to establish a connection, the remote client needs to know the address of the connector server. JSR 160 does not provide any specific API that would make it possible for a client to find the address of a connector server. The standard suggests using existing discovery and lookup infrastructures, for instance, JNDI API with an LDAP back end.

A new service added to JOnAS, the Discovery Service, allows JOnAS servers running over a LAN within the same management domain to communicate to each other the connector-server addresses they create at start-up. All the servers in the domain having the discovery service in the services list, will publish their connector-server address at start-up. The goal is to allow master servers to discover which servers are running in their domain, and to establish connections allowing them to remotely manage the slave servers by a management application deployed on a master server.

Starting with JOnAS 4.7, a server can be added to a domain via a management operation, thus allowing servers which cannot use the multicast-based discovery service to join a management domain.

The current discovery service implementation is based on MBeans (called discovery MBeans) which use:

IP Multicast protocol to communicate their connector-server addresses and state transitions to each other.
JMX notifications to update domain management information within a master server.

J2EEDomain MBean

Domain management information can be retrieved from J2EEDomain MBean.

The J2EEDomain MBean has the following JMX ObjectName:

		domainName:j2eeType=J2EEDomain,name=domainName

Where domainName is the name of the domain.

A J2EEDomain MBean contains several management attributes and operations related to servers management in a domain:

If the J2EEDomain is registered in a master server's MBean server, the servers attribute keeps the list of the servers known in the domain. Otherwise, it contains only one element corresponding to itself. The elements in the servers list are Strings representing J2EEServer ObjectNames associated to the running servers. By iterating over the servers list, a management application can determine the name of the servers in the domain.
The getConnectorServerURLs operation returns the connector server addresses for a server in the list.
In order to administer a given server in the domain, any management application has to create an MBeanServerConnection object using one of the connector-server addresses corresponding to that server (see the j2eemanagement sample).

Starting with JOnAS 4.7, the J2EEDomain MBean was enriched in order to support adding/removing a server to/from the domain, creating a cluster in the domain, listing the clusters in a domain.

The following is a list of some of the new management attributes and operations exposed by a J2EEDomain MBean:

addServer(String serverName, String[] connectorServerURLs)
getServerState(serverName)
createCluster(String clusterName)
clusters attribute gives the String representation of the J2EEDomain ObjectNames associated to the clusters created in this domain (or in this cluster if the current J2EEDomain MBean is associated to a cluster).

In JOnAS 4.7, clusters are implemented as "sub-domains", they have associated J2EEDomain type MBeans. For example, if a cluster named clust is created in the domain named domainName, it has an associated MBean with the following JMX ObjectName:

		domainName:j2eeType=J2EEDomain,name=domainName,clusterName=clust,parentClusterName=domainName

MBeans for domain/cluster monitoring

An important re-engineering of domain management mechanisms was conducted in JOnAS 4.8 to support cluster and domain monitoring. Clusters aren't implemented as sub-domains anymore; there is only one J2EEDomain MBean registered in the master's MBean server. Another important point is that the servers' state is no longer accessible in the J2EEDomain MBean (getServerState(serverName) operation doesn't exist anymore). On the other hand, new MBeans were created:

ServerProxy MBeans. For every server known in the domain, a ServerProxy MBean is created with the following JMX ObjectName:
```
		domainName:type=ServerProxy,name=serverName
	
```
It contains a State attribute which gives the server state as known by the master, and a set of monitoring information like memory usage, threads, transactions, etc. which are periodically updated. The servers may be in one of the following states:
- INIT - corresponds to a server that was added to the domain. This is a temporary state.
- UNREACHABLE - corresponds to a server that was added to the domain but the master is unable to access the managed server (the MBeanServerConnection could not be established, or it doesn't work anymore)
- RUNNING - corresponds to a server that is running and controlled by the master
- STOPPED - corresponds to a server which was stopped with the jonas stop command (if the server has enabled its discovery service), or if the server was stopped using the remote control mechanisms (via the cluster daemon).
- FAILED - corresponds to a particular case of UNREACHABLE state. The master decides to pass a server in the FAILED state if it is UNREACHABLE for a while, and if it has some information allowing to infer that the server is failed: First, the managed server had its discovery service enabled (otherwise, the server stop can't be detected, so can't distinguish STOPPED from FAILED). Moreover, the master can use information provided by the cluster daemon, if the server has an associated, running, cluster daemon.
Cluster MBeans. For every cluster created in the domain, a LogicalCluster MBean is created with the following JMX ObjectName:
```
		domainName:type=LogicalCluster,name=clusterName
	
```
The master may have registered other types of cluster MBeans which correspond to physical clusters. They are automatically created by the management system if managed servers are identified as being members of supported physical cluster types.
A cluster MBean contains a State attribute and a member's list. A cluster may be in one of the following states, which depends on the state of its members.
- INIT - corresponds to a cluster that was created in the domain. This is a temporary state.
- UP - all the members are in RUNNING state
- DOWN - all the members are STOPPED
- FAILED - all the members are FAILED
- PARTIALLY_FAILED - at least one member is FAILED
- PARTIALLY_UP - at least one member is RUNNING, there is no failed member
- PARTIALLY_DOWN- at least one member is STOPPED, there is no FAILED member, there is no RUNNING member

ClusterMember MBeans. A member MBean's role is to make the link between a cluster MBean and the ServerProxy MBean corresponding to the server which belongs to that cluster. Any member MBean has a ServerProxy attribute equal to the corresponding ServerProxy MBean OBJECT_NAME. If a server belongs to several clusters, there is a member MBean for each cluster, and all these members have the same value for the ServerProxy attribute. There are several types of member MBeans. They correspond to the several cluster types. The LogicalClusterMember MBeans are used to represent LogicalCluster members.

Physical clusters

Logical clusters are created by the administrator to facilitate management administration on a group of servers. Physical clusters are created automatically if servers running in the domain have particular properties which depend on the cluster type.

Mod_jk load balancing clusters. The members of such a cluster are workers defined in the mod_jk configuration file workers.properties. The workers correspond to JOnAS server instances having the web service based on Tomcat and a specific configuration (server.xml having a Connector XML element for AJP1.3 protocol defined, as well as the Engine element having the jvmRoute attribute set to the worker name).
The ObjectNames corresponding to Mod_jk clusters and Mod_jk cluster members are:
```
		domainName:type=JkCluster,name=clusterName
	
```
```
		domainName:type=JkClusterMember,name=workerName,JkCluster=clusterName
	
```
The workerName and clusterName are defined in the workers.properties configuration file.
Tomcat session replication clusters. The members are JOnAS-Tomcat cluster servers having a Cluster element defined in server.xml file. The management infrastructure detects Tomcat cluster members by looking for Tomcat Cluster MBeans in the managed server's MBean server.
The ObjectNames corresponding to Tomcat session replication clusters and Tomcat cluster members are:
```
		domainName:type=TomcatCluster,name=clusterName
	
```
```
		domainName:type=TomcatClusterMember,name=serverName,TomcatCluster=clusterName
	
```
The clusterName can be defined in the server.xml configuration file.
CMI clusters. The members are JOnAS cluster servers sharing the same CMI configuration for the JNDI replication and the EJB clustering.
The ObjectNames corresponding to the CMI clusters and cluster members are:
```
		domainName:type=CmiCluster,name=clusterName
	
```
```
		domainName:type=CmiClusterMember,name=serverName,CmiCluster=clusterName
	
```
The clusterName can be defined in the carol.properties configuration file and represents a JavaGroups group name.
HA clusters. The members are JOnAS cluster servers sharing the same HA service configuration for the EJB replication.
The ObjectNames corresponding to the HA clusters and cluster members are:
```
		domainName:type=HaCluster,name=clusterName
	
```
```
		domainName:type=HaClusterMember,name=serverName,HaCluster=clusterName
	
```
The clusterName can be defined in the jonas.properties configuration file and represents a JavaGroups group name.

How to start manageable servers in a Domain

The servers must adhere to these rules:

The servers must be started on hosts connected by a LAN.
The servers must be configured with the discovery service activated (see Configuring Discovery Service).
There must be at least one master server in the domain. The role of a server, slave or master, is determined by the discovery service configuration.
The servers must have different names, but the same domain name.

Also note the following:

Servers may be started in any order.
Servers may join or leave the domain at any time. Domain management information is updated on the master server(s).
A slave server may become master by invoking the startDiscoveryMaster management operation exposed by the discovery service. This operation can be invoked using the JonasAdmin management console deployed on a master.

Example

Consider a scenario in which there are three JOnAS servers named jonas, j1 and j2. Assume that they have discovery service configured with at least one of the servers playing the role of master.

Start the server named jonas in domain jonas (by default, the domain name is given by the server's name)

		jonas start

Start the server named j1 in domain jonas

		jonas start -n j1 -Ddomain.name=jonas

Start the server named j2 in domain jonas

		jonas start -n j2 -Ddomain.name=jonas

How to define domain configuration using domain.xml file

A default domain configuration is provided in $JONAS_ROOT/conf/domain.xml. This configuration corresponds to a domain named jonas managed by a master server also named jonas. The location tag defines a JMX remote connector server URL for this server.