4 Administering GlassFish Server Clusters

Protocols and Transports for GMS

You can specify that GMS should use one of the following combinations of protocol and transport for broadcasting messages:

• User Datagram Protocol (UDP) multicast

• Transmission Control Protocol (TCP) without multicast

Even if GMS should use UDP multicast for broadcasting messages, you must ensure that TCP is enabled. On Windows systems, enabling TCP involves enabling a protocol and port for security when a firewall is enabled.

If GMS should use UDP multicast for broadcasting messages and if GlassFish Server instances in a cluster are located on different hosts, the following conditions must be met:

• The DAS host and all hosts for the instances must be on the same subnet.

• UDP multicast must be enabled for the network. To test whether multicast is enabled, use the validate-multicast(1) subcommand.

If GMS should use TCP without multicast, you must configure GMS to locate the instances to use for discovering the cluster. For more information, see Discovering a Cluster When Multicast Transport Is Unavailable.

GMS Configuration Settings

GlassFish Server has the following types of GMS settings:

• GMS cluster settings — These are determined during cluster creation. For more information about these settings, see To Create a Cluster.

• GMS configuration settings — These are determined during configuration creation and are explained here.

The following GMS configuration settings are used in GMS for group discovery and failure detection:

group-discovery-timeout-in-millis

Indicates the amount of time (in milliseconds) an instance's GMS module will wait during instance startup for discovering other members of the group.

The group-discovery-timeout-in-millis timeout value should be set to the default or higher. The default is 5000.

max-missed-heartbeats

Indicates the maximum number of missed heartbeats that the health monitor counts before the instance can be marked as a suspected failure. GMS also tries to make a peer-to-peer connection with the suspected member. If the maximum number of missed heartbeats is exceeded and peer-to-peer connection fails, the member is marked as a suspected failure. The default is 3.

heartbeat-frequency-in-millis

Indicates the frequency (in milliseconds) at which a heartbeat is sent by each server instance to the cluster.

The failure detection interval is the max-missed-heartbeats multiplied by the heartbeat-frequency-in-millis. Therefore, the combination of defaults, 3 multiplied by 2000 milliseconds, results in a failure detection interval of 6 seconds.

Lowering the value of heartbeat-frequency-in-millis below the default would result in more frequent heartbeat messages being sent out from each member. This could potentially result in more heartbeat messages in the network than a system needs for triggering failure detection protocols. The effect of this varies depending on how quickly the deployment environment needs to have failure detection performed. That is, the (lower) number of retries with a lower heartbeat interval would make it quicker to detect failures.

However, lowering this value could result in false positives because you could potentially detect a member as failed when, in fact, the member's heartbeat is reflecting the network load from other parts of the server. Conversely, a higher timeout interval results in fewer heartbeats in the system because the time interval between heartbeats is longer. As a result, failure detection would take a longer. In addition, a startup by a failed member during this time results in a new join notification but no failure notification, because failure detection and verification were not completed.

The default is 2000.

verify-failure-waittime-in-millis

Indicates the verify suspect protocol's timeout used by the health monitor. After a member is marked as suspect based on missed heartbeats and a failed peer-to-peer connection check, the verify suspect protocol is activated and waits for the specified timeout to check for any further health state messages received in that time, and to see if a peer-to-peer connection can be made with the suspect member. If not, then the member is marked as failed and a failure notification is sent. The default is 1500.

verify-failure-connect-timeout-in-millis

Indicates the time it takes for the GMS to detect a hardware or network failure of a server instance. Be careful not to set this value too low. The smaller this timeout value is, the greater the chance of detecting false failures. That is, the instance has not failed but doesn't respond within the short window of time. The default is 10000.

The heartbeat frequency, maximum missed heartbeats, peer-to-peer connection-based failure detection, and the verify timeouts are all needed to ensure that failure detection is robust and reliable in GlassFish Server.

For the dotted names for each of these GMS configuration settings, see Dotted Names for GMS Settings. For the steps to specify these settings, see To Preconfigure Nondefault GMS Configuration Settings.

Dotted Names for GMS Settings

Below are sample get subcommands to get all the GMS configuration settings (attributes associated with the referenced mycfg configuration) and GMS cluster settings (attributes and properties associated with a cluster named mycluster).

asadmin> get "configs.config.mycfg.group-management-service.*"
configs.config.mycfg.group-management-service.failure-detection.heartbeat-frequency-in-millis=2000
configs.config.mycfg.group-management-service.failure-detection.max-missed-heartbeats=3
configs.config.mycfg.group-management-service.failure-detection.verify-failure-connect-timeout-in-millis=10000
configs.config.mycfg.group-management-service.failure-detection.verify-failure-waittime-in-millis=1500
configs.config.mycfg.group-management-service.group-discovery-timeout-in-millis=5000

asadmin> get clusters.cluster.mycluster
clusters.cluster.mycluster.config-ref=mycfg
clusters.cluster.mycluster.gms-bind-interface-address=${GMS-BIND-INTERFACE-ADDRESS-mycluster}
clusters.cluster.mycluster.gms-enabled=true
clusters.cluster.mycluster.gms-multicast-address=228.9.245.47
clusters.cluster.mycluster.gms-multicast-port=9833
clusters.cluster.mycluster.name=mycluster

asadmin> get "clusters.cluster.mycluster.property.*"
clusters.cluster.mycluster.property.GMS_LISTENER_PORT=${GMS_LISTENER_PORT-mycluster}
clusters.cluster.mycluster.property.GMS_MULTICAST_TIME_TO_LIVE=4
clusters.cluster.mycluster.property.GMS_LOOPBACK=false
clusters.cluster.mycluster.property.GMS_TCPSTARTPORT=9090
clusters.cluster.mycluster.property.GMS_TCPENDPORT=9200

The last get subcommand displays only the properties that have been explicitly set.

For the steps to specify these settings, see To Preconfigure Nondefault GMS Configuration Settings and To Change GMS Settings After Cluster Creation.

To Preconfigure Nondefault GMS Configuration Settings

You can preconfigure GMS with values different than the defaults without requiring a restart of the DAS and the cluster.

1. Create a configuration using the copy-config subcommand.

   For example:

   asadmin> copy-config default-config mycfg
   

   For more information, see To Create a Named Configuration.

2. Set the values for the new configuration's GMS configuration settings.

   For example:

   asadmin> set configs.config.mycfg.group-management-service.group-discovery-timeout-in-millis=8000
   asadmin> set configs.config.mycfg.group-management-service.failure-detection.max-missed-heartbeats=5
   

   For a complete list of the dotted names for these settings, see Dotted Names for GMS Settings.

3. Create the cluster so it uses the previously created configuration.

   For example:

   asadmin> create-cluster --config mycfg mycluster
   

   You can also set GMS cluster settings during this step. For more information, see To Create a Cluster.

4. Create server instances for the cluster.

   For example:

   asadmin> create-instance --node localhost --cluster mycluster instance01
   
   asadmin> create-instance --node localhost --cluster mycluster instance02
   

5. Start the cluster.

   For example:

   asadmin> start-cluster mycluster
   

See Also

You can also view the full syntax and options of a subcommand by typing asadmin help subcommand at the command line.

To Change GMS Settings After Cluster Creation

To avoid the need to restart the DAS and the cluster, configure GMS configuration settings before cluster creation as explained in To Preconfigure Nondefault GMS Configuration Settings.

To avoid the need to restart the DAS and the cluster, configure the GMS cluster settings during cluster creation as explained in To Create a Cluster.

Changing any GMS settings using the set subcommand after cluster creation requires a domain administration server (DAS) and cluster restart as explained here.

1. Ensure that the DAS and cluster are running.

   Remote subcommands require a running server.

2. Use the get subcommand to determine the settings to change.

   For example:

   asadmin> get "configs.config.mycfg.group-management-service.*"
   configs.config.mycfg.group-management-service.failure-detection.heartbeat-frequency-in-millis=2000
   configs.config.mycfg.group-management-service.failure-detection.max-missed-heartbeats=3
   configs.config.mycfg.group-management-service.failure-detection.verify-failure-connect-timeout-in-millis=10000
   configs.config.mycfg.group-management-service.failure-detection.verify-failure-waittime-in-millis=1500
   configs.config.mycfg.group-management-service.group-discovery-timeout-in-millis=5000
   

   For a complete list of the dotted names for these settings, see Dotted Names for GMS Settings.

3. Use the set subcommand to change the settings.

   For example:

   asadmin> set configs.config.mycfg.group-management-service.group-discovery-timeout-in-millis=6000
   

4. Use the get subcommand again to confirm that the changes were made.

   For example:

   asadmin> get configs.config.mycfg.group-management-service.group-discovery-timeout-in-millis
   

5. Restart the DAS.

   For example:

   asadmin> stop-domain domain1
   
   asadmin> start-domain domain1
   

6. Restart the cluster.

   For example:

   asadmin> stop-cluster mycluster
   
   asadmin> start-cluster mycluster
   

See Also

You can also view the full syntax and options of a subcommand by typing asadmin help subcommand at the command line.

To Check the Health of Instances in a Cluster

The get-health subcommand only works when GMS is enabled. This is the quickest way to evaluate the health of a cluster and to detect if cluster is properly operating; that is, all members of the cluster are running and visible to DAS.

If multicast is not enabled for the network, all instances could be running (as shown by the list-instances subcommand), yet isolated from each other. The get-health subcommand does not show the instances if they are running but cannot discover each other due to multicast not being configured properly. See To Validate That Multicast Transport Is Available for a Cluster.

1. Ensure that the DAS and cluster are running.

   Remote subcommands require a running server.

2. Check whether server instances in a cluster are running by using the get-health subcommand.

asadmin> get-health cluster1
instance1 started since Wed Sep 29 16:32:46 EDT 2010
instance2 started since Wed Sep 29 16:32:45 EDT 2010
Command get-health executed successfully.

See Also

You can also view the full syntax and options of the subcommand by typing asadmin help get-health at the command line.

To Validate That Multicast Transport Is Available for a Cluster

Before You Begin

To test a specific multicast address, multicast port, or bind interface address, get this information beforehand using the get subcommand. Use the following subcommand to get the multicast address and port for a cluster named c1:

asadmin> get clusters.cluster.c1
clusters.cluster.c1.config-ref=mycfg
clusters.cluster.c1.gms-bind-interface-address=${GMS-BIND-INTERFACE-ADDRESS-c1}
clusters.cluster.c1.gms-enabled=true
clusters.cluster.c1.gms-multicast-address=228.9.174.162
clusters.cluster.c1.gms-multicast-port=5383
clusters.cluster.c1.name=c1

Use the following subcommand to get the bind interface address of a server instance named i1that belongs to a cluster named c1, if this system property has been set:

asadmin> get servers.server.i1.system-property.GMS-BIND-INTERFACE-ADDRESS-c1
servers.server.i1.system-property.GMS-BIND-INTERFACE-ADDRESS-c1.name=GMS-BIND-INTERFACE-ADDRESS-c1
servers.server.i1.system-property.GMS-BIND-INTERFACE-ADDRESS-c1.value=10.12.152.30

For information on how to set this system property, see Using the Multi-Homing Feature With GMS.

Check whether multicast transport is available for a cluster by using the validate-multicast subcommand.

asadmin> validate-multicast
Will use port 2048
Will use address 228.9.3.1
Will use bind interface null
Will use wait period 2,000 (in milliseconds)

Listening for data...
Sending message with content "sr1" every 2,000 milliseconds
Received data from sr1 (loopback)
Received data from sr2
Exiting after 20 seconds. To change this timeout, use the --timeout command line option.
Command validate-multicast executed successfully.

asadmin> validate-multicast
Will use port 2048
Will use address 228.9.3.1
Will use bind interface null
Will use wait period 2,000 (in milliseconds)

Listening for data...
Sending message with content "sr2" every 2,000 milliseconds
Received data from sr2 (loopback)
Received data from sr1
Exiting after 20 seconds. To change this timeout, use the --timeout command line option.
Command validate-multicast executed successfully.

Next Steps

As long as all machines see each other, multicast is validated to be working properly across the machines. If the machines are not seeing each other, set the --bindaddress option explicitly to ensure that all machines are using interface on same subnet, or increase the --timetolive option from the default of 4. If these changes fail to resolve the multicast issues, ask the network administrator to verify that the network is configured so the multicast messages can be seen between all the machines used to run the cluster.

See Also

You can also view the full syntax and options of the subcommand by typing asadmin help get-health at the command line.

Discovering a Cluster When Multicast Transport Is Unavailable

When multicast transport is unavailable, GlassFish Server instances that are joining a cluster cannot rely on broadcast messages from GMS to discover the cluster. Instead, an instance that is joining a cluster uses a running instance or the DAS in the cluster to discover the cluster.

Therefore, when multicast transport is unavailable, you must provide the locations of instances in the cluster to use for discovering the cluster. You are not required to provide the locations of all instances in the cluster. However, for an instance to discover the cluster, at least one instance whose location you provide must be running. To increase the probability of finding a running instance, provide the locations of several instances.

If the DAS will be left running after the cluster is started, provide the location of the DAS first in the list of instances. When a cluster is started, the DAS is running before any of the instances in the cluster are started.

The locations of the instances to use for discovering a cluster are part of the configuration data that you provide when creating the cluster. How to provide this data depends on how instances are distributed, as explained in the following subsections:

• To Discover a Cluster When Multiple Instances in a Cluster are Running on a Host

• To Discover a Cluster When Each Instance in a Cluster Is Running on a Different Host

asadmin> create-system-properties GMS_LISTENER_PORT-tcpcluster=9090
Command create-system-properties executed successfully.
asadmin> restart-domain
Successfully restarted the domain
Command restart-domain executed successfully.
asadmin> create-cluster --properties GMS_DISCOVERY_URI_LIST=
tcp'\\:'//10.152.23.224'\\:'9090,
tcp'\\:'//10.152.23.224'\\:'9091,
tcp'\\:'//10.152.23.224'\\:'9092 tcpcluster
Command create-cluster executed successfully.
asadmin> create-local-instance 
--systemproperties GMS_LISTENER_PORT-tcpcluster=9091 --cluster tcpcluster 
instance101
Rendezvoused with DAS on localhost:4848.
Port Assignments for server instance instance101:
JMX_SYSTEM_CONNECTOR_PORT=28686
JMS_PROVIDER_PORT=27676
HTTP_LISTENER_PORT=28080
ASADMIN_LISTENER_PORT=24848
JAVA_DEBUGGER_PORT=29009
IIOP_SSL_LISTENER_PORT=23820
IIOP_LISTENER_PORT=23700
OSGI_SHELL_TELNET_PORT=26666
HTTP_SSL_LISTENER_PORT=28181
IIOP_SSL_MUTUALAUTH_PORT=23920
Command create-local-instance executed successfully.
asadmin> create-local-instance 
--systemproperties GMS_LISTENER_PORT-tcpcluster=9092 --cluster tcpcluster 
instance102
Rendezvoused with DAS on localhost:4848.
Using DAS host localhost and port 4848 from existing das.properties for node
localhost-domain1. To use a different DAS, create a new node using 
create-node-ssh or create-node-config. Create the instance with the new node and
correct host and port:
asadmin --host das_host --port das_port create-local-instance --node node_name 
instance_name.
Port Assignments for server instance instance102:
JMX_SYSTEM_CONNECTOR_PORT=28687
JMS_PROVIDER_PORT=27677
HTTP_LISTENER_PORT=28081
ASADMIN_LISTENER_PORT=24849
JAVA_DEBUGGER_PORT=29010
IIOP_SSL_LISTENER_PORT=23821
IIOP_LISTENER_PORT=23701
OSGI_SHELL_TELNET_PORT=26667
HTTP_SSL_LISTENER_PORT=28182
IIOP_SSL_MUTUALAUTH_PORT=23921
Command create-local-instance executed successfully.

asadmin> create-cluster --properties 'GMS_DISCOVERY_URI_LIST=
10.152.23.225,10.152.23.226,10.152.23.227,10.152.23.228:
GMS_LISTENER_PORT=9090' ipcluster
Command create-cluster executed successfully.

asadmin> create-cluster --properties 'GMS_DISCOVERY_URI_LIST=generate:
GMS_LISTENER_PORT=9090' gencluster
Command create-cluster executed successfully.

Using the Multi-Homing Feature With GMS

Multi-homing enables GlassFish Server clusters to be used in an environment that uses multiple Network Interface Cards (NICs). A multi-homed host has multiple network connections, of which the connections may or may not be the on same network. Multi-homing provides the following benefits:

• Provides redundant network connections within the same subnet. Having multiple NICs ensures that one or more network connections are available for communication.

• Supports communication across two or more different subnets. The DAS and all server instances in the same cluster must be on the same subnet for GMS communication, however.

• Binds to a specific IPv4 address and receives GMS messages in a system that has multiple IP addresses configured. The responses for GMS messages received on a particular interface will also go out through that interface.

• Supports separation of external and internal traffic.

To Create a Cluster

Use the create-cluster subcommand in remote mode to create a cluster.

To ensure that the GMS can detect changes in cluster membership, a cluster's GMS settings must be configured correctly. To avoid the need to restart the DAS and the cluster, configure a cluster's GMS settings when you create the cluster. If you change GMS settings for an existing cluster, the DAS and the cluster must be restarted to apply the changes.

When you create a cluster, GlassFish Server automatically creates a Message Queue cluster for the GlassFish Server cluster. For more information about Message Queue clusters, see Using Message Queue Broker Clusters With GlassFish Server.

Before You Begin

If the cluster is to reference an existing named configuration, ensure that the configuration exists. For more information, see To Create a Named Configuration. If you are using a named configuration to preconfigure GMS settings, ensure that these settings have the required values in the named configuration. For more information, see To Preconfigure Nondefault GMS Configuration Settings.

If you are configuring the cluster's GMS settings when you create the cluster, ensure that you have the following information:

• The address on which GMS listens for group events

• The port number of the communication port on which GMS listens for group events

• The maximum number of iterations or transmissions that a multicast message for GMS events can experience before the message is discarded

• The lowest port number in the range of ports from which GMS selects a TCP port on which to listen

• The highest port number in the range of ports from which GMS selects a TCP port on which to listen

If the DAS is running on a multihome host, ensure that you have the Internet Protocol (IP) address of the network interface on the DAS host to which GMS binds.

1. Ensure that the DAS is running.

   Remote subcommands require a running server.

2. Run the create-cluster subcommand.

   Note:

   Only the options that are required to complete this task are provided in this step. For information about all the options for configuring the cluster, see the create-cluster(1) help page.

   • If multicast transport is available, run the create-cluster subcommand as follows:

     asadmin> create-cluster --config configuration 
     --multicastaddress multicast-address --multicastport multicast-port 
     --properties GMS_MULTICAST_TIME_TO_LIVE=max-iterations:
     GMS_TCPSTARTPORT=start-port:GMS_TCPENDPORT=end-port cluster-name
     

   • If multicast transport is not available, run the create-cluster subcommand as follows:

     asadmin> create-cluster --config configuration 
     --properties GMS_DISCOVERY_URI_LIST=discovery-instances:
     GMS_LISTENER_PORT=gms-port
     cluster-name
     

   configuration

   An existing named configuration that the cluster is to reference.

   multicast-address

   The address on which GMS listens for group events.

   multicast-port

   The port number of the communication port on which GMS listens for group events.

   max-iterations

   The maximum number of iterations or transmissions that a multicast message for GMS events can experience before the message is discarded.

   discovery-instances

   Instances to use for discovering the cluster. For more information, see Discovering a Cluster When Multicast Transport Is Unavailable.

   gms-port

   The port number of the port on which the cluster listens for messages from GMS.

   start-port

   The lowest port number in the range of ports from which GMS selects a TCP port on which to listen. The default is 9090.

   end-port

   The highest port number in the range of ports from which GMS selects a TCP port on which to listen. The default is 9200.

   cluster-name

   Your choice of name for the cluster that you are creating.

3. If necessary, create a system property to represent the IP address of the network interface on the DAS host to which GMS binds.

   This step is necessary only if the DAS is running on a multihome host.

   asadmin> create-system-properties 
   GMS-BIND-INTERFACE-ADDRESS-cluster-name=das-bind-address
   

   cluster-name

   The name that you assigned to the cluster in Step 2.

   das-bind-address

   The IP address of the network interface on the DAS host to which GMS binds.

asadmin> create-cluster 
--systemproperties IIOP_SSL_LISTENER_PORT=1169 
ltscluster
Command create-cluster executed successfully.

asadmin> create-cluster --config clusterpresets 
--multicastaddress 228.9.3.1 --multicastport 2048 
--properties GMS_MULTICAST_TIME_TO_LIVE=3:
GMS_TCPSTARTPORT=10000:GMS_TCPENDPORT=10100 pmdcluster
Command create-cluster executed successfully.

Next Steps

After creating a cluster, you can add GlassFish Server instances to the cluster as explained in the following sections:

• To Create an Instance Centrally

• To Create an Instance Locally

See Also

• To Create a Named Configuration

• To Preconfigure Nondefault GMS Configuration Settings

• Using Message Queue Broker Clusters With GlassFish Server

• create-cluster(1)

• create-system-properties(1)

You can also view the full syntax and options of the subcommands by typing the following commands at the command line:

• asadmin help create-cluster

• asadmin help create-system-properties

To List All Clusters in a Domain

Use the list-clusters subcommand in remote mode to obtain information about existing clusters in a domain.

1. Ensure that the DAS is running.

   Remote subcommands require a running server.

2. Run the list-clusters subcommand.

   asadmin> list-clusters
   

asadmin> list-clusters
pmdclust not running
ymlclust not running
Command list-clusters executed successfully.

asadmin> list-clusters sj01
ymlclust not running
Command list-clusters executed successfully.

See Also

list-clusters(1)

You can also view the full syntax and options of the subcommand by typing asadmin help list-clusters at the command line.

To Delete a Cluster

Use the delete-cluster subcommand in remote mode to remove a cluster from the DAS configuration.

If the cluster's named configuration was created automatically for the cluster and no other clusters or unclustered instances refer to the configuration, the configuration is deleted when the cluster is deleted.

Before You Begin

Ensure that following prerequisites are met:

• The cluster that you are deleting is stopped. For information about how to stop a cluster, see To Stop a Cluster.

• The cluster that you are deleting contains no GlassFish Server instances. For information about how to remove instances from a cluster, see the following sections:

  • To Delete an Instance Centrally

  • To Delete an Instance Locally

1. Ensure that the DAS is running.

   Remote subcommands require a running server.

2. Confirm that the cluster is stopped.

   asadmin> list-clusters cluster-name
   

   cluster-name

   The name of the cluster that you are deleting.

3. Confirm that the cluster contains no instances.

   asadmin> list-instances cluster-name
   

   cluster-name

   The name of the cluster that you are deleting.

4. Run the delete-cluster subcommand.

   asadmin> delete-cluster cluster-name
   

   cluster-name

   The name of the cluster that you are deleting.

asadmin> list-clusters adccluster
adccluster not running
Command list-clusters executed successfully.
asadmin> list-instances adccluster
Nothing to list.
Command list-instances executed successfully.
asadmin> delete-cluster adccluster
Command delete-cluster executed successfully.

See Also

• To Stop a Cluster

• To Delete an Instance Centrally

• To Delete an Instance Locally

• delete-cluster(1)

• list-clusters(1)

• list-instances(1)

You can also view the full syntax and options of the subcommands by typing the following commands at the command line:

• asadmin help delete-cluster

• asadmin help list-clusters

• asadmin help list-instances