Points of contact: Dale Wilson and Steve Totten
Object Computing, Inc. (OCI) and the Distributed Object Computing (DOC) group at Vanderbilt University's Institute for Software Intensive Systems (ISIS) collaborated on a research and development (R&D) effort to demonstrate the viability of the OMG FT CORBA specification (defined in Chapter 23 of the CORBA 3.0 specification), with some extensions, as a platform for building fault-tolerant DRE applications.
The OCI team designed, implemented, and tested FT service-level entities and other components needed to support an FT infrastructure in TAO, and a sample application to demonstrate TAO's FT capabilities. The ISIS team provided enhancements to TAO's ORB Core to support fault tolerance in applications, including implementing ORB-core-level features defined in the FT CORBA specification.
Extensions to the FT CORBA specification investigated during the project included:
SEMI_ACTIVE replication style similar to
that described here.The Replication Manager is perhaps the most visible of FT CORBA's infrastructure components. Fault tolerant services interact with the replication manager to create object groups, manage an object group's properties, control an object group's membership, and so forth. The Replication Manager is also solely responsible for the creation and maintenance of Interoperable Object Group References (IOGRs). According to the FT CORBA specification, the Replication Manager's operations are defined by three separate interfaces:
Note: The Data Parallel (DP) CORBA final adopted specification defines
a new PortableGroup module, including the three
interfaces listed above, to share common interfaces and their
supporting types among DP CORBA, FT CORBA, Load Balancing, and other
specifications. It is identical to a subset of the FT CORBA
specification with a few changes to make it more generic to group
management. TAO already has an implementation of the PortableGroup
module that we adapted for reuse by the Replication Manager's
implementation for this project.
In addition, the Replication Manager serves the role of a consumer for fault report events propagated to it via the Fault Notifier, so it must realize the Structured Push Consumer interface from the CosNotifyComm module (defined in the OMG's Notification Service specification (formal/02-08-04). Our design provides a Fault Consumer/Analyzer framework and concrete fault consumer and fault analyzer implementations that are tailored for use by the Replication Manager. The classes making up this framework, and the relationships among them, are shown in the figure below.
The OCI team also added a Factory Registry interface that is also realized by the Replication Manager to allow various types of factories, such as Replica Factories and Fault Detector Factories (all of which implement the Generic Factory interface) to register with the Replication Manager. The Replication Manager then uses these factories when necessary to add members (replicas) to object groups or to create a new Fault Detector at a specific location to monitor a replica.
The Replication Manager's interfaces are defined in FT_ReplicationManager.idl. The Replication Manager also supports interfaces and types from the PortableGroup module that is defined in PortableGroup.idl and additional interfaces and types from the FT module that is defined in FT_CORBA.idl. Source code for the Replication Manager's implementation is found in the FT_ReplicationManager directory.
FT CORBA's Fault Notifier is based upon a subset of the OMG's Notification Service specification (formal/02-08-04). The Fault Notifier typically gathers fault reports from Fault Detectors as well as from application- or platform-specific fault detectors.
The Fault Notifier can support an arbitrary number of Fault Detectors and consumers because it is based upon the Notification Service. Components interested in receiving fault reports assume the role of push consumer with respect to the Fault Notifier. The Replication Manager is one such component, as described above; an application may provide its own fault analysis capability by connecting an application-specific fault analyzer as a consumer to the Fault Notifier. (In fact, a real-world application will likely participate intimately in identifying and analyzing faults. One way this could be done is to "plug-in" an application-specific fault analyzer to the Replication Manager, using the Fault Consumer/Analyzer framework described above.)
The Fault Notifier's interfaces are defined in FT_Notifier.idl. Source code for the Fault Notifier's implementation is found in the Fault_Notifier directory.
The Fault Detector is the basic component in FT CORBA for monitoring a
fault tolerant system's software components, processes, and processing
nodes and reporting faults. The FT CORBA specification defines a
single monitoring style, the pull monitoring style, in which
a Fault Detector periodically issues a CORBA request
(is_alive) to monitored objects and reports faults for
those objects that fail to respond. A fault detector that monitors a
single replica may be co-located on the same host as that replica. If
the replica fails (defined as failure to reply to the detector's
is_alive invocation within a prescribed time-out period),
the detector issues a fault report to the Fault Notifier, which it
finds via the Replication Manager. In our example application, the
detector then exits since the replica that it was monitoring no longer
exists. Fault detectors can also be deployed on other nodes and used
to monitor other FT CORBA infrastructure components, such as a Fault
Detector or Fault Detector Factory on another host. The pull
monitoring style is used to monitor these components as well.
Fault Detectors are created and managed by a Fault Detector Factory. There may be many Fault Detector Factories deployed in a typical fault tolerant system. The Fault Detector Factory implements the Generic Factory interface.
Fault Detectors are created in the same process as their Fault Detector Factory. Each Fault Detector runs in its own thread and monitors its replica according to its prescribed monitoring interval (defined by a property on the object group). The Fault Detector Factory owns the thread manager for these threads. If a replica member is removed from an object group, the Fault Detector Factory that "owns" the Fault Detector that is monitoring that replica can cause the detector to "quit," thereby causing it to clean up any resources and its thread to exit.
Fault Detector Factories register with the Replication Manager via the Factory Registry interface. The Replication Manager then uses these Fault Detector Factories to create new Fault Detectors as needed to monitor replicas as they are created.
The Fault Detector Factory's interfaces are defined in FT_FaultDetectorFactory.idl. Source code for the Fault Detector and Fault Detector Factory implementations is found in the Fault_Detector directory.
To achieve fault tolerance, a system must not have a single point of failure. This includes not only application services, but infrastructure services as well. In the case of this project, the following FT infrastructure services need to be made fault tolerant via redundancy:
One of the initial goals of this project was to provide redundant implementations of each of these services after first providing basic non-redundant implementations. Unfortunately, due to complexities encountered during implementation and the relatively short time frame for the project, we did not complete development of redundant versions of the various FT infrastructure services. We encourage this to be given a high priority for any follow-on work.
Note that making the Replication Manager redundant will require direct access to the lower-level state synchronization mechanism (i.e., via a synchronization strategy) while other FT infrastructure services can likely be made fault tolerant using the full range of FT CORBA mechanisms.
A Replica Factory is an application-defined entity that implements the Generic Factory interface. There may be many Replica Factories deployed in a typical fault tolerant application. Each Replica Factory acts as an agent for the Replication Manager to create and manage replica members of object groups of a specific type at a specific location. Replica Factories register with the Replication Manager via the Factory Registry interface. The Replication Manager then uses these Replica Factories to create new replicas as needed when creating object groups or adding new members to existing object groups.
We have provided a sample implementation of a Replica Factory as part of our example application for this project. It implements the Generic Factory interface from the FT module defined in FT_CORBA.idl. Source code for the example application's Replica Factory is found in the FT_App directory.
A Replica is an application object that serves as a member of an object group. Each replica implements an application-defined interface. In addition, each replica must implement the Pull Monitorable interface so it can be monitored by a Fault Detector. Replicas are created by Replica Factories by the Replication Manager or by another application. Each new replica is then added to an object group and managed by the Replication Manager.
We have provided a sample implementation of a Replica as part of our
example application for this project. A Replica must implement the
PullMonitorable, Checkpointable, and
Updateable interfaces, which are defined in FT_Replica.idl.
For our example application, a test replica interface is defined in FT_App/FT_TestReplica.idl.
The implementation of the test replica is also in the FT_App
directory.
The Object Group Creator is a utility for creating an object group.
It can be used by an application to create an initial set of objects
in a system. The Object Group Creator finds the Replication Manager
and uses its Factory Registry interface to get a list of factories it
can use to create objects of the desired type. The Object Group
Creator can be used in different ways depending upon if the object
group's MembershipStyle property value is
application-controlled membership or infrastructure-controlled
membership.
add_member operation.set_type_properties operation of the Replication
Manager, then calls Replication Manager's create_object
operation to create an object group.After creating the object group, the Object Group Creator can optionally write the group's IOGR to a file or bind it in the Naming Service so it can be accessed by clients.
The Object Group Creator can exist as a stand-alone utility or it can be integrated with an application. Our example application includes an implementation of the Object Group Creator in the FT_App directory.
A client application obtains the object group reference from a file or
from the Naming Service and invokes operations on it as it would a
normal IOR. In the SEMI_ACTIVE replication style, only the primary
replica receives and processes each request. The state
synchronization strategy developed by the ISIS team for this project
is used synchronize state between the primary and backup replicas with
the completion of each request. If the primary replica fails, the
transparent reinvocation mechanism inherent in the FT ORB (also
developed by the ISIS team) causes the client's failed request to be
automatically reinvoked on a backup replica. Meanwhile, the fault
detection mechanisms described above are used to notify the
Replication Manager of the fault and the Replication Manager takes the
necessary actions to maintain the object group's integrity. Our
example application includes a simple client in the FT_App
directory.
The figure below shows the architecture of a prototypical FT system and the relationships among the various FT infrastructure and application-defined components described above.
The steps involved in orderly start-up and operation of an FT system are numbered in Figure 2 and described below:
The FT CORBA specification requires the Replication Manager to create
and maintain IOGRs. It also requires the FT ORB to perform
most-recent IOGR processing, whereby the FT ORB can update a
client using an old IOGR, by means of a LOCATION_FORWARD
reply, with a new IOGR. However, the specification fails to define a
way for the Replication Manager to propagate revised IOGRs to the FT
ORBs of object group members. Therefore, the OCI and ISIS teams
agreed upon a simple interface (tao_update_iogr) by which
the Replication Manager can propagate revised IOGRs to the FT ORB for
each member of an object group (e.g., after failure of a primary
replica, selection of a new primary member, and generation of a new
IOGR by the Replication Manager). While this interface is
TAO-specific, it accomplishes one of our research goals of
investigating formal protocols by which different ORB implementations
of FT CORBA could be made interoperable. The ISIS team implemented
this interface and the OCI team incorporated its use within the
Replication Manager.
To support FT CORBA, the Replication Manager must be able to create and manipulate IOGRs. For example, the Replication Manager's realization of the Generic Factory interface must return an IOGR. Also, upon receiving a fault report on an object group, the Replication Manager may need to remove a member, designate a member as the new primary replica, and generate a new IOGR that can then be propagated to each member of the object group.
For the purposes of this project, we used TAO's existing IORManipulation library for creating and managing IOGRs. However, the IORManipulation library lacked certain features that were needed. We worked with the ISIS team to define extensions to the IORManipulation library to:
While the IORManipulation library can be used to create and manipulate IOGRs, a longer term approach may be to use a specialized implementation of the Object Reference Template and IORInterceptor abstractions defined in sections 21.5.3 and 21.5.4 of the CORBA 3.0 specification, respectively.
FT CORBA infrastructure and application components must collaborate to achieve fault tolerance. To do so, infrastructure and application components must be started and initial objects and object groups created in an orderly fashion. Much of this "bootstrapping" can be accomplished by scripting. However, an entity to control the creation of initial objects and object groups can greatly simplify certain aspects of the bootstrapping process.
The sample application provided as part of this project uses an Object Group Creator utility to create initial objects and object groups. The Object Group Creator is implemented as a library that can be easily integrated with other parts of an application. A simple wrapper is also provided allowing the Object Group Creator to be used as a stand-alone executable.
During the course of this research, we uncovered several areas for further research and development, including:
