Documentation
 
 
 

Chapter 21. EnterpriseDB Replication

21.1. Oracle to EnterpriseDB Replication

21.1.1. Introduction

The EnterpriseDB Replication Server is an asynchronous one master to one slave replication system. The replication framework is integrated and installed as part of EnterpriseDB DBA Management Server.

The EnterpriseDB Replication Server is a trigger-based replication framework that provides one-way asynchronous data replication from a master Oracle database to slave EnterpriseDB databases. The framework supports "snapshot" and "synchronize" replication modes, such that in "snapshot" mode a copy of the source table data is replicated to the destination database whereas "synchronize" mode transfers incremental changes.

21.1.2. EnterpriseDB Replication Concepts

Transactional Replication

The EnterpriseDB Replication Framework uses a trigger-based approach to perform replication. A shadow table is created for each of the publication tables and post-insert, post-update, and post-delete triggers are used to capture changes in the source tables. The trigger implementation is kept simple to avoid overhead - it simply inserts the required data in the shadow table.

When synchronization occurs, the changes recorded in the shadow tables are applied to the subscription tables as batch updates. Within each batch, changes may be applied using either an individual SQL statement for each change, or a set of changes may be applied using a prepared SQL statement. Generally, if it is determined that there is a sequence of consecutive changes to occur to the same table using the same type of operation (insert, update, or delete), then these changes will be applied using a prepared statement. If these conditions are not true, then each change will be applied with its own individual SQL statement.

The use of a prepared statement is also controlled by a parameter named, "busBatchThresholdCount". The number of consecutive changes using the same table and SQL statement type must exceed "busBatchThresholdCount" before a prepared statement will be used. Thus, setting this threshold to a low value will encourage higher use of prepared statements while setting it to a high value will limit the use of prepared statements.

The user can customize this value as illustrated below.

After specifying the value for the threshold count, the value is saved by pressing the "Save" button. The value of property busBatchThresholdCount takes effect on the next start-up of the publication service.

Snapshot-Only Replication

In this type of replication, any existing data in the subscription table is truncated and a complete copy of the publication table data is replicated. If there are no frequent changes in the source table(s), then "snapshot" replication is recommended as triggers and shadow tables will not be created against source tables.

21.1.3. Application Architecture

The EnterpriseDB Replication Framework consists of the following three components:

21.1.3.1. Publication Server

The publication server interacts with the publication source database (Oracle) and exposes the RMI (Remote Method Invocation) interface for replication management (i.e., publication creation and snapshot/synchronize operations) that is consumed by the subscription server and replication console. Additionally it communicates with the subscription database (EnterpriseDB) and replicates database changes over a JDBC layer.

The publication server has a one-to-one mapping with the source Oracle database server and manages replication for one or more schemas residing on the associated database instance. Multiple subscription servers can communicate and subscribe to one or more publications available on a publication server, however a single publication cannot have more than one subscription.

21.1.3.2. Subscription Server

The subscription server interacts with the subscription database (EnterpriseDB) and exposes an RMI interface for replication management (i.e., subscription creation and snapshot/synchronize operations) that is consumed by the replication console.

The subscription server handles replication for a single instance of the EnterpriseDB database server and can contain subscriptions from one or more publication servers.

21.1.3.3. Replication Console

The EnterpriseDB replication console is a desktop application that provides an easy to use graphical console for replication administration. The user can register a publication and/or subscription server and perform the available operations that include publication/subscription creation, replication schedule management, and snapshot/synchronize operations.

The EnterpriseDB replication console is a Java Swing based application designed to run seamlessly on all platforms that support Java Runtime Environment 5 and above. Briefly, the console provides the following operations for replication management:

Publication Servers

  • Registering/unregistering multiple publication servers.

  • Creating/removing publications in the publication server.

  • Adding/removing tables to/from publications.

Subscription Servers

  • Registering/unregistering multiple subscription servers.

  • Creating/updating/removing subscriptions in the subscription server.

  • Snapshot/synchronize data from a publication to a subscription.

  • Configuring/removing schedules for snapshot/synchronize jobs for subscriptions.

The console communicates with publication and subscription servers using the Java Remote Method Invocation (RMI) technology which allows the console to be placed on a different computer in a network.

21.1.4. Installation

The EnterpriseDB Replication Server and replication console are installed and configured as part of the DBA Management Server installation.

21.1.4.1. Prerequisites

The machine on which the EnterpriseDB Replication application will run must meet the following conditions:

  • Java Runtime Environment 1.5 should be installed on the machine and $JAVA_HOME/bin is available in the system PATH environment variable.

  • The publication and subscription database servers (i.e., EnterpriseDB and/or Oracle) must be running on the same machine or the machines must have network access to the system(s) that will host the Publication and Subscription databases.

21.1.5. Key Features

Multi-Database Server Publication and Subscription

The publication and subscription servers in the EnterpriseDB Replication Server can manage more than one database server. The available publications and subscriptions are grouped under their respective publication and subscription database nodes.

Replication Support for Oracle Views

The EnterpriseDB Replication Server also supports the publishing of Oracle views which are then mapped to tables in EnterpriseDB.

Connection Verification Option

The user can optionally verify the database connectivity when an Oracle or EnterpriseDB database is registered for publication or subscription.

Horizontal Partitioning

Horizontal partitioning, also known as row-level partitioning, is used to selectively limit the rows of a publication table that are available for replication. The filter clause is used to provide this capability.

Create TxSet Job

Transaction sets are created to group unprocessed transactions during a synchronize operation. However if the publication server is unavailable for a long period of time, the unprocessed transaction volume could grow and transaction set creation could slow down the database when the next synchronization occurs. A set creation job is scheduled to run every 2 hours in the database server context..

Create TxSet Job is performed on the database level over which the user has no control. In the DBA Management Server, it's performed every 10 minutes.

21.1.6. Data Type Support

In EnterpriseDB, data replication (snapshot and synchronize) is not supported for the following Redwood data types.

  • MLSLABEL

  • XMLTYPE

  • BFILE

  • BINARY_DOUBLE

  • BINARY_FLOAT

The following data types are not supported in synchronize (transactional) replication:

  • BLOB

  • LONG RAW

  • RAW

  • LONG

  • CLOB

  • NCLOB

The large objects types e.g. BLOB, RAW, LONG, LONG RAW, CLOB and NCLOB are either restricted e.g. data type LONG is not captured by triggers, or are not recommended e.g. there may be multi-giga bytes of data in a single CLOB column. Any tables with these data types are normally less transactional in nature and are recommended to be replicated via snapshot publication. The user may choose transactional publication for such tables if partial data replication (excluding the large object type columns) is desired.

21.1.7. EnterpriseDB Replication Framework Configuration

21.1.7.1. Replication Servers Life Cycle

The publication and subscription services' life cycles are bound to that of the EnterpriseDB DBA Management Server. The DBA Management Server remembers the last state of the publication/subscription service at server shutdown time and starts the publication and/or subscription service accordingly when it is started next time.

21.1.7.2. Replication Cluster Configuration with an Example

The user can either run all the components on a single machine or each component can be configured and run on a separate node.

21.1.7.2.1. Replication Admin User Creation

The replication framework creates different control schema objects such as shadow tables, triggers, sequences, and packages in the publication source database that are used to record the replicated table transactions and maintain the replication life cycle. The replication user needs certain privileges to create the control schema objects during replication configuration as described below.

It is recommended to create a separate replication admin user (e.g., rrepadmin) in Oracle to host the control schema objects so that the published tables source schemas are not over-populated with redundant information. The replication admin user should be granted the required privileges as listed below:

  • Create a user (e.g., "rrepadmin") in the source Oracle database server.

  • Grant the following privilege to rrepadmin user:

    GRANT CREATE ANY TRIGGER TO rrepadmin

  • For each of the tables that need to be published, grant SELECT privilege. For example to publish all the "scott" schema tables, issue the following commands:

    	 GRANT SELECT ON scott.dept TO rrepadmin 
             GRANT SELECT ON scott.emp TO rrepadmin 
             GRANT SELECT ON scott.bonus TO rrepadmin
    	 

21.1.7.2.2. Start Replication Console

Select the "Replication Console (for Oracle)" menu option from the EnterpriseDB program menu to launch the replication console desktop application.

21.1.7.2.3. Managing Publication/Subscription Servers

Right click on the Replication Servers tree node and select the Register Management Server menu option. In the Register Management Server dialog box, provide the IP address and port of the server on which the DBA Management Server is running.

Since the replication cluster may be configured in a distributed Intranet or Internet environment, authentication is enforced to eliminate any security issues. As part of the registration process, the user is required to provide super user login credentials for the DBA Management Server. Once the required information is entered click Register to register the DBA Management Server.

The DBA Management Server hosts one publication and one subscription service. The following sections describe how to configure and run publication and subscription services:

21.1.7.2.4. Start Publication Service

The publication service is in "stop" mode by default. Right click on the Publication Service node and select Start Service to start the service.

Choose the Add Database context menu option by right clicking on the Publication Service node that displays the Add Database dialog box. Specify publication database connectivity information, as per your system environment.

Host=10.90.1.121 Port=1521 User=rrepadmin Password=rrepadmin SID=XE

After providing the required information for adding the database, test the connection to the database and then save it.

21.1.7.2.5. Publication Service Configuration

After the database has been added, it can then be verified by using the Publication Databases tab as shown in the following screenshot:

Next, proceed to create a publication. EnterpriseDB allows the user to create two different types of publications:

  • Transactional Publication

  • Snapshot-Only Publication

21.1.7.2.6. Transactional Publication

To create a transactional publication, choose the Create Publication context menu option by right clicking on the Publication Service node that displays the Create Publication dialog box. Specify the database tables that are to be published. This is shown in the following screenshots:

The EnterpriseDB replication console also provides the user with the ability to provide a Filter Clause for a specific table that is to be published while creating a publication as shown in the following screen shot:

21.1.7.2.6.1. Managing Row-Level Filtering (Horizontal Partitioning)

A filter clause is added to a publication table if row-level filtering is desired. Once a filter clause is defined for a specified table, the user can view/update/remove it using the following options:

  • View Filter Clause

  • Update Filter Clause

  • Remove Filter Clause

Each of the above options is described in the following sections.

View Filter Clause

The user can view the filer clause expression for a specific table via the "Filter clause" property displayed under the General tab of the table as shown below.

Alternatively the user can select the View Filter Clause menu option that displays the clause expression in a dialog box.

Update Filter Clause

The Update Filter Clause selection provides the ability to update an existing filter clause for a specific table. The user can also check the correctness of the filter clause by pressing the Validate button. This functionality is shown in the following screenshot:

Remove Filter Clause

The user can remove an existing filter clause for a specific table by selecting the Remove Filter Clause option as shown in the following screenshots:

21.1.7.2.7. Snapshot-Only Publication

To create a snapshot-only publication, choose the Create Publication context menu option by right clicking on the Publication Service node that displays the Create Publication dialog box. Specify the database tables that are to be published. To create a snapshot-only publication, check the Snapshot-only replication check box. This is shown in the following screenshots:

The user can also publish Oracle "VIEWS". This option is available for a "snapshot-only" publication.

21.1.7.2.8. Start Subscription Service

  • The subscription service is stopped by default. Right click on the Subscription Service node and select Start Service to start the service.

  • Choose the Add Database context menu option by right clicking on the Subscription Service node that displays the Add Database dialog box. Specify the subscription database connectivity information including the IP address of the system running the subscription service, as per your system environment.

    Host=10.90.1.17 Port=5444 User=enterprisedb Password=edb Database=edb

21.1.7.2.9. Subscription Service Configuration

After the database has been added, it can then be verified by using the Subscription Databases tab as shown in the following screenshot:

After adding a database, create a subscription.

21.1.7.2.10. Creating a Subscription

To create a subscription, choose the Create Subscription context menu option by right clicking on the Subscription Service node that displays the Create Subscription dialog box. Specify the publication to which the user is subscribing. This is shown in the following screen shots:

After taking a snapshot in a snapshot-only transaction, the replication history for a specific table can be viewed:

For a transactional publication, after the initial snapshot has been taken, the synchronize operation is performed by right clicking on the subscription and choosing the "synchronize" option as shown in the following screenshot:

21.1.7.2.11. Replication History

The replication framework maintains replication history for all publications. The replication history is a record of when and how many times a replication (snapshot and/or synchronize) was performed over a certain period of time for a given publication. The summary history is maintained and accessible at the publication node level with a detailed table level history available against each table node as described below:

Viewing Publication Replication History

EnterpriseDB provides the user with the facility of viewing the "publication-level replication history". Publication-level replication history is a sum of all the replications that are performed for each table in that specific publication and tells about the replication performed with respect to a publication. These replications could either be transactional/synchronize or snapshot only:

Viewing Table Replication History

EnterpriseDB also provides the user with the facility of viewing the table-level replication history. Table-level replication history is separate for each table and tells about the replication performed with respect to an individual table:

21.1.7.2.12. Scheduling a Replication

After creating a subscription, the user can also schedule a replication so that the replication operation is performed repetitively after specific intervals. The schedule is configured by right clicking on the subscription and choosing the "Configure Schedule" option from the menu as shown below:

In the Scheduled Task Wizard, various parameters are set for the replication to be scheduled:

21.1.7.2.13. Shadow Table Cleanup Job

The transactions processed against the subscription tables are kept in the shadow tables to maintain synchronization replication history. Since the shadow tables may grow to contain large volumes of data, a cleanup job (implemented as an Oracle database job using the DBMS_JOB package) is scheduled to run every Sunday at 12:00 AM (midnight) by default. The user can disable this job or customize its default scheduling period via the Replication Console.

Shadow table cleanup is performed using either the DELETE command or the TRUNCATE command. The cleanup job performs a TRUNCATE on the shadow table if there are no pending transactions in the shadow table; otherwise, the cleanup job uses a DELETE operation.

Shadow table cleanup can be explicitly performed by the user from the EnterpriseDB Replication Console. The Cleanup Shadow Table History option deletes processed transactions from the shadow table(s). This functionality is shown in the following screenshots:

There is a subtle difference between replication history and shadow table history. Replication history is the record of when and how many times a replication (snapshot and/or synchronization) was performed over a certain period of time. The shadow table history is a record of the actual transactions processed against a subscription table when a synchronization occurs. These transaction records are viewable by clicking on the View Data link of a table's Replication History tab.

If the user opts to cleanup shadow table history, the replication history will still be available. However when the user selects the View Data option, the replicated rows will not be available for view as these are deleted from the underlying shadow tables.

21.1.7.3. Troubleshooting

21.1.7.3.1. KNOWN ISSUES

21.1.7.3.1.1. ISSUE: 01

When running replication servers on a machine with Linux and the Replicator Console on another machine, the console may raise a ConnectException (e.g., unable to connect to 127.0.0.1). Some Linux distributions set up the /etc/hosts file so that the hostname resolves to the loopback address rather than the host address. If you issue the "hostname -i" command and it returns 127.0.0.1 then you may have problems with RMI that is used by the EnterpriseDB Replication Server.

SOLUTION 1

In the Replicator Console, right click on the Publication/Subscription Service node and select Advanced JVM Options... that displays the Service JVM Options dialog box. Click Insert and specify the following JVM option.

    -Djava.rmi.server.hostname=192.168.0.1 
    

Change the IP address as per your system. The change will be effective when the service is started next time.

SOLUTION 2

Editing the /etc/hosts file so that the hostname resolves to the host network IP address should also fix the issue. A sample snippet of the /etc/hosts file is illustrated below.

	192.168.0.1 demo-server
        
	127.0.0.01 localhost.localdomain
	

21.1.7.3.1.2. ISSUE: 02

When replicating data from Oracle to EnterpriseDB, problem occurs if any of the data in Oracle table columns contains special characters, for instance, single quote characters (') - e.g., "John O' Hara". In that case the SQL INSERT statement that EnterpriseDB Replication Server attempts to use in Batch Update using Simple Statement (BUS) mode merely throws single quotes around the entire literal ( INSERT INTO ... VALUES ('John O'Hara')) whereas the quotes within the string are left as it is which either will cause an error or result in fewer quotes being inserted (if the Oracle data contained multiple consecutive quotes).

The following approach should be used in order to handle the above scenario:

  • Make use of the PreparedStatement mode

In case a problem is encountered during the Synchronize operation while performing Transactional Replication,the user can always force to use the Batch Update using Prepared Statement approach by specifying the following JVM options via the EnterpriseDB Replication Console using publication service JVM Options dialog box:

  • -DdefaultBatchUpdateMode=BUP

  • -DswitchBatchUpdateMode=false

The above functionality is illustrated using the following screenshots:

After specifying the above options, the value is saved by pressing the Save button. The values of the above properties will take effect on next start-up of the publication service.

 
 ©2004-2007 EnterpriseDB All Rights Reserved