Home | Table of Contents | Data Partitioning (Parallel Running) | CloverETL 4.6.1
--- | --- | ---
Prev | ETL Graphs | Next

# Chapter 41. Data Partitioning (Parallel Running)

This chapter describes way to speed up graph runs with help of data partitioning.

| Note
Data partitioning is available in Corporate Server and Cluster. It is not available in local projects.

#### What Is Data Partitioning

Data partitioning runs parts of graph in parallel. A component that is a bottleneck of a graph is run in multiple instances and each instance processes one part of the original data stream.

Figure 41.1. Parallel Run

The processing can be further scaled to cluster without modification to the graph.

##### Partitioned Sandboxes

In CloverETL Cluster, you can partition files with temporary data to multiple cluster nodes using Partitioned sandboxes. A file stored in partitioned sandbox is split into several parts. Each part of the file is on a different cluster node. This way, you can partition both: processing and data. It reduces amount of data being transferred between cluster nodes.

#### When to Use Data Partitioning

Data partitioning is convenient to speed up processing when:

- components communicate over the network with high latency (HTTPConnector, WebServiceClient)

- some component is significantly slower than other components in a graph (AddressDoctor 5)

#### Way To Speed Up Processing

The way to speed up the run is to partition the data and run the slow component in parallel.

##### Designer and Server

In Designer and Server, you can speed up processing with copying the slow component and running it in parallel.

Figure 41.2. Parallel Run

##### Scalable Solution in Corporate Server and Cluster

There is a better solution that avoids copying components and is scalable.

Replace Partition with ParallelPartition and SimpleGather with ParallelSimpleGather.

Set allocation to the components positioned between the cluster components: right click the component and choose Set Allocation.

Figure 41.3. Parallel Run with Cluster Components

In Component Allocation dialog choose By number of workers and enter number of parallel workers.

Figure 41.4. Component Allocation

Components in your graph will contain text denoting the allocation.

Figure 41.5. Component Allocation

#### How Does the Data Partitioning Work

Data partitioning runs part of a graph in parallel. The number of parallel workers is configured without copying the components. Data-partitioned graphs can take advantage of CloverETL cluster without modification.

#### Benefits of Data Partitioning

- Clean design, no duplication. Avoid copying parts of graph to speed-up the processing. Set the number of parallel workers with a single option.

- Scales to Cluster. You can use the same graph on a multi-node Cluster without any additional modifications.

- Maximize use of available hardware. Take advantage of parallel processing on multi-core processors.

#### Things to Consider when Going Parallel

- When you run some component in parallel, you should be aware of limits of hardware and other systems.

- If you run parallel a component that does many I/O operations (e.g FastSort), you might be limited by speed of hard drive.

- If you run parallel a component that opens many files (e.g FastSort), you might reach limit on number of opened files.

- If you run parallel a component that connects to a web service, you may reach the limit of parallel connections to the service or run out of the quota on number of requests.

- Consider other jobs running on server. Too many jobs running in parallel may slow down run of other graphs.

- Some tasks cannot be easily parallelized.

Prev | Up | Next
Common Java Interfaces | Home | Table of Contents | Subgraphs