Transformers have both input and output ports. They can:
put together more data flows with the same metadata (Concatenate, SimpleGather, and Merge);
remove duplicate records (Dedup);
filter data records (Filter and EmailFilter);
create samples from input records (DataSampler), sort data records (ExtSort, FastSort and SortWithinGroups);
multiplicate existing data flow (SimpleCopy);
split one data flow into more data flows (Partition at all, but optionally also Dedup, Filter and Reformat);
intersect two data flows (even with different metadata on inputs) (DataIntersection), aggregate data information (Aggregate);
and perform much more complicated transformations of data flows (Reformat, Denormalizer, Normalizer, Rollup and XSLTransformer).
Metadata can be propagated through some of these transformers, whereas the same is not possible in such components that transform data flows in a more complicated manner. You must have the output metadata defined prior to configuring these components.
Some of these transformers use transformations that have been described above. See Defining Transformations for detailed information about how transformation should be defined.
Some Transformers can have a transformation attribute defined, it may be optional or required. For information about transformation templates for transformations written in CTL see: CTL Templates for Transformers.
Some Transformers can have a transformation attribute defined, it may be optional or required. For information about transformation interfaces that must be implemented in transformations written in Java see: Java Interfaces for Transformers.
Below is an overview of all Transformers:
Table 55.1. Transformers Comparison
Component | Same input metadata | Sorted inputs | Inputs | Outputs | Java | CTL | Auto-propagated metadata |
---|---|---|---|---|---|---|---|
Aggregate | - | 1 | 1 | ||||
Concatenate | 1-n | 1 | |||||
DataIntersection | 2 | 3 | |||||
DataSampler | - | 1 | n | ||||
Dedup | - | 1 | 1-2 | ||||
Denormalizer | - | 1 | 1 | ||||
ExtSort | - | 1 | 1-n | ||||
FastSort | - | 1 | 1-n | ||||
Filter | - | 1 | 1-2 | ||||
LoadBalancingPartition | - | 1 | 1-n | ||||
Merge | 2-n | 1 | |||||
MetaPivot | - | 1 | 1 | ||||
Normalizer | - | 1 | 1 | ||||
Partition | - | 1 | 1-n | [1] | [1] | ||
Pivot | - | 1 | 1 | ||||
Reformat | - | 1 | 1-n | ||||
Rollup | - | 1 | 1-n | ||||
SimpleCopy | - | 1 | 1-n | ||||
SimpleGather | 1-n | 1 | |||||
SortWithinGroups | - | 1 | 1-n | ||||
XSLTransformer | - | 1 | 1 | ||||
[1] Partition can use either the transformation or two other attributes (Ranges or Partition key). A transformation must be defined unless one of these is specified. |