Home | Table of Contents | Normalizer | CloverETL 4.6.1
--- | --- | ---
Prev | Transformers | Next

## Normalizer

# Not available in Community Designer

Short Description
Ports
Metadata
Normalizer Attributes
Details
CTL Interface
Java Interface
Examples
Best Practices
See also

### Short Description

Normalizer creates one or more output records from each single input record. Input records do not have to be sorted.

Component | Same input metadata | Sorted inputs | Inputs | Outputs | Java | CTL | Auto-propagated metadata
--- | --- | --- | --- | --- | --- | --- | ---
Normalizer | - |  | 1 | 1 |  |  |

### Icon

### Ports

Port type | Number | Required | Description | Metadata
--- | --- | --- | --- | ---
Input | 0 |  | For input data records | Any1
Output | 0 |  | For normalized data records | Any2

### Metadata

Normalizer does not propagate metadata.

Normalizer does not have any metadata template.

Normalizer does not require any specific metadata fields.

### Normalizer Attributes

Attribute | Req | Description | Possible values
--- | --- | --- | ---
Basic
Normalize | [1] | Definition of the way how records should be normalized written in the graph in CTL or Java. |
Normalize URL | [1] | Name of external file, including path, containing the definition of the way how records should be normalized written in CTL or Java. |
Normalize class | [1] | Name of external class defining the way how records should be normalized. |
Normalize source charset |  | Encoding of external file defining the transformation. The default encoding depends on DEFAULT_SOURCE_CODE_CHARSET in defaultProperties. | E.g. UTF-8
Deprecated
Error actions |  | Definition of the action that should be performed when the specified transformation returns some Error code. See Return Values of Transformations. |
Error log |  | URL of the file to which error messages for specified Error actions should be written. If not set, they are written to Console. |
[1] One of these must specified.

### Details

Normalizer requires transformation. The transformation can be defined in CTL (see CTL Interface) or in Java (see Java Interface).

The transformation is defined using several functions. Each of them has it's own purpose. The order of function calls is depicted in diagram below.

Figure 50.3. Normalizer code workflow

Number of calls of `transform()` function depends on result of the `count()` (or `countOnError()`) function.

### CTL Interface

CTL Templates for Normalizer
Access to input and output fields

The transformation written in CTL uses a CTL template for Normalizer. Only the functions `count()` and `transform()` are mandatory. Other functions are optionals.

Once you have written your transformation, you can also convert it to Java language code using the corresponding button in the upper right corner of the tab.

#### CTL Templates for Normalizer

Table 50.4. Functions in Normalizer

CTL Template Functions
boolean init()
Required | No
Description | Initialize the component, setup the environment, global variables
Invocation | Called before processing the first record
Returns | `true` | `false` (in case of `false` graph fails)
integer count()
Required | yes
Input Parameters | none
Returns | The returned number defines the number of new output records that will be created by the `transform()` function. If the `count()` function returns `0`, the subsequent call of `transform()` is skipped (`transform()` is called zero times).
Invocation | Called repeatedly, once for each input record
Description | For each input record it generates the number of output records that will be created from this input. If `count()` fails and user has not defined any `countOnError()`, the whole graph will fail. If any of the input records causes fail of the `count()` function, and if user has defined another function (`countOnError()`), processing continues in this `countOnError()` at the place where `count()` failed. The `countOnError()` function gets the information gathered by `count()` that was get from previously successfully processed input records. Also error message and stack trace are passed to `countOnError()`.
Example |
    function integer count() {
       customers = split($in.0.customers,"-");
       return length(customers);
    }
integer transform(integer idx)
Required | yes
Input Parameters | `integer idx` integer numbers from `0` to `count-1` (Here `count` is the number returned by the `count()` function.)
Returns | Integer number. The number corresponds to the return value of transformation. See Return Values of Transformations.
Invocation | Called repeatedly, once for each output record. The number of calls is defined by return value of function `count()`.
Description | It creates output records. If `transform()` fails and user has not defined any `transformOnError()`, the whole graph will fail. If any part of the `transform()` function for some output record causes fail of the `transform()` function, and if user has defined another function (`transformOnError()`), processing continues in this `transformOnError()` at the place where `transform()` failed. The `transformOnError()` function gets the information gathered by `transform()` that was get from previously successfully processed code. Also error message and stack trace are passed to `transformOnError()`.
Example |
    function integer transform(integer idx) {
       myString = customers[idx];
       $out.0.OneCustomer = str2integer(myString);
       $out.0.RecordNo = $in.0.recordNo;
       $out.0.OrderWithinRecord = idx;
       return OK;
    }
void clean()
Required | no
Input Parameters | none
Returns | void
Invocation | Called repeatedly, once for each input record. The function is called after the corresponding call(s) of `transform()` function.
Description | Returns the component to the initial settings
Example |
    function void clean() {
       clear(customers);
    }
integer countOnError(string errorMessage, string stackTrace)
Required | no
Input Parameters | `string errorMessage`
`string stackTrace`
Returns | The returned number defines the number of new output records that will be created by the `transform()` function. If the `count()` function returns `0` the subsequent call of `transform()` is skipped.
Invocation | Called if `count()` throws an exception.
Description | For each input record it generates the number of output records that will be created from this input. If any of the input records causes fail of the `count()` function, and if user has defined another function (`countOnError()`), processing continues in this `countOnError()` at the place where `count()` failed.
Example |
    function integer countOnError(
                      string errorMessage,
                      string stackTrace) {
       printErr(errorMessage);
       return 1;
    }
integer transformOnError(string errorMessage, string stackTrace, integer idx)
Required | no
Input Parameters | `string errorMessage`
`string stackTrace`
`integer idx`
Returns | Integer numbers. See Return Values of Transformations for detailed information.
Invocation | Called if `transform()` throws an exception.
Description | It creates output records. If any part of the `transform()` function for some output record causes fail of the `transform()` function, and if user has defined another function (`transformOnError()`), processing continues in this `transformOnError()` at the place where `transform()` failed. The `transformOnError()` function gets the information gathered by `transform()` that was get from previously successfully processed code. Also error message and stack trace are passed to `transformOnError()`.
Example |
    function integer transformOnError(
                      string errorMessage,
                      string stackTrace,
                      integer idx) {
       printErr(errorMessage);
       printErr(stackTrace);
       $out.0.OneCustomerOnError = customers[idx];
       $out.0.RecordNo = $recordNo;
       $out.0.OrderWithinRecord = idx;
       return OK;
    }
string getMessage()
Required | No
Description | Prints error message specified and invoked by user
Invocation | Called in any time specified by user (called only when either `count()`, `transform()`, `countOnError()`, or `transformOnError()` returns value less than or equal to -2).
Returns | `string`
void preExecute()
Required | No
Input parameters | None
Returns | `void`
Description | May be used to allocate and initialize resources required by the transform. All resources allocated within this function should be released by the `postExecute()` function.
Invocation | Called during each graph run before the transform is executed.
void postExecute()
Required | No
Input parameters | None
Returns | `void`
Description | Should be used to free any resources allocated within the `preExecute()` function.
Invocation | Called during each graph run after the entire transform was executed.

#### Access to input and output fields

##### Input records or fields

Input records or fields are accessible within the `count()`, `countOnError()`, `transform()` and `transformOnError()` functions only.

##### Output records or fields

Output records or fields are accessible within the `transform()` and `transformOnError()` functions only.

| Warning
All of the other CTL template functions allow to access neither inputs nor outputs. Remember that if you do not hold these rules, NPE will be thrown!

### Java Interface

The transformation implements methods of the `RecordNormalize` interface and inherits other common methods from the `Transform` interface. See Common Java Interfaces. See Public Clover API.

Following are the methods of `RecordNormalize` interface:

- `boolean init(Properties parameters, DataRecordMetadata sourceMetadata, DataRecordMetadata targetMetadata)`

  Initializes normalize class/function. This method is called only once at the beginning of normalization process. Any object allocation/initialization should happen here.

- `int count(DataRecord source)`

  Returns the number of output records which will be created from specified input record.

- `int countOnError(Exception exception, DataRecord source)`

  Called only if `count(DataRecord)` throws an exception.

- `int transform(DataRecord source, DataRecord target, int idx)`

  `idx` is a sequential number of output record (starting from 0). See Return Values of Transformations for detailed information about return values and their meaning. In Normalizer, only `ALL`, `0`, `SKIP`, and Error codes have some meaning.

- `int transformOnError(Exception exception, DataRecord source, DataRecord target, int idx)`

  Called only if `transform(DataRecord, DataRecord, int)` throws an exception.

- `void clean()`

  Finalizes current round/clean after current round - called after the transform method was called for the input record.

### Examples

#### Converting multivalue fields to multiple records

Input records contain group name and list of users of the group. Convert records into tuples having group name and one username.

    accounting  | [johnsmith, elisabethtaylor]
    development | [georgegreen, janegreen, peterbrown]

##### Solution

Define the transformation using Normalize attribute.

    //#CTL2

    function integer count() {
     	return length($in.0.users);
    }

    function integer transform(integer idx) {
     	$out.0.group = $in.0.group;
     	$out.0.user = $in.0.users[idx];
     	return OK;
    }

Normalizer will return following records:

    accounting |johnsmith
    accounting |elisabethtaylor
    development|georgegreen
    development|janegreen
    development|peterbrown

### Best Practices

If the transformation is specified in an external file (with Normalize URL), we recommend users to explicitly specify Normalize source charset.

### See also

Denormalizer
Rollup
Common Properties of Components
Specific Attribute Types
Common Properties of Transformers
Transformers Comparison
Defining Transformations

Prev | Up | Next
MetaPivot | Home | Table of Contents | Partition