Not available in Community Designer

Instead of reading tons of material, why not take a look at a video walkthrough. After watching it, you should have a clear notion of how to use and configure the ComplexDataReader component.

See ComplexDataReader example video:

http://www.cloveretl.com/resources/repository/complexdatareader-shipments

To start designing the machine, edit the Transform attribute. A new window opens offering these tabs: States, Overview, Selector, Source and other tabs representing states labeled $stateNo stateLabel, e.g. "$0 myFirstState".

On the left side of the States tab, you can see a pane with all the Available metadata your graph works with. In this tab, you design new states by dragging metadata to the States pane on the right. At the bottom, you can set the Initial state (the first state) and the Final state (the machine switches to it shortly before terminating its execution or if you call Flush and finish). The final state can serve mapping data to the output before the automaton terminates (especially handy for treating the last records of your input). Finally, in the center there is the Expression editor pane, which supports Ctrl+Space content assist and lets you directly edit the code.

In the Overview tab, the machine is graphically visualized. Here you can Export Image to an external file or Cycle View Modes to see other graphical representations of the same machine. If you click Undock, the whole view will open in a separate window that is regularly refreshed.

In state tabs (e.g. "$0 firstState") you define the outputs in the Output ports pane. What you see in Output field is in fact the (fixed) output metadata. Next, you define Actions and work with the Transition table at the bottom pane in the tab. Inside the table, there are Conditions which are evaluated top-down and Target states assigned to them. These are these values for Target states:

The Selector tab allows you to implement your own selector or supply it in an external file/Java class.

Finally, the Source tab shows the code the machine performs. For more information, see CTL in ComplexDataReader

There are the counterStateNo variables which store the number of times a state has been accessed. There is one such variable for each state and their numbering starts with 0. So e.g. counter2 stores how many times state $2 was accessed. The counter can be reset in Actions.

integer initialState() - determines which state of the automaton is the first one initiated. If you return ALL, it means Let selector decide, i.e. it passes the current state to the selector that determines which state will be next (if it cannot do that, the machine fails)

integer finalState(integer lastState) - specifies the last state of the automaton. If you return STOP, it means the final state is not defined.

Each state has two major functions describing it:

integer nextState_stateNo() returns a number saying which state follows the current state (stateNo). If you return ALL, it means Let selector decide. If you return STOP, it means Flush and finish.

nextState_0() {
	if(counter0 > 5) {
		return 1;	// if state 0 has been accessed more than five times since 
            // the last counter reset, go to state 1 
	}
	return 0;  // otherwise stay in state 0
}      	
		

nextOutput_stateNo(integer seq) - the main output function for a particular state (stateNo). It calls the individual nextOutput_stateNo_seq() service functions according to the value of seq. The seq is a counter which stores how many times the nextOutput_stateNo function has been called so far. At last, it calls nextOutput_stateNo_default(integer seq) which typically returns STOP meaning everything has been sent to the output and the automaton can change to the next state.

integer nextOutput_stateNo_seq() - maps data to output ports. In particular, the function can look like e.g. integer nextOutput_1_0() meaning it defines mapping for state $1 and seq equal to 0 (i.e. this is the first time the function has been called). The function returns a number. The number says which port has been served by this function.

integer nextState(integer state)) - calls individual nextState() functions according to the current state

integer nextOutput(integer state, integer seq) - calls individual nextOutput() functions according to the current state and the value of seq.

Three programs log actions to one logging facility and the facility produces single file. Each program uses different log format. Categorize lines of a log file for further processing.

Record produced by program A1:

programName|action|time|message

Record produced by program A2:

programName|time|user

Record produced by program A3:

programName|severity|code

Input data:

A1|received|2015-02-28|Message received
A2|2015-03-11 10:12:00|smithj
A1|received|2015-03-11|Message received
A3|INFO|login
A3|INFO|password changed
A2|2015-03-11 10:15:30|taylorg

Solution

Use File URL to define source file name(s).

Open dialog in the Transforms attribute.

On the States tab, add three states with corresponding metadata. You can either add states using plus button and subsequently add metadata from combo box or you can drag metadata from metadata pane onto pane with fields.

Switch to the Selector tab. Set up Maximum prefix length to 2. Add states using the plus button and fill in the Prefix and Go to state columns in the tab: A1 for state $0, A2 for state $1 and A3 for state $2.

Define the mapping on tabs with particular states: on $0 map record with metadata for A1, on $1 map record with metadata for A2 and on $2 map record with metadata for A3.

The following records will be sent to particular output ports.

A1|received|2015-02-28|Message received
A1|received|2015-03-11|Message received

A2|2015-03-11 10:12:00|smithj
A2|2015-03-11 10:15:30|taylorg

A3|INFO|login
A3|INFO|password changed

See data from previous example. There is an unnecessary field in output produced in the last example. Process the same file, but remove prefixes A1, A2 and A3 from the result.

Solution

On the selector tab set prefix to "A1|", "A2|" and "A3|" and check Consume prefix checkbox. Note: you need to modify metadata.

Input file contains records about shipments. Data of one shipment occupies one line. Each shipment has identifier (11 chars), source (7) and destination (7).

One shipment consists of zero or more pieces. Each piece is described by a record having header (4) beginning with 300B, piece identifier (3) and description (7).

There can be zero or more charges related to the shipment or particular pieces. Each charge consists of header (4) beginning with 400B, piece identifier in case of charge related to the piece (3), amount (7) and description (27).

Pieces and charges are on the same line with shipment.

Parse records, each type of record should be passed to different output port.

The source file example:

1426089255 London Nice   300B001paper  400B0011000.00good too heavy surcharge   400B001  50.00special delivery conditions
1426089256 Narvik Rome   300B001snow   400B001 500.00bulk good surcharge        300A002ski    400B002 100.00long good surcharge        
1426089257 Nimes  Miami  300B001wine   400B001 500.00dangerous good surcharge   

Input file contains fixed-length data. As a result we need fixed-length metadata. Create fixed-length metadata Shipment, Piece and Charge. All these fixed-length metadata have to have empty record delimiter.

Create auxiliary delimited metadata Aux with one string field. This will be used to read line ends.

Use File URL and Transform attributes.

On States tab create states $0 with Shipment metadata, $1 with Piece metadata, $2 with Charge metadata and $3 with Aux metadata.

Set up the Selector:

Set up output mapping of particular states: produce output on port 0 in state $0, on port 1 in state $1 and on port 2 in state $2. Do not produce any output in state $3.

Results on particular output ports are following:

1426089255 London Nice   
1426089256 Narvik Rome   
1426089257 Nimes  Miami  

300B001paper  
300B001snow   
300B001wine   

400B0011000.00good too heavy surcharge
400B001 500.00bulk good surcharge     
400B001 500.00dangerous good surcharge

Records read by ComplexDataReader in the previous have lost mutual binding. It is impossible to track which charges or pieces are connected to which shipment. Add identifier from Shipment record to other records.

Solution

The solution is built on solution from previous example.

Add the identifier field to Piece metadata and to Charge metadata.

On tab $1 add mapping of identifier from Shipment to the identifier in Piece. Do the same of $2 tab with Charge.