Home | Table of Contents | Developing Subgraphs | CloverETL 4.6.1
--- | --- | ---
Prev | Subgraphs | Next

# Chapter 44. Developing Subgraphs

There are two ways how to create a subgraph.

- Wrapping

- Creating from Scratch .

## Wrapping

Subgraph wrapping is a way to convert section of existing ETL graph into subgraph. Wizard will let you copy additional graph elements (metadata, connections, lookups) from parental graph to subgraph.

###### Wrapping in Steps

1. Select components you would like to move into a new subgraph.

Figure 44.1. Original graph without subgraphs

2. Use the right mouse button and choose Wrap As Subgraph .

Figure 44.2. Wrapping components into a subgraph

3. Enter name of the subgraph file ( `*.sgrf` ) and order of its input and output ports.

Figure 44.3. Wrapping Subgraph Wizard

4. A new Subgraph component replaced the wrapped components in the parent graph.

Figure 44.4. CloverETL Graph with the Subgraph Component

Continue with Making Subgraph Configurable .

## Creating from Scratch

New subgraph can be created from scratch. The new subgraph will have an initial structure - it will contain a debug input component, SubgraphInput and SubgraphOutput components and a sample of the subgraph body. The initial structure is a template to help you design the subgraph.

1. Choose in the main menu File → New → Subgraph .

Figure 44.5. A new Subgraph

2. Design the subgraph body - implement the subgraph's logic in the central body section of the subgraph, using components, other subgraphs etc.

3. Connect subgraph body with SubgraphInput and SubgraphOutput components.

Continue with Making Subgraph Configurable .

## Making Subgraph Configurable

Optional Ports
Color of Subgraph
Icon of Subgraph

Each attribute of component in a subgraph can be exposed as an attribute of corresponding subgraph component using public parameter. It makes possible to develop more generic subgraphs.

Example 44.1. Using public parameter

You have a subgraph filtering and aggregating records. You need to use the subgraph on several places but with different filter expression. Export the filter expression as a public parameter and let the user of the subgraph to set it up per subgraph component.

##### Exporting an Attribute as Subgraph Parameter

To export attribute of a component as a parameter of subgraph choose the attribute of a component of subgraph and use the Export as subgraph parameter button.

Figure 44.6. Export as subgraph parameter button

The public parameter appears as a subgraph component attribute.

Figure 44.7. Public parameter appeared as a subgraph component attribute

One public parameter can be used in more components of the same subgraph. For example two Filter components can share the filter expression exported from the subgraph component as a public parameter.

##### Using Existing Public Graph Parameter

Any existing public graph parameter can be used as an attribute value of components in a subgraph.

To use an existing public parameter as a value of attribute of a component choose the attribute and use the Use graph parameter as value button.

Figure 44.8. Use graph parameter as value button

### Optional Ports

Input and output ports of a subgraph can be marked as optional. It lets you create a component with ports that do not require a connected edge.

It can be set up in Outline within the subgraph. Right click the port in Outline and choose the corresponding option.

Figure 44.9. Setting up an Optional Port

You can set up optional ports from Context menu in the subgraph editor too. Move mouse cursor on the optional port and do right click to open the Context menu .

This way is available only if there is an edge connected to the port.

Figure 44.10. Setting up an Optional Port in Graph Editor

There are three options:

- required
- optional - edge receives zero records
- optional - edge is removed

You can use Optional ports to conditionally enable or disable component within subgraph. See Enable/Disable Component .

##### Required

If a port of subgraph is marked as required , an edge has to be connected to the port of subgraph component.

E.g. first output port of Filter is required and the port of subgraph will work in the same way.

##### Optional - Edge Receives Zero Records

An edge does not have to be connected to the port of subgraph component. If the edge is not connected to the port of subgraph component, subgraph itself assumes port to be connected and zero records to be received through the port.

This is similar to any input port of SimpleGather .

This is useful e.g. in case of merging input data streams within the subgraph.

##### Optional - Edge Is Removed

An edge does not have to be connected to the port of subgraph component. If the edge is not connected to the port of subgraph component, it works like there is no edge connected to the port within the subgraph.

This is similar to optional input ports of readers as it changes behaviour of the component.

This case is useful e.g. in case of wrapping reader with optional port in the subgraph.

##### Optional - Edge Discards All Records

An edge does not have to be connected to the output port of a subgraph component. If the edge is connected to output port of subgraph component, records are sent out to the port from of a subgraph component. If the edge is not connected to output port of subgraph component, records to be sent out to the port are silently discarded within a subgraph.

It is similar to second output port of Filter .

This is useful, if you convert records to several output formats using subgraph and let user of subgraph to choose one or more output format to use.

### Color of Subgraph

You can set up color to the subgraph component arising from the subgraph. The subgraph can be assigned to the category of components (Readers, Writers, etc.). The subgraph component will have the color of the assigned category.

The component category is set up on Properties Tab of the subgraph in a section Subgraph in property Category .

Figure 44.11. Setting up Color of Subgraph

### Icon of Subgraph

Subgraph can have own icon assigned. Three sizes of icons can be defined:

- Small (16x16)

- Medium (32x32)

- Large (64x64)

Path to the icons is defined in Subgraph section on Properties tab of the subgraph.

Figure 44.12. Setting up Icon of Subgraph

Suggested place for subgraph icons is in `${PROJECT}/icons` .

As an icon, `.png` and `.gif` files can be used.

Continue with Developing and Testing Subgraphs .

## Developing and Testing Subgraphs

Subgraph can be launched and tested standalone, without being run from a parent graph. You can run and debug it the same as an ordinary graph.

###### Useful tips:

- You can run the subgraph as an ordinary graph via the Run As item in toolbar or right-click context menu.

- Debugging tools such as View Data can be used as in an ordinary graph.

- You can use all the ordinary graph elements in the subgraph - connections, metadata, lookup tables, phases etc.

- Prepare test data in the Debug Input section of the subgraph - components that produce data to input ports of SubgraphInput are executed only when running the subgraph standalone, not when it's used from a parent graph. These components are used to generate testing and development data for the subgraph for easy development and testing without the need to use it in parent graph.

- Parameterize subgraph components using public parameters if necessary.

Note, that subgraph elements like graph parameters, connections etc. are independent of graph elements of parent graph. There is no connection between them like inheritance. If you need some elements of parent subgraph, map it using Input mapping or Output mapping attributes.

### Filling Required Parameters

If subgraph contains required parameters, you are asked to fill them in using Dialog for Filling Required Parameters before graph run. This way, you can easily test the subgraph with various values of required parameters.

- Dialog opens just before the graph (subgraph) is run from designer if the graph has any required parameters.

- It shows only required parameters.

- Values of parameters are shown in the dialog. If no value is defined for the parameter, it is prefilled with value that was used last time the graph was run.

Figure 44.13. Dialog for Filling Required Parameters

## Metadata Propagation

You can use Auto-propagated Metadata in the same way as in the graphs. Metadata can be propagated between parent graph and subgraph.

### Subgraph Providing Metadata

A subgraph can define explicit metadata in its definition and propagate them to SubgraphOutput component. When the subgraph is used in a parent graph, these metadata will be propagated via subgraph's output edge to the parent graph.

Typical use-case is reader subgraph that not only reads a data source, but also provides metadata of the data source (e.g. orders). In the example below, we define explicit metadata on the output of SpreadsheetDataReader component for records containing `orders` . Metadata on the output of the Filter component are set to be auto-propagated, which propagates the `orders` metadata to the output of the subgraph (as defined by SubgraphOutput ). When using such subgraph in a parent graph, the `orders` metadata are auto-propagated on the output of the subgraph.

Figure 44.14. Subgraph providing metadata

Figure 44.15. Metadata propagated from Subgraph component

### Subgraph Requiring Metadata

Subgraph can require specific metadata when used in a parent graph by defining explicit metadata on the outputs of SubgraphInput . When the subgraph is used, its input metadata in the parent graph must match the metadata defined inside the subgraph.

Typical use-case is a writer subgraph that requires some specific metadata (e.g. customers) to store records in a service. In the example below, we explicitly define `customers` metadata on the output of the SubgraphInput . When the subgraph is used, its input metadata in the parent graph must match the customers metadata.

Figure 44.16. Subgraph explicitly defines input metadata for customers

Figure 44.17. Using subgraph with matching metadata

### Metadata Acquired from Parent

Subgraph can be quite generic and not specify any explicit metadata, only use auto-propagated metadata. The subgraph will acquire metadata from its parent graph.

To develop and test such a subgraph, we recommend that you define explicit metadata in the Debug Input (or Debug Output ) section of the subgraph. These metadata will make the subgraph valid for testing it.

Typical use-case is a generic filter graph that performs filtering on a specific (or user defined) fields, and copies all other fields. In the example below, all edges of the subgraph body are set to be auto-propagated. When the subgraph is used in a parent graph, the customers metadata are propagated through the subgraph.

Figure 44.18. Generic subgraph not defining explicit metadata in its body

Figure 44.19. Metadata propagate through the Subgraph component

See also Auto-propagated Metadata for details on metadata propagation.

Prev | Up | Next
Using Subgraphs | Home | Table of Contents | Design Patterns