Chapter 44. Developing Subgraphs
There are two ways how to create a subgraph.
Wrapping
Subgraph wrapping is a way to convert section of existing ETL graph into subgraph. Wizard will let you copy additional graph elements (metadata, connections, lookups) from parental graph to subgraph.
Wrapping in Steps
Select components you would like to move into a new subgraph.
Figure 44.1. Original graph without subgraphs
Use the right mouse button and choose Wrap As Subgraph .
Figure 44.2. Wrapping components into a subgraph
Enter name of the subgraph file (
*.sgrf) and order of its input and output ports.
Figure 44.3. Wrapping Subgraph Wizard
A new Subgraph component replaced the wrapped components in the parent graph.
Figure 44.4. CloverETL Graph with the Subgraph Component
Continue with Making Subgraph Configurable .
Creating from Scratch
New subgraph can be created from scratch. The new subgraph will have an initial structure - it will contain a debug input component, SubgraphInput and SubgraphOutput components and a sample of the subgraph body. The initial structure is a template to help you design the subgraph.
Choose in the main menu → → .
 |
Figure 44.5. A new Subgraph
Design the subgraph body - implement the subgraph's logic in the central body section of the subgraph, using components, other subgraphs etc.
Connect subgraph body with SubgraphInput and SubgraphOutput components.
Continue with Making Subgraph Configurable .
Making Subgraph Configurable
| Optional Ports |
| Color of Subgraph |
| Icon of Subgraph |
Each attribute of component in a subgraph can be exposed as an attribute of corresponding subgraph component using public parameter. It makes possible to develop more generic subgraphs.
Example 44.1. Using public parameter
You have a subgraph filtering and aggregating records. You need to use the subgraph on several places but with different filter expression. Export the filter expression as a public parameter and let the user of the subgraph to set it up per subgraph component.
Exporting an Attribute as Subgraph Parameter
To export attribute of a component as a parameter of subgraph choose the attribute of a component of subgraph and use the Export as subgraph parameter button.
Figure 44.6. Export as subgraph parameter button
The public parameter appears as a subgraph component attribute.
 |
Figure 44.7. Public parameter appeared as a subgraph component attribute
One public parameter can be used in more components of the same subgraph. For example two Filter components can share the filter expression exported from the subgraph component as a public parameter.
Using Existing Public Graph Parameter
Any existing public graph parameter can be used as an attribute value of components in a subgraph.
To use an existing public parameter as a value of attribute of a component choose the attribute and use the Use graph parameter as value button.
Figure 44.8. Use graph parameter as value button
Optional Ports
Input and output ports of a subgraph can be marked as optional. It lets you create a component with ports that do not require a connected edge.
It can be set up in Outline within the subgraph. Right click the port in Outline and choose the corresponding option.
Figure 44.9. Setting up an Optional Port
You can set up optional ports from Context menu in the subgraph editor too. Move mouse cursor on the optional port and do right click to open the Context menu .
This way is available only if there is an edge connected to the port.
Figure 44.10. Setting up an Optional Port in Graph Editor
There are three options:
- required
- optional - edge receives zero records
- optional - edge is removed
You can use Optional ports to conditionally enable or disable component within subgraph. See Enable/Disable Component .
Required
If a port of subgraph is marked as required , an edge has to be connected to the port of subgraph component.
E.g. first output port of Filter is required and the port of subgraph will work in the same way.
Optional - Edge Receives Zero Records
An edge does not have to be connected to the port of subgraph component. If the edge is not connected to the port of subgraph component, subgraph itself assumes port to be connected and zero records to be received through the port.
This is similar to any input port of SimpleGather .
This is useful e.g. in case of merging input data streams within the subgraph.
Optional - Edge Is Removed
An edge does not have to be connected to the port of subgraph component. If the edge is not connected to the port of subgraph component, it works like there is no edge connected to the port within the subgraph.
This is similar to optional input ports of readers as it changes behaviour of the component.
This case is useful e.g. in case of wrapping reader with optional port in the subgraph.
Optional - Edge Discards All Records
An edge does not have to be connected to the output port of a subgraph component. If the edge is connected to output port of subgraph component, records are sent out to the port from of a subgraph component. If the edge is not connected to output port of subgraph component, records to be sent out to the port are silently discarded within a subgraph.
It is similar to second output port of Filter .
This is useful, if you convert records to several output formats using subgraph and let user of subgraph to choose one or more output format to use.
Color of Subgraph
You can set up color to the subgraph component arising from the subgraph. The subgraph can be assigned to the category of components (Readers, Writers, etc.). The subgraph component will have the color of the assigned category.
The component category is set up on Properties Tab of the subgraph in a section Subgraph in property Category .
Figure 44.11. Setting up Color of Subgraph
Icon of Subgraph
Subgraph can have own icon assigned. Three sizes of icons can be defined:
Small (16x16)
Medium (32x32)
Large (64x64)
Path to the icons is defined in Subgraph section on Properties tab of the subgraph.
Figure 44.12. Setting up Icon of Subgraph
Suggested place for subgraph icons is in
${PROJECT}/icons
.
As an icon,
.png
and
.gif
files can be used.
Continue with Developing and Testing Subgraphs .
Developing and Testing Subgraphs
Subgraph can be launched and tested standalone, without being run from a parent graph. You can run and debug it the same as an ordinary graph.
Useful tips:
You can run the subgraph as an ordinary graph via the Run As item in toolbar or right-click context menu.
Debugging tools such as View Data can be used as in an ordinary graph.
You can use all the ordinary graph elements in the subgraph - connections, metadata, lookup tables, phases etc.
Prepare test data in the Debug Input section of the subgraph - components that produce data to input ports of SubgraphInput are executed only when running the subgraph standalone, not when it's used from a parent graph. These components are used to generate testing and development data for the subgraph for easy development and testing without the need to use it in parent graph.
Parameterize subgraph components using public parameters if necessary.
Note, that subgraph elements like graph parameters, connections etc. are independent of graph elements of parent graph. There is no connection between them like inheritance. If you need some elements of parent subgraph, map it using Input mapping or Output mapping attributes.
Filling Required Parameters
If subgraph contains required parameters, you are asked to fill them in using Dialog for Filling Required Parameters before graph run. This way, you can easily test the subgraph with various values of required parameters.
Dialog opens just before the graph (subgraph) is run from designer if the graph has any required parameters.
It shows only required parameters.
Values of parameters are shown in the dialog. If no value is defined for the parameter, it is prefilled with value that was used last time the graph was run.
Figure 44.13. Dialog for Filling Required Parameters
Metadata Propagation
You can use Auto-propagated Metadata in the same way as in the graphs. Metadata can be propagated between parent graph and subgraph.
Subgraph Providing Metadata
A subgraph can define explicit metadata in its definition and propagate them to SubgraphOutput component. When the subgraph is used in a parent graph, these metadata will be propagated via subgraph's output edge to the parent graph.
Typical use-case is reader subgraph that not only reads a data
source,
but also provides metadata of the data source (e.g. orders).
In the example
below, we define explicit metadata on the output of
SpreadsheetDataReader
component for records containing
orders
.
Metadata on the output of the
Filter
component
are set to be auto-propagated, which propagates the
orders
metadata to the output
of the subgraph (as defined by
SubgraphOutput
).
When using
such subgraph in a parent graph, the
orders
metadata
are auto-propagated on the output of the subgraph.
Figure 44.14. Subgraph providing metadata
Figure 44.15. Metadata propagated from Subgraph component
Subgraph Requiring Metadata
Subgraph can require specific metadata when used in a parent graph by defining explicit metadata on the outputs of SubgraphInput . When the subgraph is used, its input metadata in the parent graph must match the metadata defined inside the subgraph.
Typical use-case is a writer subgraph that requires some specific
metadata (e.g. customers) to store records in a service. In the
example below,
we explicitly define
customers
metadata on the output of the
SubgraphInput
.
When the subgraph is used, its input
metadata in the parent graph must match the customers metadata.
Figure 44.16. Subgraph explicitly defines input metadata for customers
Figure 44.17. Using subgraph with matching metadata
Metadata Acquired from Parent
Subgraph can be quite generic and not specify any explicit metadata, only use auto-propagated metadata. The subgraph will acquire metadata from its parent graph.
To develop and test such a subgraph, we recommend that you define explicit metadata in the Debug Input (or Debug Output ) section of the subgraph. These metadata will make the subgraph valid for testing it.
Typical use-case is a generic filter graph that performs filtering on a specific (or user defined) fields, and copies all other fields. In the example below, all edges of the subgraph body are set to be auto-propagated. When the subgraph is used in a parent graph, the customers metadata are propagated through the subgraph.
 |
Figure 44.18. Generic subgraph not defining explicit metadata in its body
Figure 44.19. Metadata propagate through the Subgraph component
See also Auto-propagated Metadata for details on metadata propagation.