Manipulating large volumes of data in a single record is always an issue. In CloverETL Designer, sending big data along graph edges means this:

By default, the maximum size of a record sent along an edge is 33554432 bytes (32 MB). This value can be increased, theoretically, up to tens of MBs by setting the Record.RECORD_LIMIT_SIZE property (Chapter 18, Engine Configuration). Record.FIELD_LIMIT_SIZE can also be 33554432 bytes (32 MB) by default. Naturally, all fields in total cannot use more memory than Record.RECORD_LIMIT_SIZE.

There is no harm in increasing Record.RECORD_LIMIT_SIZE to whatever size you want. The only reason for keeping it smaller is an early error detection. For instance, if you start appending to a string field and forget to reset record (after each record), the field size can break the limits.

	Note
	Let us look a little deeper into what happens in the memory. Initially, a record starts with 65536 (64kB) of memory allocated to it. If there is a need to transfer huge data, its size can dynamically grow up to the value of Record.RECORD_LIMIT_SIZE. If you ever wondered how much memory a record could consume, then the answer is between 65536 (64k) and Record.RECORD_LIMIT_SIZE.

In your ETL graph, edges which are more 'memory greedy' look like regular edges. They have no special visual effects.

Measuring and Estimating Edge Memory Demands

To estimate how memory-greedy your graph is even before executing it, consult the table below (note: computations are simplified). In general, a graph's memory demands depend on the input data, components used and edge types. In this place, we contribute to understanding the last one. See how much memory approx. your graph takes before its execution and to what extent memory demands can rise.

The following table depicts memory demands for particular edge types in MB and in the multiples of record initial size and record limit size. The limits can be enlarged if necessary.