Chapter 4. Postinstallation Configuration

Current implementation of Java Virtual Machine allows only global configuration of memory for the JVM system process. Thus whole application server, together with WARs and EARs running on it, share one memory space.

Default JVM memory settings is too low for running application container with CloverETL Server. Some application servers, like IBM WebSphere, increase JVM defaults themselves, however they still may be too low.

The optimal memory limits depend on many conditions, i.e. transformations which CloverETL should execute. Please note that maximum limit isn't amount of permanently allocated memory, but limit which can't be exceeded. If the limit is exhausted, the OutOfMemoryError is raised.

JVM Memory Areas

JVM memory consists of several areas: heap memory, PermGen space, direct memory and stack memory. Since JVM memory is not just HEAP memory, you should not set the HEAP limit too high; in case it consumes whole RAM, JVM won't be able to allocate direct memory and stack for new threads.

Configuring Memory

You can set the minimum and maximum memory heap size by adjusting the "Xms" and "Xmx" JVM parameters. There are more ways to change the settings depending on the used application container.

Recommended Heap Memory Configuration

If you are not sure about the memory requirements for the transformations, a maximum of 1-2 GB heap memory is recommended. This limit may be increased during transformations development when OutOfMemoryError occurs.

Heap limit is not a limit of memory used by JVM. If you do not understand JVM in details, you should not assign more than 50% of main memory to heap.

Memory Configuration in Different Java Versions

In Java 7 and earlier, the memory space for loading classes (so called "PermGen space") is separated from the heap memory, and can be set by the JVM parameter "-XX:MaxPermSize". By default, it is just 64 MB which is not enough for enterprise applications. Again, suitable memory limit depends on various criteria, but 512 MB should be enough in most cases. If the PermGen space maximum is too low, OutOfMemoryError: PermGen space may occur.

In Java 8, memory space for loading classes (so called "Metaspace") is separated from heap, and can be set by the JVM parameter -XX:MaxMetaspaceSize. The default maximum Metaspace size is unlimited.

Please see the specific container section for details on memory settings.

Codecache Size

Some CloverETL Server installations can occasionally run into performance issue: JVM is running more than hundred times slower. The issue can be caused by a full code cache (https://docs.oracle.com/javase/8/embedded/develop-apps-platforms/codecache.htm). Reserved code cache size is platform dependent and can be too small for CloverETL Server. It is highly recommended to increase code cache size using the following JVM argument:

-XX:ReservedCodeCacheSize=256m

When using resource-demanding components, such as FastSort, or when running a large number of graphs concurrently, you may reach the system limit on simultaneously open files. This is usually indicated by the java.io.IOException: Too many open files exception.

The default limit is fairly low in many Linux distributions (e.g. 4096 in Ubuntu). Such a limit can be easily exceeded, considering that one FastSort component can open up to 1,000 files when sorting 10 million records. Furthermore, some application containers recommend increasing the limit themselves (8192 for IBM WebSphere).

Therefore, it is recommended to increase the limit for production systems. Reasonable limits vary from 10,000 to about 100,000 depending on the expected load of CloverETL Server and the complexity of your graphs.

The current limit can be displayed in most UNIX-like systems using the ulimit -Hn command.

The exact way of increasing the limit is OS-specific and is beyond the scope of this manual.

In order to function properly, CloverETL Server requires an outside communication. The table below describes both incoming and outgoing communication of CloverETL Server. Please, configure your firewall exceptions accordingly.