Home | Table of Contents | CopyFiles | CloverETL 4.6.1
--- | --- | ---
Prev | File Operations | Next

## CopyFiles

# Jobflow Component

Short Description
Ports
Metadata
CopyFiles Attributes
Details
Examples
See also

### Short Description

CopyFiles can be used to copy files and directories.

CopyFiles can copy multiple sources into one destination directory, or a regular source file to a target file. Directories can be copied recursively. Optionally, existing files may be skipped or updated based on the modification date of the files.

| Note
To be able to use this component, you need a separate jobflow license.

Component | Inputs | Outputs | Auto-propagated metadata
--- | --- | --- | ---
CopyFiles | 0-1 | 0-2 |

### Icon

### Ports

Port type | Number | Required | Description | Metadata
--- | --- | --- | --- | ---
Input | 0 |  | Input data records to be mapped to component attributes. | Any
Output | 0 |  | Results | Any
1 |  | Errors | Any

### Metadata

CopyFiles does not propagate metadata from left to right or from right to left.

This component has metadata template available. See Details for details on template fields of CopyFiles or Metadata Templates for general details on metadata templates.

### CopyFiles Attributes

Attribute | Req | Description | Possible values
--- | --- | --- | ---
Basic
Source file URL | yes [1] | Path to the source file or directory (see Supported URL Formats for File Operations). |
Target file URL | yes [1] | Path to the destination file or directory (see Supported URL Formats for File Operations). When it points to a directory, the source will be copied into the directory. It must be a path to a single file or directory. |
Recursive | no | Copies directories recursively. | false (default) | true
Overwrite | no | Specifies whether existing files shall be overwritten. In always mode, the target will be overwritten. In update mode, the target will be overwritten only when the source file is newer than the destination file. In never mode, the target will not be overwritten. | always (default) | update | never
Create parent directories | no | Attempts to create non-existing parent directories. When the Create parent directories option is enabled and the Target file URL ends with a slash ('/'), it is treated as the parent directory, i.e. the source directory or file is copied into the target directory, even if it does not exist. | false (default) | true
Input mapping | [2] | Defines mapping of input records to component attributes. |
Output mapping | [2] | Defines mapping of results to the standard output port. |
Error mapping | [2] | Defines mapping of errors to the error output port. |
Redirect error output | no | If enabled, errors will be sent to the output port instead of the error port. | false (default) | true
Verbose output | no | If enabled, one input record may cause multiple records to be sent to the output (e.g. as a result of wildcard expansion). Otherwise, each input record will yield just one cumulative output record. | false (default) | true
Advanced
Stop processing on fail | no | By default, a failure causes the component to skip all subsequent input records and send the information about skipped input records to the error output port. This behavior can be turned off by this attribute. | true (default) | false
[1] The attribute is required, unless specified in the Input mapping. [2] Required if the corresponding edge is connected.

### Details

Editing any of the Input, Output or Error mapping opens the Transform Editor.

##### Input mapping

The editor allows you to override selected attributes of the component with the values of the input fields.

Field Name | Attribute | Type | Possible values
--- | --- | --- | ---
sourceURL | Source file URL | string |
targetURL | Target file URL | string |
recursive | Recursive | boolean | true | false
overwrite | Overwrite | string | "always" | "update" | "never"
makeParentDirs | Create parent directories | boolean | true | false

##### Output mapping

The editor allows you to map the results and the input data to the output port.

If Output mapping is empty, fields of the input record and result record are mapped to the output by name.

Field Name | Type | Description
--- | --- | ---
sourceURL | string | URL of the source file.
targetURL | string | URL of the destination.
resultURL | string | A new URL of the successfully copied file. Only set in Verbose output mode.
result | boolean | True if the operation has succeeded (can be false when Redirect error output is enabled).
errorMessage | string | If the operation has failed, the field contains the error message (used when Redirect error output is enabled).
stackTrace | string | If the operation has failed, the field contains the stack trace of the error (used when Redirect error output is enabled).

##### Error mapping

The editor allows you to map the errors and the input data to the error port.

If Error mapping is empty, fields of the input record and result record are mapped to the output by name.

Field Name | Type | Description
--- | --- | ---
result | boolean | Will always be set to false.
errorMessage | string | The error message.
stackTrace | string | The stack trace of the error.
sourceURL | string | URL of the source file.
targetURL | string | URL of the destination.

#### Stop processing on fail

If you get file URLs from the input port having Stop processing on fail set to true and error occurs, the record causing error and the following ones are skipped.

### Examples

Copying Files from one Directory to Another
Using Stop Processing on Fail
Non-recursive copying directory with mixed content

#### Copying Files from one Directory to Another

Copy `.txt` files from `${DATAIN_DIR}/FO/` to `${DATAOUT_DIR}/FO/`. The target directory may not exist. If the files in the target directory exist, they should be overwritten.

##### Solution

Use the attributes Source file URL, Target file URL and Create parent directories.

Attribute | Value
--- | ---
Source file URL | ${DATAIN_DIR}/FO/*.txt
Target file URL | ${DATAOUT_DIR}/FO/
Create parent directories | true

#### Using Stop Processing on Fail

Copy files `f1.txt` `f2.txt` and `f3.txt`. If you cannot copy a file, continue with the following one.

    -rw-rw-r--. 1 clover clover 0 Feb 23 18:11 f1.txt
    -rw-------. 1 user23 user23 0 Feb 23 18:11 f2.txt
    -rw-rw-r--. 1 clover clover 0 Feb 23 18:11 f3.txt

##### Solution with Source File URL attribute

The solution without reading file URLs from an input edge is same as in the previous example. Use the attributes Source file URL, Target file URL and Create parent directories.

Note that only files `f1.txt` and `f2.txt` will be copied. The file `f2.txt` cannot be copied as the user 'clover' does not have read access to the file.

##### Solution with Input Edge

If you read URLs of files from an input edge one by one, use the attributes Target file URL, Create parent directories Input mapping and Stop processing on fail.

Attribute | Value
--- | ---
Target file URL | ${DATAOUT_DIR}/FO/
Create parent directories | true
Input mapping |
Stop processing on fail | false

In Input mapping, you map Source URL.

Note that you need to uncheck the attribute Stop processing on fail. Otherwise only the file `f1.txt` would be copied. (Assuming that the files come in ascending order.)

#### Non-recursive copying directory with mixed content

This example shows non-recursive copying of files from one directory to another.

Copy files and directories from `${DATAIN_DIR}/abc` to `${DATAOUT_DIR}/def`. If a directory in `${DATAIN_DIR}/abc` contains a file or a directory, these files and directories should not be copied.

Source directory

    ${DATAIN_DIR}/abc/file.txt
    ${DATAIN_DIR}/abc/dirA/file2.txt

Expected result

    ${DATAIN_DIR}/def/file.txt
    ${DATAIN_DIR}/def/dirA

##### Solution

Use ListFiles to list the files and directories to be copied. In ListFiles set File URL attribute to the source directory.

Attribute | Value
--- | ---
File URL | ${DATAIN_DIR}/abc/

In CopyFiles, copy the files and directories from the list received from an input edge.

Attribute | Value
--- | ---
Target file URL | ${DATAIN_DIR}/def/
Create parent directories | true
Input mapping |
    //#CTL2

    // Transforms input record into output record.
    function integer transform() {
    	$out.0.sourceURL = $in.0.URL;

    	return ALL;
    }

| Note
The ListFiles component is necessary to list all files and directories. If you non-recursively copied the files with CopyFiles only, the copying would stop when a first subdirectory is reached.

### See also

MoveFiles
Common Properties of Components
Specific Attribute Types
Common Properties of File Operations
File Operations Comparison

Prev | Up | Next
Supported URL Formats for File Operations | Home | Table of Contents | CreateFiles