This component provides access to remote file systems over the FTP and SFTP protocols.
Consuming from remote FTP server | |
---|---|
Make sure you read the section titled Default when consuming files further below for details related to consuming files. |
ftp://[username@]hostname[:port]/directoryname[?options] sftp://[username@]hostname[:port]/directoryname[?options] ftps://[username@]hostname[:port]/directoryname[?options]
Where directoryname represents the underlying directory. Can contain nested folders.
If no username is provided, then
anonymous
login is attempted using no password. If no port number is provided, Fuse Mediation Router will provide default values
according to the protocol (ftp = 21, sftp = 22, ftps = 21).
This component uses two different libraries for the actual FTP work. FTP and FTPS use Apache Commons Net while SFTP uses JCraft JSCH.
You can append query options to the URI in the following format,
?option=value&option=value&...
The options below are exclusive to the FTP2 component:
Name | Default Value | Description |
---|---|---|
username
|
null
|
Specifies the username to use to log in to the remote file systen. |
password
|
null
|
Specifies the password to use to log in to the remote file system. |
binary
|
false
|
Specifies the file transfer mode, BINARY or ASCII. Default is ASCII
(false ). |
disconnect
|
false
|
Camel 2.2: Whether or not to disconnect from remote FTP server right after use. Can be used for both consumer and producer. Disconnect will only disconnect the current connection to the FTP server. If you have a consumer which you want to stop, then you need to stop the consumer/route instead. |
localWorkDirectory
|
null
|
When consuming, a local work directory can be used to store the remote file content directly in local files, to avoid loading the content into memory. This is beneficial, if you consume a very big remote file and thus can conserve memory. See below for more details. |
passiveMode
|
false
|
FTP only: Specifies whether to use passive mode
connections. Default is active mode {false ). |
securityProtocol
|
TLS
|
FTPS only: Sets the underlying security protocol.
The following values are defined: |
disableSecureDataChannelDefaults
|
false
|
Camel 2.4: FTPS only: Whether or not to disable using
default values for execPbsz and execProt when using
secure data transfer. You can set this option to true if you want to be
in absolute full control what the options execPbsz and
execProt should be used. |
execProt
|
null
|
Camel 2.4: FTPS only: Will by default use option
|
execPbsz
|
null
|
Camel 2.4: FTPS only: This option specifies the buffer
size of the secure data channel. If option useSecureDataChannel has
been enabled and this option has not been explicit set, then value 0 is
used. |
isImplicit
|
false
|
FTPS only: Sets the security mode(implicit/explicit).
Default is explicit (false ). |
knownHostsFile
|
null
|
SFTP only: Sets the known_hosts
file, so that the SFTP endpoint can do host key verification. |
privateKeyFilePassphrase
|
null
|
SFTP only: Set the private key file passphrase to that the SFTP endpoint can do private key verification. |
privateKeyFilePassphrase
|
null
|
SFTP only: Set the private key file passphrase to that the SFTP endpoint can do private key verification. |
strictHostKeyChecking
|
no
|
SFTP only:Camel 2.2:
Sets whether to use strict host key checking. Possible values are: no ,
yes and ask . ask does not make
sense to use as Camel cannot answer the question for you as its meant for human
intervention. Note: The default in Camel 2.1 and below
was ask . |
maximumReconnectAttempts
|
3 | Specifies the maximum reconnect attempts Fuse Mediation Router performs when it tries to connect to the remote FTP server. Use 0 to disable this behavior. |
reconnectDelay
|
1000 | Delay in millis Fuse Mediation Router will wait before performing a reconnect attempt. |
connectTimeout
|
10000
|
Camel 2.4: Is the connect timeout in millis. This
corresponds to using ftpClient.connectTimeout for the FTP/FTPS. For
SFTP this option is also used when attempting to connect. |
soTimeout
|
null
|
FTP and FTPS Only:Camel
2.4: Is the SocketOptions.SO_TIMEOUT value in millis. Note
SFTP will automatic use the connectTimeout as the
soTimeout . |
timeout
|
30000
|
FTP and FTPS Only:Camel
2.4: Is the data timeout in millis. This corresponds to using
ftpClient.dataTimeout for the FTP/FTPS. For SFTP there is no data
timeout. |
throwExceptionOnConnectFailed
|
false
|
Camel 2.5: Whether or not to thrown an exception if a
successful connection and login could not be establish. This allows a custom
pollStrategy to deal with the exception, for example to stop the
consumer or the likes. |
siteCommand
|
null
|
FTP and FTPS Only:Camel
2.5: To execute site commands after successful login. Multiple site commands
can be separated using a new line character (\n). Use help site to see
which site commands your FTP server supports. |
stepwise
|
true
|
When consuming directories, specifies whether or not to use stepwise mode for traversing the directory tree. Stepwise means that it will CD one directory at a time. For more details, see Stepwise changing directories. |
separator
|
Auto
|
Camel 2.6: Dictates what path separator
char to use when uploading files. Auto means use the
path provided without altering it. UNIX means use UNIX style path
separators. Windows means use Windows style path separators. |
ftpClient
|
null
|
FTP and FTPS Only:Camel
2.1: Allows you to use a custom
org.apache.commons.net.ftp.FTPClient instance. |
ftpClientConfig
|
null
|
FTP and FTPS Only:Camel
2.1: Allows you to use a custom
org.apache.commons.net.ftp.FTPClientConfig instance. |
serverAliveInterval
|
0
|
SFTP Only:Camel 2.8 Allows you to set the serverAliveInterval of the sftp session |
serverAliveCountMax
|
1
|
SFTP Only:Camel 2.8 Allows you to set the serverAliveCountMax of the sftp session |
ftpClient.trustStore.file
|
null
|
FTPS Only: Sets the trust store file, so that the FTPS client can look up for trusted certificates. |
ftpClient.trustStore.type
|
JKS
|
FTPS Only: Sets the trust store type. |
ftpClient.trustStore.algorithm
|
SunX509
|
FTPS Only: Sets the trust store algorithm. |
ftpClient.trustStore.password
|
null
|
FTPS Only: Sets the trust store password. |
ftpClient.keyStore.file
|
null
|
FTPS Only: Sets the key store file, so that the FTPS client can look up for the private certificate. |
ftpClient.keyStore.type
|
JKS
|
FTPS Only: Sets the key store type. |
ftpClient.keyStore.algorithm
|
SunX509
|
FTPS Only: Sets the key store algorithm. |
ftpClient.keyStore.password
|
null
|
FTPS Only: Sets the key store password. |
ftpClient.keyStore.keyPassword
|
null
|
FTPS Only: Sets the private key password. |
FTPS component default trust store | |
---|---|
By default, the FTPS component trust store accept all certificates. If you only want
trust selective certificates, you have to configure the trust store with the
|
You can configure additional options on the ftpClient
and
ftpClientConfig
from the URI directly by using the
ftpClient.
or ftpClientConfig.
prefix.
For example to set the setDataTimeout
on the
FTPClient
to 30 seconds you can do:
from("ftp://foo@myserver?password=secret&ftpClient.dataTimeout=30000") .to("bean:foo");
You can mix and match and have use both prefixes, for example to configure date format or timezones.
from("ftp://foo@myserver?password=secret&ftpClient.dataTimeout=30000&ftpClientConfig.serverLanguageCode=fr") .to("bean:foo");
You can have as many of these options as you like.
See the documentation of the Apache Commons FTP FTPClientConfig for possible options and more details. And as well for Apache Commons FTP FTPClient.
If you do not like having many and long configuration in the url you can refer to the
ftpClient
or ftpClientConfig
to use by letting Camel
lookup in the Registry for it.
For example:
<bean id="myConfig" class="org.apache.commons.net.ftp.FTPClientConfig"> <property name="lenientFutureDates" value="true"/> <property name="serverLanguageCode" value="fr"/> </bean>
And then let Camel lookup this bean when you use the # notation in the url.
from("ftp://foo@myserver?password=secret&ftpClientConfig=#myConfig").to("bean:foo");
Important | |
---|---|
See File2 as all the options there also applies for this component. |
Here are some examples of FTP endpoint URIs:
ftp://[email protected]/public/upload/images/holiday2008?password=secret&binary=true ftp://[email protected]:12049/reports/2008/password=secret&binary=false ftp://publicftpserver.com/download
FTP Consumer does not support concurrency | |
---|---|
The FTP consumer (with the same endpoint) does not support concurrency (the backing FTP client is not thread safe). You can use multiple FTP consumers to poll from different endpoints. It is only a single endpoint that does not support concurrent consumers. The FTP producer does not have this issue, it supports concurrency. |
The FTP consumer will by default leave the consumed
files untouched on the remote FTP server. You have to configure it explicit if you want it to
delete the files or move them to another location. For example you can use
delete=true
to delete the files, or use move=.done
to
move the files into a hidden done sub directory.
The regular File consumer is different as it will by
default move files to a .camel
sub directory. The reason Camel does
not do this by default for the FTP consumer is that it may
lack permissions by default to be able to move or delete files.
The option readLock can be used to force Fuse Mediation Router not to consume files that are currently in the process of being written. However, this option is turned off by default, as it requires that the user has write access. There are other solutions to avoid consuming files that are currently being written over FTP; for instance, you can write to a temporary destination and move the file after it has been written.
The ftp producer does not support appending to existing files. Any existing files on the remote server will be deleted before the file is written.
The following message headers can be used to affect the behavior of the component
Header | Description |
---|---|
CamelFileName
|
Specifies the output file name (relative to the endpoint directory) to be used for the output message when sending to the endpoint. If this is not present and no expression either, then a generated message ID is used as the filename instead. |
CamelFileNameProduced
|
The actual absolute filepath (path + name) for the output file that was written. This header is set by Fuse Mediation Router and its purpose is providing end-users the name of the file that was written. |
CamelFileBatchIndex
|
Current index out of total number of files being consumed in this batch. |
CamelFileBatchSize
|
Total number of files being consumed in this batch. |
CamelFileHost
|
The remote hostname. |
CamelFileLocalWorkPath
|
Path to the local work file, if local work directory is used. |
The two sets of libraries (see above) have different APIs for setting the timeout. You can
use the connectTimeout
option for both of them to set a timeout in
milliseconds to establish a network connection. An individual soTimeout
can
also be set on the FTP/FTPS, which corresponds to using
ftpClient.soTimeout
. Notice SFTP will automatically use
connectTimeout
as its soTimeout
. The
timeout
option only applies for FTP/FTSP as the data timeout, which
corresponds to the ftpClient.dataTimeout
value. All timeout values are in
milliseconds.
Fuse Mediation Router supports consuming from remote FTP servers and downloading the files directly
into a local work directory. This avoids reading the entire remote file content into memory as
it is streamed directly into the local file using FileOutputStream
.
Fuse Mediation Router will store to a local file with the same name as the remote file, though with
.inprogress
as extension while the file is being downloaded. Afterwards,
the file is renamed to remove the .inprogress
suffix. And finally, when the
Exchange is complete the local file is deleted.
So if you want to download files from a remote FTP server and store it as files then you need to route to a file endpoint such as:
from("ftp://[email protected]?password=secret&localWorkDirectory=/tmp").to("file://inbox");
Optimization by renaming work file | |
---|---|
The route above is ultra efficient as it avoids reading the entire file content into
memory. It will download the remote file directly to a local file stream. The
|
Camel FTP can operate in two modes in terms of traversing directories when consuming files (for example, downloading) or producing files (for example, uploading):
stepwise
not stepwise
You may want to pick either one depending on your situation and security issues. Some Camel end users can only download files if they use stepwise, while others can only download if they do not. At least you have the choice to pick.
Note that stepwise changing of directory will in most cases only work when the user is
confined to it's home directory and when the home directory is reported as
/
.
The difference between the two of them is best illustrated with an example. Suppose we have the following directory structure on the remote FTP server we need to traverse and download files:
/ /one /one/two /one/two/sub-a /one/two/sub-b
And that we have a file in each of sub-a
(a.txt
) and
sub-b
(b.txt
) folder.
The following log shows the conversation between the FTP endpoint and the remote FTP server when the FTP endpoint is operating in stepwise mode:
TYPE A 200 Type set to A PWD 257 "/" is current directory. CWD one 250 CWD successful. "/one" is current directory. CWD two 250 CWD successful. "/one/two" is current directory. SYST 215 UNIX emulated by FileZilla PORT 127,0,0,1,17,94 200 Port command successful LIST 150 Opening data channel for directory list. 226 Transfer OK CWD sub-a 250 CWD successful. "/one/two/sub-a" is current directory. PORT 127,0,0,1,17,95 200 Port command successful LIST 150 Opening data channel for directory list. 226 Transfer OK CDUP 200 CDUP successful. "/one/two" is current directory. CWD sub-b 250 CWD successful. "/one/two/sub-b" is current directory. PORT 127,0,0,1,17,96 200 Port command successful LIST 150 Opening data channel for directory list. 226 Transfer OK CDUP 200 CDUP successful. "/one/two" is current directory. CWD / 250 CWD successful. "/" is current directory. PWD 257 "/" is current directory. CWD one 250 CWD successful. "/one" is current directory. CWD two 250 CWD successful. "/one/two" is current directory. PORT 127,0,0,1,17,97 200 Port command successful RETR foo.txt 150 Opening data channel for file transfer. 226 Transfer OK CWD / 250 CWD successful. "/" is current directory. PWD 257 "/" is current directory. CWD one 250 CWD successful. "/one" is current directory. CWD two 250 CWD successful. "/one/two" is current directory. CWD sub-a 250 CWD successful. "/one/two/sub-a" is current directory. PORT 127,0,0,1,17,98 200 Port command successful RETR a.txt 150 Opening data channel for file transfer. 226 Transfer OK CWD / 250 CWD successful. "/" is current directory. PWD 257 "/" is current directory. CWD one 250 CWD successful. "/one" is current directory. CWD two 250 CWD successful. "/one/two" is current directory. CWD sub-b 250 CWD successful. "/one/two/sub-b" is current directory. PORT 127,0,0,1,17,99 200 Port command successful RETR b.txt 150 Opening data channel for file transfer. 226 Transfer OK CWD / 250 CWD successful. "/" is current directory. QUIT 221 Goodbye disconnected.
As you can see when stepwise is enabled, it will traverse the directory structure using CD xxx.
The following log shows the conversation between the FTP endpoint and the remote FTP server when the FTP endpoint is operating in non-stepwise mode:
230 Logged on TYPE A 200 Type set to A SYST 215 UNIX emulated by FileZilla PORT 127,0,0,1,4,122 200 Port command successful LIST one/two 150 Opening data channel for directory list 226 Transfer OK PORT 127,0,0,1,4,123 200 Port command successful LIST one/two/sub-a 150 Opening data channel for directory list 226 Transfer OK PORT 127,0,0,1,4,124 200 Port command successful LIST one/two/sub-b 150 Opening data channel for directory list 226 Transfer OK PORT 127,0,0,1,4,125 200 Port command successful RETR one/two/foo.txt 150 Opening data channel for file transfer. 226 Transfer OK PORT 127,0,0,1,4,126 200 Port command successful RETR one/two/sub-a/a.txt 150 Opening data channel for file transfer. 226 Transfer OK PORT 127,0,0,1,4,127 200 Port command successful RETR one/two/sub-b/b.txt 150 Opening data channel for file transfer. 226 Transfer OK QUIT 221 Goodbye disconnected.
As you can see when not using stepwise, there are no CD operation invoked at all.
In the sample below we set up Fuse Mediation Router to download all the reports from the FTP server once every hour (60 min) as BINARY content and store it as files on the local file system.
protected RouteBuilder createRouteBuilder() throws Exception { return new RouteBuilder() { public void configure() throws Exception { // we use a delay of 60 minutes (eg. once pr. hour we poll the FTP server long delay = 60 * 60 * 1000L; // from the given FTP server we poll (= download) all the files // from the public/reports folder as BINARY types and store this as files // in a local directory. Fuse Mediation Router will use the filenames from the FTPServer // notice that the FTPConsumer properties must be prefixed with "consumer." in the URL // the delay parameter is from the FileConsumer component so we should use consumer.delay as // the URI parameter name. The FTP Component is an extension of the File Component. from("ftp://tiger:scott@localhost/public/reports?binary=true&consumer.delay=" + delay). to("file://target/test-reports"); } }; }
And the route using Spring DSL:
<route> <from uri="ftp://scott@localhost/public/reports?password=tiger&inary=true&elay=60000"/> <to uri="file://target/test-reports"/> </route>
The FTP consumer is built as a scheduled consumer to be used in the from route. However, if you want to start consuming from an FTP server triggered within a route, use a route like the following:
from("seda:start") // set the filename in FILE_NAME header so Fuse Mediation Router know the name of the remote file to poll .setHeader(Exchange.FILE_NAME, header("myfile")) .pollEnrich("ftp://admin@localhost:21/getme?password=admin&binary=false") .to("mock:result");
from("ftps://admin@localhost:2222/public/camel?password=admin&securityProtocol=SSL&isImplicit=true &ftpClient.keyStore.file=./src/test/resources/server.jks &ftpClient.keyStore.password=password&ftpClient.keyStore.keyPassword=password") .to("bean:foo");
from("ftps://admin@localhost:2222/public/camel?password=admin&ftpClient.trustStore.file=./src/test/resources/server.jks&ftpClient.trustStore.password=password") .to("bean:foo");
Fuse Mediation Router supports pluggable filtering strategies. You define a filter strategy by
implementing the org.apache.camel.component.file.GenericFileFilter
interface in Java. You can then configure the endpoint with the filter to skip certain
files.
In the following sample we define a filter that only accepts files whose filename starts
with report
.
public class MyFileFilter implements GenericFileFilter { public boolean accept(GenericFile file) { // we only want report files return file.getFileName().startsWith("report"); } }
And then we can configure our route using the filter
attribute to reference our filter (using #
notation) that we have defined
in the spring XML file:
<!-- define our sorter as a plain spring bean --> <bean id="myFilter" class="com.mycompany.MyFileFilter"/> <route> <from uri="ftp://[email protected]?password=secret&filter=#myFilter"/> <to uri="bean:processInbox"/> </route>
The ANT path matcher is a filter that is shipped out-of-the-box in the camel-spring jar. So you need to depend on camel-spring if you are using Maven. The reason is that we leverage Spring's AntPathMatcher to do the actual matching.
The file paths are matched with the following rules:
?
matches one character*
matches zero or more characters**
matches zero or more directories in a path
The sample below demonstrates how to use it:
<camelContext xmlns="http://camel.apache.org/schema/spring"> <template id="camelTemplate"/> <!-- use myFilter as filter to allow setting ANT paths for which files to scan for --> <endpoint id="myFTPEndpoint" uri="ftp://admin@localhost:20123/antpath?password=admin&ecursive=true&elay=10000&nitialDelay=2000&ilter=#myAntFilter"/> <route> <from ref="myFTPEndpoint"/> <to uri="mock:result"/> </route> </camelContext> <!-- we use the AntPathMatcherRemoteFileFilter to use ant paths for includes and exlucde --> <bean id="myAntFilter" class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter"> <!-- include and file in the subfolder that has day in the name --> <property name="includes" value="**/subfolder/**/*day*"/> <!-- exclude all files with bad in name or .xml files. Use comma to seperate multiple excludes --> <property name="excludes" value="**/*bad*,**/*.xml"/> </bean>