Simple HTTP API

Home \| Table of Contents	Simple HTTP API	CloverETL 4.7.0
Prev	API	Next

The Simple HTTP API is a basic Server automation tool that lets you control the Server from external applications using simple HTTP calls.

Most of operations is accessible using the HTTP GET method and return plain text. Thus, both “request” and “response” can be conveniently sent and parsed using very simple tools (wget, grep, etc.).

If global security is “on” (on by default), Basic HTTP authentication is used. Authenticated operations will require valid user credentials with corresponding permissions.

Note that the ETL graph-related operations "graph_run", "graph_status" and "graph_kill" also work for jobflows and Data Profiler jobs.

The generic pattern for a request URL:

http://[domain]:[port]/[context]/[servlet]/[operation]?[param1]=[value1]&[param2]=[value2]...

For a wget client, you can use following command line:

wget --user=$USER --password=$PASS -O ./$OUTPUT_FILE $REQUEST_URL

Operation help

parameters

returns

list of possible operations and parameters with its descriptions

example

http://localhost:8080/clover/request_processor/help

Operation graph_run

Call this operation to start execution of the specified job. The operation is called graph_run for backward compatibility, however it may execute ETL graph, jobflow or profiler job.

parameters

Table 25.1. Parameters of graph_run

parameter name	mandatory	default	description
graphID	yes	-	A file path to the job file, relative to the sandbox root.
sandbox	yes	-	Text ID of sandbox.
additional job parameters	no		Any URL parameter with "param_" prefix is passed to executed job and may be used in transformation XML as a placeholder, but without the "param_" prefix. e.g. "param_FILE_NAME" specified in URL may be used in the XML as ${FILE_NAME}. These parameters are resolved only during loading of XML, so it cannot be pooled.
additional config parameters	no		URL Parameters prefixed with "config_" can set some of the execution parameters. For graphs, the following parameters are supported: `config_skipCheckConfig` - when set to "false", graph configuration will be checked before the execution. `config_logLevel` - log level of the executed graph, one of OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, ALL. `config_clearObsoleteTempFiles` - when set to "true", temp files of previous runs of this graph will be deleted before the execution. `config_debugMode` - when set to "true", debug mode for given graph will be enabled. See Job Config Properties for more info.
nodeID	no	-	In cluster mode it's ID of a node which should execute the job. However it's not final. If the graph is distributed, or the node is disconnected, the graph may be executed on some another node.
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should possible error message be.

returns

run ID: incremental number, which identifies each execution request

example

http://localhost:8080/clover/request_processor/graph_run?graphID=graph/graphDBExecute.grf&sandbox=mva

Operation graph_status

Call this operation to obtain status of specified job execution. The operation is called graph_status for backward compatibility, however it may return status of an ETL graph or jobflow.

parameters

Table 25.2. Parameters of graph_status

parameter name	mandatory	default	description
runID	yes	-	Id of each graph execution
returnType	no	STATUS	STATUS \| STATUS_TEXT \| DESCRIPTION \| DESCRIPTION_XML
waitForStatus	no	-	Status code which we want to wait for. If it is specified, this operation will wait until the graph is in the required status.
waitTimeout	no	0	If waitForStatus is specified, it will wait only the specified amount of milliseconds. Default 0 means forever, but it depends on an application server configuration. When the specified timeout expires and graph run still isn't in required status, the server returns code 408 (Request Timeout). 408 code may be also returned by an application server if its HTTP request timeout expires before.
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should possible error message be.

returns

Status of a specified graph. It may be number code, text code or a complex description in dependence on optional parameter returnType. Description is returned as plain text with a pipe as a separator, or as XML. A schema describing XML format of the XML response is accessible on CloverETL Server URL: http://[host]:[port]/clover/schemas/executions.xsd In dependence on waitForStatus parameter it may return result immediately or wait for a specified status.

example

http://localhost:8080/clover/request_processor/graph_status?runID=123456&returnType=DESCRIPTION&waitForStatus=FINISHED&waitTimeout=60000

Operation graph_kill

Call this operation to abort/kill job execution. The operation is called graph_kill for backward compatibility, however it may abort/kill ETL graph, jobflow or profiler job.

parameters

Table 25.3. Parameters of graph_kill

parameter name	mandatory	default	description
runID	yes	-	Id of each graph execution
returnType	no	STATUS	STATUS \| STATUS_TEXT \| DESCRIPTION
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should possible error message be.

returns

Status of the specified graph after an attempt to kill it. It may be number code, text code or a complex description in dependence on optional parameter.

example

http://localhost:8080/clover/request_processor/graph_kill?runID=123456&returnType=DESCRIPTION

Operation server_jobs

parameters

returns

List of runIDs of currently running jobs.

example

http://localhost:8080/clover/request_processor/server_jobs

Operation sandbox_list

parameters

returns

List of all sandbox text IDs. In next versions will return only accessible ones.

example

http://localhost:8080/clover/request_processor/sandbox_list

Operation sandbox_content

parameters

Table 25.4. Parameters of sandbox_content

parameter name	mandatory	default	description
sandbox	yes	-	text ID of sandbox
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should possible error message be.

returns

A list of all elements in the specified sandbox. Each element may be specified as a file path relative to the sandbox root.

example

http://localhost:8080/clover/request_processor/sandbox_content?sandbox=mva

Operation executions_history

parameters

Table 25.5. Parameters of executions_history

parameter name	mandatory	default	description
sandbox	yes	-	text ID of sandbox
from	no		Lower datetime limit of start of execution. The operation will return only records after (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
to	no		Upper datetime limit of start of execution. The operation will return only records before (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
stopFrom	no		Lower datetime limit of stop of execution. The operation will return only records after (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
stopTo	no		Upper datetime limit of stop of execution. The operation will return only records before (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
status	no		Current execution status. The operation will return only records with specified STATUS. Meaningful values are RUNNING \| ABORTED \| FINISHED_OK \| ERROR
sandbox	no		Sandbox code. The operation will return only records for graphs from the specified sandbox.
graphId	no		Text Id, which is unique in a specified sandbox. The file path is relative to the sandbox root.
orderBy	no		Attribute for list ordering. Possible values: id \| graphId \| status \| startTime \| stopTime. By default, there is no ordering.
orderDescend	no	true	A switch which specifies ascending or descending ordering. If it is true (which is default), ordering is descending.
returnType	no	IDs	Possible values are: IDs \| DESCRIPTION \| DESCRIPTION_XML
index	no	0	Index of the first returned records in whole record set. (starting from
records	no	infinite	Max amount of returned records.
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should possible error message be.

returns

List of executions according to filter criteria.

For returnType==IDs returns a simple list of runIDs (with new line delimiter).

For returnType==DESCRIPTION returns complex response which describes current status of selected executions, their phases, nodes and ports.

execution|[runID]|[status]|[username]|[sandbox]|[graphID]|[startedDatetime]|[finishedDatetime]|[clusterNode]|[graphVersion]
phase|[index]|[execTimeInMilis]
node|[nodeID]|[status]|[totalCpuTime]|[totalUserTime]|[cpuUsage]|[peakCpuUsage]|[userUsage]|[peakUserUsage]
port|[portType]|[index]|[avgBytes]|[avgRows]|[peakBytes]|[peakRows]|[totalBytes]|[totalRows]

example of request

http://localhost:8080/clover/request_processor/executions_history?from=&to=2008-09-16+16%3A40&status=&sandbox=def&graphID=&index=&records=&returnType=DESCRIPTION

example of DESCRIPTION (plain text) response

execution|13108|FINISHED_OK|clover|def|test.grf|2008-09-16 11:11:19|2008-09-16 11:11:58|nodeA|2.4
phase|0|38733
node|DATA_GENERATOR1|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
port|Output|0|0|0|0|0|130|10
node|TRASH0|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
port|Input|0|0|0|5|0|130|10
node|SPEED_LIMITER0|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
port|Input|0|0|0|0|0|130|10
port|Output|0|0|0|5|0|130|10
execution|13107|ABORTED|clover|def|test.grf|2008-09-16 11:11:19|2008-09-16 11:11:30
phase|0|11133
node|DATA_GENERATOR1|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
port|Output|0|0|0|0|0|130|10
node|TRASH0|RUNNING|0|0|0.0|0.0|0.0|0.0
port|Input|0|5|0|5|0|52|4
node|SPEED_LIMITER0|RUNNING|0|0|0.0|0.0|0.0|0.0
port|Input|0|0|0|0|0|130|10
port|Output|0|5|0|5|0|52|4

For returnType==DESCRIPTION_XML returns complex data structure describing one or more selected executions in XML format. A schema describing XML format of the XML response is accessible on CloverETL Server URL: http://[host]:[port]/clover/schemas/executions.xsd

Operation suspend

Suspends server or sandbox (if specified). Suspension means, that no graphs may me executed on suspended server/sandbox.

parameters

Table 25.6. Parameters of suspend

parameter name	mandatory	default	description
sandbox	no	-	Text ID of a sandbox to suspend. If not specified, it suspends the whole server.
atonce	no		If this param is set to true, running graphs from suspended server (or just from sandbox) are aborted. Otherwise it can run until it is finished in common way.

returns

Result message

Operation resume

parameters

Table 25.7. Parameters of resume

parameter name	mandatory	default	description
sandbox	no	-	Text Id of a sandbox to resume. If not specified, the server will be resumed.
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should the possible error message be.

returns

Result message

Operation sandbox_create

This operation creates a specified sandbox. If it is sandbox of "partitioned" or "local" type, it also creates locations by "sandbox_add_location" operation.

parameters

Table 25.8. Parameters of sandbox create

parameter name	mandatory	default	description
sandbox	yes	-	Text Id of sandbox to be created.
path	no	-	Path to the sandbox root if server is running in standalone mode.
type	no	shared	Sandbox type: shared \| partitioned \| local. For standalone server may be left empty, since the default "shared" is used.
createDirs	no	true	Switch whether to create directory structure of the sandbox (only for standalone server or "shared" sandboxes in cluster environment).
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should possible error message be.

returns

Result message

Operation sandbox_add_location

This operation adds a location to the specified sandbox. Can be only used with partitioned or local sandboxes.

parameters

Table 25.9. Parameters of sandbox add location

parameter name	mandatory	default	description
sandbox	yes	-	Sandbox which we want to add location to.
nodeId	yes	-	Location attribute - node which has direct access to the location.
path	yes	-	Location attribute - path to the location root on the specified node.
location	no	-	Location attribute - location storage ID. If it's not specified, new one will be generated.
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should possible error message be.

returns

Result message

Operation sandbox_remove_location

This operation removes a location from the specified sandbox. Only sandboxes of type partitioned or local can have locations asociated.

parameters

Table 25.10. Parameters of sandbox add location

parameter name	mandatory	default	description
sandbox	yes	-	Removes specified location from its sandbox.
location	yes	-	Location storage ID. If the specified location isn't attached to the specified sandbox, sandbox won't be changed.
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should possible error message be.

returns

Result message

Operation download_sandbox_zip

This operation downloads content of a specified sandbox as a ZIP archive.

parameters

Table 25.11. Parameters

parameter name	mandatory	default	description
sandbox	yes	-	Code of the sandbox to be downloaded.

returns

Content of the specified sandbox as a ZIP archive

example

 wget --http-user=username --http-password=password http://localhost:8080/clover/simpleHttpApi/download_sandbox_zip?sandbox=my-sandbox

Operation upload_sandbox_zip

This operation uploads content of a ZIP archive into a specified sandbox.

parameters

Table 25.12. Parameters

parameter name	mandatory	default	description
sandbox	yes	-	Code of the sandbox the ZIP file will be expanded to.
zipFile	yes	-	The ZIP archive file.
overwriteExisting	no	false	If `true`, the files already present in the sandbox will be overwritten.
deleteMissing	no	false	If `true`, the files not present in the ZIP file will be deleted from the sandbox.
fileNameEncoding	no	UTF-8	The encoding that was used to store file names in the ZIP archive.

returns

Result message

example of request (with using curl CLI tool (http://curl.haxx.se/))

curl -u username:password -F "overwriteExisting=true"
    -F "zipFile=@/tmp/my-sandbox.zip"
    http://localhost:8080/clover/simpleHttpApi/upload_sandbox_zip

Operation cluster_status

This operation displays cluster's nodes list.

parameters

returns

Cluster's nodes list.

Operation export_server_config

This operation exports a current server configuration in XML format.

parameters

Table 25.13. Parameters of server configuration export

parameter name	mandatory	default	description
include	no	all	Selection which items will be included in the exported XML file; the parameter may be specified multiple times. Possible values are: `all` - include items of all types `users` - include list of users `userGroups` - include list of user groups `sandboxes` - include list of sandboxes `jobConfigs` - include list of job configuration parameters `schedules` - include list of schedules `eventListeners` - include list of event listeners `launchServices` - include list of launch services `tempSpaces` - include list of temporary spaces

returns

Current server configuration as an XML file.

example

wget http://localhost:8080/clover/simpleHttpApi/export_server_config

Operation import_server_config

This operation imports server configuration.

parameters

Table 25.14. Parameters of server configuration import

parameter name	mandatory	default	description
xmlFile	yes	-	An XML file with server's configuration.
dryRun	no	true	If `true`, a dry run is performed with no actual changes written.
verbose	no	MESSAGE	MESSAGE \| FULL - how verbose should the response be: MESSAGE for a simple message, FULL for a full XML report.
newOnly	no	false	If `true` only new items will imported to the server; the items already present on the server will be left untouched.
include	no	all	Selection which items will be imported from the XML; the parameter may be specified multiple times. Possible values are: `all` - import items of all types `users` - import users `userGroups` - import user groups `sandboxes` - import sandboxes `jobConfigs` - import job configuration parameters `schedules` - import schedules `eventListeners` - import listeners `launchServices` - import launch services `tempSpaces` - import temporary spaces

returns

Result message or XML report

example of request (with using curl CLI tool (http://curl.haxx.se/))

curl -u username:password -F "dryRun=true" -F "verbose=FULL"
    -F "xmlFile=@/tmp/clover_configuration_2013-07-10_14-03-23+0200.xml"
    http://localhost:8080/clover/simpleHttpApi/import_server_config