Home | Table of Contents | Simple HTTP API | CloverETL 4.6.1
--- | --- | ---
Prev | API | Next

## Simple HTTP API

The Simple HTTP API is a basic Server automation tool that lets you control the Server from external applications using simple HTTP calls.

Most of operations is accessible using the HTTP GET method and return plain text. Thus, both “request” and “response” can be conveniently sent and parsed using very simple tools (wget, grep, etc.).

If global security is “on” (on by default), Basic HTTP authentication is used. Authenticated operations will require valid user credentials with corresponding permissions.

Note that the ETL graph-related operations "graph_run", "graph_status" and "graph_kill" also work for jobflows and Data Profiler jobs.

The generic pattern for a request URL:

    http://[domain]:[port]/[context]/[servlet]/[operation]?[param1]=[value1]&[param2]=[value2]...

For a wget client, you can use following command line:

    wget --user=$USER --password=$PASS -O ./$OUTPUT_FILE $REQUEST_URL

- Operation help

- Operation graph_run

- Operation graph_status

- Operation graph_kill

- Operation server_jobs

- Operation sandbox_list

- Operation sandbox_content

- Operation executions_history

- Operation suspend

- Operation resume

- Operation sandbox_create

- Operation sandbox_add_location

- Operation sandbox_remove_location

- Operation download_sandbox_zip

- Operation upload_sandbox_zip

- Operation cluster_status

- Operation export_server_config

- Operation import_server_config

### Operation help

parameters

no

returns

list of possible operations and parameters with its descriptions

example

    http://localhost:8080/clover/request_processor/help

### Operation graph_run

Call this operation to start execution of the specified job. The operation is called graph_run for backward compatibility, however it may execute ETL graph, jobflow or profiler job.

parameters

Table 25.1. Parameters of graph_run

parameter name | mandatory | default | description
--- | --- | --- | ---
graphID | yes | - | A file path to the job file, relative to the sandbox root.
sandbox | yes | - | Text ID of sandbox.
additional job parameters | no |  | Any URL parameter with "param_" prefix is passed to executed job and may be used in transformation XML as a placeholder, but without the "param_" prefix. e.g. "param_FILE_NAME" specified in URL may be used in the XML as ${FILE_NAME}. These parameters are resolved only during loading of XML, so it cannot be pooled.
additional config parameters | no |  | URL Parameters prefixed with "config_" can set some of the execution parameters. For graphs, the following parameters are supported: `config_skipCheckConfig` - when set to "false", graph configuration will be checked before the execution. `config_logLevel` - log level of the executed graph, one of OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, ALL. `config_clearObsoleteTempFiles` - when set to "true", temp files of previous runs of this graph will be deleted before the execution. `config_debugMode` - when set to "true", debug mode for given graph will be enabled. See Job Config Properties for more info.
nodeID | no | - | In cluster mode it's ID of a node which should execute the job. However it's not final. If the graph is distributed, or the node is disconnected, the graph may be executed on some another node.
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should possible error message be.

returns

run ID: incremental number, which identifies each execution request

example

    http://localhost:8080/clover/request_processor/graph_run?graphID=graph/graphDBExecute.grf&sandbox=mva

### Operation graph_status

Call this operation to obtain status of specified job execution. The operation is called graph_status for backward compatibility, however it may return status of an ETL graph or jobflow.

parameters

Table 25.2. Parameters of graph_status

parameter name | mandatory | default | description
--- | --- | --- | ---
runID | yes | - | Id of each graph execution
returnType | no | STATUS | STATUS | STATUS_TEXT | DESCRIPTION | DESCRIPTION_XML
waitForStatus | no | - | Status code which we want to wait for. If it is specified, this operation will wait until the graph is in the required status.
waitTimeout | no | 0 | If waitForStatus is specified, it will wait only the specified amount of milliseconds. Default 0 means forever, but it depends on an application server configuration. When the specified timeout expires and graph run still isn't in required status, the server returns code 408 (Request Timeout). 408 code may be also returned by an application server if its HTTP request timeout expires before.
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should possible error message be.

returns

Status of a specified graph. It may be number code, text code or a complex description in dependence on optional parameter returnType. Description is returned as plain text with a pipe as a separator, or as XML. A schema describing XML format of the XML response is accessible on CloverETL Server URL: http://[host]:[port]/clover/schemas/executions.xsd In dependence on waitForStatus parameter it may return result immediately or wait for a specified status.

example

    http://localhost:8080/clover/request_processor/graph_status?runID=123456&returnType=DESCRIPTION&waitForStatus=FINISHED&waitTimeout=60000

### Operation graph_kill

Call this operation to abort/kill job execution. The operation is called graph_kill for backward compatibility, however it may abort/kill ETL graph, jobflow or profiler job.

parameters

Table 25.3. Parameters of graph_kill

parameter name | mandatory | default | description
--- | --- | --- | ---
runID | yes | - | Id of each graph execution
returnType | no | STATUS | STATUS | STATUS_TEXT | DESCRIPTION
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should possible error message be.

returns

Status of the specified graph after an attempt to kill it. It may be number code, text code or a complex description in dependence on optional parameter.

example

    http://localhost:8080/clover/request_processor/graph_kill?runID=123456&returnType=DESCRIPTION

### Operation server_jobs

parameters

no

returns

List of runIDs of currently running jobs.

example

    http://localhost:8080/clover/request_processor/server_jobs

### Operation sandbox_list

parameters

no

returns

List of all sandbox text IDs. In next versions will return only accessible ones.

example

    http://localhost:8080/clover/request_processor/sandbox_list

### Operation sandbox_content

parameters

Table 25.4. Parameters of sandbox_content

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | yes | - | text ID of sandbox
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should possible error message be.

returns

A list of all elements in the specified sandbox. Each element may be specified as a file path relative to the sandbox root.

example

    http://localhost:8080/clover/request_processor/sandbox_content?sandbox=mva

### Operation executions_history

parameters

Table 25.5. Parameters of executions_history

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | yes | - | text ID of sandbox
from | no |  | Lower datetime limit of start of execution. The operation will return only records after (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
to | no |  | Upper datetime limit of start of execution. The operation will return only records before (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
stopFrom | no |  | Lower datetime limit of stop of execution. The operation will return only records after (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
stopTo | no |  | Upper datetime limit of stop of execution. The operation will return only records before (and equal) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).
status | no |  | Current execution status. The operation will return only records with specified STATUS. Meaningful values are RUNNING | ABORTED | FINISHED_OK | ERROR
sandbox | no |  | Sandbox code. The operation will return only records for graphs from the specified sandbox.
graphId | no |  | Text Id, which is unique in a specified sandbox. The file path is relative to the sandbox root.
orderBy | no |  | Attribute for list ordering. Possible values: id | graphId | status | startTime | stopTime. By default, there is no ordering.
orderDescend | no | true | A switch which specifies ascending or descending ordering. If it is true (which is default), ordering is descending.
returnType | no | IDs | Possible values are: IDs | DESCRIPTION | DESCRIPTION_XML
index | no | 0 | Index of the first returned records in whole record set. (starting from
records | no | infinite | Max amount of returned records.
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should possible error message be.

returns

List of executions according to filter criteria.

For `returnType==IDs` returns a simple list of runIDs (with new line delimiter).

For `returnType==DESCRIPTION` returns complex response which describes current status of selected executions, their phases, nodes and ports.

    execution|[runID]|[status]|[username]|[sandbox]|[graphID]|[startedDatetime]|[finishedDatetime]|[clusterNode]|[graphVersion]
    phase|[index]|[execTimeInMilis]
    node|[nodeID]|[status]|[totalCpuTime]|[totalUserTime]|[cpuUsage]|[peakCpuUsage]|[userUsage]|[peakUserUsage]
    port|[portType]|[index]|[avgBytes]|[avgRows]|[peakBytes]|[peakRows]|[totalBytes]|[totalRows]

example of request

    http://localhost:8080/clover/request_processor/executions_history?from=&to=2008-09-16+16%3A40&status=&sandbox=def&graphID=&index=&records=&returnType=DESCRIPTION

example of DESCRIPTION (plain text) response

    execution|13108|FINISHED_OK|clover|def|test.grf|2008-09-16 11:11:19|2008-09-16 11:11:58|nodeA|2.4
    phase|0|38733
    node|DATA_GENERATOR1|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
    port|Output|0|0|0|0|0|130|10
    node|TRASH0|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
    port|Input|0|0|0|5|0|130|10
    node|SPEED_LIMITER0|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
    port|Input|0|0|0|0|0|130|10
    port|Output|0|0|0|5|0|130|10
    execution|13107|ABORTED|clover|def|test.grf|2008-09-16 11:11:19|2008-09-16 11:11:30
    phase|0|11133
    node|DATA_GENERATOR1|FINISHED_OK|0|0|0.0|0.0|0.0|0.0
    port|Output|0|0|0|0|0|130|10
    node|TRASH0|RUNNING|0|0|0.0|0.0|0.0|0.0
    port|Input|0|5|0|5|0|52|4
    node|SPEED_LIMITER0|RUNNING|0|0|0.0|0.0|0.0|0.0
    port|Input|0|0|0|0|0|130|10
    port|Output|0|5|0|5|0|52|4

For `returnType==DESCRIPTION_XML` returns complex data structure describing one or more selected executions in XML format. A schema describing XML format of the XML response is accessible on CloverETL Server URL: http://[host]:[port]/clover/schemas/executions.xsd

### Operation suspend

Suspends server or sandbox (if specified). Suspension means, that no graphs may me executed on suspended server/sandbox.

parameters

Table 25.6. Parameters of suspend

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | no | - | Text ID of a sandbox to suspend. If not specified, it suspends the whole server.
atonce | no |  | If this param is set to true, running graphs from suspended server (or just from sandbox) are aborted. Otherwise it can run until it is finished in common way.

returns

Result message

### Operation resume

parameters

Table 25.7. Parameters of resume

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | no | - | Text Id of a sandbox to resume. If not specified, the server will be resumed.
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should the possible error message be.

returns

Result message

### Operation sandbox_create

This operation creates a specified sandbox. If it is sandbox of "partitioned" or "local" type, it also creates locations by "sandbox_add_location" operation.

parameters

Table 25.8. Parameters of sandbox create

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | yes | - | Text Id of sandbox to be created.
path | no | - | Path to the sandbox root if server is running in standalone mode.
type | no | shared | Sandbox type: shared | partitioned | local. For standalone server may be left empty, since the default "shared" is used.
createDirs | no | true | Switch whether to create directory structure of the sandbox (only for standalone server or "shared" sandboxes in cluster environment).
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should possible error message be.

returns

Result message

### Operation sandbox_add_location

This operation adds a location to the specified sandbox. Can be only used with partitioned or local sandboxes.

parameters

Table 25.9. Parameters of sandbox add location

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | yes | - | Sandbox which we want to add location to.
nodeId | yes | - | Location attribute - node which has direct access to the location.
path | yes | - | Location attribute - path to the location root on the specified node.
location | no | - | Location attribute - location storage ID. If it's not specified, new one will be generated.
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should possible error message be.

returns

Result message

### Operation sandbox_remove_location

This operation removes a location from the specified sandbox. Only sandboxes of type partitioned or local can have locations asociated.

parameters

Table 25.10. Parameters of sandbox add location

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | yes | - | Removes specified location from its sandbox.
location | yes | - | Location storage ID. If the specified location isn't attached to the specified sandbox, sandbox won't be changed.
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should possible error message be.

returns

Result message

### Operation download_sandbox_zip

This operation downloads content of a specified sandbox as a ZIP archive.

parameters

Table 25.11. Parameters

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | yes | - | Code of the sandbox to be downloaded.

returns

Content of the specified sandbox as a ZIP archive

example

     wget --http-user=username --http-password=password http://localhost:8080/clover/simpleHttpApi/download_sandbox_zip?sandbox=my-sandbox

### Operation upload_sandbox_zip

This operation uploads content of a ZIP archive into a specified sandbox.

parameters

Table 25.12. Parameters

parameter name | mandatory | default | description
--- | --- | --- | ---
sandbox | yes | - | Code of the sandbox the ZIP file will be expanded to.
zipFile | yes | - | The ZIP archive file.
overwriteExisting | no | false | If `true`, the files already present in the sandbox will be overwritten.
deleteMissing | no | false | If `true`, the files not present in the ZIP file will be deleted from the sandbox.
fileNameEncoding | no | UTF-8 | The encoding that was used to store file names in the ZIP archive.

returns

Result message

example of request (with using curl CLI tool (http://curl.haxx.se/))

    curl -u username:password -F "overwriteExisting=true"
        -F "zipFile=@/tmp/my-sandbox.zip"
        http://localhost:8080/clover/simpleHttpApi/upload_sandbox_zip

### Operation cluster_status

This operation displays cluster's nodes list.

parameters

no

returns

Cluster's nodes list.

### Operation export_server_config

This operation exports a current server configuration in XML format.

parameters

Table 25.13. Parameters of server configuration export

parameter name | mandatory | default | description
--- | --- | --- | ---
include | no | all | Selection which items will be included in the exported XML file; the parameter may be specified multiple times. Possible values are: `all` - include items of all types `users` - include list of users `userGroups` - include list of user groups `sandboxes` - include list of sandboxes `jobConfigs` - include list of job configuration parameters `schedules` - include list of schedules `eventListeners` - include list of event listeners `launchServices` - include list of launch services `tempSpaces` - include list of temporary spaces

returns

Current server configuration as an XML file.

example

    wget http://localhost:8080/clover/simpleHttpApi/export_server_config

### Operation import_server_config

This operation imports server configuration.

parameters

Table 25.14. Parameters of server configuration import

parameter name | mandatory | default | description
--- | --- | --- | ---
xmlFile | yes | - | An XML file with server's configuration.
dryRun | no | true | If `true`, a dry run is performed with no actual changes written.
verbose | no | MESSAGE | MESSAGE | FULL - how verbose should the response be: MESSAGE for a simple message, FULL for a full XML report.
newOnly | no | false | If `true` only new items will imported to the server; the items already present on the server will be left untouched.
include | no | all | Selection which items will be imported from the XML; the parameter may be specified multiple times. Possible values are: `all` - import items of all types `users` - import users `userGroups` - import user groups `sandboxes` - import sandboxes `jobConfigs` - import job configuration parameters `schedules` - import schedules `eventListeners` - import listeners `launchServices` - import launch services `tempSpaces` - import temporary spaces

returns

Result message or XML report

example of request (with using curl CLI tool (http://curl.haxx.se/))

    curl -u username:password -F "dryRun=true" -F "verbose=FULL"
        -F "xmlFile=@/tmp/clover_configuration_2013-07-10_14-03-23+0200.xml"
        http://localhost:8080/clover/simpleHttpApi/import_server_config

Prev | Up | Next
API | Home | Table of Contents | JMX mBean