Data Processing API¶
Cluster templates¶
A cluster template configures a Hadoop cluster. A cluster template lists node groups with the number of instances in each group. You can also define cluster-scoped configurations in a cluster template.
Shows details for a cluster template.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_template_id | path | string | The unique identifier of the cluster template. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description (Optional) | body | string | Description of the cluster template |
| use_autoconfig | body | boolean | If set to true, the cluster is auto
configured. |
| cluster_configs | body | object | A set of key and value pairs that contain the cluster configuration. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| default_image_id | body | string | The default ID of the image. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| plugin_name | body | string | The name of the plugin. |
| is_default | body | boolean | If set to true, the cluster is the default
cluster. |
| is_protected | body | boolean | If set to true, the object is protected. |
| shares | body | string | The sharing of resources in the cluster. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
| tenant_id | body | string | The UUID of the tenant. |
| node_groups | body | object | The detail properties of the node in key-value pairs. |
| is_public | body | boolean | If set to true, the object is public. |
| hadoop_version | body | string | The version of the Hadoop used in the cluster. |
| id | body | string | The UUID of the cluster template. |
| name | body | string | The name of the cluster template. |
Response Example¶
{
"cluster_templates": [
{
"is_public": false,
"anti_affinity": [],
"name": "cluster-template",
"created_at": "2015-09-14T10:38:44",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {},
"shares": null,
"id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": null,
"is_default": false,
"updated_at": null,
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "1751c04e-8f39-467e-a421-480961172d4b",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "3ee85068-c455-4391-9db2-b54a20b99df3",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"domain_name": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"description": null,
"is_protected": false
},
{
"is_public": true,
"anti_affinity": [],
"name": "asd",
"created_at": "2015-08-18T08:39:39",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {
"general": {}
},
"shares": null,
"id": "5a9c787c-2078-4f7d-9a66-27759be9051b",
"default_image_id": null,
"is_default": false,
"updated_at": "2015-09-14T08:41:15",
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": true,
"availability_zone": "",
"count": 1,
"flavor_id": "2",
"id": "a65864dd-3f99-4d29-a011-f7711cc23fa0",
"security_groups": [],
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-08-18T08:39:39",
"node_group_template_id": "42ce49de-1b8f-41d5-8f4a-244ec0826d92",
"updated_at": null,
"volumes_per_node": 1,
"is_proxy_gateway": false,
"name": "asd",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"jobtracker"
],
"volumes_size": 10,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": null,
"domain_name": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"description": "",
"is_protected": false
}
],
"markers": {
"prev": null,
"next": "2c76e0d3-56cd-4d28-bb4f-4808e538c7b9"
}
}
Updates a cluster template.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_template_id | body | string | The UUID of the cluster template. |
Request Example¶
{
"description": "Updated template",
"plugin_name": "vanilla",
"hadoop_version": "2.7.1",
"name": "vanilla-updated",
"cluster_configs": {
"HDFS": {
"dfs.replication": 2
}
}
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description (Optional) | body | string | Description of the cluster template |
| use_autoconfig | body | boolean | If set to true, the cluster is auto
configured. |
| cluster_configs | body | object | A set of key and value pairs that contain the cluster configuration. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| default_image_id | body | string | The default ID of the image. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| plugin_name | body | string | The name of the plugin. |
| is_default | body | boolean | If set to true, the cluster is the default
cluster. |
| is_protected | body | boolean | If set to true, the object is protected. |
| shares | body | string | The sharing of resources in the cluster. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
| tenant_id | body | string | The UUID of the tenant. |
| node_groups | body | object | The detail properties of the node in key-value pairs. |
| is_public | body | boolean | If set to true, the object is public. |
| hadoop_version | body | string | The version of the Hadoop used in the cluster. |
| id | body | string | The UUID of the cluster template. |
| name | body | string | The name of the cluster template. |
Deletes a cluster template.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_template_id | body | string | The UUID of the cluster template. |
Lists available cluster templates.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| limit (Optional) | path | integer | Maximum number of objects in response data. |
| marker (Optional) | path | string | ID of the last element on the list which won’t be in response. |
| sort_by (Optional) | path | string | The field for sorting cluster templates.
this parameter accepts the following values:
name, plugin_name, hadoop_version,
created_at, updated_at, id. Also
this values can started with - prefix for
descending sort. For example: -name. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| markers (Optional) | body | object | The markers of previous and following pages of data.
This field exists only if limit is passed to
request. |
| prev (Optional) | body | string | The marker of previous page. May be null if
previous page is first or if current page is first. |
| next (Optional) | body | string | The marker of next page of list data. |
| description (Optional) | body | string | Description of the cluster template |
| use_autoconfig | body | boolean | If set to true, the cluster is auto
configured. |
| cluster_configs | body | object | A set of key and value pairs that contain the cluster configuration. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| default_image_id | body | string | The default ID of the image. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| plugin_name | body | string | The name of the plugin. |
| is_default | body | boolean | If set to true, the cluster is the default
cluster. |
| is_protected | body | boolean | If set to true, the object is protected. |
| shares | body | string | The sharing of resources in the cluster. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
| tenant_id | body | string | The UUID of the tenant. |
| node_groups | body | object | The detail properties of the node in key-value pairs. |
| is_public | body | boolean | If set to true, the object is public. |
| hadoop_version | body | string | The version of the Hadoop used in the cluster. |
| id | body | string | The UUID of the cluster template. |
| name | body | string | The name of the cluster template. |
{
"cluster_templates": [
{
"is_public": false,
"anti_affinity": [],
"name": "cluster-template",
"created_at": "2015-09-14T10:38:44",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {},
"shares": null,
"id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": null,
"is_default": false,
"updated_at": null,
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "1751c04e-8f39-467e-a421-480961172d4b",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "3ee85068-c455-4391-9db2-b54a20b99df3",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"domain_name": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"description": null,
"is_protected": false
},
{
"is_public": true,
"anti_affinity": [],
"name": "asd",
"created_at": "2015-08-18T08:39:39",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {
"general": {}
},
"shares": null,
"id": "5a9c787c-2078-4f7d-9a66-27759be9051b",
"default_image_id": null,
"is_default": false,
"updated_at": "2015-09-14T08:41:15",
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": true,
"availability_zone": "",
"count": 1,
"flavor_id": "2",
"id": "a65864dd-3f99-4d29-a011-f7711cc23fa0",
"security_groups": [],
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-08-18T08:39:39",
"node_group_template_id": "42ce49de-1b8f-41d5-8f4a-244ec0826d92",
"updated_at": null,
"volumes_per_node": 1,
"is_proxy_gateway": false,
"name": "asd",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"jobtracker"
],
"volumes_size": 10,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": null,
"domain_name": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"description": "",
"is_protected": false
}
],
"markers": {
"prev": null,
"next": "2c76e0d3-56cd-4d28-bb4f-4808e538c7b9"
}
}
Creates a cluster template.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | body | string | The UUID of the project. |
Request Example¶
{
"plugin_name": "vanilla",
"hadoop_version": "2.7.1",
"node_groups": [
{
"name": "worker",
"count": 3,
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251"
},
{
"name": "master",
"count": 1,
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae"
}
],
"name": "cluster-template"
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description (Optional) | body | string | Description of the cluster template |
| use_autoconfig | body | boolean | If set to true, the cluster is auto
configured. |
| cluster_configs | body | object | A set of key and value pairs that contain the cluster configuration. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| default_image_id | body | string | The default ID of the image. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| plugin_name | body | string | The name of the plugin. |
| is_default | body | boolean | If set to true, the cluster is the default
cluster. |
| is_protected | body | boolean | If set to true, the object is protected. |
| shares | body | string | The sharing of resources in the cluster. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
| tenant_id | body | string | The UUID of the tenant. |
| node_groups | body | object | The detail properties of the node in key-value pairs. |
| is_public | body | boolean | If set to true, the object is public. |
| hadoop_version | body | string | The version of the Hadoop used in the cluster. |
| id | body | string | The UUID of the cluster template. |
| name | body | string | The name of the cluster template. |
Clusters¶
A cluster is a group of nodes with the same configuration.
Lists available clusters.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| limit (Optional) | path | integer | Maximum number of objects in response data. |
| marker (Optional) | path | string | ID of the last element on the list which won’t be in response. |
| sort_by (Optional) | path | string | The field for sorting clusters.
this parameter accepts the following values:
name, plugin_name, hadoop_version,
status, id. Also this values can
started with - prefix for descending sort.
For example: -name. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| markers (Optional) | body | object | The markers of previous and following pages of data.
This field exists only if limit is passed to
request. |
| prev (Optional) | body | string | The marker of previous page. May be null if
previous page is first or if current page is first. |
| next (Optional) | body | string | The marker of next page of list data. |
| count | body | integer | The number of nodes in the cluster. |
| info | body | object | A set of key and value pairs that contain cluster information. |
| cluster_template_id | body | string | The UUID of the cluster template. |
| is_transient | body | boolean | If set to true, the cluster is transient. |
| provision_progress | body | array | A list of the cluster progresses. |
| status | body | string | The status of the cluster. |
| neutron_management_network | body | string | The UUID of the neutron management network. |
| clusters | body | array | The list of clusters. |
| management_public_key | body | string | The SSH key for the management network. |
| status_description | body | string | The description of the cluster status. |
| trust_id | body | integer | The id of the trust. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
{
"clusters": [
{
"is_public": false,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"domain_name": null,
"status_description": "",
"plugin_name": "vanilla",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"info": {
"YARN": {
"Web UI": "http://172.18.168.115:8088",
"ResourceManager": "http://172.18.168.115:8032"
},
"HDFS": {
"Web UI": "http://172.18.168.115:50070",
"NameNode": "hdfs://vanilla-cluster-master-0:9000"
},
"JobFlow": {
"Oozie": "http://172.18.168.115:11000"
},
"MapReduce JobHistory Server": {
"Web UI": "http://172.18.168.115:19888"
}
},
"user_keypair_id": "apavlov",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"security_groups": null,
"use_autoconfig": true,
"instances": [
{
"created_at": "2015-09-14T10:57:36",
"id": "4867d92e-cc7b-4cde-9a1a-149e91caa491",
"management_ip": "172.18.168.115",
"updated_at": "2015-09-14T10:57:39",
"instance_id": "b9f16a07-88fc-423e-83a3-489598fe6737",
"internal_ip": "10.50.0.60",
"instance_name": "vanilla-cluster-master-0"
}
],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": "2015-09-14T10:57:36",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"security_groups": null,
"use_autoconfig": true,
"instances": [
{
"created_at": "2015-09-14T10:57:37",
"id": "f3633b30-c1e4-4144-930b-ab5b780b87be",
"management_ip": "172.18.168.118",
"updated_at": "2015-09-14T10:57:39",
"instance_id": "0cf1ee81-aa72-48da-be2c-65bc2fa51f8f",
"internal_ip": "10.50.0.63",
"instance_name": "vanilla-cluster-worker-0"
},
{
"created_at": "2015-09-14T10:57:37",
"id": "0d66fd93-f277-4a94-b46a-f5866aa0c38f",
"management_ip": "172.18.168.117",
"updated_at": "2015-09-14T10:57:40",
"instance_id": "4a937391-b594-4ad0-9a53-00a99a691383",
"internal_ip": "10.50.0.62",
"instance_name": "vanilla-cluster-worker-1"
},
{
"created_at": "2015-09-14T10:57:37",
"id": "0982cefd-5c58-436e-8f1e-c1d0830f18a7",
"management_ip": "172.18.168.116",
"updated_at": "2015-09-14T10:57:40",
"instance_id": "839b1d56-6d0d-4aa4-9d05-30e029c276f8",
"internal_ip": "10.50.0.61",
"instance_name": "vanilla-cluster-worker-2"
}
],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": "2015-09-14T10:57:37",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"provision_progress": [
{
"created_at": "2015-09-14T10:57:18",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "0a6d95f9-30f4-4434-823a-a38a7999a5af",
"step_type": "Engine: create cluster",
"step_name": "Create Heat stack",
"updated_at": "2015-09-14T10:57:38",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:16",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "29f2b587-c34c-4871-9ed9-9235b411cd9a",
"step_type": "Engine: create cluster",
"step_name": "Configure instances",
"updated_at": "2015-09-14T10:58:22",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:27",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "36f1efde-90f9-41c1-b409-aa1cf9623e3e",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): Oozie",
"updated_at": "2015-09-14T11:01:15",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:22",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "602bcc27-3a2d-42c8-8aca-ebc475319c72",
"step_type": "Plugin: configure cluster",
"step_name": "Configure instances",
"updated_at": "2015-09-14T10:59:21",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:21",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "7e291df1-2d32-410d-ae89-33ab6f83cf17",
"step_type": "Plugin: configure cluster",
"step_name": "Configure topology data",
"updated_at": "2015-09-14T10:59:37",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:01",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "8ab7933c-ad61-4a4f-88db-23ce78ee10f6",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): DataNodes, NodeManagers",
"updated_at": "2015-09-14T11:00:11",
"successful": true,
"total": 3,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:11",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "9c8dc016-8c5b-4e80-9857-80c41f6bd971",
"step_type": "Plugin: start cluster",
"step_name": "Await DataNodes start up",
"updated_at": "2015-09-14T11:00:21",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:21",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "c6327532-222b-416c-858f-73dbb32b8e97",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): HistoryServer",
"updated_at": "2015-09-14T11:00:27",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:57:41",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "d3eca726-8b44-473a-ac29-fba45a893725",
"step_type": "Engine: create cluster",
"step_name": "Wait for instance accessibility",
"updated_at": "2015-09-14T10:58:14",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:14",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "d7a875ff-64bf-41aa-882d-b5061c8ee152",
"step_type": "Engine: create cluster",
"step_name": "Mount volumes to instances",
"updated_at": "2015-09-14T10:58:15",
"successful": true,
"total": 0,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:55",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "ded7d227-10b8-4cb0-ab6c-25da1462bb7a",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): ResourceManager",
"updated_at": "2015-09-14T11:00:00",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:38",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "e1701ff5-930a-4212-945a-43515dfe24d1",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): NameNode",
"updated_at": "2015-09-14T10:59:54",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:57:38",
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"id": "eaf0ab1b-bf8f-48f0-8f2c-fa4f82f539b9",
"step_type": "Engine: create cluster",
"step_name": "Assign IPs",
"updated_at": "2015-09-14T10:57:41",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
}
],
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"trust_id": null,
"description": null,
"created_at": "2015-09-14T10:57:11",
"is_protected": false,
"updated_at": "2015-09-14T11:01:15",
"is_transient": false,
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"anti_affinity": [],
"name": "vanilla-cluster",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"status": "Active"
}
]
}
Creates a cluster.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
Request Example¶
{
"plugin_name": "vanilla",
"hadoop_version": "2.7.1",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"user_keypair_id": "test",
"name": "vanilla-cluster",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd"
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| count | body | integer | The number of nodes in the cluster. |
| info | body | object | A set of key and value pairs that contain cluster information. |
| cluster_template_id | body | string | The UUID of the cluster template. |
| is_transient | body | boolean | If set to true, the cluster is transient. |
| provision_progress | body | array | A list of the cluster progresses. |
| status | body | string | The status of the cluster. |
| neutron_management_network | body | string | The UUID of the neutron management network. |
| management_public_key | body | string | The SSH key for the management network. |
| status_description | body | string | The description of the cluster status. |
| trust_id | body | integer | The id of the trust. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
Creates multiple clusters.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
Request Example¶
{
"plugin_name": "vanilla",
"hadoop_version": "2.6.0",
"cluster_template_id": "9951f86d-57ba-43d6-9cb0-14ed2ec7a6cf",
"default_image_id": "bc3c3d3c-2684-4bf8-a9fa-388fb71288a9",
"user_keypair_id": "test",
"name": "def-cluster",
"count": 2,
"cluster_configs": {},
"neutron_management_network": "7e31648b-4b2e-4f32-9b0a-113581c27076"
}
Shows details for a cluster, by ID.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_id | path | string | The ID of the cluster |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| count | body | integer | The number of nodes in the cluster. |
| info | body | object | A set of key and value pairs that contain cluster information. |
| cluster_template_id | body | string | The UUID of the cluster template. |
| is_transient | body | boolean | If set to true, the cluster is transient. |
| provision_progress | body | array | A list of the cluster progresses. |
| status | body | string | The status of the cluster. |
| neutron_management_network | body | string | The UUID of the neutron management network. |
| management_public_key | body | string | The SSH key for the management network. |
| status_description | body | string | The description of the cluster status. |
| trust_id | body | integer | The id of the trust. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
Response Example¶
{
"cluster": {
"is_public": false,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"domain_name": null,
"status_description": "",
"plugin_name": "vanilla",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"info": {},
"user_keypair_id": "test",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"provision_progress": [],
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"trust_id": null,
"description": null,
"created_at": "2015-09-14T10:57:11",
"is_protected": false,
"updated_at": "2015-09-14T10:57:12",
"is_transient": false,
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"anti_affinity": [],
"name": "vanilla-cluster",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"status": "Validating"
}
}
Deletes a cluster.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_id | path | string | The ID of the cluster |
Scales a cluster.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_id | body | string | The UUID of the cluster. |
Request Example¶
{
"add_node_groups": [
{
"count": 1,
"name": "b-worker",
"node_group_template_id": "bc270ffe-a086-4eeb-9baa-2f5a73504622"
}
],
"resize_node_groups": [
{
"count": 4,
"name": "worker"
}
]
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| count | body | integer | The number of nodes in the cluster. |
| info | body | object | A set of key and value pairs that contain cluster information. |
| cluster_template_id | body | string | The UUID of the cluster template. |
| is_transient | body | boolean | If set to true, the cluster is transient. |
| provision_progress | body | array | A list of the cluster progresses. |
| status | body | string | The status of the cluster. |
| neutron_management_network | body | string | The UUID of the neutron management network. |
| management_public_key | body | string | The SSH key for the management network. |
| status_description | body | string | The description of the cluster status. |
| trust_id | body | integer | The id of the trust. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
Updates a cluster.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_id | path | string | The ID of the cluster |
Request Example¶
{
"name": "public-vanilla-cluster",
"is_public": true
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| count | body | integer | The number of nodes in the cluster. |
| info | body | object | A set of key and value pairs that contain cluster information. |
| cluster_template_id | body | string | The UUID of the cluster template. |
| is_transient | body | boolean | If set to true, the cluster is transient. |
| provision_progress | body | array | A list of the cluster progresses. |
| status | body | string | The status of the cluster. |
| neutron_management_network | body | string | The UUID of the neutron management network. |
| management_public_key | body | string | The SSH key for the management network. |
| status_description | body | string | The description of the cluster status. |
| trust_id | body | integer | The id of the trust. |
| domain_name (Optional) | body | string | Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
Shows provisioning progress for a cluster.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_id | path | string | The ID of the cluster |
Response Example¶
{
"status": "Error",
"neutron_management_network": "7e31648b-4b2e-4f32-9b0a-113581c27076",
"is_transient": false,
"description": "",
"user_keypair_id": "vgridnev",
"updated_at": "2015-03-31 14:10:59",
"plugin_name": "spark",
"provision_progress": [
{
"successful": false,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:20",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:35",
"events": [
{
"instance_name": "sample-worker-spark-004",
"successful": false,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": "Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"instance_id": "b5ba5ba8-e9c1-47f7-9355-3ce0ec0e449d",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "34afcfc7-bdb0-43cb-b142-283d560dc6ad"
},
{
"instance_name": "sample-worker-spark-001",
"successful": true,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": null,
"instance_id": "c532ab71-38da-475a-95f8-f8eb93b8f1c2",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "4ba50414-5216-4161-bc7a-12716122b99d"
}
],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Wait for instances to become active",
"total": 5,
"id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6"
},
{
"successful": true,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:12",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:19",
"events": [],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Run instances",
"total": 5,
"id": "407ba50a-c799-46af-9dfb-6aa5f6ade426"
}
],
"anti_affinity": [],
"node_groups": [],
"management_public_key": "Sahara",
"status_description": "Creating cluster failed for the following reason(s): Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"hadoop_version": "1.0.0",
"id": "c26ec982-ba6b-4d75-1f8c-a50240164af0",
"trust_id": null,
"info": {},
"cluster_template_id": "5a9a09a3-9349-43bd-9058-16c401fad2d5",
"name": "sample",
"cluster_configs": {},
"created_at": "2015-03-31 14:10:07",
"default_image_id": "e6a6c5da-67be-4017-a7d2-81f466efe67e",
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda"
}
Data sources¶
A data source object defines the location of input or output for MapReduce jobs and might reference different types of storage.
The Data Processing service does not validate data source locations.
Shows details for a data source.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| data_source_id | path | string | The UUID of the data source. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description | body | string | The description of the data source object. |
| url | body | string | The url of the data source object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| type | body | string | The type of the data source object. |
| id | body | string | The UUID of the data source. |
| name | body | string | The name of the data source. |
Response Example¶
{
"data_source": {
"is_public": false,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:18:10.691493",
"id": "953831f2-0852-49d8-ac71-af5805e25256",
"updated_at": null,
"name": "swift_input",
"description": "This is input",
"url": "swift://container/text",
"type": "swift"
}
}
Deletes a data source.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| data_source_id | path | string | The UUID of the data source. |
Updates a data source.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| data_source_id | path | string | The UUID of the data source. |
Request Example¶
{
"description": "This is public input",
"is_protected": true
}
Lists all data sources.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| limit (Optional) | path | integer | Maximum number of objects in response data. |
| marker (Optional) | path | string | ID of the last element on the list which won’t be in response. |
| sort_by (Optional) | path | string | The field for sorting data sources.
this parameter accepts the following values:
id, name, type, created_at,
updated_at. Also this values can started
with - prefix for descending sort.
For example: -name. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| markers (Optional) | body | object | The markers of previous and following pages of data.
This field exists only if limit is passed to
request. |
| prev (Optional) | body | string | The marker of previous page. May be null if
previous page is first or if current page is first. |
| next (Optional) | body | string | The marker of next page of list data. |
| description | body | string | The description of the data source object. |
| url | body | string | The url of the data source object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| type | body | string | The type of the data source object. |
| id | body | string | The UUID of the data source. |
| name | body | string | The name of the data source. |
{
"data_sources": [
{
"is_public": false,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:18:10",
"id": "953831f2-0852-49d8-ac71-af5805e25256",
"name": "swift_input",
"updated_at": null,
"description": "This is input",
"url": "swift://container/text",
"type": "swift"
},
{
"is_public": false,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:09:36",
"id": "d7fffe9c-3b42-46a9-8be8-e98f586fa7a9",
"name": "hdfs_input",
"updated_at": null,
"description": "This is hdfs input",
"url": "hdfs://test-master-node:8020/user/hadoop/input",
"type": "hdfs"
}
]
}
Creates a data source.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
Request Example¶
{
"description": "This is hdfs input",
"url": "hdfs://test-master-node:8020/user/hadoop/input",
"type": "hdfs",
"name": "hdfs_input"
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description | body | string | The description of the data source object. |
| url | body | string | The url of the data source object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| type | body | string | The type of the data source object. |
| id | body | string | The UUID of the data source. |
| name | body | string | The name of the data source. |
Event log¶
The event log feature provides information about cluster provisioning. In the event of errors, the event log shows the reason for the failure.
Shows provisioning progress of cluster.
Normal response codes: 200 Error response codes:
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| cluster_id | body | string | The UUID of the cluster. |
Response Example¶
{
"status": "Error",
"neutron_management_network": "7e31648b-4b2e-4f32-9b0a-113581c27076",
"is_transient": false,
"description": "",
"user_keypair_id": "vgridnev",
"updated_at": "2015-03-31 14:10:59",
"plugin_name": "spark",
"provision_progress": [
{
"successful": false,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:20",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:35",
"events": [
{
"instance_name": "sample-worker-spark-004",
"successful": false,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": "Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"instance_id": "b5ba5ba8-e9c1-47f7-9355-3ce0ec0e449d",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "34afcfc7-bdb0-43cb-b142-283d560dc6ad"
},
{
"instance_name": "sample-worker-spark-001",
"successful": true,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": null,
"instance_id": "c532ab71-38da-475a-95f8-f8eb93b8f1c2",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "4ba50414-5216-4161-bc7a-12716122b99d"
}
],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Wait for instances to become active",
"total": 5,
"id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6"
},
{
"successful": true,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:12",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:19",
"events": [],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Run instances",
"total": 5,
"id": "407ba50a-c799-46af-9dfb-6aa5f6ade426"
}
],
"anti_affinity": [],
"node_groups": [],
"management_public_key": "Sahara",
"status_description": "Creating cluster failed for the following reason(s): Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"hadoop_version": "1.0.0",
"id": "c26ec982-ba6b-4d75-1f8c-a50240164af0",
"trust_id": null,
"info": {},
"cluster_template_id": "5a9a09a3-9349-43bd-9058-16c401fad2d5",
"name": "sample",
"cluster_configs": {},
"created_at": "2015-03-31 14:10:07",
"default_image_id": "e6a6c5da-67be-4017-a7d2-81f466efe67e",
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda"
}
Image registry¶
Use the image registry tool to manage images, add tags to and remove tags from images, and define the user name for an instance operating system. Each plugin lists required tags for an image. To run remote operations, the Data Processing service requires a user name with which to log in to the operating system for an instance.
Adds tags to an image.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| tags | body | array | List of tags to add. |
| image_id | body | string | The UUID of the image. |
Request Example¶
{
"tags": [
"vanilla",
"2.7.1",
"some_other_tag"
]
}
Shows details for an image.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| image_id | path | string | The UUID of the image. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| status | body | string | The status of the cluster. |
| username | body | string | The name of the user for the image. |
| updated | body | string | The date and time when the image was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
| description | body | string | The description of the image. |
| created | body | string | The date and time when the image was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
| image | body | object | A set of key and value pairs that contain image properties. |
| tags | body | array | List of tags to add. |
| minDisk | body | integer | The minimum disk space, in GB. |
| name | body | string | The name of the operating system image. |
| progress | body | integer | A progress indicator, as a percentage value, for the amount of image content that has been processed. |
| minRam | body | integer | The minimum amount of random access memory (RAM) for the image, in GB. |
| id | body | string | The UUID of the image. |
| metadata | body | object | A set of key and value pairs that contain image metadata. |
Response Example¶
{
"image": {
"updated": "2015-02-03T10:29:32Z",
"metadata": {
"_sahara_username": "ubuntu",
"_sahara_tag_vanilla": true,
"_sahara_tag_2.6.0": true
},
"id": "bb8d12b5-f9bb-49f0-aecb-739b8a9bec89",
"minDisk": 0,
"status": "ACTIVE",
"tags": [
"vanilla",
"2.6.0"
],
"minRam": 0,
"progress": 100,
"username": "ubuntu",
"created": "2015-02-03T10:28:39Z",
"name": "sahara-vanilla-2.6.0-ubuntu-14.04",
"description": null,
"OS-EXT-IMG-SIZE:size": 1101856768
}
}
Registers an image in the registry.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| username | body | string | The name of the user for the image. |
| description | body | string | The description of the image. |
| image_id | path | string | The UUID of the image. |
Request Example¶
{
"username": "ubuntu",
"description": "Ubuntu image for Hadoop 2.7.1"
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| status | body | string | The status of the cluster. |
| username | body | string | The name of the user for the image. |
| updated | body | string | The date and time when the image was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
| description | body | string | The description of the image. |
| created | body | string | The date and time when the image was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
| image | body | object | A set of key and value pairs that contain image properties. |
| tags | body | array | List of tags to add. |
| minDisk | body | integer | The minimum disk space, in GB. |
| name | body | string | The name of the operating system image. |
| progress | body | integer | A progress indicator, as a percentage value, for the amount of image content that has been processed. |
| minRam | body | integer | The minimum amount of random access memory (RAM) for the image, in GB. |
| id | body | string | The UUID of the image. |
| metadata | body | object | A set of key and value pairs that contain image metadata. |
Removes an image from the registry.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| image_id | path | string | The UUID of the image. |
Removes tags from an image.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| tags | body | array | List of tags to add. |
| image_id | path | string | The UUID of the image. |
Request Example¶
{
"tags": [
"some_other_tag"
]
}
Lists all images registered in the registry.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| tags | body | array | List of tags to add. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| status | body | string | The status of the cluster. |
| username | body | string | The name of the user for the image. |
| updated | body | string | The date and time when the image was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
| description | body | string | The description of the image. |
| created | body | string | The date and time when the image was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
| image | body | object | A set of key and value pairs that contain image properties. |
| tags | body | array | List of tags to add. |
| minDisk | body | integer | The minimum disk space, in GB. |
| name | body | string | The name of the operating system image. |
| images | body | array | The list of images and their properties. |
| progress | body | integer | A progress indicator, as a percentage value, for the amount of image content that has been processed. |
| minRam | body | integer | The minimum amount of random access memory (RAM) for the image, in GB. |
| id | body | string | The UUID of the image. |
| metadata | body | object | A set of key and value pairs that contain image metadata. |
Response Example¶
{
"images": [
{
"name": "ubuntu-vanilla-2.7.1",
"id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"created": "2015-08-06T08:17:14Z",
"metadata": {
"_sahara_tag_2.7.1": true,
"_sahara_username": "ubuntu",
"_sahara_tag_vanilla": true
},
"username": "ubuntu",
"progress": 100,
"OS-EXT-IMG-SIZE:size": 998716928,
"status": "ACTIVE",
"minDisk": 0,
"tags": [
"vanilla",
"2.7.1"
],
"updated": "2015-09-04T09:35:09Z",
"minRam": 0,
"description": null
},
{
"name": "cdh-latest",
"id": "ff74035b-9da7-4edf-981d-57f270ed337d",
"created": "2015-09-04T11:56:44Z",
"metadata": {
"_sahara_username": "ubuntu",
"_sahara_tag_5.4.0": true,
"_sahara_tag_cdh": true
},
"username": "ubuntu",
"progress": 100,
"OS-EXT-IMG-SIZE:size": 3281453056,
"status": "ACTIVE",
"minDisk": 0,
"tags": [
"5.4.0",
"cdh"
],
"updated": "2015-09-04T12:46:42Z",
"minRam": 0,
"description": null
}
]
}
Job binaries¶
Job binary objects represent data processing applications and libraries that are stored in either the internal database or the Object Storage service.
Lists the available job binaries.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| limit (Optional) | path | integer | Maximum number of objects in response data. |
| marker (Optional) | path | string | ID of the last element on the list which won’t be in response. |
| sort_by (Optional) | path | string | The field for sorting job binaries.
this parameter accepts the following values:
id, name, created_at, updated_at.
Also this values can started with - prefix
for descending sort. For example: -name. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| markers (Optional) | body | object | The markers of previous and following pages of data.
This field exists only if limit is passed to
request. |
| prev (Optional) | body | string | The marker of previous page. May be null if
previous page is first or if current page is first. |
| next (Optional) | body | string | The marker of next page of list data. |
| description | body | string | The description of the job binary object. |
| url | body | string | The url of the data source object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| binaries | body | array | The list of job binary internal objects. |
| id | path | string | The UUID of the job binary. |
| name | body | string | The name of the object. |
{
"binaries": [
{
"is_public": false,
"description": "",
"url": "internal-db://d2498cbf-4589-484a-a814-81436c18beb3",
"tenant_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 12:36:59.375060",
"updated_at": null,
"id": "84248975-3c82-4206-a58d-6e7fb3a563fd",
"name": "example.pig",
"is_protected": false
},
{
"is_public": false,
"description": "",
"url": "internal-db://22f1d87a-23c8-483e-a0dd-cb4a16dde5f9",
"tenant_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 12:43:52.265899",
"updated_at": null,
"id": "508fc62d-1d58-4412-b603-bdab307bb926",
"name": "udf.jar",
"is_protected": false
},
{
"is_public": false,
"description": "",
"url": "swift://container/jar-example.jar",
"tenant_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 14:25:04.970513",
"updated_at": null,
"id": "a716a9cd-9add-4b12-b1b6-cdb71aaef350",
"name": "jar-example.jar",
"is_protected": false
}
]
}
Creates a job binary.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
Request Example¶
{
"url": "swift://container/jar-example.jar",
"name": "jar-example.jar",
"description": "This is a job binary",
"extra": {
"password": "swordfish",
"user": "admin"
}
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description | body | string | The description of the job binary object. |
| url | body | string | The url of the data source object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| id | path | string | The UUID of the job binary. |
| name | body | string | The name of the object. |
Shows details for a job binary.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_binary_id | path | string | The UUID of the job binary. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description | body | string | The description of the job binary object. |
| url | body | string | The url of the data source object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| id | path | string | The UUID of the job binary. |
| name | body | string | The name of the object. |
Response Example¶
{
"job_binary": {
"is_public": false,
"description": "an example jar file",
"url": "swift://container/jar-example.jar",
"tenant_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 14:25:04.970513",
"updated_at": null,
"id": "a716a9cd-9add-4b12-b1b6-cdb71aaef350",
"name": "jar-example.jar",
"is_protected": false
}
}
Deletes a job binary.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_binary_id | path | string | The UUID of the job binary. |
Updates a job binary.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_binary_id | path | string | The UUID of the job binary. |
Request Example¶
{
"url": "swift://container/new-jar-example.jar",
"name": "new-jar-example.jar",
"description": "This is a new job binary"
}
Shows data for a job binary.
The response body shows the job binary raw data and the response headers show the data length.
Example response:
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 161
Content-Type: text/html; charset=utf-8
Date: Sat, 28 Mar 2016 02:42:48 GMT
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage();
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_binary_id | path | string | The UUID of the job binary. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| Content-Length | header | string | The length of the data, in bytes. |
Response Example¶
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage();
Job executions¶
A job execution object represents a Hadoop job that runs on a cluster. A job execution polls the status of a running job and reports it to the user. Also a user can cancel a running job.
Refreshes the status of and shows information for a job execution.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_execution_id | path | string | The UUID of the job execution. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| info | body | object | A set of key and value pairs that contain cluster information. |
| output_id | body | string | The UUID of the output of job execution object. |
| start_time | body | string | The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| job_id | body | string | The UUID of the job object. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| args | body | array | The list of arguments. |
| data_source_urls | body | object | The data source URLs. |
| return_code | body | string | The code returned after job has executed. |
| oozie_job_id | body | string | The UUID of the oozie_job. |
| is_protected | body | boolean | If set to true, the job execution object is
protected. |
| cluster_id | body | string | The UUID of the cluster. |
| end_time | body | string | The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| params | body | object | The mappings of values to the parameters. |
| is_public | body | boolean | If set to true, the job execution object is
public. |
| input_id | body | string | The UUID of the input. |
| configs | body | object | The mappings of the job tasks. |
| job_execution | body | object | A set of key and value pairs that contain the job object. |
| id | body | string | The UUID of the job execution object. |
Response Example¶
{
"job_execution": {
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"is_protected": false,
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"created_at": "2015-09-15T09:49:24",
"end_time": "2015-09-15T12:50:46",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"is_public": false,
"updated_at": "2015-09-15T09:50:46",
"return_code": null,
"data_source_urls": {
"3e1bc8e6-8c69-4749-8e52-90d9341d15bc": "swift://ap-cont/input",
"52146b52-6540-4aac-a024-fee253cf52a9": "swift://ap-cont/output"
},
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"start_time": "2015-09-15T12:49:43",
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"oozie_job_id": "0000001-150915094349962-oozie-hado-W",
"info": {
"user": "hadoop",
"actions": [
{
"name": ":start:",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": "job-node",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"errorCode": null,
"id": "0000001-150915094349962-oozie-hado-W@:start:",
"consoleUrl": "-",
"errorMessage": null,
"toString": "Action name[:start:] status[OK]",
"stats": null,
"type": ":START:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "job-node",
"trackerUri": "http://172.18.168.119:8032",
"externalStatus": "FAILED/KILLED",
"status": "ERROR",
"externalId": "job_1442310173665_0002",
"transition": "fail",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "JA018",
"id": "0000001-150915094349962-oozie-hado-W@job-node",
"consoleUrl": "http://ap-cluster-all-0:8088/proxy/application_1442310173665_0002/",
"errorMessage": "Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]",
"toString": "Action name[job-node] status[ERROR]",
"stats": null,
"type": "pig",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "fail",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": null,
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "E0729",
"id": "0000001-150915094349962-oozie-hado-W@fail",
"consoleUrl": "-",
"errorMessage": "Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]",
"toString": "Action name[fail] status[OK]",
"stats": null,
"type": ":KILL:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"externalChildIDs": null,
"cred": "null"
}
],
"createdTime": "Tue, 15 Sep 2015 09:49:58 GMT",
"status": "KILLED",
"group": null,
"externalId": null,
"acl": null,
"run": 0,
"appName": "job-wf",
"parentId": null,
"conf": "<configuration>\r\n <property>\r\n <name>user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>oozie.use.system.libpath</name>\r\n <value>true</value>\r\n </property>\r\n <property>\r\n <name>mapreduce.job.user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>nameNode</name>\r\n <value>hdfs://ap-cluster-all-0:9000</value>\r\n </property>\r\n <property>\r\n <name>jobTracker</name>\r\n <value>http://172.18.168.119:8032</value>\r\n </property>\r\n <property>\r\n <name>oozie.wf.application.path</name>\r\n <value>hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml</value>\r\n </property>\r\n</configuration>",
"id": "0000001-150915094349962-oozie-hado-W",
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"appPath": "hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml",
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"toString": "Workflow id[0000001-150915094349962-oozie-hado-W] status[KILLED]",
"lastModTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"consoleUrl": "http://ap-cluster-all-0.novalocal:11000/oozie?job=0000001-150915094349962-oozie-hado-W"
}
}
}
Lists available job executions.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| limit (Optional) | path | integer | Maximum number of objects in response data. |
| marker (Optional) | path | string | ID of the last element on the list which won’t be in response. |
| sort_by (Optional) | path | string | The field for sorting job executions.
this parameter accepts the following values:
id, job_template, cluster,
status. Also this values can started
with - prefix for descending sort.
For example: -cluster. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| markers (Optional) | body | object | The markers of previous and following pages of data.
This field exists only if limit is passed to
request. |
| prev (Optional) | body | string | The marker of previous page. May be null if
previous page is first or if current page is first. |
| next (Optional) | body | string | The marker of next page of list data. |
| info | body | object | A set of key and value pairs that contain cluster information. |
| output_id | body | string | The UUID of the output of job execution object. |
| start_time | body | string | The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| job_id | body | string | The UUID of the job object. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| args | body | array | The list of arguments. |
| data_source_urls | body | object | The data source URLs. |
| return_code | body | string | The code returned after job has executed. |
| oozie_job_id | body | string | The UUID of the oozie_job. |
| is_protected | body | boolean | If set to true, the job execution object is
protected. |
| cluster_id | body | string | The UUID of the cluster. |
| end_time | body | string | The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| params | body | object | The mappings of values to the parameters. |
| is_public | body | boolean | If set to true, the job execution object is
public. |
| input_id | body | string | The UUID of the input. |
| configs | body | object | The mappings of the job tasks. |
| job_execution | body | object | A set of key and value pairs that contain the job object. |
| id | body | string | The UUID of the job execution object. |
| job_executions | body | array | The list of job execution objects. |
{
"job_executions": [
{
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"is_protected": false,
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"created_at": "2015-09-15T09:49:24",
"end_time": "2015-09-15T12:50:46",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"is_public": false,
"updated_at": "2015-09-15T09:50:46",
"return_code": null,
"data_source_urls": {
"3e1bc8e6-8c69-4749-8e52-90d9341d15bc": "swift://ap-cont/input",
"52146b52-6540-4aac-a024-fee253cf52a9": "swift://ap-cont/output"
},
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"start_time": "2015-09-15T12:49:43",
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"oozie_job_id": "0000001-150915094349962-oozie-hado-W",
"info": {
"user": "hadoop",
"actions": [
{
"name": ":start:",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": "job-node",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"errorCode": null,
"id": "0000001-150915094349962-oozie-hado-W@:start:",
"consoleUrl": "-",
"errorMessage": null,
"toString": "Action name[:start:] status[OK]",
"stats": null,
"type": ":START:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "job-node",
"trackerUri": "http://172.18.168.119:8032",
"externalStatus": "FAILED/KILLED",
"status": "ERROR",
"externalId": "job_1442310173665_0002",
"transition": "fail",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "JA018",
"id": "0000001-150915094349962-oozie-hado-W@job-node",
"consoleUrl": "http://ap-cluster-all-0:8088/proxy/application_1442310173665_0002/",
"errorMessage": "Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]",
"toString": "Action name[job-node] status[ERROR]",
"stats": null,
"type": "pig",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "fail",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": null,
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "E0729",
"id": "0000001-150915094349962-oozie-hado-W@fail",
"consoleUrl": "-",
"errorMessage": "Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]",
"toString": "Action name[fail] status[OK]",
"stats": null,
"type": ":KILL:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"externalChildIDs": null,
"cred": "null"
}
],
"createdTime": "Tue, 15 Sep 2015 09:49:58 GMT",
"status": "KILLED",
"group": null,
"externalId": null,
"acl": null,
"run": 0,
"appName": "job-wf",
"parentId": null,
"conf": "<configuration>\r\n <property>\r\n <name>user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>oozie.use.system.libpath</name>\r\n <value>true</value>\r\n </property>\r\n <property>\r\n <name>mapreduce.job.user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>nameNode</name>\r\n <value>hdfs://ap-cluster-all-0:9000</value>\r\n </property>\r\n <property>\r\n <name>jobTracker</name>\r\n <value>http://172.18.168.119:8032</value>\r\n </property>\r\n <property>\r\n <name>oozie.wf.application.path</name>\r\n <value>hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml</value>\r\n </property>\r\n</configuration>",
"id": "0000001-150915094349962-oozie-hado-W",
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"appPath": "hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml",
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"toString": "Workflow id[0000001-150915094349962-oozie-hado-W] status[KILLED]",
"lastModTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"consoleUrl": "http://ap-cluster-all-0.novalocal:11000/oozie?job=0000001-150915094349962-oozie-hado-W"
}
}
]
}
Shows details for a job execution, by ID.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_execution_id | path | string | The UUID of the job execution. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| info | body | object | A set of key and value pairs that contain cluster information. |
| output_id | body | string | The UUID of the output of job execution object. |
| start_time | body | string | The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| job_id | body | string | The UUID of the job object. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| args | body | array | The list of arguments. |
| data_source_urls | body | object | The data source URLs. |
| return_code | body | string | The code returned after job has executed. |
| oozie_job_id | body | string | The UUID of the oozie_job. |
| is_protected | body | boolean | If set to true, the job execution object is
protected. |
| cluster_id | body | string | The UUID of the cluster. |
| end_time | body | string | The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| params | body | object | The mappings of values to the parameters. |
| is_public | body | boolean | If set to true, the job execution object is
public. |
| input_id | body | string | The UUID of the input. |
| configs | body | object | The mappings of the job tasks. |
| job_execution | body | object | A set of key and value pairs that contain the job object. |
| id | body | string | The UUID of the job execution object. |
Response Example¶
{
"job_execution": {
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"is_protected": false,
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"created_at": "2015-09-15T09:49:24",
"end_time": "2015-09-15T12:50:46",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"is_public": false,
"updated_at": "2015-09-15T09:50:46",
"return_code": null,
"data_source_urls": {
"3e1bc8e6-8c69-4749-8e52-90d9341d15bc": "swift://ap-cont/input",
"52146b52-6540-4aac-a024-fee253cf52a9": "swift://ap-cont/output"
},
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"start_time": "2015-09-15T12:49:43",
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"oozie_job_id": "0000001-150915094349962-oozie-hado-W",
"info": {
"user": "hadoop",
"actions": [
{
"name": ":start:",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": "job-node",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"errorCode": null,
"id": "0000001-150915094349962-oozie-hado-W@:start:",
"consoleUrl": "-",
"errorMessage": null,
"toString": "Action name[:start:] status[OK]",
"stats": null,
"type": ":START:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "job-node",
"trackerUri": "http://172.18.168.119:8032",
"externalStatus": "FAILED/KILLED",
"status": "ERROR",
"externalId": "job_1442310173665_0002",
"transition": "fail",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "JA018",
"id": "0000001-150915094349962-oozie-hado-W@job-node",
"consoleUrl": "http://ap-cluster-all-0:8088/proxy/application_1442310173665_0002/",
"errorMessage": "Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]",
"toString": "Action name[job-node] status[ERROR]",
"stats": null,
"type": "pig",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "fail",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": null,
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "E0729",
"id": "0000001-150915094349962-oozie-hado-W@fail",
"consoleUrl": "-",
"errorMessage": "Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]",
"toString": "Action name[fail] status[OK]",
"stats": null,
"type": ":KILL:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"externalChildIDs": null,
"cred": "null"
}
],
"createdTime": "Tue, 15 Sep 2015 09:49:58 GMT",
"status": "KILLED",
"group": null,
"externalId": null,
"acl": null,
"run": 0,
"appName": "job-wf",
"parentId": null,
"conf": "<configuration>\r\n <property>\r\n <name>user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>oozie.use.system.libpath</name>\r\n <value>true</value>\r\n </property>\r\n <property>\r\n <name>mapreduce.job.user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>nameNode</name>\r\n <value>hdfs://ap-cluster-all-0:9000</value>\r\n </property>\r\n <property>\r\n <name>jobTracker</name>\r\n <value>http://172.18.168.119:8032</value>\r\n </property>\r\n <property>\r\n <name>oozie.wf.application.path</name>\r\n <value>hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml</value>\r\n </property>\r\n</configuration>",
"id": "0000001-150915094349962-oozie-hado-W",
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"appPath": "hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml",
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"toString": "Workflow id[0000001-150915094349962-oozie-hado-W] status[KILLED]",
"lastModTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"consoleUrl": "http://ap-cluster-all-0.novalocal:11000/oozie?job=0000001-150915094349962-oozie-hado-W"
}
}
}
Deletes a job execution.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_execution_id | path | string | The UUID of the job execution. |
Updates a job execution.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_execution_id | path | string | The UUID of the job execution. |
Request Example¶
{
"is_public": true
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| info | body | object | A set of key and value pairs that contain cluster information. |
| output_id | body | string | The UUID of the output of job execution object. |
| start_time | body | string | The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| job_id | body | string | The UUID of the job object. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| args | body | array | The list of arguments. |
| data_source_urls | body | object | The data source URLs. |
| return_code | body | string | The code returned after job has executed. |
| oozie_job_id | body | string | The UUID of the oozie_job. |
| is_protected | body | boolean | If set to true, the job execution object is
protected. |
| cluster_id | body | string | The UUID of the cluster. |
| end_time | body | string | The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| params | body | object | The mappings of values to the parameters. |
| is_public | body | boolean | If set to true, the job execution object is
public. |
| input_id | body | string | The UUID of the input. |
| configs | body | object | The mappings of the job tasks. |
| job_execution | body | object | A set of key and value pairs that contain the job object. |
| id | body | string | The UUID of the job execution object. |
Cancels a job execution.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_execution_id | path | string | The UUID of the job execution. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| info | body | object | A set of key and value pairs that contain cluster information. |
| output_id | body | string | The UUID of the output of job execution object. |
| start_time | body | string | The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| job_id | body | string | The UUID of the job object. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| args | body | array | The list of arguments. |
| data_source_urls | body | object | The data source URLs. |
| return_code | body | string | The code returned after job has executed. |
| oozie_job_id | body | string | The UUID of the oozie_job. |
| is_protected | body | boolean | If set to true, the job execution object is
protected. |
| cluster_id | body | string | The UUID of the cluster. |
| end_time | body | string | The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| params | body | object | The mappings of values to the parameters. |
| is_public | body | boolean | If set to true, the job execution object is
public. |
| input_id | body | string | The UUID of the input. |
| configs | body | object | The mappings of the job tasks. |
| job_execution | body | object | A set of key and value pairs that contain the job object. |
| id | body | string | The UUID of the job execution object. |
Response Example¶
{
"job_execution": {
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"is_protected": false,
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"created_at": "2015-09-15T09:49:24",
"end_time": "2015-09-15T12:50:46",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"is_public": false,
"updated_at": "2015-09-15T09:50:46",
"return_code": null,
"data_source_urls": {
"3e1bc8e6-8c69-4749-8e52-90d9341d15bc": "swift://ap-cont/input",
"52146b52-6540-4aac-a024-fee253cf52a9": "swift://ap-cont/output"
},
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"start_time": "2015-09-15T12:49:43",
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"oozie_job_id": "0000001-150915094349962-oozie-hado-W",
"info": {
"user": "hadoop",
"actions": [
{
"name": ":start:",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": "job-node",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"errorCode": null,
"id": "0000001-150915094349962-oozie-hado-W@:start:",
"consoleUrl": "-",
"errorMessage": null,
"toString": "Action name[:start:] status[OK]",
"stats": null,
"type": ":START:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "job-node",
"trackerUri": "http://172.18.168.119:8032",
"externalStatus": "FAILED/KILLED",
"status": "ERROR",
"externalId": "job_1442310173665_0002",
"transition": "fail",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "JA018",
"id": "0000001-150915094349962-oozie-hado-W@job-node",
"consoleUrl": "http://ap-cluster-all-0:8088/proxy/application_1442310173665_0002/",
"errorMessage": "Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]",
"toString": "Action name[job-node] status[ERROR]",
"stats": null,
"type": "pig",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "fail",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": null,
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "E0729",
"id": "0000001-150915094349962-oozie-hado-W@fail",
"consoleUrl": "-",
"errorMessage": "Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]",
"toString": "Action name[fail] status[OK]",
"stats": null,
"type": ":KILL:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"externalChildIDs": null,
"cred": "null"
}
],
"createdTime": "Tue, 15 Sep 2015 09:49:58 GMT",
"status": "KILLED",
"group": null,
"externalId": null,
"acl": null,
"run": 0,
"appName": "job-wf",
"parentId": null,
"conf": "<configuration>\r\n <property>\r\n <name>user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>oozie.use.system.libpath</name>\r\n <value>true</value>\r\n </property>\r\n <property>\r\n <name>mapreduce.job.user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>nameNode</name>\r\n <value>hdfs://ap-cluster-all-0:9000</value>\r\n </property>\r\n <property>\r\n <name>jobTracker</name>\r\n <value>http://172.18.168.119:8032</value>\r\n </property>\r\n <property>\r\n <name>oozie.wf.application.path</name>\r\n <value>hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml</value>\r\n </property>\r\n</configuration>",
"id": "0000001-150915094349962-oozie-hado-W",
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"appPath": "hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml",
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"toString": "Workflow id[0000001-150915094349962-oozie-hado-W] status[KILLED]",
"lastModTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"consoleUrl": "http://ap-cluster-all-0.novalocal:11000/oozie?job=0000001-150915094349962-oozie-hado-W"
}
}
}
Job types¶
Each plugin that supports EDP also supports specific job types. Different versions of a plugin might actually support different job types. Configuration options vary by plugin, version, and job type.
The job types provide information about which plugins support which job types and how to configure the job types.
Lists all job types.
You can use query parameters to filter the response.
Normal response codes: 200 Error response codes:
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| plugin (Optional) | path | string | Filters the response by a plugin name. |
| version (Optional) | path | string | Filters the response by a plugin version. |
| type | body | string | The type of the data source object. |
| hints (Optional) | path | boolean | Includes configuration hints in the response. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| versions | body | array | The list of plugin versions. |
| title | body | string | The title of the plugin. |
| description | body | string | The full description of the plugin. |
| job_types | body | array | The list of plugins and their job types. |
| name | body | string | The name of the plugin. |
Response Example¶
{
"job_types": [
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Hive"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Java"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "MapReduce"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "MapReduce.Streaming"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Pig"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Shell"
},
{
"plugins": [
{
"description": "This plugin provides an ability to launch Spark on Hadoop CDH cluster without any management consoles.",
"versions": {
"1.0.0": {}
},
"title": "Apache Spark",
"name": "spark"
}
],
"name": "Spark"
}
]
}
Job binary internals¶
Job binary internal objects represent data processing applications and libraries that are stored in the internal database.
Creates a job binary internal.
Job binary internals are objects that represent data processing applications and libraries that are stored in the internal database.
Specify the file contents (raw data or script text) in the request body. Specify the file name in the URI.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| name | path | string | The name of the job binary internal. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| name | body | string | The name of the job binary internal. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| datasize | body | integer | The size of the data stored in the internal database. |
| id | body | string | The UUID of the job binary internal. |
Shows data for a job binary internal.
The response body shows the job binary raw data and the response headers show the data length.
Example response:
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 161
Content-Type: text/html; charset=utf-8
Date: Sat, 28 Mar 2016 02:21:13 GMT
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage();
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_binary_internals_id | path | string | The UUID of the job binary internal. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| Content-Length | header | string | The length of the data, in bytes. |
Response Example¶
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage()
Shows details for a job binary internal.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_binary_internals_id | path | string | The UUID of the job binary internal. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| name | body | string | The name of the job binary internal. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| datasize | body | integer | The size of the data stored in the internal database. |
| id | body | string | The UUID of the job binary internal. |
Response Example¶
{
"job_binary_internal": {
"is_public": false,
"name": "script.pig",
"tenant_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 13:17:35.994466",
"updated_at": null,
"datasize": 160,
"id": "4833dc4b-8682-4d5b-8a9f-2036b47a0996",
"is_protected": false
}
}
Deletes a job binary internal.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_binary_internals_id | path | string | The UUID of the job binary internal. |
Updates a job binary internal.
Normal respose codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_binary_internals_id | path | string | The UUID of the job binary internal. |
Request Example¶
{
"name": "public-jbi",
"is_public": true
}
Lists the available job binary internals.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| limit (Optional) | path | integer | Maximum number of objects in response data. |
| marker (Optional) | path | string | ID of the last element on the list which won’t be in response. |
| sort_by (Optional) | path | string | The field for sorting job binary internals.
this parameter accepts the following values:
id, name, created_at, updated_at.
Also this values can started with - prefix
for descending sort. For example: -name. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| markers (Optional) | body | object | The markers of previous and following pages of data.
This field exists only if limit is passed to
request. |
| prev (Optional) | body | string | The marker of previous page. May be null if
previous page is first or if current page is first. |
| next (Optional) | body | string | The marker of next page of list data. |
| binaries | body | array | The list of job binary internal objects. |
| name | body | string | The name of the job binary internal. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| is_protected | body | boolean | If set to true, the object is protected. |
| is_public | body | boolean | If set to true, the object is public. |
| datasize | body | integer | The size of the data stored in the internal database. |
| id | body | string | The UUID of the job binary internal. |
{
"binaries": [
{
"is_public": false,
"name": "example.pig",
"tenant_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 12:36:59.329034",
"updated_at": null,
"datasize": 161,
"id": "d2498cbf-4589-484a-a814-81436c18beb3",
"is_protected": false
},
{
"is_public": false,
"name": "udf.jar",
"tenant_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 12:43:52.008620",
"updated_at": null,
"datasize": 3745,
"id": "22f1d87a-23c8-483e-a0dd-cb4a16dde5f9",
"is_protected": false
}
]
}
Jobs¶
A job object lists the binaries that a job needs to run. To run a job, you must specify data sources and job parameters.
You can run a job on an existing or new transient cluster.
Runs a job.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_id | path | string | The UUID of the job. |
Request Example¶
{
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"job_configs": {
"configs": {
"mapred.map.tasks": "1",
"mapred.reduce.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
}
}
Lists all jobs.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| limit (Optional) | path | integer | Maximum number of objects in response data. |
| marker (Optional) | path | string | ID of the last element on the list which won’t be in response. |
| sort_by (Optional) | path | string | The field for sorting jobs.
this parameter accepts the following values:
id, name, type, created_at,
updated_at. Also this values can started
with - prefix for descending sort.
For example: -name. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| jobs | body | array | The list of the jobs. |
| description | body | string | The description of the job object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| mains | body | array | The list of the job object and their properties. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| libs | body | array | The list of the job object properties. |
| is_protected | body | boolean | If set to true, the object is protected. |
| interface | body | array | The interfaces of the job object. |
| is_public | body | boolean | If set to true, the object is public. |
| type | body | string | The type of the data source object. |
| id | body | string | The UUID of the job object. |
| name | body | string | The name of the job object. |
| markers (Optional) | body | object | The markers of previous and following pages of data.
This field exists only if limit is passed to
request. |
| prev (Optional) | body | string | The marker of previous page. May be null if
previous page is first or if current page is first. |
| next (Optional) | body | string | The marker of next page of list data. |
Response Example¶
..rest_method:: GET /v1.1/{project_id}/jobs?limit=2
{
"jobs": [
{
"is_public": false,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "1a674c31-9aaa-4d07-b844-2bf200a1b836",
"name": "Edp-test-job-3d60854e",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "0ff4ac10-94a4-4e25-9ac9-603afe27b100",
"name": "binary-job-339c2d1a.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-c71e6bce.sahara/binary-job-339c2d1a.jar"
}
],
"type": "MapReduce",
"mains": [],
"is_protected": false
},
{
"is_public": false,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "4d1f3759-3497-4927-8352-910bacf24e62",
"name": "Edp-test-job-6b6953c8",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "e0d47800-4ac1-4d63-a2e1-c92d669a44e2",
"name": "binary-job-6f21a2f8.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-b409ec68.sahara/binary-job-6f21a2f8.jar"
}
],
"type": "Pig",
"mains": [
{
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "e073e896-f123-4b76-995f-901d786262df",
"name": "binary-job-d4f8bd75.pig",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-b409ec68.sahara/binary-job-d4f8bd75.pig"
}
],
"is_protected": false
}
],
"markers": {
"prev": null,
"next": "c53832da-6e7b-449e-a166-9f9ce1718d03"
}
}
Creates a job object.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
Request Example¶
{
"description": "This is pig job example",
"mains": [
"90d9d5ec-11aa-48bd-bc8c-34936ce0db6e"
],
"libs": [
"320a2ca7-25fd-4b48-9bc3-4fb1b6c4ff27"
],
"type": "Pig",
"name": "pig-job-example"
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description | body | string | The description of the job object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| mains | body | array | The list of the job object and their properties. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| libs | body | array | The list of the job object properties. |
| is_protected | body | boolean | If set to true, the object is protected. |
| interface | body | array | The interfaces of the job object. |
| is_public | body | boolean | If set to true, the object is public. |
| type | body | string | The type of the data source object. |
| id | body | string | The UUID of the job object. |
| name | body | string | The name of the job object. |
Shows details for a job.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_id | path | string | The UUID of the job. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description | body | string | The description of the job object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| mains | body | array | The list of the job object and their properties. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| libs | body | array | The list of the job object properties. |
| is_protected | body | boolean | If set to true, the object is protected. |
| interface | body | array | The interfaces of the job object. |
| is_public | body | boolean | If set to true, the object is public. |
| type | body | string | The type of the data source object. |
| id | body | string | The UUID of the job object. |
| name | body | string | The name of the job object. |
Response Example¶
{
"job": {
"is_public": false,
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "1a674c31-9aaa-4d07-b844-2bf200a1b836",
"name": "Edp-test-job",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"tenant_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "0ff4ac10-94a4-4e25-9ac9-603afe27b100",
"name": "binary-job.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-c71e6bce.sahara/binary-job.jar"
}
],
"type": "MapReduce",
"mains": [],
"is_protected": false
}
}
Removes a job.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_id | path | string | The UUID of the job. |
Updates a job object.
Normal response codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| job_id | path | string | The UUID of the job. |
Request Example¶
{
"description": "This is public pig job example",
"name": "public-pig-job-example"
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| description | body | string | The description of the job object. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| mains | body | array | The list of the job object and their properties. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| libs | body | array | The list of the job object properties. |
| is_protected | body | boolean | If set to true, the object is protected. |
| interface | body | array | The interfaces of the job object. |
| is_public | body | boolean | If set to true, the object is public. |
| type | body | string | The type of the data source object. |
| id | body | string | The UUID of the job object. |
| name | body | string | The name of the job object. |
Node group templates¶
A cluster is a group of nodes with the same configuration. A node group template configures a node in the cluster.
A template configures Hadoop processes and VM characteristics, such as the number of reduced slots for task tracker, the number of CPUs, and the amount of RAM. The template specifies the VM characteristics through an OpenStack flavor.
Lists available node group templates.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| limit (Optional) | path | integer | Maximum number of objects in response data. |
| marker (Optional) | path | string | ID of the last element on the list which won’t be in response. |
| sort_by (Optional) | path | string | The field for sorting node group templates.
this parameter accepts the following values:
name, plugin_name, hadoop_version,
created_at, updated_at, id. Also
this values can started with - prefix for
descending sort. For example: -name. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| markers (Optional) | body | object | The markers of previous and following pages of data.
This field exists only if limit is passed to
request. |
| prev (Optional) | body | string | The marker of previous page. May be null if
previous page is first or if current page is first. |
| next (Optional) | body | string | The marker of next page of list data. |
| volume_local_to_instance | body | boolean | If set to true, the volume is local to the
instance. |
| availability_zone | body | string | The availability of the node in the cluster. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| use_autoconfig | body | boolean | If set to true, the cluster is auto
configured. |
| volumes_per_node | body | integer | The number of volumes for the node. |
| id | body | string | The UUID of the node group template. |
| security_groups | body | string | The security groups of the node. |
| shares | body | string | The sharing of resources in the cluster. |
| node_configs | body | object | A set of key and value pairs that contain the node configuration in the cluster. |
| auto_security_group | body | boolean | If set to True, the cluster group is
automatically secured. |
| volumes_availability_zone | body | string | The availability zone of the volumes. |
| description (Optional) | body | string | Description of the node group template |
| volume_mount_prefix | body | string | The mount point of the node. |
| plugin_name | body | string | The name of the plugin. |
| floating_ip_pool | body | string | The UUID of the pool in the template. |
| is_default | body | boolean | If set to true, the cluster is the default
cluster. |
| image_id | body | string | The UUID of the image. |
| volumes_size | body | integer | The size of the volumes in a node. |
| is_proxy_gateway | body | boolean | If set to true, the node is the proxy
gateway. |
| is_public | body | boolean | If set to true, the object is public. |
| hadoop_version | body | string | The version of the Hadoop used in the cluster. |
| name | body | string | The name of the node group template. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| volume_type | body | string | The type of volume in a node. |
| is_protected | body | boolean | If set to true, the object is protected. |
| node_processes | body | array | The list of the processes performed by the node. |
| flavor_id | body | string | The ID of the flavor. |
{
"node_group_templates": [
{
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"description": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:20:11",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
},
{
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"description": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:27:00",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
}
],
"markers": {
"prev":"39dfc852-8588-4b61-8d2b-eb08a67ab240",
"next":"eaa0bd97-ab54-43df-83ab-77a9774d7358"
}
}
Creates a node group template.
Normal response codes: 202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
Request Example¶
{
"plugin_name": "vanilla",
"hadoop_version": "2.7.1",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"name": "master",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"flavor_id": "2"
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| volume_local_to_instance | body | boolean | If set to true, the volume is local to the
instance. |
| availability_zone | body | string | The availability of the node in the cluster. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| use_autoconfig | body | boolean | If set to true, the cluster is auto
configured. |
| volumes_per_node | body | integer | The number of volumes for the node. |
| id | body | string | The UUID of the node group template. |
| security_groups | body | string | The security groups of the node. |
| shares | body | string | The sharing of resources in the cluster. |
| node_configs | body | object | A set of key and value pairs that contain the node configuration in the cluster. |
| auto_security_group | body | boolean | If set to True, the cluster group is
automatically secured. |
| volumes_availability_zone | body | string | The availability zone of the volumes. |
| description (Optional) | body | string | Description of the node group template |
| volume_mount_prefix | body | string | The mount point of the node. |
| plugin_name | body | string | The name of the plugin. |
| floating_ip_pool | body | string | The UUID of the pool in the template. |
| is_default | body | boolean | If set to true, the cluster is the default
cluster. |
| image_id | body | string | The UUID of the image. |
| volumes_size | body | integer | The size of the volumes in a node. |
| is_proxy_gateway | body | boolean | If set to true, the node is the proxy
gateway. |
| is_public | body | boolean | If set to true, the object is public. |
| hadoop_version | body | string | The version of the Hadoop used in the cluster. |
| name | body | string | The name of the node group template. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| volume_type | body | string | The type of volume in a node. |
| is_protected | body | boolean | If set to true, the object is protected. |
| node_processes | body | array | The list of the processes performed by the node. |
| flavor_id | body | string | The ID of the flavor. |
Shows a node group template, by ID.
Normal response codes: 200
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| node_group_template_id (Optional) | path | string | The UUID of the node group template. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| volume_local_to_instance | body | boolean | If set to true, the volume is local to the
instance. |
| availability_zone | body | string | The availability of the node in the cluster. |
| updated_at | body | string | The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| use_autoconfig | body | boolean | If set to true, the cluster is auto
configured. |
| volumes_per_node | body | integer | The number of volumes for the node. |
| id | body | string | The UUID of the node group template. |
| security_groups | body | string | The security groups of the node. |
| shares | body | string | The sharing of resources in the cluster. |
| node_configs | body | object | A set of key and value pairs that contain the node configuration in the cluster. |
| auto_security_group | body | boolean | If set to True, the cluster group is
automatically secured. |
| volumes_availability_zone | body | string | The availability zone of the volumes. |
| description (Optional) | body | string | Description of the node group template |
| volume_mount_prefix | body | string | The mount point of the node. |
| plugin_name | body | string | The name of the plugin. |
| floating_ip_pool | body | string | The UUID of the pool in the template. |
| is_default | body | boolean | If set to true, the cluster is the default
cluster. |
| image_id | body | string | The UUID of the image. |
| volumes_size | body | integer | The size of the volumes in a node. |
| is_proxy_gateway | body | boolean | If set to true, the node is the proxy
gateway. |
| is_public | body | boolean | If set to true, the object is public. |
| hadoop_version | body | string | The version of the Hadoop used in the cluster. |
| name | body | string | The name of the node group template. |
| tenant_id | body | string | The UUID of the tenant. |
| created_at | body | string | The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
| volume_type | body | string | The type of volume in a node. |
| is_protected | body | boolean | If set to true, the object is protected. |
| node_processes | body | array | The list of the processes performed by the node. |
| flavor_id | body | string | The ID of the flavor. |
Response Example¶
{
"node_group_template": {
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"description": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:20:11",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
}
}
Deletes a node group template.
Normal response codes:204
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| node_group_template_id (Optional) | path | string | The UUID of the node group template. |
Updates a node group template.
Normal respose codes:202
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| node_group_template_id (Optional) | path | string | The UUID of the node group template. |
Request Example¶
{
"plugin_name": "vanilla",
"hadoop_version": "2.7.1",
"node_processes": [
"datanode"
],
"name": "new",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"flavor_id": "2"
}
Plugins¶
A plugin object defines the Hadoop or Spark version that it can install and which configurations can be set for the cluster.
Shows details for a plugin.
Normal response codes: 200 Error response codes: 400, 500
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| plugin_name (Optional) | path | string | Name of the plugin. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| versions | body | array | The list of plugin versions. |
| title | body | string | The title of the plugin. |
| description | body | string | The full description of the plugin. |
| name | body | string | The name of the plugin. |
Response Example¶
{
"plugin": {
"name": "vanilla",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"title": "Vanilla Apache Hadoop",
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component."
}
}
Lists all registered plugins.
Normal response codes: 200 Error response codes: 400, 500
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| title | body | string | The title of the plugin. |
| versions | body | array | The list of plugin versions. |
| plugins | body | array | The list of plugins. |
| description | body | string | The full description of the plugin. |
| name | body | string | The name of the plugin. |
Response Example¶
{
"plugins": [
{
"name": "vanilla",
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"title": "Vanilla Apache Hadoop"
},
{
"name": "hdp",
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": [
"1.3.2",
"2.0.6"
],
"title": "Hortonworks Data Platform"
},
{
"name": "spark",
"description": "This plugin provides an ability to launch Spark on Hadoop CDH cluster without any management consoles.",
"versions": [
"1.0.0",
"0.9.1"
],
"title": "Apache Spark"
},
{
"name": "cdh",
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": [
"5",
"5.3.0"
],
"title": "Cloudera Plugin"
}
]
}
Shows details for a plugin version.
Normal response codes: 200 Error response codes: 400, 500
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| plugin_name (Optional) | path | string | Name of the plugin. |
| version (Optional) | path | string | Filters the response by a plugin version. |
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| versions | body | array | The list of plugin versions. |
| title | body | string | The title of the plugin. |
| description | body | string | The full description of the plugin. |
| name | body | string | The name of the plugin. |
Response Example¶
{
"plugin": {
"name": "vanilla",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"required_image_tags": [
"vanilla",
"2.6.0"
],
"node_processes": {
"JobFlow": [
"oozie"
],
"HDFS": [
"namenode",
"datanode",
"secondarynamenode"
],
"YARN": [
"resourcemanager",
"nodemanager"
],
"MapReduce": [
"historyserver"
],
"Hadoop": [],
"Hive": [
"hiveserver"
]
},
"configs": [
{
"default_value": "/tmp/hadoop-${user.name}",
"name": "hadoop.tmp.dir",
"priority": 2,
"config_type": "string",
"applicable_target": "HDFS",
"is_optional": true,
"scope": "node",
"description": "A base for other temporary directories."
},
{
"default_value": true,
"name": "hadoop.native.lib",
"priority": 2,
"config_type": "bool",
"applicable_target": "HDFS",
"is_optional": true,
"scope": "node",
"description": "Should native hadoop libraries, if present, be used."
},
{
"default_value": 1024,
"name": "NodeManager Heap Size",
"config_values": null,
"priority": 1,
"config_type": "int",
"applicable_target": "YARN",
"is_optional": false,
"scope": "node",
"description": null
},
{
"default_value": true,
"name": "Enable Swift",
"config_values": null,
"priority": 1,
"config_type": "bool",
"applicable_target": "general",
"is_optional": false,
"scope": "cluster",
"description": null
},
{
"default_value": true,
"name": "Enable MySQL",
"config_values": null,
"priority": 1,
"config_type": "bool",
"applicable_target": "general",
"is_optional": true,
"scope": "cluster",
"description": null
}
],
"title": "Vanilla Apache Hadoop"
}
}
Updates details for a plugin.
Normal response codes: 202 Error response codes: 400, 500
Request¶
| Name | In | Type | Description |
|---|---|---|---|
| project_id | path | string | UUID of the project. |
| plugin_name (Optional) | path | string | Name of the plugin. |
Request Example¶
{
"plugin_labels": {
"enabled": {
"status": false
}
}
}
Response Parameters¶
| Name | In | Type | Description |
|---|---|---|---|
| title | body | string | The title of the plugin. |
| versions | body | array | The list of plugin versions. |
| description | body | string | The full description of the plugin. |
| name | body | string | The name of the plugin. |
Response Example¶
{
"plugin": {
"plugin_labels": {
"hidden": {
"status": true,
"mutable": true,
"description": "Existence of plugin or its version is hidden, but still can be used for cluster creation by CLI and directly by client."
},
"enabled": {
"status": false,
"mutable": true,
"description": "Plugin or its version is enabled and can be used by user."
}
},
"description": "It's a fake plugin that aimed to work on the CirrOS images. It doesn't install Hadoop. It's needed to be able to test provisioning part of Sahara codebase itself.",
"versions": [
"0.1"
],
"tenant_id": "993f53c1f51845e48e013aeb632358d8",
"title": "Fake Plugin",
"version_labels": {
"0.1": {
"enabled": {
"status": true,
"mutable": true,
"description": "Plugin or its version is enabled and can be used by user."
}
}
},
"name": "fake"
}
}
Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.