Short Description |
Ports |
Metadata |
SalesforceBulkReader Attributes |
Details |
Examples |
Compatibility |
See also |
SalesforceBulkReader reads records from Salesforce using Bulk API.
Which Salesforce reader? | |
---|---|
If you need to read a small number of records, read attachments, or use subqueries, use SalesforceReader. If you need to read a large number of records, use SalesforceBulkReader. |
Component | Data source | Input ports | Output ports | Each to all outputs | Different to different outputs | Transformation | Transf. req. | Java | CTL | Auto-propagated metadata |
---|---|---|---|---|---|---|---|---|---|---|
SalesforceBulkReader | database | 0 | 1 |
Port type | Number | Required | Description | Metadata |
---|---|---|---|---|
Output | 0 | SOQL query results | output0 |
SalesforceBulkReader does not propagate metadata.
SalesforceBulkReader has no metadata templates.
SalesforceBulkReader has no special requirements on metadata names or field data types.
Attribute | Req | Description | Possible values |
---|---|---|---|
Basic | |||
Connection | yes | A Salesforce connection. See Salesforce connection. | e.g. MySFConnection |
SOQL query | yes |
A query for retrieving data from Salesforce. The component allows you to use subset of SOQL language. It is the constraint of the Bulk API. If you query the records, you should use API names for objects and fields. The API name can differ from the name of an object in Salesforce web gui. You can use graph parameters in SOQL query. | e.g. SELECT Name, Website FROM Account WHERE Industry = 'Energy' |
Output mapping | Mapping from query fields to output metadata fields | Map by name (default) | |
Advanced | |||
Read mode |
Enables or disables reading deleted or archived reports. You can choose between Do not return deleted or archived records (default) and Return also deleted or archived records. This attribute is available since 4.6.0-M2. | ||
Result polling interval (seconds) |
Time between queries for result of the asynchronous calls. The default value is taken from the connection configuration. | 5 (default) |
SalesforceBulkReader reads records from Salesforce using Bulk API. Bulk API performs the operations asynchronously It sends request to Salesforce, waits several seconds and queries Salesforce for the result. The time interval is configured with Result polling interval attribute. With a small interval, you can get results quickly, but you can consume more of your Salesforce requests.
To use the component, create a Salesforce connection, enter SOQL query, and specify the output mapping. If you perform the steps in this order, the transform editor can provide you with metadata extracted from the SOQL query. Therefore, you will be able to map the fields with drag and drop.
SalesforceBulkReader uses Salesforce Object Query Language (SOQL) to query data in Salesforce. However, only a subset of the language is supported by the Bulk API. Because of that, following parts of SOQL are not supported by SalesforceBulkReader:
COUNT
ROLLUP
SUM
GROUP BY
OFFSET
Nested queries
Relationship fields
The output records come out in arbitrary order unless you use ORDER BY in your query.
If you read more than 10-15k of records, it is better to use Bulk API because it will use less API requests.
Bulk API does not allow you to read compound fields, e.g. address compound fields. See https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/compound_fields.htm
Bulk API does not allow you to read base64 fields. Therefore, attachments cannot be read using SalesforceBulkReader.
SalesforceBulkReader uses multiple API calls during its run. All of them count towards your Salesforce API request limit. The precise call flow is:
Login
Extract fields of expected result set from Salesforce object.
Create a bulk query job.
Create a batch with SOQL query.
Get job completion status. This call is repeated in interval specified by Result polling interval attribute until the job is completed.
Download list of results sets.
Download query results. The number of calls depends on size of returned data. Salesforce limits the size of a single query result to 1GB.
Close the bulk query job.
When working with SalesforceBulkReader, you can hit the limitations of the Bulk API. The following error messages should help you to determine what is going wrong.
The SOQL query asks for a compound data field (e.g. address), but this operation is not supported by the Bulk API. Query the particular fields of the compound field instead.
See https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/compound_fields_address.htm
Reading records from Salesforce |
Reading addresses |
Reading object IDs |
This example shows the basic use case of SalesforceBulkReader.
Read Account name, Industry, and Website fields from Account.
Create a Salesforce connection.
In SalesforceBulkReader, set up the Connection, SOQL Query, and Output mapping attributes.
Attribute | Value |
---|---|
Connection | Connection from the first step |
SOQL query | SELECT Name, Industry, Website FROM Account |
Output mapping | See the code below |
//#CTL2 function integer transform() { $out.0.Name = $in.0.Name; $out.0.Industry = $in.0.Industry; $out.0.Website = $in.0.Website; return ALL; }
You can use output mapping if you have specified the SOQL query attribute.
Creating a metadata with SalesforceBulkReader | |
---|---|
Connect an edge to the output port. In Output mapping, select the fields in the left side and drag them to the right side. |
This example shows way to read addresses.
Read a name and a shipping address (street, city, state/province, postal code, and country) of customers from energy industry.
Create a Salesforce connection.
In SalesforceBulkReader, fill in Connection, SOQL Query, and Output mapping attributes.
Attribute | Value |
---|---|
Connection | Connection from the first step |
SOQL query | SELECT Name, ShippingStreet, ShippingCity, ShippingPostalcode, ShippingState, ShippingCountry FROM Account WHERE Industry = 'Energy' |
Output mapping | See the code below |
//#CTL2 function integer transform() { $out.0.Name = $in.0.Name; $out.0.ShippingStreet = $in.0.ShippingStreet; $out.0.ShippingCity = $in.0.ShippingCity; $out.0.ShippingPostalCode = $in.0.ShippingPostalCode; $out.0.ShippingState = $in.0.ShippingState; $out.0.ShippingCountry = $in.0.ShippingCountry; return ALL; }
This example shows reading object IDs from Salesforce.
Read product names and object IDs of our Animal product family.
Create a Salesforce connection.
In SalesforceBulkReader, set Connection, SOQL query, and Output mapping.
Attribute | Value |
---|---|
Connection | Connection from the first step |
SOQL query | SELECT Id, Name FROM Product2 WHERE Family = 'Animal' |
Output mapping | See the code below |
//#CTL2 function integer transform() { $out.0.Id = $in.0.Id; $out.0.Name = $in.0.Name; return ALL; }
SalesforceBulkReader is available since 4.3.0-M2. It uses Salesforce Bulk API version 37.0.
Since CloverETL 4.5.0-M2, SalesforceBulkReader uses Salesforce Bulk API version 39.0.
Since CloverETL 4.6.0-M2, SalesforceBulkReader can also read deleted or archived records. This is configured with Read mode attribute.