SQL

The user can execute SQL from the actions option by selecting the connection at which he wants to execute the SQL and provide the query which he wants to execute.

Shell Script

The user can invoke a Shell Script as an Action.

To invoke the shell script there are two ways to provide the input:

- Writing the shell script in the inline editor

- The user can upload the shell script file to execute it.

Stored Procedure

With this option user can execute a stored procedure or function stored in a database.

To do so, the user has to select an appropriate JDBC connection.

Further he would be required to select the relevant stored procedure appearing in the drop-down menu.

Type Of Data

Input format of Data.

Is Header Included in Source

Option to enable header in source data.

Enable PII Masking

Option to enable PII (Personally Identifiable Information) masking. To know more, see PII Masking processor.

Max no. of Rows

Maximum no. of the rows. Sample to pull from Streaming Source.

Trigger time for Sample

Minimum wait time before system fires a query to fetch data. Example: If it is 15 Secs, then the system will first wait for 15 secs and will fetch all the rows available and create dataset out of it.

Sampling Method

Dictates how to extract sample records from the complete data fetched from the source.

Following are the ways:

-Top N: Extract top n records using limit() on dataset.

-Random Sample: Extract Random Sample records applying sample transformation and then limit to max number of rows in dataset.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Container

Provide connection name in Azure Blob storage.

ADLS Directory Path

Provide directory path for ADLS file system.

Note: User has an option to configure ADLS source with supported compressed data files.

ADD CONFIGURATIONS

User can add further configurations (Optional).

Environment Params

User can add further environment parameters. (Optional)

Enable Incremental Read

Unchecked by default, check mark this option to enable incremental read support.

Read By

Option to read data incrementally either by choosing the File Modification Time option or Column Partition option.

Upon selecting the File Modification Time option provide the below detail:

Offset

Specifies the last modified time of the file.

Note: The offset time must be lesser than the latest file modification time.

Records with timestamp value greater than the specified datetime (in UTC) will be fetched. After each pipeline run the datetime configuration will set to the most recent timestamp value from the last fetched records. The given value should be in UTC with ISO Date format as yyyy-MM-dd'T'HH:mm:ss.SSSZZZ. Ex: 2021-12-24T13:20:54.825+0000.

Upon selecting the Column Partition option provide the below details:

Read Control Type

Options to control data fetch:

None: All records in reference column with values greater than the start value will be read.

Limit by Value: All records in reference column with values greater than the start value but less than/equal to the max value that you set will be read.

Limit by Incremental Size:  

All records in reference column with values greater than the start value with specified incremental size that you set will be selected.

Note: Upon selecting the Inclusive Start Offset checkbox, the Start value will be included with selected size.

Inclusive Start Offset

Check the checkbox for enabling the Inclusive Start Offset option to include the start value for incrementally reading the schema.

Note: In case if the Limit by Value option is selected as Read Control Type, then provide the Max value along with Start value of the selected column ID.

Upon selecting the Inclusive Start Offset option the schema from Start value to the Max value will be incrementally read.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Database Name

Select the Mongo DB database source from which the data is to be fetched.

Collection Name

Name of the database collection that needs to be scanned should be selected.

Query

Filtering criteria option to choose between All Data or Match Query.

If Match Query option is selected, provide the below detail:

Filter/Query

Fetch filtered schema or read data from source as per the filter condition or query provided.

Note:

In Match Query, use only single quotes (‘’) where required.

Records Per Partition

Number of records to be read per partition.

Schema Updated Alert

Check the checkbox to receive alerts for any schema changes when the data is fetched from the source.

Add Configuration

Option to add further configuration.

Read Type

Option to fetch data from the source. Full Load, Incremental and CDC options are available to fetch data as explained below:

Full Load

Reads all the records as per the configured collection from the database during the pipeline execution.

Incremental

Reads records as per specified offset(s) or start value from the database during pipeline execution. Provide the below

Column

Select the column for incremental read. The listed columns can be integer, long, date, timestamp, decimal, etc.

Note: The selected column should have sequential, sorted (in increasing order) and unique values.

Column Includes Date

Check-mark the check-box if the column to be read is of data/stamp type.

Start Value

An offset value needs to be set for the incremental read that will be done on the selected column.

Only those column records with values greater than the offset value will be read.

Read Control Type

Provides three options to control data fetched - None, Limit by Count, and Limit by Value.

None: All the records in the reference column with values greater than the Start Value will be read.

Limit by Count: Only the mentioned number of records in the reference column with their values greater than the Start Value will be read.

Provide No. of Records.

Limit by Value: All the records in the reference column with

values greater than the Start Value but less than/equal to the Column Value that you set will be read (Column Value is inclusive).

Set the Column Value.

For None and Limit by Count, it is recommended that the table should have data in sequential and sorted (increasing) order.

CDC

Reads the records from the configured namespace or the Oplog namespace as per the specified CDC configuration during the pipeline execution.

Oplog Database

Select the Oplog Database from where the data should be read.

Oplog Collection

Select the Oplog Collection from where the data should be read.

Load From Original Collection

This option is checked by default during first time of the configuration. If the check box is unchecked, then it reads the records from Oplog collection as per the specified CDC configuration during the pipeline execution. It will get automatically disabled after pipeline is successfully executed.

If this option is unchecked, then provide the below field:

Offset

Records with timestamp value greater than the specified datetime (in UTC) will be fetched. After each pipeline run the datetime configuration will set to the most recent timestamp value from the last fetched records. The given value should be in UTC with ISO Date format as yyyy-MM-dd'T'HH:mm:ss.SSSZZZ. Ex: 2021-12-24T13:20:54.825+0000.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Capture

Data: Flag to capture data

Metadata: Flag to capture Metadata.

You can select both the options too.

Define Offset

This configurations is similar to what is used for Kafka offset.

• Latest: The starting point of the query is just from the latest offset.

• Earliest: The starting point of the query is from the starting /first offset.

Connection Retries

The number of retries for component connection. Possible values are -1, 0 or any positive number. If the value were -1 then there would be infinite retries for infinite connection.

Delay Between Connection Retries

Retry delay interval for component connection. (In milliseconds.)

Add Configuration

To add additional custom Kafka properties in key-value pairs.

Topic Name

Topic name from where consumer will read the messages.

ZK ID

Zookeeper path to store the offset value at per-consumer basis. An offset is the position of the consumer in the log.

Connection Name

Select the connection name from the available list of con­nections, from where you would like to read the data.

Container

Container name in Azure Blob.

Path

End path with * in case of directory.

For example:- outdir.*.

For Absolute path:- outdir/filename

Add Configuration

To add additional custom properties in key-value pairs.

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Container

Container name in Azure Blob.

Path

End path with * in case of directory.

For example:- outdir/*

Expecting any specific format in directory Path:- outdir/*.csv

Add Configuration

To add additional custom properties in key-value pairs.

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Override Credentials

Unchecked by default, check mark the checkbox to override credentials for user specific actions.

Key

Provide the Azure Cosmos DB key. Click TEST Connection to test the execution of the connection.

Database

Select the Cosmos Database from the drop-down list.

Container

Select the Cosmos Database from the drop-down list.

Enable Change Feed

Change feed enables to read the latest records or changed records. Effect comes only during pipeline flow.

Change Feed From Beginning

If set to True, data will be read from beginning.

CosmosDB Checkpoint Directory

It is the file path where Cosmos stores the checkpoint data for Change feed.

ADD CONFIGURATION

To add additional custom properties in key-value pairs.

Source

Select source for reading the delta file from the available options in the drop down list: HDFS, S3, GCS, DBFS and ADLS.

Provide below fields if the user selects HDFS source for reading the data:

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Override Credentials

Unchecked by default, check mark the checkbox to override credentials for user specific actions.

Username

Once the Override Credentials option is checked, provide the user name through which the Hadoop service is running.

HDFS File Path

Provide the file path of HDFS file system.

Time Travel Option

Select one of the time travel options:

- None: Option not to choose Time Travel.

- Version: Specify the version of delta file in order to fetch the older snapshot of the table with given version number.

- Timestamp: Specifies last modified time of file. All the files that have their last modified time greater than the present value should be read.

Note: The Time Travel option is not available for Streaming Delta source.

Provide below fields if the user selects S3 source for reading the data:

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Override Credentials

Unchecked by default, check mark the checkbox to override credentials for user specific actions.

AWS Key Id

Provide the S3 account access key.

Secret Access Key

Provide the S3 account secret key.

Note: Once the AWS Key Id and Secret Access Key is provided, user has an option to test the connection.

S3 Protocol

Select the S3 protocol from the drop down list. Below protocols are supported for various versions when user selects S3 connection type:

- For HDP versions, S3a protocol is supported.

- For CDH versions, S3a protocol is supported.

- For Apache versions, S3n protocol is supported.

- For GCP, S3n and S3a protocol is supported.

- For Azure S3n protocol is supported. Read/Write to Mumbai and Ohio regions is not supported.

- For EMR S3, S3n, and S3a protocol is supported.

- For AWS Databricks, s3a protocol is supported.

Bucket Name

Provide the S3 bucket name.

Path

Provide the sub-directories of the bucket name on which the data is to be written.

Time Travel Option

Select one of the time travel options:

- None: Option not to choose Time Travel.

- Version: Specify the version of delta file in order to fetch the older snapshot of the table with given version number. ()

- Timestamp: Specifies last modified time of file. All the files that have their last modified time greater than the present value should be read.

Note: The Time Travel option is not available for Streaming Delta source.

Provide below fields if the user selects GCS source for reading the data:

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Override Credentials

Unchecked by default, check mark the checkbox to override credentials for user specific actions.

sax.label.comp.serviceAccountKey

Upload the Service Account File.

User has an option to test connection.

Bucket Name

Provide the GCS bucket name.

Path

Provide the sub-directories of the bucket name on which the data is to be written.

Time Travel Option

Select one of the time travel options:

- None: Option not to choose Time Travel.

- Version: Specify the version of delta file in order to fetch the older snapshot of the table with given version number.

- Timestamp: Specifies last modified time of file. All the files that have their last modified time greater than the present value should be read.

Note: The Time Travel option is not available for Streaming Delta source.

Provide below fields if the user selects DBFS source for reading the data:

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Override Credentials

Unchecked by default, check mark the checkbox to override credentials for user specific actions.

Directory Path

Provide the DBFS parent path for check-pointing.

DBFS File Path

Provide the DBFS file path.

Provide below fields if the user selects ADLS source for reading the data:

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Container Name

Provide container name for azure delta lake storage.

ADLS File Path

Provide the directory path for azure delta lake storage file system.

Time Travel Option

Select one of the time travel options:

- None: Option not to choose Time Travel.

- Version: Specify the version of delta file in order to fetch the older snapshot of the table with given version number.

- Timestamp: Specifies last modified time of file. All the files that have their last modified time greater than the present value should be read.

Note: The Time Travel option is not available for Streaming Delta source.

ADD CONFIGURATION

To add additional custom properties in key-value pairs.

Environment Params

User can add further environment parameters. (Optional)

Type

The type of thing you will register.

Name

Name of the thing.

Attributes

Attributes of associated thing type.

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Keyspace

Cassandra keyspace name.

Table Name

Table name inside the keyspace from where we read data.

Add configuration

To add additional custom properties in key-value pairs.

Use Pluggable Database

Select the option for 12c multi-tenant databases.

Pluggable Database Name

Name of the pluggable database that contains the schema you want to use. Use only when the schema was created in a pluggable database.

Container Database Connection

Use the user account created for the Data Source. Common user accounts for Oracle.

Schema Name

Provide the schema name.

Tables

Tables on which data will be processed.

Operations

Operations for creating records.

• INSERT

• DELETE

• UPDATE

Database Time Zone

Time zone of the database. When the database operates in a different time zone from Data Collector.

Maximum Transaction Time

Time in seconds to wait for changes for a transaction. Enter the longest period that you expect a transaction to require. Default is 60 seconds.

LogMiner Session Window

Time in seconds to keep a LogMiner session open. Set to larger than the maximum transaction length. Reduce when not using local buffering to reduce LogMiner resource use. Default is 7200 seconds.

Local Buffering

TRUE:-

Using local buffers, the Data Source requests the transactions for the relevant tables and period. The Data Source buffers the resulting LogMiner redo SQL statements until it verifies a commit for a transaction. After the commit, it parses and processes the committed data. The Data Source can buffer the redo SQL statements completely in memory.

FALSE:-

When using Oracle LogMiner buffers, the Data Source requests data from Oracle LogMiner for a particular period. LogMiner then buffers all transactions for that period for all the tables in the database, rather than only the tables needed by the origin.

Max Batch Size (records)

Maximum number of records processed at one time. Keep values up to the Data Collector maximum batch size. Default is 1000.

LogMiner Fetch Size

Minimum number of records to fetch before passing a batch to the pipeline. Keep this value low to allow the origin to pass records to the pipeline as soon as possible, rather than waiting for a larger number of records to become available. Lower values can increase throughput when writes to the destination system are slow. Default is 1.

Query Timeout

Time to wait before timing out a LogMiner query. Default is 300 seconds.

Start From

Start From is the point in the LogMiner redo logs where you want to start processing from. When you start the pipeline, Oracle CDC Client starts processing from the specified initial change and continues until you stop the pipeline. It is sub- categories in the following:

1. From the latest change

The origin processes all changes that happened in the pipeline after you start the pipeline.

2. From a specified date-time:

The origin processes all changes that occurred at the specified date-time and later use the format Use the format: DD-MM-YYYY HH24:mm:ss.

3. From a specified system-change number (SCN)

The origin processes all changes that occurred in the specified SCN (specified-change number) and later. When using the specified SCN, the origin starts processing with the timestamps associated with the SCN. If the SCN cannot be found in the redo logs, the origin continues reading from the next higher SCN that is available in the redo logs.

Load From Template

Select the template from the list.

Only those templates will be visible in the drop down list that are created globally or within that workspace.

Sync With Source Template

When a template is saved and is edited, this tab is reflected. If any change is made in source template, corresponding changes will reflect in Container’s configuration.

Connection Name

Name of the connection

Bucket Name

Couchbase Bucket name.

Select the Couchbase bucket that gets generated with the Couchbase connection.

You can also create a bucket and fetch details in the fields.

Bucket password

Couchbase Bucket password

Memory Quota

Memory Quota in megabytes per server node. Memory Quota cannot be less than 100 MB.

Replicate view indexes

By checking the Replicate view indexes checkbox, you ensure that view indexes, as well as data are replicated

Flush

When flushed, all items in the bucket are removed as soon as possible.

Note: This does not immediately show in item count statistics with memcached buckets.

Record Limit

Enable the limit to fetch the number of records from Couchbase.

If you select it as Yes, the following tabs are populated, else, you can move to Fetch Data.

Max no. of Records

Maximum number of records to be fetched from the Couchbase bucket

Overflow Condition

Fetch the records based on order for the selected column. (Fetch minimum records for ascending order and fetch maximum records for descending order)

Note: With aggregation, select the grouping field in the Overflow condition.

Fetch Data

Raw Data:

Fetch the raw data from the Couchbase bucket.

Aggregated Data:

Fetch the aggregated data from the Couchbase. A new field is populated, where you can group the fields from the schema.

Grouping Fields

Field of selected message on which group by is applied.

Fields

Apply functions on the input and output fields.

Data Source Plugin

Fully qualified name of a custom code class.

Select Dataset Type

Option to select a specific type of dataset (for example: HDFS, Hive, JDBC or S3) or all types of dataset to filter and narrow down the selection of required dataset in the subsequent step.

Select Dataset & Version

Select any of the existing dataset and its version that you want to use as a channel in the pipeline.

Max No of Rows

Maximum number of sample rows to pull from the Dataset Source. 100 by default.

Trigger time for Sample

Minimum wait time before system fires a query to fetch data. Example: If it is 15 Secs, then system will first wait for 15 secs and will fetch all the rows available and create dataset out of it. 3 Seconds by default.

Sampling Method

Dictates how to extract sample records from the complete data fetched from the source.

Following are the ways:

-Top N: Extract top n records using limit() on dataset.

-Random Sample: Extract Random Sample records applying sample transformation and then limit to max number of rows in dataset.

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Note: DeltaSQL connection is required while pipeline creation. The actual pipeline execution will happen through metastore.

Override Credentials

Unchecked by default, check the checkbox to override credentials for user specific actions.

Username

Once the Override Credentials option is checked, provide the user name through which the Delta SQL service is running.

Password

Provide the password for Delta SQL override credentials.

Query

Provide the JDBC compatible SQL query to be executed in the component.

Inspect Query

Provide the SQL query to be executed in the component with a set limit in records count for inspect and schema detection.

Note:

- For inspect query field you need to provide simple query with table name like "select * from tablename"

- "As of" clause is not supported for inspect query but can be used in query field.

- Also for Inspect query it should be JDBC compatible

ADD CONFIGURATION

To add additional custom properties in key-value pairs.

Environment Params

User can add further environment parameters. (Optional)

Enable Incremental Read

Check the radio button to enable the incremental option. The available options are:

- None

- Normal

- Version

Upon selecting None option the incremental read support will be disabled.

Upon selecting Normal, provide the inputs for the below fields:

Column to Check

Select a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp, decimal types of values.

Start Value

Mention a value of the reference column, only the records whose value of the reference column is greater than this value will be read.

Read Control Type

Provides three options to control data to be fetched -None, Limit By Count, and Maximum Value.

None: All the records with value of reference column greater than offset will be read.

Limit By Count: Mentioned no. of records will be read with the value of reference column greater than offset will be read.

Note: Number of records to be read if Read Control Type is set as Limit by Count. If in the Column to Check field an integer value is selected then, provide integer value greater than 0 in the No. of Records field.

Limit by Value: All the records with value of reference column greater than offset and less than Column Value field will be read.

Note: In the Column Value field, the incremental read will only be done within the set value for the selected column. Only those column records with values less than or equal to the set column value will be read. The Column Value field appears when the Read Control Type is set as Limit by Value.

For None and Limit by count it is recommended that table should have data in sequential and sorted (increasing) order.

Upon selecting Version, provide the inputs for the below fields:

Table Name

Option to provide delta table name for querying the records.

Version

Option to provide version for querying the records. The records will be read subsequent to the selected version.

ADD CONFIGURATION

To add additional custom properties in key-value pairs.

Environment Params

User can add further environment parameters. (Optional)

File Path

The local file path from where the data will be read.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

HDFS file Path

HDFS path from where data is read. For Parquet, provide.parquet file path.

URI

HTTP or HTTPS URL to fetch data.

Request Method

HTTP request method- GET, POST.

Request Body

Request body for method.

Header

Header’s parameter name and value.

Accept Type

Content type that URI accepts. For example: JSON, CSV and Text.

Auth Type

Used to specify the authorization type associated with the URL. Includes None, Basic, Token and Oauth2 as authorization types. All the types are explained below in detail.

Path to message

JSON path expression to be evaluated to get input data as per message.Works only for Fetch From source as Json

User name

Enter the user name for accessing the URL.

Password

Enter the password for accessing the URl.

Token ID

Key with which token is referred in request.

Token

Token which is provided in order to access the URL.

AuthURL

The endpoint for authorization server, which retrieves authorization code.

ClientId   

Client ID identifier, given to client during application registration process

Secret Key

The secret key provided to the client during application registration process.

Header   

Used to specify the headers associated with AuthUrl, through which authorization code is generated.

HTTP URL

HTTP or HTTPS URI to send request to a resource.

Method Type

HTTP request method for the URI to be selected out of GET or POST.

Request Body

Request body to be provided for the POST Request Method selected above.

Design Time Attributes

This value can be different from what is given in the request body field and will be utilized only during application design time.

Header

Header’s parameter name and value.

Auth Type

Used to specify the authorization type associated with the URL. Includes None, Basic, Token and OAuth2 as authorization types. All the types are explained below in detail.

None

This option specify that URL can be accessed without any authentication.

Basic

This option specify that accessing URL requires Basic Authorization. Provide user name and password for accessing the URL.

Token Based

Token-based authentication is a security technique that authenticates the users who attempts to log in to a server, a network, or other secure system, using a security token provided by the server.

Oauth2

It is an authentication technique in which application gets a token that authorizes access to the user’s account.

If Auth Type is selected as Basic, proceed by providing inputs for below parameters:

Username

Enter the user name for accessing the URI.

Password

Enter the password for accessing the URl.

If Auth Type is selected as Token, proceed by providing inputs for below parameters:

Token ID

Key with which token is referred in request.

Token

Token which is provided in order to access the URL.

If Auth Type is selected as OAuth2, proceed by providing inputs for below parameters:

Header

Used to specify the headers associated with Auth Url, through which authorization code is generated.

Client ID

Client ID identifier, given to client during application registration process.

Secret Key

The secret key provided to the client during application registration process.

Auth URL

The endpoint for authorization server, which retrieves authorization code.

OAuth2 (Client Credential)

The grant_type is client credential in case of OAuth2 (Client Credential). You can also pass parameters to authentication URL using this authentication method.

Auth Headers

Header’s parameter name and value can be provided.

Auth Params

Auth parameter name and value can be provided.

ClientId

The client identifier that is given to the client during the application registration process should be provided.

Secret Key

The secret key that is given to the client during the application registration process should be provided.

Auth Url

The endpoint for authorization server, that retrieves the authorization code.

Use Token

Use generated token in the URL parameter or header of the request.

Custom Token

When user wants to first get a token from a custom API, that can be used in main data API to fetch the data as an authentication key.

Make sure that the response content type of the custom token request is JSON.

HTTP URL

HTTP or HTTPS URI to send request to a resource.

Method Type

HTTP request method for the URI to be selected out of GET or POST.

Request Body

Request body to be provided for the for Request Method selected above.

Header

Header’s parameter name and value.

Path to Token

JSON path expression that points to tokens.

Auth Type

Used to specify the authorization type associated with the URL. Includes None, Basic and Token.

None

This option specify that URL can be accessed without any authentication.

Basic

This option specify that accessing URL requires Basic Authorization. Provide user name and password for accessing the URL.

Token

Token-based authentication is a security technique that authenticates the users who attempts to log in to a server, a network, or other secure system, using a security token provided by the server.

Enable SSL

This option can be selected as True if SSL is enabled on Http. It is set to False by default.

If selected as true, keystore/certificate file needs to be uploaded and Keystore Password/Certificate Alias should be provided in the respective configuration fields.

There is another field “Path to Data” that will appear only when in Schema Type page, for Fetch From Source option, you select the type of data as JSON.

Path to Data

Path to data is a JSON path expression that points to arrays.

Enable Pagination

Enable this option to make paginated calls to fetch the desired data from the endpoint URI.

Paginate Using

Choose the request component that will change for each pagination Get/Post call.

URL

When the content of the URL will get updated.

Request Body

When the request body content will get updated. This option is only available for Post request methods.

Header

When the header content will get updated.

Persist across run

Enable the checkbox to retain the last offset value as the initial value for the subsequent run.

Treat as Expression

During configuration this option can be enabled at the applicable fields to consider the entered value as an expression. Else, the value will be considered as a constant string.

Paginate using URL

Option

Choose the relevant method to provide the variable for URL-based pagination.

Query Parameter

When the query parameters are changing in URL.

URL Offset

When the URL will get generated from the previous response.

Custom Expression

To provide custom logic to create URL.

Query Parameter

Parameter Name

The query parameter name should be provided that will be used in the URL.

Example: page -> This will make the URL as http://www..com?page=<query_parameter_value>

Initial Value

The initial value of the query parameter should be provided.

Example: 1 -> This will make the URL as http://www..com?page=1

Next Value

The next value expression of the query parameter should be provided.

Example: $page+1 -> This will make the URL for next call as http://www..com?page=2

URL Offset

Initial Page URL

Initial URL expression should be provided.

Example: “http://www..com”

Next Page URL

The next page URL expression should be provided.

The previous response will have a URL value, that will be used as the next URL.

Example: When response is Json, and URL is present at path “data.url”, then the next page URL expression is from_json_object($last_response_last_record,’$.data.url')

Custom Expression

URI

Custom URI expression should be provided with help of offset variables.

Example: concat($url,‘page=’,offset) where offset is the variable created in offset name section.

Offset Name

The offset name should be provided that can be used in the Custom Expression and Pagination End Condition expression.

Example: offset -> This can be used in the URI expression as concat($url,‘page=’,offset)

Initial Offset Value

The initial offset value should be provided.

Example: 1 -> This will make the initial value for the URL expression as concat($url,‘page=’,<1>)

Next Offset Value

The next offset value expression should be provided.

Example: $offset+1 -> This will make the next value for the URL expression as concat($url,‘page=’,<2>)

Paginate using Request Body

Option

Choose the relevant method to provide the variable for body-based pagination.

JSON Body

When the body is in JSON format.

Custom Expression

To provide custom logic to create a body for post-call requests.

JSON Body

Key

The path to the JSON object that will update for each pagination request should be provided.

Example 1: $body.path_to_variable_component

Example 2: $body.a.b

Initial Value

The initial value of the key in the JSON body should be provided.

Example: 10 -> This value 10 will be replaced for the first call at key path: $body.path_to_variable_component

Next Value

The next value expression of the key in the JSON body should be provided.

Example: $body.path_to_variable_component+10

Custom Expression

Body Expression

Provide custom expression to create body for each pagination request.

Example: concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Offset Name

The offset name should be provided that can be used in the Custom Expression and Pagination End Condition expression.

Example: offset -> This can be used in the body expression as concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Initial Offset Value

The initial offset value should be provided.

Example: 10 -> This can be used in the body expression as concat(’{“from”:’,10,’,“size”:100,“query”:{“match_all”:{}}}’)

Next Offset Value

The next offset value expression should be provided.

Example: $offset+1 -> This can be used in the body expression as concat(’{“from”:’,,’,“size”:100,“query”:{“match_all”:{}}}’)

Paginate using Header

Header Expression

All the headers with their keys and values (custom expression/constant) should be provided.

Key

Header key should be provided.

Value

An expression value or a constant value should be provided for each header key.

Example: $offset -> Offset value will be derived from the Offset variable fields.

Offset Name

The offset name should be provided that can be used in the Header value expression and Pagination End Condition expression.

Example: offset -> This can be used in the header value as $offset

Initial Offset Value

The initial offset value should be provided.

Example: 10 -> This can be used in the header value as 10 for a key.

Next Offset Value

The next offset value expression should be provided.

Example: $offset+10 -> This can be used in the header value as increments of 10 for each pagination request.

Pagination End Condition

An expression to stop the pagination should be provided.

Example 1: $offset>12

Example 2: $last_response_record_count=0

Example 3: get_json_object($last_response_last_record,’$.data.id’)>3

Example 4: get_json_object($last_response,’$.metadata.url’) is null

Enable Request Rate

Control the rate at which pagination requests are made.

Enable Incremental Read

Enable this option to read data incrementally.

Increment Using

Choose the request component that will change for each incremental run.

URL

When the content of the URL will get updated.

Request Body

When the request body content will get updated.

Header

When the header content will get updated.

Each incremental option is further explained in detail.

Common option

Treat as Expression

During configuration this option can be enabled at the applicable fields to consider the entered value as an expression. Else, the value will be considered as a constant string.

Increment using URL

Option

Choose the relevant method to provide the variable for URL-based incremental read.

Query Parameter

When the query parameters are changing in URL.

URL Offset

When the URL will get generated from the previous response.

Custom Expression

To provide custom logic to create URL.

Query Parameter

Parameter Name

The query parameter name should be provided that will be used in the URL.

Example: page -> This will make the URL as http://www..com?page=<query_parameter_value>

Initial Value

The initial value of the query parameter should be provided.

Example: 1 -> This will make the URL as http://www..com?page=1

Next Value

The next value expression of the query parameter should be provided.

Example: $page+1 -> This will make the URL for next call as http://www..com?page=2

URL Offset

Initial Page URL

Initial URL expression should be provided.

Example: “http://www..com”

Next Page URL

The next page URL expression should be provided.

The previous response will have a URL value, that will be used as the next URL.

Example: When response is Json, and URL is present at path “data.url”, then the next page URL expression is from_json_object($last_response_last_record,’$.data.url')

Custom Expression

URI

Custom URI expression should be provided with help of offset variables.

Example: concat($url,‘page=’,offset) where offset is the variable created in offset name section.

Offset Name

The offset name should be provided that can be used in the Custom Expression.

Example: offset -> This can be used in the URI expression as concat($url,‘page=’,offset)

Initial Offset Value

The initial offset value should be provided.

Example: 1 -> This will make the initial value for the URL expression as concat($url,‘page=’,<1>)

Next Offset Value

The next offset value expression should be provided.

Example: $offset+1 -> This will make the next value for the URL expression as concat($url,‘page=’,<2>)

Increment using Request Body

Option

Choose the relevant method to provide the variable for body-based incremental read.

JSON Body

When the body is in JSON format.

Custom Expression

To provide custom logic to create a body for post-call requests.

JSON Body

Key

The path to the JSON object that will update for each incremental run should be provided.

Example 1: $body.path_to_variable_component

Example 2: $body.a.b

Initial Value

The initial value of the key in the JSON body should be provided.

Example: 10 -> This value 10 will be replaced for the first call at key path: $body.path_to_variable_component

Next Value

The next value expression of the key in the JSON body should be provided.

Example: $body.path_to_variable_component+10

Custom Expression

Body Expression

Provide custom expression to create body for each incremental run request.

Example: concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Offset Name

The offset name should be provided that can be used in the Custom Expression.

Example: offset -> This can be used in the body expression as concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Initial Offset Value

The initial offset value should be provided.

Example: 10 -> This can be used in the body expression as concat(’{“from”:’,10,’,“size”:100,“query”:{“match_all”:{}}}’)

Next Offset Value

The next offset value expression should be provided.

Example: $offset+1 -> This can be used in the body expression as concat(’{“from”:’,,’,“size”:100,“query”:{“match_all”:{}}}’)

Increment using Header

Header Expression

All the headers with their keys and values (custom expression/constant) should be provided.

Key

Header key should be provided.

Value

An expression value or a constant value should be provided for each header key.

Example: $offset -> Offset value will be derived from the Offset variable fields.

Offset Name

The offset name should be provided that can be used in the Header value expression.

Example: offset -> This can be used in the header value as $offset

Initial Offset Value

The initial offset value should be provided.

Example: 10 -> This can be used in the header value as 10 for a key.

Next Offset Value

The next offset value expression should be provided.

Example: $offset+10 -> This can be used in the header value as increments of 10 for each incremental read request.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Query

Write a custom query for Impala.

Fetch Size

The fetch size determines the number of rows to be fetched per round trip.

Connection Name

Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for JDBC Custom earlier.

Use the Test Connection option to ensure that the connection with the JDBC Custom channel is established successfully.

A success message states that the connection is available. In case of any error in test connection, edit the connection to resolve the issue before proceeding further.

Schema Name

Source Schema name for which the list of table will be viewed.

This field will only be visible for databases that require schema to be selected.

Table Name

Source table name to be selected for which you want to view the metadata.

Query

Hive compatible SQL query to be executed in the component.

In case if a connection with Custom JDBC option is selected, this query will serve as a sub-query for the inspect query specified in the custom JDBC connection configuration.

Design Time Query

Query used to fetch limited records during Application design. Used only during schema detection and inspection.

This field should be ignored in case of custom JDBC connection.

Enable Query Partitioning

This enables parallel reading of data from the table. It is disabled by default. Tables will be partitioned if this check-box is enabled.

If query partitioning is enabled, additional fields will be displayed as described below:

No. of Partitions

Specifies the number of parallel threads to be invoked to partition the table while reading the data.

Type-in Partition Column

Select this option if the Partition Column list shown is empty or you do not see the required column in the list.

Partition on Column

This column will be used to partition the data. This has to be a numeric column, on which spark will perform partitioning to read data in parallel.

Data Type

In case if you have typed-in the partitioning column, you need to specify the data type of that column here.

Lower Bound

Value of the lower bound for partitioning column. This value will be used to decide the partition boundaries. The entire dataset will be distributed into multiple chunks depending on the values.

Upper Bound

Value of the upper bound for partitioning column. This value will be used to decide the partition boundaries. The entire dataset will be distributed into multiple chunks depending on the values.

Fetch Size

The fetch size determines the number of rows to be fetched per round trip. The default value is 1000.

Add configuration

Additional properties can be added using Add Configuration link.

Schema

Check the populated schema details.

Advanced Configuration

Optionally, you can enable incremental read.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Topic Type

Select one of the below option to fetch the records from Kafka topic(s)

• Topic name: The topic name is used to subscribe a single topic

• Topic list: The topic list is used to subscribe a comma-separated list of topics

• Pattern: The pattern is used to subscribe to topic values as Java regex:

• With Partitions80: The topic with partitions is used for specific topic(s) partitions to consume. i.e. json string {"topicA":[0,1],"topicB":[2,4]}

Schema must be same in case of Topic List/Pattern/With partition.

Topic Name

Topic in Kafka from where messages will be read.

Topic List/ Pattern/ With partitions

A topic is category or feed name to which messages will be published

Partitions

Number of partitions. Each partition is an ordered unchangeable sequence of message that is repeatedly added to a commit log.

Replication Factor

Number of replications. Replication provides stronger durability and higher availability. For example, a topic with replication factor N can tolerate up to N-1 server failures without losing any messages committed to the log.

Define Offset

Following configurations are used for Kafka offset.

• Latest: The starting point of the query is just from the latest offset.

• Earliest: The starting point of the query is from the starting /first offset.

• Custom: A json string specifying a starting and ending offset for each partition.

• startingOffsets: A JSON string specifying a starting offset for each partition i.e. {"topicA":{"0":23,"1":-1},"topicB":{"0":-1}}

• endingOffsets: A JSON string specifying a ending offset for each partition. This is an optional property with default value “latest”.i.e. {"topicA":{"0":23,"1":-1},"topicB":{"0":-1

Connection Retries

The number of retries for component connection. Possible values are -1, 0 or any positive number. If the value is -1 then there would be infinite retries for infinite connection.

Max Offset Per Trigger

Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topic Partitions of different volume.

Fail on Data Loss

Provides option of query failure in case of data loss. (For example, topics are deleted, or offsets are out of range). This may be a false alarm. You can disable it when it doesn't work as you expected. Batch queries will always fail, if it fails to read any data from the provided offsets due to data loss

Delay Between Connection Retries

Retry delay interval for component connection (in milliseconds).

ADD CONFIGURATION

To add additional custom Kafka properties in key-value pairs.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Override Credentials

Unchecked by default, check mark the checkbox to override credentials for user specific actions.

Provide AWS KeyID and Secret Access Key.

Stream Name

Name of the Kinesis stream.

EndPoint

End point is a URL that is the entry point of Kinesis services.

Region

Name of the region. For example, us-west-2

Initial Position

In the absence of Kinesis checkpoint, this is the workers initial starting position in the stream.

• The values are either the beginning of stream per Kinesis limit of 24 hours or the tip of the stream.

• TRIM_HORIZON: To read data from beginning of stream, use the command TRIM_HORIZON.

• LATESOpenness To read latest or most recent records, use the command LATEST.

Connection Retries

The number of retries for component connection. Possible values are -1, 0 or any positive number. If the value were -1 then there would be infinite retries for infinite connection.

Delay between Connection Retries

Retry delay interval for component connection. (In milliseconds.)

Maximum Delay Between Connection Retries

Maximum cool-off period between two retries.

ADD CONFIGURATION

User can add further configurations by providing values after clicking the ADD CONFIGURATION option.

Environment Params

Connection Name

Connections are the service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Table Name

Name of the table.

Add Configuration

To add additional properties in key-value pairs.

Table

Select table of which you want to view Metadata.

Column Name

Name of the column generated from the table.

Column Type

Type of the column, for example: Text, Int

Nullable

If the value of the column could be Nullable or not.

Enable Incremental Read

Check this check-box to enable incremental read support.

Column to Check

Select a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp, decimal types of values.

Start Value

Mention a value of the reference column, only the records whose value of the reference column is greater than this value will be read.

Read Control Type

Provides three options to control data to be fetched -None, Limit By Count, and Maximum Value.

None: All the records with value of reference column greater than offset will be read.

Limit By Count: Mentioned no. of records will be read with the value of reference column greater than offset will be read.

Maximum Value: All the records with value of reference column greater than offset and less than Column Value field will be read.

For None and Limit by count it is recommended that table should have data in sequential and sorted (increasing) order.

Connection Name

Connections are the service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

QueueName/TopicName

Queue/topic name from which messages will be read.

Add Configuration

To add additional properties in key-value pairs.

File Path

The local file path from where the data will be read.

Connection Name

Connections are the Service identifiers.Select the connection name from the available list of connections, from where you would like to read the data

Queue Name

Directory path of HDFS file system.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Exchange Name

RMQ producers can only send messages to an exchange. The exchange pushes messages to queues from where consumers can read the message.

Exchange Type

Defines the functionality of the exchange i.e., how messages are routed through it. Exchange types: DIRECT, TOPIC and FANOUT.

Exchange Durable

Durable exchanges remain active when a server restarts. Non-durable exchanges (transient exchanges) are purged when a server restarts.

Exchange Auto Delete

If you set this to True, the exchange is deleted when all queues finish using it.

Routing Key

Identifies a key for redirecting messages to a specific topic.

Queue Name

Name of the queue where data will be published.

Queue Durable

Durable queues remain active when a server restarts. Non- durable queues (transient queues) are purged if/when a server restarts.

Queue Auto Delete

If set, the queue is deleted when all the consumers finish using it.

Add Configuration

To add additional custom RabbitMQ properties in key-value pair.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

S3 Protocol

Protocols available are S3, S3n, S3a

End Point

S3 endpoint details should be provided if the source is Dell EMC S3.

Bucket Name

Buckets are storage units used to store objects, which consists of data and meta-data that describes the data.

Override Credentials

Unchecked by default, check mark the checkbox to override credentials for user specific actions.

Provide AWS KeyID and Secret Access Key.

Path

File or directory path from where data is to be read. The path to be added must be a directory and not absolute.

Add Configuration

To add additional custom S3 properties in a key-value pair.

User can add further configurations by the following ways:

Use key: avroSchema in case if you want to provide avro schema file and paste content (as value) in JSON format to map the schema.

- Use key: avroSchemaFilePath and provide S3 absolutepath of AVSC schema file as value.

To load the schema file from s3, IAM Role attached to Instance Profile will be used.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Override Credentials

Unchecked by default, check mark the checkbox to override credentials for user specific actions.

Provide AWS KeyID and Secret Access Key.

S3 Protocol

Available protocols of S3 are S3, S3n, S3a.

S3a protocol is supported by Databricks.

S3 and S3n are supported by EMR.

S3a is supported by Hadoop Version 3.x.

S3 and S3n protocol is supported by Hadoop Version below 3.x.

End Point

S3 endpoint details should be provided if the source is Dell EMC S3.

Bucket Name

Buckets are storage units used to store objects, which consists of data and meta-data that describes the data.

Path

File or directory path from where data is to be read.

Enable Incremental Read

Note: Incremental Read works during Pipeline run.

Offset

Specifies last modified time of file - all the files whose last modified time is greater that this value will be read.

Add Configuration

To add additional custom S3 properties in a key-value pair.

User can add further configurations by the following ways:

Use key: avroSchema in case if you want to provide avro schema file and paste content (as value) in JSON format to map the schema.

- Use key: avroSchemaFilePath and provide S3 absolutepath of AVSC schema file as value.

To load the schema file from s3, IAM Role attached to Instance Profile will be used.

Enable Incremental Read

Unchecked by default, check mark this option to enable incremental read support.

Read By

Option to read data incrementally either by choosing the File Modification Time option or Column Partition option.

Upon selecting the File Modification Time option provide the below detail:

Offset

Records with timestamp value greater than the specified datetime (in UTC) will be fetched. After each pipeline run the datetime configuration will set to the most recent timestamp value from the last fetched records. The given value should be in UTC with ISO Date format as yyyy-MM-dd'T'HH:mm:ss.SSSZZZ. Ex: 2021-12-24T13:20:54.825+0000.

Upon selecting the Column Partition option provide the below details:

Column

Select the column for incremental read. The listed columns can be integer, long, date, timestamp, decimal, etc.

Note: The selected column should have sequential, sorted (in increasing order) and unique values.

Start Value

Mention the value of reference column. Only the records whose value of the reference column is greater than this value will be read.

Read Control Type

Provides three options to control data to be fetched -None, Limit By Count, and Limit by Value.

None: All the records with value of reference column greater than offset will be read.

Limit By Count: Mentioned no. of records will be read with the value of reference column greater than offset will be read.

Limit by Value: All the records with value of reference column greater than offset and less than Column Value field will be read.

For None and Limit by count it is recommended that table should have data in sequential and sorted (increasing) order.

Fetch From Source/Upload Data File

Option to provide source schema details by either fetching from source or by uploading data file.

Connection Name

Connections are the service identifiers.

A connection name can be selected from the list if you have created and saved connection details for Salesforce earlier.

Table Name

Source table name to be selected for which you want to view the metadata.

More Configurations

Query

Salesforce Object Query Language (SOQL) to search your organization’s Salesforce data for specific information. SOQL is similar to the SELECT statement in the widely used Structured Query Language (SQL) but is designed specifically for Salesforce data. It is mandatory for reading objects like opportunity.

Note: SOQL does not support ‘*’ identifier.

Infer Schema

(Optional) Infer schema from the query results. This will find the data type of the field specified in SOQL. Sample rows will be taken to find the data type. This will work if number of records are 5 and above.

Date Format

A string that indicates the format that follow java.text.SimpleDateFormat to use when reading timestamps. This applies to Timestamp type. By default, it is null which means trying to parse timestamp by java.sql.Timestamp.valueof().

Bulk

(Optional) Flag to enable bulk query. This is the preferred method when loading large sets of data. Bulk API is based on REST principles and is optimized for loading large sets of data. You can use it to query many records asynchronously by submitting batches. Salesforce will process batches in the background. Default value is false.

Salesforce Object

(Conditional) Salesforce Objects are database tables which permit you to store data specific to organization. This is a mandatory parameter when bulk is true and it should be same as specified in SOQL.

Pk Chunking

(Optional) Flag to enable automatic primary key chunking for bulk query job.

This splits bulk queries into separate batches that of the size defined by chunkSize option. By default, it is false. Pk Chunking feature can automatically make large queries manageable when using the Bulk API.

Pk stands for Primary Key — the object’s record ID — which is always indexed. This feature is supported for all custom objects, many standard objects, and their sharing tables.

Chunk size

The size of the number of records to include in each batch. Default value is 100,000. This option can only be used when Pk Chunking is true. Maximum size is 250,000.

Version

Salesforce API version to be selected from the list.

Connection Name

Select the SFTP connection name.

File Path

Mention the file path of SFTP file system.

Note: 

In case of AWS, while providing directory the user is required to provide the wildcard character ( * ) along with the directory. For eg: /home/centos/foldername/*

Incremental Read

Check mark to read latest file in case of folder.

Parallelism

Number of parallel threads to launch in order to run to download file from SFTP.

Connection Name

The user is required to select the Snowflake connection.

Override Credentials

Unchecked by default, check the checkbox to override credentials for user specific actions.

Username

Once the Override Credentials option is checked, provide the user name through which the Delta SQL service is running.

Password

Provide the password for Delta SQL override credentials.

Create Warehouse

The user can create a new warehouse by selecting the check-box. The user will be required to upload the file to detect schema.

Note: Upon selecting the create warehouse option, the user is required to mention the warehouse configuration details.

Warehouse Name

The user is required to mention the warehouse name.

The user can select a warehouse from the existing warehouse list displayed here.

Schema Name

Snowflake schema list of the database of the selected connection is to be mentioned here.

Note: Upon selecting schema, the metadata of the table will be displayed.

Query

The user is required to write the Snowflake SQL query.

Pre-Action Query

A semicolon separated list of SQL commands that are executed before reading data.

Post Action Query

A semicolon separated list of SQL commands that are executed after reading data.

Warehouse Size

User will be required to select a warehouse size.

Maximum Cluster Count

The user will be required to specify the maximum num­ber of clusters for the warehouse.

Scaling Policy

The user is required to mention the scaling policy.

Auto Suspend

The user is required to mention the value in seconds.

Auto Resume

Specifies whether to automatically resume a warehouse when a SQL statement (e.g. query) is submitted to it.

Comments

If needed, the user can mention specific comments.

Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections, from where you would like to read the data.

Add Configuration

To add additional custom properties in key-value pairs.

Connection Name

Connection Name for creating connection.

Queue Type

Select type of the queue: Standard or FIFO.

Queue Name

Name of the queue where data will be published.

Visibility Timeout

The length of the time (in seconds) that a message received from a queue will be invisible to other receiving components.

Message Retention Period (in seconds)

The amount of time Amazon SQS will retainna message in if it does not get deleted.

Maximum Message Size (in bytes)

Maximum message size (in bytes) accepted by Amazon SQS.

Receive Message Wait Time (in seconds)

The maximum amount of time that a long polling receive call will wait for a message to become available before returning an empty house.

Delivery Delay (in seconds)

The amount of time to delay the first delivery of all messages added to this queue.

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Database

Cosmos Database.

Container

Container name in Cosmos.

Change Feed from Beginning

If set to True, data will be read from beginning.

CosmosDB Checkpoint Directory

It is the file path where Cosmos stores the checkpoint data for Change feed.

Add Configuration

To add additional custom properties in key-value pairs.

Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Messaging Model

Option to select the messaging model as either Topic or Queue.

Queue Name

The queue name should be specified.

Durable

Durable name should be specified.

Number of Consumers

Number of consumers should be specified.

Batch Size

Maximum number of records processed at one time. Keep values up to the Data Collector maximum batch size.

Connection Timeout

Connection timeout in milliseconds should be specified. 2000 by default.

Connection Retries

Maximum number of connection retries allowed should be specified. 3 by default.

HDFS Connection

Container connection name for creating connection.

Data Type

Two types of data type are supported:

EBDCIC: Extended Binary Coded Decimal Interchange Code, is an 8 bit character encoding used on IBM mainframes and AS/400s.

TEXT: Text files are also accepted as data type.

Generate Record ID

Every generated record will have a primary record id corresponding to every entry.

HDFS Copy Book Path

CopyBook is Schema file for Cobol. This is where you can use the Scope Variable using @. To know more about the same, read about Scope Variable.

HDFS Data Path

The file path in HDFS from where the data will be read.