Connection Name

All Kafka connections will be listed here. Select a connection for connecting to Advance Kafka.

Topic Name

Advance Kafka topic name where you want to emit data.

Partitions

Number of partitions to be created for a topic. Each partition is ordered, immutable sequence of messages that is contin­ually appended to a commit log.

Replication Factor

For a topic with replication factor N, Advance Kafka will toler­ate up to N-1 failures without losing any messages commit­ted to the log.

Output Format

Data Type format of the output message.

Advance Kafka Parti­tioner

Round Robin(Default): Advance Kafka uses the default partition mechanism to distribute the data.

Key Based: Advance Kafka partitioning is done on the basis of keys.

Custom: Enables to write custom partition class by imple­menting the com.streamanalytix.framework.api.spark.parti­tioner.SAXAdvance Kafka Partitioner interface. The partition method contains logic to calculate the destination partition and returns the target partition number

Enable TTL

Select the checkbox to enable TTL(Time to Live) for records to persist for that time duration

Output Mode

Output mode to be used while writing the data to Streaming emitter. Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trigger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggregation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emitter.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected. Processing Time is the trigger time interval in min­utes or seconds.

CheckPoint directory

Kafka offsets will be stored in the given checkpoint location.

Connection Name

All Cassandra connections will be listed here. Select a connection for connecting to Advance Kafka.

keySpace

Cassandra keyspace name. If keySpace name does not exist in Cassandra, then it will create new keyspace.

Replication Strategy

A replication strategy specifies the implementation class for determining the nodes where replicas are placed. Possible strategies are SimpleStrategy and Net­workTopologyStrategy.

Replication Factor

Replication factor used to make additional copies of data

Table Name Expres­sion

Casandra table name. If the table name does not exist on the keyspace then it will create a new table.

Consistency Level

Consistency level refers to how up-to-date and syn­chronized a row of Cassandra data is on all its replicas.

Consistency levels are as follows:

ONE

Only a single replica must respond.

TWO

Two replicas must respond.

THREE

Three replicas must respond.

QUORUM

A majority (n/2 + 1) of the replicas must respond.

ALL

All of the replicas must respond.

LOCAL_QUORUM

A majority of the replicas in the local datacenter (which­ever datacenter the coordinator is in) must respond.

EACH_QUORUM

A majority of the replicas in each datacenter must respond.

LOCAL_ONE

Only a single replica must respond. In a multi-datacen­ter cluster, this also guarantees that read requests are not sent to replicas in a remote datacenter.

Output Fields

Output messages fields

key Columns

A single/compound primary key consists of the parti­tion key and one or more additional columns that deter­mine clustering.

Output Mode

Output mode to be used while writing the data to Streaming emitter. Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trigger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggregation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emitter.

Save Mode

Save operation specifies how to handle existing data if present.

ErrorifExist: When saving a DataFrame to a data source, if data already exists, an exception is expected to be thrown.

Append: When saving a DataFrame to a data source, if data/table already exists, contents of the DataFrame are expected to be appended to existing data.

Overwrite: Overwrite mode means that when saving a DataFrame to a data source, if data/table already exists, existing data is expected to be overwritten by the contents of the DataFrame.

Ignore: Ignore mode means that when saving a Data­Frame to a data source, if data already exists, the save operation is expected to not save the contents of the DataFrame and to not change the existing data. This is similar to a CREATE TABLE IF NOT EXISTS in SQL.

Enable TTL

Select the checkbox to enable TTL (Time to Live) for records to persist for that time duration.

TTL Value

It will appear only when Enable TTL checkbox is selected.

Provide TTL value in seconds.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected. Processing Time is the trigger time interval in minutes or seconds.

CheckPoint directory

Location where checkpoint data is stored.

Add Configuration

Enables to configure additional cassandra proper­ties.

Implementation Class

Foreach implementation class to which control will be passed in order to process incoming data flow.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trig­ger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once. For example, queries with only select, where map, flatMap, filter, join etc will support Append mode.

Complete: The whole Result Table will be delivered to the emitter after every trigger. This is supported for aggregation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emitter.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

ADD CONFIGURATION

Additional properties can be added using ADD CON­FIGURATION link.

Load From Template

Select the template from the list.

Only those templates will be visible in the drop down list that are created globally or within that workspace.

Sync With Source Template

When a template is saved and is edited, this tab is reflected. If any change is made in source template, corresponding changes will reflect in Container’s configuration.

Connection Name

Name of the connection

Bucket Name

Couchbase Bucket name.

Select the Couchbase bucket that gets generated with the Couchbase connection.

Max Buckets per cluster is 10, by default.

Document Id

Unique document key for Couchbase to persist the docu­ment in Couchbase. If document id is not unique then docu­ment with the same id gets overridden every time.

Check Point Directory

It is the HDFS path where the Spark Application stores the checkpoint data.

Retention Enable

When selected, each newly created item lives for the num­ber of seconds specified by retention policy. After the expi­ration time is reached item will be deleted by Couchbase server.

Retention Policy

TTL for the document to be persisted in the Couchbase.

Output Fields

Fields in the message that needs to be a part of the output data.

Output Mode

Output mode to be used while writing the data to Streaming emitter. Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trigger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggregation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emitter.

Enable Trigger

Trigger defines how frequently a streaming query should be exe­cuted.

Processing Time

It will appear only when Enable Trigger checkbox is selected. Pro­cessing Time is the trigger time interval in minutes or seconds.

Component Mode

Determines whether the emitter will act as a streaming or a batch emitter.

Streaming: If the channel used in the pipeline is of type Streaming, select the component mode as Streaming and the emitter will act as a Streaming emitter.

Batch: If the channel used in the pipeline is of type Batch, select the component type as Batch and the emitter will act as a Batch emitter.

If Component Mode selected is Streaming, you will view the fields listed below:

Use Spark API

Select this checkbox if you wish to use Spark inbuilt API’s.For example, “memory” and “Kafka”.

If selected, you will view one additional field Sink Provider.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trigger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once. For example, queries with only select, where map, flatMap, filter, join etc will sup­port Append mode.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggregation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emit­ter.

checkpointLocation

Location where the checkpoint data is stored.

Sink Provider

Enables to use Spark native implementation and custom emitter for storing data.

In case of Spark native implementation, specify short name of the emitter where you wish to store data. For example, if you specify Kafka as Sink Provider, data will be stored in Kafka.

If you wish to use your own custom emitter, provide fully qualified name of the class which implements StreamSink­Provider and DataSourceRegister interfaces.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

If Component Mode selected is Batch, you will view only one field “Emitter Plugin”.

Emitter Plugin

Provide fully qualified class name which implements Strea­mAnalytix CustomSSEmitter interface.

You can download sample project from StreamAnalytix UI (Data Pipeline page) and refer SampleCustomEmitter class.

ADD CONFIGURATION

Enables to configure additional properties.

public class CustomSinkProvider implements StreamSinkProvider, DataSourceRegis­ter { 

public Sink createSink(SQLContext sqlContext, Map<String, String> options, Seq<String> partitionCols, OutputMode outputMode) {

return new MySink("mysink1");
}

public String shortName() {
return "mysink";
}

}

public class MySink implements Sink {

String message;

public MySink(String msg) {
message = msg;
}

public void addBatch(long batchId, Dataset<Row> dataset) {
ds.write().format("json").option("path", "localhost:/user/sax/blankTest/custom/data").save();

 

}

Connection Name

All ElasticSearch connections will be listed here.Select a connection for connecting to ElasticSearch.

Output Message

Output message which is to be indexed.

Index Nested JSON

Select the checkbox if nested JSON fields are to be indexed.

If unchecked, three additional fields are populated: Index Number of Shards, Index Replication Factor and Output Fields.

If selected, these three fields will be hidden and follow­ing note be displayed.

“Index will be created with Elastic Search default 5 shards and 1 replication factor.”

Note: If this checkbox is selected, you can use Curly Brackets to avail schema fields in the Index name.(shown below in the diagram)

Index Number of Shards

Number of shards to be created in Index Store.

Index Replication Factor

Number of additional copies of data.

Index Name

Specify the index name where data is to be indexed.

Verify dynamic index creation works only when below condition is fulfill.

 1. Index Nested JSON check box should be selected on ES emitter.

 2. Action.auto_create_index: true should be set in Elasticsearch cluster.

3. The field data should always be in lower case other­wise pipeline would fail.

Check Point Directory

HDFS Path where the Spark application stores the checkpoint data.

ID Field Name

Specify a name for the generated ID field.

ID Generator Type

Enables to generate the ID field.

Following type of ID generators are available:

UUID: Universally unique identifier.

Field Values based: In this case, ID is generated by appending the values of selected fields.

If you select this option then an additional field – “Key Fields” will be displayed, where you need to select the fields you want to combine. The fields will be appended in the same order as selected on the user interface.

Custom: In this case, you can write your custom logic to create the ID field. For example, if you wish to use an UUID key but want to prefix it with “HSBC”, then you can write the logic in a Java class.

If you select this option then an additional field - “Class Name” will be displayed on user interface where you need to mention the fully qualified class name of your Java class.

You can download the sample project from the “Data Pipeline” landing page and refer Java class com.yourcompany.custom.keygen.SampleKeyGener­ator to write the custom code.

Output Fields

Output message fields.

Connection Retries

Number of retries for component connection. Possible values are -1, 0 or positive number. -1 denotes infinite retries.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Append Mode: Only the new rows added to the Result Table since the last trigger will be delivered to the emit­ter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be out­put only once.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

ADD CONFIGURATION

Enables to configure additional ElasticSearch proper­ties.

Connection Name

All HBase connections will be listed here.Select a connec­tion for connecting to HBase.

Table Name Expression

Javascript expression used to evaluate table name.

The keyspace will be formed as ns_+{tenanatId}. For exam­ple, ns_1

Compression

Provides the facility to compress the message before stor­ing it. The algorithm used is Snappy.

when selected true, enables compression on data

Region Splitting Definition

This functionality defines how the HBase tables should be pre-split. The default value is ‘No pre-split’. The supported options are:

Default: No Pre-Split- Only one region will be created ini­tially.

Based on Region Boundaries: Regions are created based on given key boundaries. For example, if your key is a hexa­decimal key and you provide a value ‘4, 8, d’, it will create four regions as follows:

1st region for keys less than 4

2nd region for keys greater than 4 and less than 8

3rd region for keys greater than 8 and less than d

4th region for keys greater than d

Encoding

Data encoding type either UTF-8 (base encoding) or BASE 64(64 bit encoding).

Row Key Generator Type

Enables to generate the custom row key.

Following type of key generators are available:

UUID: Universally unique identifier.

Field Values Based: In this case, key is generated by appending the values of selected fields.

If you select this option then an additional field – “Key Fields” will be displayed where you need to select the keys you want to combine. The keys will be appended in the same order as selected on the user interface.

Custom: In this case, you can write your custom logic to create the row key. For example, if you want to use an UUID key but want to prefix it with HSBC, then you can write the logic in a Java class.

If you select this option then an additional field - “Class Name” will be displayed on UI where you need to mention the fully qualified class name of your Java class. You can download the sample project from the “Data Pipeline” land­ing page and refer Java class “com.yourcompany.cus­tom.keygen.SampleKeyGenerator” to write the custom code.

Column Family

Specify the name of column family that will be used while saving your data in a HBase table.   

Output Fields

Fields in the message that needs to be a part of the output message.

Replication

Enables to copy your data on underlying Hadoop file sys­tem. For example, if you specify “2” as Replication, then two copies will be created on HDFS

Ignore Missing Values

Ignore or persist empty or null values of message fields in emitter.

when selected true, ignores null value of message fields.

Connection Retries

The number of retries for component connection. Possible values are -1, 0 or positive number. -1 denotes infinite retries

Delay Between Connection Retries

Defines the retry delay intervals for component connection in millis.

Enable TTL

Specifies the life time of a record.when selected, record will persist for that time duration which you specify in TTL field text box.

TTL Value

Provide TTL value in seconds.

Checkpoint Directory

Location where the checkpoint data is stored.

Output Mode

Output mode to be used while writing the data to Stream­ing sink.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trigger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggregation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emit­ter.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

ADD CONFIGURATION

Enables to configure additional properties.

Connection Name

All Hive connections will be listed here. Select a con­nection for connecting to Hive.

Check Point Directory

HDFS path where Spark application stores checkpoint data.

Database Name

HIVE database name.

Table Name

HIVE table name.

Output Fields

Fields in the schema that needs to be a part of the out­put data.

Partitioning Required

Check this option if HIVE table is to be partitioned. You will view Partition List input box.

Format

PARQUET: Columnar storage format and includes com­pression of file.

TEXT: Stores information as plain text.

Note: Space’ ’ Delimiter is not supported in TEXT format.

Delimiter

Message field separator.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Append Mode: Only the new rows added to the Result Table since the last trigger will be delivered to the emit­ter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be out­put only once.

Save Mode

Save operation specifies how to handle existing data if present.

ErrorifExist: When saving a DataFrame to a data source, if data already exists, an exception is expected to be thrown.

Append: When saving a DataFrame to a data source, if data/table already exists, contents of the DataFrame are expected to be appended to existing data.

Overwrite: Overwrite mode means that when saving a DataFrame to a data source, if data/table already exists, existing data is expected to be overwritten by the contents of the DataFrame.

Ignore: Ignore mode means that when saving a Data­Frame to a data source, if data already exists, the save operation is expected to not save the contents of the DataFrame and to not change the existing data. This is similar to a CREATE TABLE IF NOT EXISTS in SQL.

Replication

Enables to copy your data on underlying Hadoop file system. For example, if you specify “2” as Replication, then two copies will be created on HDFS.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.g

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

Add Configuration

Enables to configure custom properties.

Connection Name

All JDBC connections will be listed here. Select a con­nection for connecting to JDBC.

Message Name

Message used in the pipeline

Is batch Enable

Parameter used to batch multiple messages.

batch Size

Number of messages to be batched together.

Schema Name

Existing database Schema Names whose tables are fetched.(for MSSQL and POSTGRES)

Table Name

Existing database tablename whose schema is to be fetched.

Connection Retries

Number of retries for component connection

Delay Between Connec­tion Retries

Defines the retry delay intervals for component con­nection in millis.

CheckPoint Directory

Location where the checkpoint data is stored.

Output Mode

Output mode to be used while writing the data to Streaming emitter.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trig­ger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggre­gation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the sink.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

ADD CONFIGURATION

Enables to configure custom properties.

Connection Name

All Kafka connections will be listed here.Select a connection for con­necting to Kafka.

Topic Name

Kafka topic name where you want to emit data.

Partitions

Number of partitions to be created for a topic. Each partition is ordered, immutable sequence of messages that is continually appended to a commit log.

Replication Factor

For a topic with replication factor N, Kafka will tolerate up to N-1 fail­ures without losing any messages committed to the log.

Producer Type

Specifies whether the messages are sent asynchronously or syn­chronously in a background thread. Valid Values are async for asyn­chronous send and sync for synchronous send

Output Format

Data type format of the output message.

Output Fields

Message fields which will be a part of output data.

Kafka Partitioner

Round Robin(Default): Kafka uses the default partition mechanism to distribute the data.

Key Based: kafka partitioning is done on the basis of keys.

Custom: Enables to write custom partition class by implementing the com.streamanalytix.framework.api.spark.partitioner.SAXKafkaParti­tioner interface.The partition method contains logic to calculate the destination partition and returns the target partition number.

Enable TTL

Select the checkbox to enable TTL(Time to Live) for records to per­sist for that time duration.

Checkpoint Direc­tory

Location where the checkpoint data is stored.

Output Mode

Output mode to be used while writing the data to Streaming emitter.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trigger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggregation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emitter.

Enable Trigger

Trigger defines how frequently a streaming query should be exe­cuted.

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

Connection Name

All Kinesis connections will be listed here. Select a Kinesis connection for connecting to Kinesis.

Stream Name

Name of the Kinesis Stream.

Shard Count

Number of shards required to create the stream, in case stream is not present.

Region

Name of the region. For example, us-west-2

Partition Key

Used by Amazon Kinesis to distribute data across shards.

Output Format

Datatype format of the output.

Output Fields

Fields in the message that needs to be a part of the out­put data.

Checkpoint Directory

Location where the checkpoint data is stored.

Output Mode

Mode to be used while writing the data to Streaming sink.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trig­ger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggre­gation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emitter.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

ADD CONFIGURATION

Enables to configure additional properties.

Connection Name

All Mqtt connections will be listed here. Select a con­nection for connecting to Mqtt.

Queue Name/Topic Name

Queue or topic name to which messages will be pub­lished.

Output Format

Datatype format of the output.

JSON and Delimited.

If Delimited is selected, provide a delimiter as well.

Output Fields

Fields in the message that needs to be a part of the out­put data.

Check Point Directory

It is the HDFS path where the checkpoint data is saved.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trig­ger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggre­gation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the sink

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

Trigger time interval in minutes or seconds.

ADD CONFIGURATION

Enables to configure additional properties.

Connection Name

All Native HDFS connections will be listed here.Select a connection for connecting to HDFS.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Append Mode: In this mode, the new rows added to the Result Table since the last trigger will be delivered to the sink. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Save Mode

Save operation specifies how to handle existing data if present.

ErrorifExist: When saving a DataFrame to a data source, if data already exists, an exception is expected to be thrown.

Append: When saving a DataFrame to a data source, if data/table already exists, contents of the DataFrame are expected to be appended to existing data.

Overwrite: Overwrite mode means that when saving a DataFrame to a data source, if data/table already exists, existing data is expected to be overwritten by the contents of the DataFrame.

Ignore: Ignore mode means that when saving a DataFrame to a data source, if data already exists, the save operation is expected to not save the contents of the DataFrame and to not change the existing data. This is similar to a CREATE TABLE IF NOT EXISTS in SQL.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

Trigger time interval in minutes.

HDFS Path

Directory path on HDFS where data has to be written.

Output Fields

Fields which will be part of output data.

Partitioning Required

if checked, table will be partitioned.

Partition List

Select the fields on which table is to be partitioned.

Output Type

Output format of the request.Supported formats are CSV and JSON.

Delimiter

Delimiter character is used to separate two fields.

Check Point Directory

HDFS path where Spark application stores the checkpoint data.

Block Size

Size of each block(in Bytes)   allocated in HDFS.

Replication

Enables to make additional copies of data.

Compression Type

Algorithm used to compress the data.

ADD CONFIGURATION

Enables to configure additional ElasticSearch properties.

Connection Name

All OpenJms connections will be listed here.Select a connection for connecting to the OpenJMS.

Queue Name

Queue or topic name to which messages will be pub­lished.

Output Format

Select the data format in which OpenJMS should write the data.

Output Fields

Select the fields which should be a part of the output data.

Output Mode

Mode to be used while writing the data to Streaming sink

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trig­ger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggre­gation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emitter.

Checkpoint Directory

Location where the checkpoint data is stored.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

It will appear only when Enable Trigger checkbox is selected.

Processing Time is the trigger time interval in minutes or seconds.

ADD CONFIGURATION

Enables to configure additional properties.

Connection Name

All RabbitMQ connections will be listed here.Select a connection for connecting to the RabbitMQ server.

Exchange Name

Rabbit MQ Exchange name

Exchange Type

Specifies how messages are routed through it.

Direct: Delivers message to queues based on a mes­sage routing key.

Fanout: Routes message to all of the queues that are bound to it.

Topic: Does a wildcard match between the routing key and the routing pattern specified in the binding.

Exchange Durable

Specifies whether exchange will be deleted or remain active on server restart.

TRUE: Exchange will not be deleted if you restart Rab­bitMQ server.

FALSE: Exchange will be deleted if you restart Rab­bitMQ server.

Routing Key

Select RabbitMQ routing Key where data will be pub­lished.

Queue Name

RabbitMQ queue name where data will be published.

Queue Durable

Specifies whether queue will remain active or deleted on server restart.

TRUE: Queue will not be deleted if you restart Rab­bitMQ.

FALSE: Queue will be deleted if you restart RabbitMQ.

Output Format

Select the data format in which RabbitMQ should write the data.

Output Fields

Select the fields which should be a part of the output data.

Enable Message TTL

when selected, message will be discarded to TTL exchange specified.

Message TTL

Time to live in seconds after which message will be dis­carded to TTL Exchange specified.

TTL Exchange

Exchange to which message will be sent once time to live expires.

TTL Queue

Queue on which message will be sent once time to live expires.

TTL Routing Key

Routing key used to bind TTL queue with TTL exchange.

Check Point Directory

Location where the checkpoint data is stored.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trig­ger will be delivered to the emitter. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggre­gation queries.

Update: Only the rows in the Result Table that were sent since the last trigger will be delivered to the sink.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

Trigger time interval in minutes or seconds.

Add Configuration

Enables to configure additional RabbitMQ properties.

Stream Id

Exchange name on which streaming messages will be sent.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Select the output mode from the given three options:

Append Mode: This is the default mode, where only the new rows added to the Result Table since the last trig­ger will be delivered to the sink. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Complete: The whole Result Table will be delivered to the sink after every trigger. This is supported for aggre­gation queries.

Update: Only the rows in the Result Table that were updated since the last trigger will be delivered to the emitter.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.

Processing Time

Trigger time interval in minutes or seconds.

ADD CONFIGURATION

Enables to configure additional custom properties.

Connection Name

All Solr connections are listed here. Select a connection for connecting to Solr.

Across Field Search Enabled

Specifies if full text search is to be enabled across all fields.

Index Number of Shards

Specifies number of shards to be created in index store.

Index Replication Fac­tor

Specifies number of additional copies of data to be kept across nodes. Should be less than n-1, where n is the number of nodes in the cluster.

Index Expression

The MVEL Expression is used to evaluate the index name. This can help you leverage field based partition­ing.

For example consider the expression below:

@{'ns_1_myindex' + Math.round(<MessageName>.time­stamp 3600*1000))}

Here a new index will be created with one-hour time range and data will be dynamically indexed based on field whose field alias name is 'timestamp'.

Routing Required

This specifies if custom dynamic routing is to be enabled. If enabled, a routing policy json needs to be defined as shown in the below figure.

ID Generator Type

Enables to generate the ID field.

Following type of ID generators are available:

UUID: Universally unique identifier.

Field Values based: In this case, ID is generated by appending the values of selected fields.

If you select this option then an additional field – “Key Fields” will be displayed, where you need to select the fields you want to combine. The fields will be appended in the same order as selected on the user interface.

Custom: In this case, you can write your custom logic to create the ID field. For example, if you wish to use an UUID key but want to prefix it with “HSBC”, then you can write the logic in a java class.

If you select this option then an additional field - “Class Name” will be displayed on user interface where you need to mention the fully qualified class name of your Java class.

You can download the sample project from the “Data Pipeline” landing page and refer Java class com.yourcompany.custom.keygen.SampleKeyGener­ator to write the custom code.

Output Fields

Fields of the output message.

Ignore Missing Values

Ignore or persist empty or null values of message fields in emitter.

Connection Retries

Number of retries for component connection. Possible values are -1, 0 or positive number. -1 denotes infinite retries.

If Routing Required =true, then:

Routing Policy - A json defining the custom routing pol­icy. Example: {"1":{"company":{“Goo­gle”:20.0,"Apple":80.0}}} where 1 is the timestamp after which custom routing policy will be active, 'company' is the field name and the value 'Google' takes 20% shards and value 'Apple' takes 80% shards.

Delay Between Connec­tion Retries

Defines the retry delay intervals for component con­nection in milliseconds.

Enable TTL

when selected, message will be discarded to TTL exchange specified.

Checkpoint Directory

Location where the checkpoint data is stored.

Output Mode

Output mode to be used while writing the data to Streaming sink.

Append Mode: In this mode, the new rows added to the Result Table since the last trigger will be delivered to the sink. This is supported for only those queries where rows added to the Result Table are never going to change. Hence, this mode guarantees that each row will be output only once.

Enable Trigger

Trigger defines how frequently a streaming query should be executed.