RabbitMQ ETL Target

The RabbitMQ emitter is used when you want to write data to RabbitMQ.

Data formats supported are JSON and DELIMITED (CSV, TSV, PSV, etc).

Target Configuration #

Configure the data emitter parameters as explained below.

Connection Name #

Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for RabbitMQ earlier. Or create one as explained in the topic - RabbitMQ Connection →

Use the Test Connection option to ensure that the connection with the RabbitMQ channel is established successfully.

A success message states that the connection is available. In case of any error in test connection, edit the connection to resolve the issue before proceeding further.

Exchange Name #

RabbitMQ Exchange name. RabbitMQ producers can only send messages to an exchange. The exchange pushes messages to queues from where consumers can read the message.

Exchange Type #

Specifies how messages are routed through Exchange.

Direct: A message goes to the queue(s) whose binding key exactly matches the routing key of the message.

Fanout: The fanout copies and routes a received message to all queues that are bound to it regardless of routing keys or pattern matching as with direct and topic exchanges. Keys provided will simply be ignored.

Topic: Topic exchanges route messages to queues based on wildcard matches between the routing key and something called the routing pattern specified by the queue binding. Messages are routed to one or many queues based on a matching between a message routing key and this pattern.

Exchange Durable #

Specifies whether exchange will be deleted or will remain active on server restart.

TRUE: Exchange will not be deleted if you restart RabbitMQ server.

FALSE: Exchange will be deleted if you restart RabbitMQ server.

Routing Key #

RabbitMQ routing key where data will be published. The routing key is used for routing messages depending on the exchange configuration.

Queue Name #

Queue name where data will be published.

Queue Durable #

Specifies whether queue will remain active or deleted on server restart.

TRUE: Queue will not be deleted if you restart RabbitMQ.

FALSE: Queue will be deleted if you restart RabbitMQ.

Delivery Mode #

Choose how RabbitMQ handles message storage.

Persistent (default): Stores messages on disk, delivers them during regular operation, and preserves them for delivery if RabbitMQ restarts.

Non-Persistent: Stores messages in memory, delivers them during regular operation, but loses them if RabbitMQ restarts.

Output Format #

Select the data format in which RabbitMQ should write the data. The supported formats are JSON and DELIMITED.

Output Fields #

Select the fields which should be a part of the output data.

Enable Message TTL #

RabbitMQ allows you to set TTL (time to live) for messages.

TTL Value #

Time to live in seconds after which message will be discarded to the specified TTL Exchange.

TTL Exchange #

Name of the Exchange, on which message will be sent once time to live expires.

TTL Queue #

Name of the Queue on which message will be sent once time to live expires.

TTL Routing Key #

Routing key used to bind TTL queue with TTL exchange.

Checkpoint Storage Location #

Select the checkpointing storage location. Available options are HDFS, S3, and EFS.

Checkpoint Connections #

Select the connection. Connections are listed corresponding to the selected storage location.

Checkpoint Directory #

It is the path where Spark Application stores the checkpointing data.

For HDFS and EFS, enter the relative path like /user/hadoop/, checkpointingDir system will add suitable prefix by itself.

For S3, enter an absolute path like: S3://BucketName/checkpointingDir

Time-Based Check Point #

Select checkbox to enable timebased checkpoint on each pipeline run i.e. in each pipeline run above provided checkpoint location will be appended with current time in millis.

Output Mode #

Output mode to be used while writing the data to Streaming sink.

Select the output mode from the given three options:

Append Mode:

Output Mode in which only the new rows in the streaming data will be written to the sink.

Complete Mode:

Output Mode in which all the rows in the streaming data will be written to the sink every time there are some updates.

Update Mode:

Output Mode in which only the rows that were updated in the streaming data will be written to the sink every time there are some updates.

Enable Trigger #

Trigger defines how frequently a streaming query should be executed.

Processing Time #

Trigger time interval in minutes or seconds.

Add Configuration: Additional properties can be added using this option as key-value pairs to configure additional RabbitMQ properties.

Post Action #

Post actions are available with batch data sources.

To understand how to provide SQL queries or Stored Procedures that will be executed during pipeline run, see Post-Actions →

Notes #

Optionally, enter notes in the Notes → tab and save the configuration.