Unified Processor

Unified Schema Generator is used for generating unified schema for different type of input events. these input events can have input files with different schema.

Processor Configuration #

Provide the Unified Processor’s configuration details as mentioned below.

Connection Type #

Select connection type. Available options are: HDFS and S3.

Connection Name #

Connection name for creating connection.

Schema File Path #

Path of unified schema file. For S3 please enter an absolute path like: s3a://bucketName/filePath

Extra Fields Expression File Path #

Path of extra fields properties file. For s3 please enter an absolute path like: s3a://bucketName/filePath

Ignore Action If Empty #

This option gives you control to ignore the child component actions if current micro batch does not have any data.

Output Mode #

Output mode specifies how to write the data. Available options are Append, Complete, Update

Checkpoint Storage Location #

Select the checkpointing storage location. Available options are: HDFS, S3, GCS.

Checkpoint Connections #

Select connections that are listed corresponding to the selected storage location.

Check Point Directory #

It is the HDFS path where the Spark application stores the checkpoint data. For S3 and GCS enter an absolute path. Example: S3://bucketName/checkpointgDir and gs://bucketName/checkpointDir

Time-based Check Point #

Select checkbox to enable timebased checkpoint on each pipeline run i.e., in each pipeline run the above provided checkpoint location will be appended with the current time in millis.

Enable Trigger #

Trigger defines how frequently a streaming query should be executed.

Trigger Type #

Supported Trigger Types are:

One-Time Micro Batch: A trigger that process only one batch of data in a streaming query then terminates the query.

Fixed Interval Micro-Batches: A trigger policy that runs a query periodically based on an interval in processing time.

Processing Time #

Specifies the time interval that governs the trigger policy.

Add Configuration: To add additional custom ADLS properties in a key-value pair.