File Writer Ingestion Target
This target is available only in Apache environment. A local file target can save data to local file system. Local file System is the File System where Gathr is deployed.
File Writer Target Configuration
Configure the target parameters that are explained below.
To add a File Writer target to your pipeline, connect it to a Data Source or processor, and right-click on it to configure.
| Field | Description |
|---|---|
| File Path | Path where the data file will be read/saved. |
| Output Fields | Fields to be included in the Output File. |
| Output Format | Format in which the output file will be saved. |
| Delimiter | Separator to separate different fields. |
| Checkpoint Storage Location | Select the check pointing storage location. Available options are HDFS, S3, and EFS. |
| Checkpoint Connections | Select the connection. Connections are listed corresponding to the selected storage location. |
| Checkpoint Directory | It is the path where Spark Application stores the checkpointing data. For HDFS and EFS, enter the relative path like /user/hadoop/, checkpointingDir system will add suitable prefix by itself. For S3, enter an absolute path like: S3://BucketName/checkpointingDir |
| Time-Based Check Point | Select checkbox to enable timebased checkpoint on each pipeline run i.e. in each pipeline run above provided checkpoint location will be appended with current time in millis. |
| Output Mode | Mode in which File writer will run. Output mode to be used while writing the data to Streaming target. Select the output mode from the given three options: Append: Output Mode in which only the new rows in the streaming data will be written to the sink. Complete Mode: Output Mode in which all the rows in the streaming data will be written to the sink every time there are some updates. Update Mode: Output Mode in which only the rows that were updated in the streaming data will be written to the sink every time there are some updates. |
ADD CONFIGURATION: Enables to configure additional properties.
If you have any feedback on Gathr documentation, please email us!