https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/Recent content in Transformations onHugo -- gohugo.ioSun, 11 Dec 2022 19:37:55 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/about_transformations/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/about_transformations/Gathr has built-in operators for processing the data by performing various transformations and analytical operations. Each operator is responsible for performing a specific action. You can select multiple operators and stitch them together to implement the business logic in an application. In the sequence of creating an ingestion application when you reach the transformations section, you can choose to add or edit transformations before proceeding to the target configuration. The transformations page has an intuitive design, you can choose to organize the source columns as well as data by adding value to it and add the supported transformations either on the preview page itself or on the canvas by clicking Add Transformations option.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/advanced_sort/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/advanced_sort/Advanced Sort processor sorts the output fields before further processing. Each field can then be sorted in Ascending or Descending order. You can specify multiple sort criteria on multiple fields to achieve secondary/tertiary and n-level sorting on input data. Processor Configuration # The Column Order can be selected and corresponding to the same is the Sort Order, which can be set to either Ascending and Descending. The output of this processor will arrange the column data in the selected sort order.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/aggregation/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/aggregation/The aggregation operator performs operations like min, max, average, sum, and count. Example: For an IOT device that generates an event comprising three fields: id, time, and action. You can calculate how many actions for each event happened in a window of 15 seconds. You can use the Aggregation (COUNT) operation to calculate the number of actions of each type from total events received in a window of 15 seconds.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/alert-processor/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/alert-processor/Alert processor notifies users about the occurrence of a specific event during pipeline execution as configured by the user. Alert Processor Configuration # To add an Alert processor to your pipeline, drag the processor to the canvas and right-click on it to configure. Field Description Alert Name Specify a name for the alert. The alert is triggered with this name when the criteria is met. Alert Type Depending upon the severity of the alert, you can provide an Alert Type.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/binary_avro_parser_processor/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/binary_avro_parser_processor/Convert binary AVRO data into JSON strings. Simply drag the processor onto the canvas and click to configure it. Binary Data Column # Specify the binary data column to be parsed in string format. Is Input Base64 Encoded # Whether Input Column data is Base64 encoded or not. Avro Schema # Provide the Avro schema details in JSON format to convert data from binary format. Parsed Column Name # Provide the new column name to store the parsed data and convert from Binary to String format to produce the JSON string.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/cache/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/cache/Processor Configuration # Storage Level # Select the storage for cache memory. The storage level option are: MEMORY ONLY: Stores data in memory without serialization. MEMORY ONLY 2: Stores data in memory with higher replication. MEMORY ONLY SER: Stores serialized data in memory. MEMORY ONLY SER 2: Stores serialized data in memory with higher replication. DISK ONLY: Stores data on disk without using memory. DISK ONLY 2: Stores data on disk with higher replication.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/custom-processor/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/custom-processor/This processor enables you to create a custom business logic. Implement the com.Gathr.framework.api.processor.JSONProcessor interface and provide custom business logic in implemented methods. You will be able to perform operations on datasets that are not supported by other processors. Prerequisites To use a custom processor, you need to create a jar file containing your custom code. Then upload the jar file as a registered component. To write a custom logic for your custom processor, download the sample component from Register Entities > Components page.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/data-cleansing/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/data-cleansing/The Data Cleansing Processor is used to cleanse the dataset using the metadata. Processor Configuration # Columns included while Extract Schema Column names that are mentioned in processor will be used in the data cleansing process. Connection Type Select connection type from where the user wants to read the metadata files. The available connection types are RDS and S3. Connection Name Select the connection name to fetch the metadata file.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/data_quality/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/data_quality/Data Quality operator defines the quality rules for data and performs necessary action on the faulty data. It performs series of validations on individual data field along with actions on functions, in case if applied validation fails. Varieties of validations are available for each Data Type such as Not Null, Contains, and Starts With and Ends With. Data Quality operator has seven categories of Validation functions, one for each Data Type.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/decoder/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/transformations/decoder/Fields are decoded to hide sensitive data. Base64 is a group of binary-to-text encoding schemes that represent binary data in an ASCII string format by translating it into a radix-64 representation. Base64 encoding schemes are commonly used when there is a need to encode binary data that needs be stored and transferred over media that are designed to deal with textual data. This is to ensure that the data remains intact without modification during transport.