https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/Recent content in Data Ingestion onHugo -- gohugo.ioSun, 11 Dec 2022 19:37:55 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/data_ingestion_applications/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/data_ingestion_applications/Data Ingestion is one of the most crucial steps of the data handling process. It involves transferring data from a source to a target after applying any desired transformations. Gathr offers a very user-friendly interface to create data ingestion applications on its SaaS platform and trigger them on either Gathr cluster or user preferred clusters that are registered with Gathr using Compute Setup → Useful references for Data Ingestion Applications:https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/create_data_ingestion_application/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/create_data_ingestion_application/Gathr supports a wide range of sources and targets that can be configured while creating an ingestion application. There are prebuilt operators and functions that can help you transform your data as desired before you transfer it to the supported target data warehousing solutions. Notes: The steps given below captures only a basic flow on how to create an ingestion application. You can create an ingestion application with any source and target as per requirement by following similar configuring steps as described below.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_data_sources/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_data_sources/Gathr data sources are built-in components, that you can configure to read data from a source location. Data source could be a NoSQL store, a flat file, web data, relationa- database, or a third party BI tool. Typically, you start creating an application by selecting a Data Source for reading data. Data Source can also help you to infer schema from Schema Type directly from the selected source or by uploading a sample data file.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_transformations/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_transformations/The supported Ingestion Transformations are listed below: Aggregation Processor→ Cache Processor→ Data Quality Processor→ Decryption Processor→ Drop Processor→ Select Processor→ Encryption Processor→ ExpressionEvaluator Processor→ ExpressionFilter Processor→ Field Flattener Processor→ HTTP Processor → Masking Processor→ Pivot Processor→ Repartition Processor→ Rename Processor→ SQL Processor→https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_targets/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_targets/Targets define the destination stage of a data ingestion application in Gathr. Targets could be a NoSQL store, indexer, relational database, or a third party BI tool. After configuring a data source and required transformations, you will need to select a target for emitting the transformed data. Gathr supported data ingestion targets are listed below: ActiveCampaign Ingestion Target→ Acumatica Ingestion Target→ ADLS Ingestion Target→ Airtable Ingestion Target→ Amazon Athena Ingestion Target→https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/configure_ingestion_application/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/configure_ingestion_application/On the Configure page, provide the details for each field as described below: Application Name: A unique name will be generated for the ingestion application. You can edit the name as per your choice. Data Ingestion applications should start with an alphabet and may include alphanumeric characters and special characters such as !@$-;:()-_?=~/*<>’ in their names. Description: Option to add supporting description about the ingestion application that is being created.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_application_details/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_application_details/The details tab shows the following information: Current Job Progress # The Job Progress Tracking feature allows you to monitor the progress of your active tasks directly from the user interface. It offers valuable insights into the status of ongoing processes, ensuring you can stay informed about the job’s progress. 👉 The Job Progress Tracking feature will not show current job progress if an application is running on a registered cluster.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_application_history/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_application_history/History tab will show the application trigger details, status and credit points utilized during each trigger. On expanding each entry, you can see the source, target and transformations used along with their record count and size details. Error Indicator: This gives you an indication of the total number of Errors for the pipeline run. For more details, please check the Error Search tab. 👉 Error indicators are available for the last 3 runs of a pipeline.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_application_error_search/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/data_ingestion/ingestion_application_error_search/Error search tab will show the application errors that can be filtered with either a time range or a time duration search.