https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/Recent content in ETL Pipelines onHugo -- gohugo.ioSun, 11 Dec 2022 19:37:55 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_applications/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_applications/Create and manage robust ETL applications with Gathr: Design data pipelines. Easily drag, drop, and connect processors to build applications. Monitor detailed metrics for each task and instance. Harness the efficiency of Gen AI capabilities. Simplify ETL application creation by providing natural language inputs to the Gathr’s AI Assistant. All ETL applications created within your project will be listed on the ETL Applications page. Manage ETL Listing Page # Here’s an overview of the actions available on the ETL listing page.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/create_etl_application/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/create_etl_application/Build ETL applications in Gathr. Effortlessly select data sources, processors, and emitters to extract, transform, and load data seamlessly. Utilize prebuilt operators for efficient data integration and transformation before loading to target solutions. Steps to Create an ETL Application # Steps to create an ETL application: Select any source or target and proceed by following the given steps. The configuration parameters for each connection, source, transformation (operators), function, and emitter are explained individually in the relevant topics.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_canvas/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_canvas/Using Gathr’s interactive ETL canvas, you can visually design and build your ETL applications, connect data sources, apply transformations, convert and interpret complex data through intuitive visual representations, and load data into target systems. Options that are available on the ETL applications design page are described below. Operators # In the ETL canvas, operators enable users to interact with Gathr’s diverse range of connectors. Operators are categorized into: Data Sources: The first operator in an ETL application.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_data_sources/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_data_sources/Gathr data sources are built-in components, that you can configure to read data from a source location. Data source could be a NoSQL store, a flat file, web data, relationa- database, or a third party BI tool. Typically, you start creating an application by selecting a Data Source for reading data. Data Source can also help you to infer schema from Schema Type directly from the selected source or by uploading a sample data file.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_data_targets/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_data_targets/Targets define the destination stage of an ETL application in Gathr. Targets could be a NoSQL store, indexer, relational database, or a third party BI tool. After configuring a data source and required transformations, you will need to select a target for emitting the transformed data. Gathr supported ETL Targets are listed below: ActiveCampaign ETL Target → Acumatica ETL Target → ADLS ETL Target → Advanced HDFS ETL Target →https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/provide_schema_for_etl_source/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/provide_schema_for_etl_source/Designing ETL applications in Gathr involves configuring schema settings to accurately process and transform data. 👉 Schema configuration applies to all data sources, but configuration parameters may vary based on the nature of the selected data source. Refer to tooltips and instructions relevant to your chosen data source for accurate configuration. Schema Type # Customize how your ETL app handles data on the Schema Type page. Select a method to design your application, specify data formats, and fine-tune options as needed.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/externalize-etl-component-configurations/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/externalize-etl-component-configurations/Configure ETL component(s) externally and parameterize their values. 👉 To use externalization feature, you need to first configure an external store in your project. Add an External Store # As a prerequisite to externalizing ETL component configurations, you need to first configure an external store in your project. From Settings, click the External Configurations option. Select the type of external store where the configurations or schema to be provided externally are stored.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_pre_actions/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_pre_actions/Pre-Actions in an ETL application are tasks executed before the main ETL process starts. They are crucial for setting up the environment, validating data, and preparing resources necessary for a successful ETL job execution. Ignore Action Error # If checked, the ETL application will continue execution even if an error occurs during the pre-action. Useful for non-critical actions or when errors can be handled downstream. Ignore While Execution # If checked, this action will be ignored while the ETL application is running, often used for debugging purposes or temporarily disabling pre-actions.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_post_actions/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_post_actions/There are certain actions that you may want to perform before the execution of target components. A Post Action tab is available at the emitters for all batch pipelines. These actions could be performed on the targets: SQL # Option to execute SQL at target level during runtime. The target records will get updated based on the query and connection details provided in the Post Action tab. Stored Procedure # Option to invoke a stored procedure or function stored in a database.https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_auto_inspection/Sun, 11 Dec 2022 21:01:49 +0530https://docs.gathr.ai/gathr-unlimited/7.6.0/docs/etl_pipelines/etl_auto_inspection/Utilize Reload Data option to ensure data consistency and accuracy while designing applications. It allows you to validate data propagation from source components downstream during design time. How Reload Data Works # When using Reload Data, the system loads the last saved sample schema from cache memory for data sources. For transformations and emitters, it processes and fetches the latest results. If a component’s Data Preparation Window is used to reload data, the system fetches data directly from the actual input source: