ETL Reload Data
Utilize Reload Data option to ensure data consistency and accuracy while designing applications.
It allows you to validate data propagation from source components downstream during design time.
How Reload Data Works
When using Reload Data, the system loads the last saved sample schema from cache memory for data sources.
For transformations and emitters, it processes and fetches the latest results.
If a component’s Data Preparation Window is used to reload data, the system fetches data directly from the actual input source:
If a file upload was selected, data is reloaded from the uploaded file.
If fetch from source was configured, data is reloaded from the source.
Error Handling During Reload Data
If an error is detected during reload data, an alert message will appear along with an Incident ID.
The Incident ID can be shared with Gathr Support to resolve the issue.
Managing Data Reload Progress
Closing the Data Reload Progress window stops the reload process.
Data reload will run only for fully configured components, meaning their configurations must be completely provided for all the required steps.
Unconfigured components will be marked in red with an error in the Data Reload Progress window.
Reloading Data for Individual Components
- Each component’s data can be reloaded individually by clicking the eye icon displayed on top of the respective component.
Resolve Component Level Schema Changes
Ensure that schema changes due to configuration modifications are seamlessly propagated to subsequent components, reducing manual intervention and potential execution issues.
If schema changes occur in the application due to configuration modifications, you can propagate them to downstream components with a one-click operation.
Steps to Resolve Schema Changes
When a configuration change alters a previously detected schema while editing the application, follow these steps:
- Upon completing the configuration of such a component, an indication will appear on the impacted component stating that changes have been detected.
- The affected component will display an alert about the detected schema changes.
- To propagate the changes, you can:
Right-click on the component and select Propagate Downstream.
Use the Resolve Schema Changes option from the design canvas to automatically propagate changes to downstream components.
- A confirmation dialog will appear, allowing you to choose how to handle targets:
Check the Manually configure targets options to mark the target(s) as stale for later reconfiguration.
Uncheck the Manually configure targets options to not mark the target(s) as stale.
For the stale targets columns with schema changes should to be added or removed manually.
Ignoring schema updates on Target may impact the application’s execution.
Session Preference for Editing an Application After Login
If the Auto Create Session preference is disabled, you will be prompted to either Start a new session or Continue without a session when editing an application.
It is recommended to edit an application within an active session to accommodate any schema changes. Editing without a session may impact the application’s execution.
Steps to Start a New Session Before Editing Application
- Click on Start a new session.
The available engines will be displayed at the bottom-left corner.
- Click on the preferred engine to establish a new session.
Once the session is active, you can proceed with editing the application.
If you have any feedback on Gathr documentation, please email us!