Externalize ETL Component Configurations
Configure ETL component(s) externally and parameterize their values.
Add an External Store
As a prerequisite to externalizing ETL component configurations, you need to first configure an external store in your project.
From Settings, click the External Configurations option.
Select the type of external store where the configurations or schema to be provided externally are stored. Supported types are:
S3
JDBC
RDS
Add S3 as External Store
Configuration details:
| Field | Description |
|---|---|
| Store Type | Select S3 as the external store type. |
| Store Connection | Choose an existing connection or create a new one, providing access to the external configuration location. To see connection details, refer to Amazon S3 Connection. |
| Bucket Name | Select or type the name of S3 bucket that contains the external configuration directory. |
| Path | Provide the path in the bucket up to (but not including) the folder that contains schema.json and config.json. That folder will be selected separately as the configuration key in the component. |
Example:
For an S3 bucket etl-config-bucket containing schema.json and config.json under the directory path customer/onboarding/US, set the S3 Path as customer/onboarding.
The final folder US should be selected as the Configuration Key in the ETL component to load the specific configuration.
Add JDBC as External Store
Configuration details:
| Field | Description |
|---|---|
| Store Type | Select JDBC as the external store type. |
| Store Connection | Choose an existing connection or create a new one, providing access to the external configuration location. To see connection details, refer to JDBC Connection. |
| Schema Name | Select the database schema name where the external configuration table resides. |
| Table Name | Select the name of the table containing the external configuration. The table must include columns: extkey, extschema, and extconfig. |
Example: For a table named external_etl_config with the following TEXT columns:
extkey: Serves as the configuration key used to retrieve the external ETL configurations.
extschema: Contains the schema definition for the ETL components in JSON format.
extconfig: Contains the configuration details for the ETL fields, also in JSON format.
Add RDS as External Store
Configuration details:
| Field | Description |
|---|---|
| Store Type | Select RDS as the external store type. |
| Store Connection | Choose an existing connection or create a new one, providing access to the external configuration location. To see connection details, refer to RDS Connection. |
| Schema Name | Select the database schema name where the external configuration table resides. |
| Table Name | Select the name of the table containing the external configuration. The table must include columns: extkey, extschema, and extconfig. |
Example: For a table named external_etl_config with the following TEXT columns:
extkey: Serves as the configuration key used to retrieve the external ETL configurations.
extschema: Contains the schema definition for the ETL components in JSON format.
extconfig: Contains the configuration details for the ETL fields, also in JSON format.
Update External Store Configuration
After adding the external store configuration, click UPDATE to save the changes.
Disable External Store Configuration
Select the Store Type as None and click UPDATE to disable the external store configuration.
Externalize Schemas and Configurations
Users can set up configurations and schemas for ETL Applications from an external service. These configurations and schemas are stored outside the application and allow the ETL Applications to adjust and operate based on the external settings.
Steps to Externalize Schemas and Configurations
While creating or editing an ETL Application, you can externalize the schema and configuration of the components.
Right-click the component for which you want to externalize the schema and/or configuration.
Click the Externalize option from the context menu.
The external store details configured for the project will be displayed.
Based on preference, select the checkbox to Externalize Configurations and/or Externalize Schema.
Click the option Retrieve Configuration Keys to fetch the configuration keys from the configured external store.
Select the key that contains the external configuration and/or schema for this component.
For S3, this refers to the folder containing schema.json and config.json.
For a database, this is the value in the extkey column for the row that holds the schema and configuration JSON.
- Click APPLY to apply the external configuration and/or schema.
Upload a Component’s Configuration and Schema for Reuse
At the data preparation stage of an ETL application, you can upload a component’s configuration and schema in an external store for reuse in other ETL applications.
Once the design-time data of a component is displayed, along with the data preparation options available there will also be an option to Upload Configurations & Schema of that component to an external store configured in Gathr.
Download external schema and configuration of the component
Update the downloaded file as per requirement.
Upload it by providing a key to a configured external store.
Useful Links:
Add an External Store in Gathr:
Store an ETL component’s configurations and schema externally.
Retrieve stored configurations and schema to externalize an ETL component’s configurations and schema.
Download or Upload Configurations & Schema:
- Learn how to download a component’s configurations and schema from Gathr and upload them to an external store.
If you have any feedback on Gathr documentation, please email us!