Templates are the pipelines in which user can add variables and parameters. Instance(s) are created within the template once value(s)/parameter(s) are added within a template.
In the data pipeline listing page, user can create a template by clicking at the Template tab.
In the template listing page, click the +icon to Create New Template. User can also upload a template by clicking at the Upload Template icon available on the template listing page.
Note: A template can also be create by clicking at the ellipses of a particular pipeline tile and click Create Template.
The icons to Create New Template and Upload Template are shown below:
To create a template, user will have to create/use an existing configured data pipeline.
On the canvas, choose the preferred data source from within the Components tab available at the top right side of the canvas.
Configure the selected data source and add the preferred processor(s) to the create the pipeline.
Once the processor(s) in the pipeline are configured, emit the data in a preferred sink/emitter.
For more on pipeline creation, see Data Pipelines and for Components configuration details refer to Data Sources, Processors and Emitters.
Once all the components are configured, then the user can add parameter(s) to create a template.
Under the Parameters tab, click +Add Parameter.
Create the parameter by providing the below fields:
Field | Description |
|---|---|
Name | Provide a name to create the new parameter. |
Value | Provide the parameter value. To create parameter, ensure that the value to be provided here is same as provided in the configuration of the existing pipeline in order to create a template. |
Default Value | Provide the default value for the new parameter. |
Click Create to complete the parameter creation. Once the parameter is created, it can be used in configured components.
Right-click the component and select the Parameterize option. In the parameterize table, select the variable from the drop-down option available under the parameterize column of the table.
Validate the applied parameter. The configuration value and the parameter value provided in the parameterize column must be same. Click Apply to confirm the changes.
Once the parameterization is done, click Save to save the Template.
Note: A template can be created from an existing pipeline by clicking at the ellipses on the available pipeline tile and then click Create Template as shown below:
Once the template is created, user can perform various actions by clicking the ellipses of the template tile as explained below:
Field | Description |
|---|---|
Edit | Click this option to edit the template. |
View Instance | Click this option to view all the instance(s) that are created within the template. Note: User can create or upload an instance from the Instance(s) listing page. Enter the instance listing page by clicking the View Instance option available upon clicking the ellipses of the template tile. Click Create New Instance. Similarly, option to upload an instance is available. |
Create Instance: Click this option to create an instance within the template. Provide the below details for instance creation: | |
Name | Provide a Unique name of the instance to be create. Note: The template name will be prefixed before the instance name. |
Parameter Details | Provide value(s) of the parameter for which the instance has to be created. Note: Under the Component Using column, the variables used in the listed components are mentioned. Click Create to create the instance. |
The other actions available on the template tile are given below: | |
Download | Click this option to download the template. |
Delete | Click this option to delete the template. |
Once the instance is created, user can perform various actions by clicking the ellipses of the instance tile as explained below:
Field | Description |
|---|---|
Edit | Click this option to edit the instance. |
Configure Job | Click this option to configure job. It is similar to configure job in the cloud environment. |
Inspect | Click this option to inspect the instance. The data source(s) with configuration details will be listed which can be updated as per requirement. |
Download | Click this option to download/export the instance. |
Delete | Click this option to delete the instance. |
Schedule | Click this option to schedule the instance. |
Monitor | Click this option to monitor the instance. |
Run | Click this option to run the instance. |
User has the option to configure the component(s) externally and parameterize the values set during configuration.
The configuration details are mentioned below:
Field | Description |
|---|---|
Configure | Option to configure the component. |
Externalize The user can add configuration and schema to the components of the pipelines externally. The External configurations and schema will be fetched from a service deployed externally and these pipelines will be able to inspect and run on the bases of external configurations and schema. The option to External Configuration are explained below: | |
External Configuration | Select the check box for enabling External Configuration option. |
Store | The external configuration of components can be done using the below types of store: - File System - Database (RDS) - S3 Note: The user can add external configuration to the components of the pipelines. Externally configured pipelines will be able to inspect and run on the bases of the external configurations. The External configurations will be fetched from a service, deployed externally. |
Keys | Select the relevant key of the selected store from the drop-down list. This is a dynamic field as the user can create a customized key as per requirement. The keys contain the configuration and schema of the component. |
External Schema | Select the check box for enabling External Schema option. |
Store | The external schema of components can be configured using the below types of store: - File System - Database (RDS) - S3 Note: The user can add external schema to the components of the pipelines. The External schema will be fetched from a service, deployed externally. |
Keys | Select the relevant key of the selected store from the drop-down list. This is a dynamic field as the user can create a customized key as per requirement. The keys contain the configuration and schema of the component. |
Note: Once the inspect of a particular component is done only then the user will be able to download external schema and configuration of the component. The user can download the zip configuration and schema file from the component's configuration window as shown below: The downloaded file can be updated as per the user's requirement and the updated file can be further placed as external service. For more details on how to configure external service, read the README file available on the external service. Other options available on the component upon right-clicking it are explained below: | |
Inspect | Option to inspect the component. |
Parameterize | The Parameterize option can be used to change the value(s) with new parameter(s). Provide the parameterize value. User can Validate/Reset the value. |
Rename | Option to rename the component. |
Node Alias | An alias can be assigned to a template component. If a pipeline uses the template with this component as a connecting point, its alias will be preferred over the component name. If this template component is deleted, an option will be available while saving the template to substitute the alias point with a different template component, which will be considered as a new connecting point in the pipeline. This feature is further explained in Template as Component in a pipeline below. For details, refer to Node Alias. |
Delete | Option to delete the component from the pipeline. |
Template as Component in a Pipeline
In certain scenarios the user might require a pipeline with similar data flow pattern at multiple times. In case if the user requires to change a few processors in the existing pipeline, then the option of using Processor Group functionality is used. However, Gathr provides its users with option to change all components (source, processor, emitter) by using Template as Component feature.
A pipeline Template may contain multiple flows, therefore user will be allowed to select partial flow of template to be changed/edited of its pipeline.
Key Functionality:
- User can utilize a Template as channel, processor and emitter.
- User can choose the part of a template that is to be used in pipeline as per requirement.
- In case if a template is updated, the effective changes will reflect once the pipeline is started.
- User will be notified via. an alert/email whenever the changes in a template are applied.
Steps:
- On the Pipeline canvas, user has an option to utilize the listed templates that are created in the existing workspace within the pipeline.
- Drag and drop any data-source or use a template as a data-source, processor or emitter.
- As shown in the below image, for an instance; once any component (source/processor/emitter) is connected to the Template, then the window opens displaying the template view.
Note: To edit the template components connecting points for the inbound/outbound pipeline flow/entry or exit point, click on the Tem icon (marked in yellow)
Upon clicking the Temp icon, the template flow will appear.
The window appears reflecting the template’s components details. Now, select the component in the template that needs to be connected to the component on the pipeline canvas. i.e., Select a template component and the flow next to it will adjoin <pipeline_component_name>. For example, in the image, Union has been selected as an entry point and is connected with Azure Blob Batch, S3 Batch, Rabbit MQ, RabbitMQ components in the pipeline canvas.
Configure Template as Component
Click on the template to configure it.
Note: Union will be connected in the pipeline with the Azure Blob Batch component of the pipeline. The pipeline flow will connect with Union and prefixed components (i.e. HDFS) from the Template flow will be skipped as shown in the above reference image.
Customize Template
User has an option to customize the template. Right click on the template and select Customize option:
Field | Description |
|---|---|
Key | Name of the parameter. |
Value | Value associated with the parameter is reflected. The value can be edited by hovering over the value and clicking Edit. An editable window opens where the value can be edited and saved. |
Default Value | The default value of the parameter is reflected. |
Action | In case of external configuration, the key used will be reflected in red color. Download sample JSON to update the external configuration of the parameter. Please add <template_name> as prefix if template component name is used in the configuration. For an example, suppose an SQL processor is externalized and has query "select * from S3BatchChannel_101494" and s3 batch is template component then in that case you will have to replace S3BatchChannel_101494 with template name followed by '_'and S3BatchChannel_101494 so the resultant query will be "select * from <templatename>_S3BatchChannel_101494". |
On the template flow window, right-click the component that needs to be terminated or skipped from the template flow. The options available upon right-clicking are: Terminate Succeeding Flow, Terminate Preceding Flow, Skip Component.
The template components are given color coding as per their relationship with the components present in the pipeline.
Note: Carefully choose the template components to be skipped or terminated. Pipeline flow will be disrupted if any connecting points are skipped or deselected.
-User has an option to skip component. Template component can be skipped from data flow when used in pipeline as template as component.
Upon terminating the succeeding or preceding flow the terminated components will be grayed out while the skipped component will be reflected in brown color.
Color coding is as follows:
UI Legends | Description |
|---|---|
| The template component(s) reflected as red are terminated from the pipeline flow along with the chosen succeeding or preceding components in template flow. |
| The template component(s) in brown are skipped from the pipeline flow. However, its succeeding/preceding component(s) will be included in the flow unless you choose to terminate the succeeding/preceding component(s) or skip them from the flow. |
After the template has been customized as per requirement, select the processor(s) and join it with the template.
Once the template flow opens, select the exit point component that will mark as the end point of the template.
As shown below, Dedup is selected as an exit point and is connected with the pipeline component. You can Add new flow or replace an existing flow from the exit point (example: Dedup). If you select any existing flow, it will not be the part of your pipeline.
Select Email will be replaced as pipeline component. The pipeline flow will not consider suffixed components (i.e. Email Filter, Email Evaluator and others marked in red) from the Template flow as shown in the reference image. All the components in red color are terminated from the pipeline flow.
In certain scenario the option to pair two templates by connecting them in serial order is required. In that case, option to select another template and pair it with the pipeline is available. 
Upon clicking Pair, the window appears showing details of both Template 1 and Template 2. In Template 1 the exit point of the flow is selected and in Template 2 the entry point of the template flow is selected.
Note:
- Template as component and Processor Group cannot be used in the same pipeline.
- When user connects two templates in a sequence, then only a single stream of data can flow between the connected templates.
Entry point: Multiple inputs can be provided using Union, Join (only two) and SQL processors. Likewise, for Exit point: Multiple exit points can be provided using Add New Flow option.
Using SQL and Router processor as exit point, if user selects Add new flow option, then provide the query name.
In gathr, while using template as a component within a pipeline, you select connecting point to connect it with pipeline’s component.
For an example in certain scenario, you may want to update a particular template that is utilized in a pipeline. In that particular template you may want to remove the component that is a connecting point for the pipeline.
The pipeline in that case will not be able to find the missing/removed component in the updated template.
Thus, with Node Alias option, you can handle the scenario of missing template component by creating a Node Alias of the pipeline’s connecting point component.
Therefore, upon removing the component of the template which is also the connecting point of a pipeline, the user will have to assign new component to that alias so that the pipeline can connect to the newly defined component. Below are images explaining steps for Node Alias.
In template component, click upon the component and select Node Alias:
Provide details of Alias and click OK.
On the template canvas, click the ellipses at the top right of the screen to see the Alias List.
The Alias name and Component name is reflected in the Alias List window.
As shown in the below images, the selected component in Alias List can be connected with the template flow that will be used as connecting point in the pipeline. 
