Register Entities

Register Entities allows you to register custom components i.e. custom parsers, data sources and processors to be used in the pipelines.

There are three types of entities:

Each entity is explained below.

Register Components

Use Register Component to register a custom component (Channel and Processor) by uploading a customized jar. Those custom components can be used in data pipe­lines.

Register Components tab comes under Register Entities side bar option.

Download a sample jar from Data Pipelines page, customize it as per your require­ment, and upload the same on Register Components page.

Custom Code Implementation

StreamAnalytix allows you implement your custom code in the platform to extend functionalities for:

Channel: To read from any source.

Processor: To perform any operation on data-in-motion.

Custom code implementation allows importing custom components and versioning.

You can download a Maven based project that contains all the necessary StreamAnalytix dependencies for writing custom code and sample code for reference.

Pre-requisites for custom code development

1. JDK 1.7 or higher

2. Apache Maven 3.x

3. Eclipse or any other IDE

Steps for Custom Code Implementation

Build Custom Code

Provide all the dependencies required for the custom components in pom.xml avail­able in the project.

• Build project using mvn clean install.

• Use jar-with-dependencies.jar for component registration.

Register custom code

 

The list of custom components is displayed on the page shown below and the prop­erties are described below:

Perform following operation on uploaded custom components.

l Change scope of custom components (i.e. Global/Local)

l Change icon of custom components.

l Add extra configuration properties.

l Update or delete registered custom components.

Version Support (Versioning) in component registration

Register multiple versions of a registered component and use any version in your pipeline.

 

NOTE: If you have used any registered component in the pipeline, make sure that all the registered components (ones registered with single jar) should be of the same version. If you have registered a component with a fully qualified name, then that component cannot be registered with another jar in the same workspace.

Same FQN and same jar will create new version of that component.

Functions

Functions enables you to enrich an incoming message with additional data that is not provided by the source.

System Defined Functions

StreamAnalytix provides a rich library of system-defined functions as explained in the Functions Guide.

Variables

Allows you to use variables in your pipelines at runtime as per the scope.

To add a variable, click on Add New Variable and provide details as explained below.

Variables.PNG

For example, if you create the following variables: Name, Salary and Average.

Then by calling the following code, you will get all the variables in the varMap in its implementation class.

Map<String, ScopeVariable> varMap = (Map<String, ScopeVariable>) config­Map.get(svMap);

If you want to use the Name variable that you have created by calling the following code you will get all the details of the scope variables.

The variable object has all the details of the variable Name, Value, Datatype and Scope.

ScopeVariable variable = varMap.get(Name);

String value = variable.getValue();

Scope Variable

You can now add Scope Variable so that you can use these variables to reuse and update them as and when needed on pipeline and pipeline components.

Scope Variable Support is added for below components with their respective loca­tion where the scope variable will be populated with the help of @.

Cobol (Data Source) --> copybookPath --> dataPath

Native/Advance HDFS (Data Source) --> file path

HIVE (Data Source) --> Query

JDBC (Data Source) -- > Query

File Reader/Native (Data Source)--> File Path

File Writer (Emitter)--> File Path

Formats supported are:

@{Pipeline.filepath} = /user/hdfs

@{Workspace.filepath} = /user/hdfs

@{Global.filepath}/JSON/demo.json = /user/hdfs/JSON/demo.json

@{Pipeline.filepath + '/JSON/demo.json'} = /user/hdfs/JSON/demo.json

@{Workspace.filepath + “/JSON/demo.json”} = /user/hdfs/JSON/demo.json

@{Global.lastdecimal + 4} // will add number = 14.0

Models

The home page of a Register Models is shown below:

NOTE: If you use a registered model in your data pipeline, make sure that the Spark and H20 version must be same.

Register Spark ML Model

Steps to register a Decision Tree Model built on Spark Pipeline API for a classifica­tion use -case.

Step 1: Train the model on sample dataset and save it at any location on HDFS or local file system.

Step 2: Click on Register Model tab and provide a model name.

Step 3: Provide model API that is ML in our case.

Step 4: Provide the model type i.e. Decision Tree.

Step 5: Mention if the model to be registered should be a Spark Pipeline model or not.

Step 6: Mention model algorithm i.e. Classification

Step 7: Select model source. If model is stored on locally, then select Local and if model is stored on HDFS then select HDFS.

Below explained is a table to show you the process flow with HDFS and Local.

 

 

When the model is successfully registered with the application, you can avail it in your data pipelines.

Register H2O Models

StreamAnalytix provides its users with the option to register H2O POJO (Plain Old Java Objects) and MOJO (Model Object Optimization) models that are trained outside of StreamAnalytix. After the successful registration, the user can use the model for doing predictions.

In the left navigation pane, click Register Entities. Click the Models tab.

To register a new model, click the + icon on the right side of the screen.

In the Register Model window, mention the following properties:

Click the Validate button.

Once, the model is validated successfully, the Register button next to Validate will be enabled.

Click Register. After the model is registered successfully, you can view the model in the models page.

Once the model is registered it can be utilized for scoring over H2O processor in pipelines over StreamAnalytix.

Register Scikit model

StreamAnalytix provides its users with the option to register Scikit model that is trained outside of StreamAnalytix. After the successful registration, the user can use the model for making predictions.

In the left navigation pane, click Register Entities. Click the Models tab to register a model.

On the right side of the screen, click the + icon to register a new model.

In the Register Model window, mention the name of the model, specify the model API. Here, we choose Scikit as API.

Select one of the below model categories:

• Classification

• Clustering

• Pipeline

• Regression

The Feature list field enlists the features that are used to train the model. You can specify the feature names either by entering them manually or by uploading a .csv file. Here, the header row of the file will be used for feature names.

You can provide the model source either by uploading the scikit model or by provid­ing the HDFS file path.

Note: If you choose HDFS as the model source, provide the connection name and model path.

Click the Validate button. Once, the model is validated successfully, the Register button next to Validate will be enabled. Click Register. After the model is registered successfully, you can view the model in the models page. Now you can use the reg­istered model in the data pipeline for making predictions.

Now, you can use the registered scikit model for doing predictions by selecting the scikit processor to create a Data Pipeline. Refer to the image below:

 

 

Calendars

You can create holiday calendars from Register Entities < Calendar< Calendar listing page. There will be a + icon to create the calendar.

On the calendar configuration page you will be asked to select the holiday dates to create calendar.

You can either select multiple dates from the complete year of multiple year calen­dar or upload the holiday dates for one or multiple calendar years.

These calender can be used in the Workflow.

Following is the configuration page, their properties and descriptions: