ExpressionEvaluator Processor

This operator is responsible for performing transformation operations on incoming dataset, e.g., replace, format, trim, uppercase, lowercase, etc. It uses spark expression language for preparing transformation query.

Processor Configuration #

Configure the processor by providing configuration values as detailed below.

Expression #

Provide SQL expression using SparkSQL functions, that evaluates to a column. Evaluated value can be updated to an existing column or can be assigned to a new column.

Example: ‘John’, ‘Suite 130 Los Gatos, CA 95032, US’

Download Sample #

A CSV file with a few sample expressions can be downloaded to understand how various expressions can be applied on multiple columns using the upload option.

Upload #

The sample CSV file can be modified with custom expressions and required column names and uploaded using this option.

Once it is uploaded, the intended expressions will get updated in the Expression Evaluator transformation.

Add Expression #

Option to add more expression entries in the Expression Evaluator transformation.

Validate #

Option to validate the correctness of all the expressions that have been added in the transformation.

All the columns of the schema are populated and you can apply any transformation on it. The functions that can be applied are all listed in the right table.

Once you are ready with the required expression, click on the Validate button to check its correctness.

Ask Assistant to Fix #

If your drafted expression fails the validation check, you can utilize the ‘Ask Assistant to Fix’ option to receive a more accurate response. The chat assistant will provide a corrected expression that you can use.

Utilize the AI Assistant feature to generate expressions for your use cases.

Ask AI Assistant #

Use the AI assistant feature to simplify the creation of custom expressions.

Note: To activate this feature, Superuser must enable it from the Open AI settings by entering the required information. For more information, please refer the Manage Open AI Properties.

It allows you to generate complex expressions effortlessly, using natural language inputs as your guide.

Describe your desired expression in plain, conversational language. The AI assistant will understand your instructions and transform them into a functional expression.

Tailor expressions to your specific requirements, whether it’s for data transformation, filtering, calculations, or any other processing task.

Note: Press Ctrl + Space to list input columns and Ctrl + Enter to submit your request.

Input Example:

Create a column called inactive_days by calculating difference between last_login_date and current date.