Script Engine Preprocessor

Script Engine Preprocessor helps you to perform preliminary steps before executing the actual Bolt. It's a pre-step where you set certain conditions by writing codes in JavaScript/Python, and your actual Bolt will run based on those conditions.

Step by Step to Use Script Engine Preprocessor

Step 1: Add Integration

• First, add an Integration that you want to integrate to Bolt

Step 2: Use Request Processor

• Enable Use Request Processor slider to open a Script Editor

List of the Following tasks you can perform in Preprocessor Engine before creating an actual Bolt-

1. Execute Pre Bolt: You can write scripts in the Preprocessor Engine to be executed as Bolt's prior step before executing the actual Bolt

2. Pass Context Parameters: You can make the Bolt dynamic by defining multiple context parameters in the script editor. The context parameter provides context information for the pipeline batch being processed. This information includes:

Field	Description	How to use it?
Trace_id	The id is used for additional information such as files names in error rows, jobs, and schedules that are linked to the current Bolt execution	context.getTraceId()
Parameters_value	The values for the parameters used in the source query or the bolt	context.getParamterValues() context.setParameterValue('key', 'value')
Pipeline_meta	The meta information of the pipeline* Name

• Description

• Tags

• Audit | context.getMeta() context. getMeta().getName() context.getMeta().getAudit() | | Pipeline_options | Additional Configuration information of the pipeline* Mono

• Display meta

• Parameters | context.getPipelineOptions() context.getPipelineOptions().getMono() context.getPipelineOptions().getParameters() | | Error | Set an error if a pipeline gets failed | context.failPipeline('failure reason') |

** 2. Bolt Chaining:** You can call another Bolt from your Bolt to combine and compare the data and other operations-

• Defining the parameters' values based on the results of the previous Bolt

• The actual Bolt's execution depends on the previous Bolt's execution. For example, If the previous Bolt execution fails, the actual Bolt will get failed.

Read this doc to understand Bolt Chaining.

Step 3: Test Script

• Click the Test to run the script

Preprocessor Script Engine Example

Example 1: Process Data Updated Successfully

Suppose you have data containing a Datetime column. And you want to query data that has been updated/processed within the last 10 minutes. For that you pass the datetime parameter values in the preprocessor script.

The start time and end time parameters define a time period for filtering rows updated/processed within the last 10 minutes.

var moment = require("moment");

/*
datetimeFormat: datetime format for the parameter values

For more info on datetime formats, 
refer https://momentjscom.readthedocs.io/en/latest/moment/04-displaying/01-format/
*/
const datetimeFormat = "MM/DD/YYYY h:mm"


/*
context: context information for pipeline execution request
*/
function preprocess(context) {
    var end_time = moment()
    var start_time = moment().subtract(10, "minutes")

    context.setParameterValue("start_time", start_time.format(datetimeFormat))
    context.setParameterValue("end_time", end_time.format(datetimeFormat))
}

Example 2: Call Another Bolt as a Prerequisite

You plan to perform processing on your BigQuery data, like cleaning it for reporting and analysis. But first, ensure that the most recent data has been successfully migrated to BigQuery. Execute the migration Bolt using the preprocessor, and if it fails for any reason, the current Bolt will get failed as well.

function preprocess(context) {
    var response = bolt.withName("Migrate Database to BigQuery").execute()
    if (response.error != null) {
        context.failPipeline(response.error.reason)
    }
}

Any Question? 🤓

We are always an email away to help you resolve your queries. If you need any help, write to us at - 📧 support@boltic.io