Skip to main content

Script Engine FAQ

This doc listed all the questions that users have asked about Script Engine.

Q1. I don’t know how to write code, but I want to perform transformations on my data. Where do I begin?​

It’s not necessary to use custom scripts; you may check out UI transformations (link). However, if the transformation you intend to perform is unavailable there, we have a set of predefined script templates that may help you, which doesn’t require much programming knowledge. Refer Custom Script Examples to learn how to use these templates.

If you want us to add a new transformation or template, contact our support team with your use case and we’ll add more such transformations which are used frequently.

Q2. I intend to import some external libraries and use them to process my data. How and where do I import them? What are the supported libraries?​

For importing libraries:

Q3. The library I intend to use is not present in the list of supported libraries. So how do I still import it?​

You may contact our support team and request the library and the use case you intend to use it for.

Q4. I’m facing some issues while executing my script; how do I resolve them?​

Some errors may occur due to invalid or uncleaned data. For example, like parsing values from a string to an integer for a given column, the values are invalid and not numeric for some rows. Refer Error Handling in Script Engine.

For other common errors, refer to Troubleshooting Script Errors.

Q5. What is the context parameter in the process function? How do I use it?​

The context parameter provides context information for the pipeline batch being processed. This information includes:

FieldDescriptionHow to Use?
ColumnsThe list of columns in the data.context.getColumns()
Row_processedThe total number of rows processed in the pipeline till nowcontext.getRowsProcessed()
Query_parameterThe parameters used in query, visual query to fetch the data from the sourcecontext.getQueryParameters()
Trace_idThe id used for additional information like in error rows file name, jobs, schedules which is linked to the current bolt executionContext.getTraceId()
Parameters_valuesThe values for the parameters used in the source query or the boltcontext.getParameterValues()
Pipeline_metaThe meta information of the pipeline - name - description - tags - markers - auditcontext.getMeta() context. getMeta().getName() coontext.getMeta().getAudit()
Batch_noThe current batch number being processedcontext.getBatchNo()
is_final_batchThe flag that tells if the current batch being processed is the last batch or notcontext.isFinalBatch()
Batch_sizeThe total batch size determining the maximum numbers of rows that will be processed in a batchcontext.getBatchSize()
Pipeline_optionsSome additional pipeline level config options like - mono - display_meta - parameterscontext.getPipelineOptions() context.getPipelineOptions(). getMono() context.getPipelineOptions().getParameters()

Examples​

Doing some additional processing in case of the final batch before sending out your results

function processRow(rowNo, row, context) {
    row["batch_no"] = context.getBatchNo()
    row["pipeline_name"] = context.getPipelineMeta().getName()
    // Write your code here
    return row
}


function process(rows, context) {
    if (context.isFinalBatch()) {
        // Add additional steps here
    }
    // Write your code here
    return processedRows
}

Adding pipeline meta or current batch information in data to identify them later

function processRow(rowNo, row, context) {
    row["batch_no"] = context.getBatchNo()
    row["pipeline_name"] = context.getPipelineMeta().getName()
    // Write your code here
    return row
}


function process(rows, context) {
    const processedRows = []
    for (rowNo = 0; rowNo < rows.size(); rowNo++) {
        try {
            const row = rows[rowNo]
            let processedRow = processRow(rowNo, row, context)
            if (processRow !== null) {
                processedRows.push(processedRow)
            }
        } catch (err) {
            error_rows.capture(400, row, err.message)
        }
    }
    return processedRows
}

Creating variables by using values from parameters and using them in the script to create dynamic behavior

function process(rows, context) {
    const groupByColumn = context.getParameterValues().get("group_by_key")
    var results = lodash(rows)
        .map(row => parseValues(row, context))
        .groupBy(groupByColumn)
        .map((rows, key) => aggregateRows(rows, key, context))
        .value()
    return results
}

Any Question? 🤓​

We are always an email away to help you resolve your queries. If you need any help, write to us at - 📧 support@boltic.io