Local variables

Local variables are variables that you can use to store data specific to a single task run. They can be initialized with an initial value and updated or used throughout the task run. Each task maintains its own set of local variables, which ensures that data is isolated between different task runs.

Local variables can receive data dynamically from task step outputs or be manually initialized with predefined values. This makes them versatile for both automatic data processing and custom configuration.

Key features

Task-Specific data storage: Local variables are unique to each task run, ensuring data is not shared across runs.
Nullable by design: All local variables are treated as nullable, regardless of their nullable flag.
Flexible initialization: Local variables can be initialized with manual inputs at setup and dynamically updated with output data from steps through schema mapping.
Data type compatibility: During mapping, output and input data types must match
Schema selection: Local variables are represented on a simple schema and exclude complex/table columns, limiting them to primitive data types only.

Performance impact of Local variables

Using the Local variables improves data flow accessibility within a task run, but it can cause an increased processing load on the application. This may result in performance slowdowns, particularly in more complex tasks with multiple steps.

Ways to use the Local variables

Manual input: You can set initial values for local variables within the selected schema in the Design view. These values are available throughout the task run and may be updated with data from the assigned output schema.

Schema output assignment: Assign output data from steps into local variables by clicking the Local Variables symbol on the desired step. You can view the stored values for each step in the History view by selecting the variable symbol.

Combined approach: Local variables can be initially set up with manual values and later updated based on output data from other steps through assignments. This allows flexibility while maintaining the structure defined by the local variable schema.

Accessing the Local variables

JS Mapper, Node.js, or Python connectors

You can access local variables using the predefined localVariable object, which provides dynamic data access. For example:

// accessing local variables via predefined object
log.warn("Counter: " + localVariable.Counter);

// logging entire variable object
log.warn(JSON.stringify(localVariable));

After starting a run, switching to the history view, and opening the Task Run Logs, you will see the logged details of the task execution.

SQL connectors

In the SQL Postgres step, local variables are represented as a temporary table called LocalVariable. This table is created at the start of the task run and is dropped when the run ends. You can access the variables as follows:

-- Select all local variables
select * from LocalVariable;

This query will return all local variables as a single row in the table. Column names in the database query must match the fields in the output schema.

To retrieve specific columns from the local variables:

-- Select specific variables
select "ID", "Name" from LocalVariable;

Other connectors

In non-programming connectors, local variables are accessible via placeholders ${LocalVariable.<Name>}, dynamically replacing values at runtime.

// applying local variable placeholder in any configuration
log.warn('${LocalVariable.Counter}');

These placeholders represent the variable's value and are replaced with the corresponding values at runtime. For example, when processing data row by row, placeholders are replaced for each row.

Commented-out variables

Commented-out variables are processed as active, so any errors or typos within them will still cause the step to fail.

Best practices

Consistent data types: Ensure that the data types of local variables match their mapped schema to prevent errors.
Placeholders in JavaScript: Avoid using placeholders in JavaScript steps; prefer accessing variables through the localVariable object.
Monitor mapping: Check for the blue dot to confirm correct variable mapping within steps.
Usage indicator: The blue dot next to the Local variable symbol indicates whether local variables are actively used.

Conclusion

Local variables provide a powerful mechanism to store and manipulate data specific to a task run. Whether used in JavaScript or Python steps, SQL queries, or non-programming connectors, they enable dynamic interactions between output schemas and task execution. By understanding how to initialize, map, and retrieve these variables, you can enhance the flexibility and reusability of your tasks.