steps

Steps are a linear sequence of operations that make up a job. They are listed in an array under the keyword steps.

Each step in the array uses the following syntax for its properties:

step: [ name | notes | condition | skip_warning | variables | action | params | ignore_fail | cache | loop | while | pause ]

Each step has an action which defines what the step does when it runs. Actions are run using adapters and each adapter has its own set available actions. The full format for referencing and action is <adapter>.<action>. If the adapter is missing, the dataflow uses the default adapter which is stdout.

Examples

The following YAML snippet shows how steps can be defined:

steps:
  - name: step_one
    notes: This step does nothing.
    action: force_pass

  - name: step_two
    notes: Display message on console.
    params:
        output: "Step two done"

step properties

• step.name
• step.notes
• step.condition
• step.skip_warning
• step.variables
• step.action
• step.params
• step.ignore_fail
• step.cache
• step.loop
• step.output
• step.while
• step.pause

Running steps

The primary function of steps is to run actions.

A step is run if condition is True (the default).

If loop is present, the action may be repeated for each item in a list. If while is present, the action may be repeated until the while clause is False.

The sequence is:

1. run the action one or more times
2. cache the data from the action response, if cache present
3. trigger output, if there is are step output keywords configured (e.g. set_parent_variables_enabled or show_context)

steps runtime variables

Context reference format:

steps.<name>.<runtime_variable>

The following variables are set at runtime:

• data: Data generated by the action. This includes the response from the adapter plus and values added through the cache. See data cache
• error: Error message, if any, from run action
• fail: Boolean indicating if the step failed (convenience value, opposite of pass)
• pass: Boolean indicating if the step passed
• metrics: DataFrame with one row of metrics for every run of the step action
• response: Last run action result
• runs: Integer counting the number of times and action was run for the step
• status: Dictionary with status information
• success: Boolean indicating if the step passed (legacy - synonymous with pass)

To see details from the namespace at run-time, see show_context.