condition

String or boolean indicating if a step should be run or not.

condition: string | boolean

Condition can either be set to fixed values (e.g. set to True, False, or on_error), or can be text strings with Jinja templates with expressions that are evaluated at runtime.

The most common format for runtime conditionals is to use inline if expressions. These follow the general syntax of True if <something is true> else False. The evaluation should return either the True or False.

For example, the following dataflow has a disabled step that will not run:

steps:
  - name: disabled_step
    condition: False
    skip_warning: True
    params:
        output: "This text will NOT be rendered because the step does not run."

The following example shows a jinja condition being used to decide if the step will be run. If rvrdata is called on the command-line with the cli option --run_switch on, the conditional step will run, otherwise it will not.

variables:
    run_switch: "{{ kwargs.run_switch|default('off', true) }}"

steps:
  - name: conditional_step
    condition: "{{ (run_switch == 'on') }}"
    skip_warning: True
    params:
        output: |
          Conditional step run with '--run_switch on' is appended to the rvrdata cli

  - name: negative_conditional_step
    condition: "{{ not(run_switch == 'on') }}"
    skip_warning: True
    params:
        output: |
          This text will be rendered by default - use `--run_switch on` to run the conditional_step.

skip_warning

In all of the examples above, a boolean skip_warning was set to “True”. By default, a warning log message will be reported if a step is skipped. This highlights when expected steps are not run.

But in the case where the expected behavior is to not have a step run, enabling skip_warning surpresses the warning message so that log is cleaner.

on_error

There is a special type of condition on_error which will run only if dataflow pipeline has failed.

The following example shows the on_error condition. The on_error step is skipped over and not run unless there is an error in the pipeline. To simulate this, the “stop” step uses force_fail to trigger a failure. Note that pipeline_quickexit is set to True so that the “Never_Run” step is never reached due to the previous failed step.

variables:
    pipeline_quickexit: True

steps:
  - name: on_error_step
    condition: on_error
    params:
        output: |
          "Error from {{ first_error.context_ref }}: {{ first_error.message }}"

  - name: Happy_Step
    params: "This step always runs first."

  - name: stop
    action: force_fail
    notes: This step stops the dataflow.
    params:
        message: "This dataflow was stopped on purpose."

  - name: Never_Run
    notes: This step never runs because the pipeline is stopped and quickexit is set to True.
    params: "Never Step"