pipelines

Assembly of jobs possible grouped in stages.

pipelines: [ name | notes | variables | stages | jobs | steps ]

For simple dataflows, typically pipelines are not used explicitly. If no pipelines property, the name property at the root of the YAML is deemed to be the pipeline name.

Examples

Simple dataflow format with a single pipeline:

name: my_pipeline
notes: The pipelines property is implied.
steps:
  - name: step01
    notes: The stages and jobs properties are implied.

Below is a example of a more complex dataflow with multiple pipelines:

pipelines:
  - name: first_pipeline
    jobs:
      - name: first_job
        steps:
          - name: step_in_first_pipeline
  - name: second_pipeline
    jobs:
      - name: next_job
        steps:
          - name: step_in_second_pipeline

pipelines properties

• pipelines.name
• pipelines.notes
• pipelines.variables
• pipelines.stages
• pipelines.stages
• pipelines.stages

pipeline_quickexit

A special variable is use to control how pipelines react to errors. By default pipeline_quickexit is set to False. This means that when an error is encountered, rvrdata will continue to run through the pipeline to see if any steps have condition overrides that cause them to run.

When the variable pipeline_quickexit is enabled, rvrdata will exit the dataflow on a failed step. Any steps after the failed step will be skipped. Steps with ‘on_error’ conditions will still be run.

To enable this behavior, add the variable to the dataflow as shown below:

variables:
    pipeline_quickexit: True