Skip to content

config_files

Config files can be referenced at the top of a dataflow (new in rvrdata-1.1.0). This allows variables from one or more configuration files to be imported.

The YAML syntax is similar to the following where each item is a pointer to a config file.

config_files:
  - "filepath/config_file1.yml"
  - "filepath/config_file2.yml"

Default Config file

A default configuration file can be setup by creating a file named rvrdata_config.yml. The settings from this default config will be used for all dataflow runs unless disabled.

On startup, rvrdata searches for the default config file in sequence in the following locations. The first match is used:

  1. If ROVER_CONFIG_DIR environment variable is set, rvrdata check this directory.
  2. CHeck ‘./data/local` local directory
  3. Check ‘./’ current directory
  4. Check ‘./.config’ directory
  5. Check ‘data/engine/.config’ directory (used for rover installs)

To disable this behavior, set default_config_file_disabled to True using a cli argument.

Order of config file variables import

Only the variables: section of the config files is imported. These variables are imported in sequence with each update overriding previous values, if any are present:

  1. Defaults from a default config file, if any is present
  2. Variables from config files listed in the config_files: of a dataflow, if any
  3. Variables from the config file referenced in the CLI config file argument
  4. Allow CLI input keywords to override of variables, if allowed
  5. Use the variables set in the dataflow (each can be set to allow kwargs inputs or not).

Special handling for dataflow config_files

Note that if multiple config files set the same variable, the string value from last config file loaded wins. This is not rendering jinja for config file variables until the dataflow is run.

Config files referenced on the cli generate an error if not present. If a config file path referenced in a dataflow is not present, it will be ignored with a WARNING log message by default. This allows dataflows to run even if a config file is not present.

To force a dataflow to require a config file, use the syntax shown below:

config_files:
  necessary_config:
    filename: "{{ 'somedir/config_filename.yml'|required }}"
    required: True