rvrdata
rvrdata is a command-line utility used to run dataflows.
It comes bundled in each rover install.
Running dataflows from the CLI
Dataflows can be run from the command line by calling rvrdata with a YAML file:
The command-line interface has a number of options to control how rvrdata is run. Use the following command to see help:
Config file
A config file is special version of dataflow YAML that can be used to pass input variables.
Dataflow with config file:
In this case, the variables: section from the config file will be treated as keyword arguments (kwargs) passed in to the dataflow.
Using a configuration file is especially useful when there are multiple dataflows that need the same variables. Setting these values in a common file avoids having to change values in multiple locations.
Config file with dataflow
The schema of config files follows the same format as regular dataflows.
As an option, config files can include a key dataflow: with a dataflow file as the value.
This allows rvrdata to be called just with the config file:
rvrdata CLI Options
The CLI has a number of switches that can be used to control how rvrdata runs.
The -cf option for the config file is one example.
In general, short-form options with a single dash '-' are used to by rvrdata to denote options.
Long-form options with double dash '--' used by rvrdata typically have a dash somewhere in the option (e.g. --log-level).
There are some exceptions to this rule however such as --help.
For a full list of available options, use the following command to see the rvrdata CLI help:
rvrdata CLI Input Keywords
The rvrdata CLI accepts user-defined keywords as inputs.
Any option provided on the CLI with a double dash '--' that is not in the list of rvrdata options is treated as an input keyword.
For example, consider the following command:
This will run the dataflow with kwargs.key1 set to value1 and kwargs.key2 set to value2.
If there are spaces in the value, surround it with single or double quotes:
Recommendations for Keywords
Care should be taken to avoid conflict with any rvrdata options. The convention of including dashes in rvrdata long-form options is intended to help with this. Avoid using dashes in input keywords to minimize the change of conflict.
Keyword inputs should also be strings without spaces. Consider the use of underscores for the keys to break apart words. Built-in rvrdata options will not use underscores so this is an easy way to avoid conflict.
One final note, if there is conflict between rvrdata built-in options and an input keyword, the rvrdata option takes precedence. To address this situation, consider using a config file to pass input variables instead and refer to the config file on the command line.
Environment Variables
It should be noted that some rvrdata adapters accept environment variables as inputs.
For example, the rvrvault adapter can use the environment variables VAULT_ADDR and VAULT_TOKEN to access the vault for secrets.