Skip to content

rvrdata.adapters.adapter

Common adapter interface for rvrdata.

class RvrdataAdapterResults

Return results for adapter actions.

  • Parameters:
  • success (bool) – Boolean indicating if the action passed.
  • status (dict) – Dictionary with status results including ‘code’.
  • response (Any) – Action result that is tied to the adapter type. See individual adapters for details on what is in the response.
  • error (dict) – Dictionary with error details (empty if no errors).
  • start (str) – ISO8601 timestamp when action initialized
  • end (str) – ISO8601 timestamp when action stopped
  • metrics (pandas.DataFrame) – data series with a row for each run of an action

These adapter results are summarized into steps runtime variables.

Adapter Metrics

Adapter metrics provide useful statistics regarding how actions ran for a step (added in rvrdata-1.1.0). They are structured using pandas.DataFrame where each row in the series corresponds to a run of an action. If no loop or while is used in a step, there will only be one row in the series.

Any of the features of DataFrames can be used on the metrics. This is useful for analyzing results including generating summaries and charting. As a simple example len(steps..metrics.index) shows the number of runs. For more examples of generating summary statistics, see how to calculate summary statistics # noqa:E501

All adapter metrics include the following series:

  • rvrdata.dataflow.action.duration: execution duration in seconds for each action.
  • rvrdata.dataflow.action.fail_count: number of actions that failed. Use the row count minus this to get the number that passed.

See the individual adapters for details on the unique metrics for each type of adapter.

class RvrdataAdapterDataCacheTarget

Data caching targets and aliasing.

When an action is run in a dataflow step, the response returned is automatically in stored in the “.data” property of the step. Often it is useful to change how the data is captured by aliasing it, changing the source of where the data comes from, promoting it up to a parent (e.g. to a job’s data), or running it through a series of functions to transform the data.

Controls for are set under the “.cache” property of a step. Each step can have multiple cache targets, each with its own settings.

  • Parameters:
  • key (str) – alias for the data. Each target can have its own key. This makes it easy to reference the cached data using the format “steps..data.”.
  • source (str) – Defaults to the full response from the adapter. Can be set to a property or method of the response. If source is a method, “args” can be used to pass positional arguments and “kwargs” can be used to pass keyword arguments to it.
  • compose (list) – List of functions to run the data through in sequence.
  • data_type (str) – Use this to cast data to either “list”, “dict” or “str”.
  • disabled (bool) – Allows caching to be turned off for a target (defaults to False).
  • extend (bool) – Extends an array of data for each set of results in a loop.
  • overwrite (bool) – Flag indicating all data should be overwritten (defaults to False).
  • parent (str) – Enables promotion of data to a “job”, “stage”, or “pipeline” (defaults to “step”).

extend: True

When looping through a set of actions that return list data, there may be a need to extend the list by appending items to it. For example, paging through requests.get actions where each GET returns an array.

Example: Trivial loop to show how extend works.

This loop prints each item and stores them in step data with the key “items”. The context ref “steps.loop_items.data.items” contains a list of the three items.

steps:
  - name: loop_items
    params:
        output: {{ item }}
    pause: 0
    loop:
      - item1
      - item2
      - item3
    cache:
      - key: items
        extend: True

Example: Request Paging

A hypothetical API returns JSON objects in the following format:

{
    "objects": [
        {"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}, {"id": 5}
    ]
}

The following would loop through the first 3 pages of results with 3 GET requests and return a list called “object_list” of the first 15 records.

steps:
  - name: get_objects
    notes: Call API to get JSON.
    action: requests.get
    params:
        url: "https://host/api/objects?limit=5&page={{ item }}"
    loop: [1, 2, 3]
    cache:
      - key: object_list
        source: json
        compose:
          - get: "objects"
        extend: True

class RvrdataAdapter

Data adapter interface with common methods for all adapters.

adapter_type()

Get type of current data adapter.

All adapter classes are prefixed with the string “Rvrdata”. This returns the type of the adapter. It is derived from the lowercase class name with “Rvrdata” removed.

For dataflows, this is the first part of the ‘action’ property. It follows the format of “.” where adapter is the adapter_type, and action is the method name.

force_fail()

Fake unsuccessful adapter action.

This action can be used to stop a dataflow by forcing it to fail.

  • Parameters:
  • message (str) – Optional message override.
  • status (dict) – Status code for failure. Defaults to {“code”: 1}.
  • type (str) – Type of failure (Defaults to “force_fail”)

Example: Dataflow step that fails

steps:
  - name: stop
    action: force_fail
    params:
        message: "This dataflow was stopped on purpose."

abstract force_pass()

Fake successful action that does nothing.

This is an abstract method replaced by a concrete implementation on each adapter that returns the proper data type.

See force_pass in stdout adapter.