rvrdata.adapters.requests

Data adapter for HTTP/HTTPS Requests.

This adapter wraps the requests python library. There are many features available for working with http requests. See the requests docs for a full inventory.

It is highly recommended that tools such as Postman and curl be used to test out how best to structure http requests and discover the format of responses.

Once this is well understood, running these requests with dataflows should enable the requests to be integrated along with other operations.

Making Requests

The most common actions are

requests.get
requests.post
requests.put
requests.patch
requests.delete

All use the url parameter as the request target.

Structuring requests

In addition to the url, requests accept other keywords.

These can be used under the params: property in a step:

params is used to pass query string values as a dictionary
data accepts a dictionary to be sent in the body of the request
headers is an optional dictionary of HTTP headers

auth can be used to supply credentials

For basic auth, supply a username and password.

steps:
  - name: get_url_with_basic_auth
    action: requests.get
    params:
        url: "https://example.com"
        auth:
            username: "myuser"
            password: "mypass"

For bearer authentication, add an Authorization header.

steps:
  - name: get_url_with_bearer_token
    action: requests.get
    params:
        url: "https://example.com"
        headers:
            Authorization: "Bearer TOKENGOESHERE"

For digest authentication, set the auth type to “digest” and supply a username and password.

steps:
  - name: get_url_with_digest_auth
    action: requests.get
    params:
        url: "https://example.com"
        auth:
            type: digest
            username: "myuser"
            password: "mypass"

- timeout identifies how many seconds to wait before giving up - verify can be set to False to stop verification of the server’s certificate

For a full listing of available keywords, see https://requests.readthedocs.io/en/latest/api/#requests.request

Responses

Requests return a response object and this is stored in the .data property of the step.

The following properties are available from the response:

data.text: content of the response in unicode
data.json: json-encoded content of the response, if any
data.status_code: integer code of the responded HTTP status (e.g. 404 or 200)
data.content: content of the response in bytes

For more detail on the response object, see https://requests.readthedocs.io/en/latest/api/#requests.Response

JSON Response

Calling APIs is the most common use of requests and often the response body contains JSON. A useful pattern is to cache the JSON in .data under a supplied key using the property source: json.

Example: Dataflow to GET JSON data from an API and save it.

steps:
  - name: get_todo
    notes: Call API to get JSON.
    action: requests.get
    params:
        url: "https://jsonplaceholder.typicode.com/todos/1"
    cache:
      - source: json
        key: todo

  - name: save_todo
    notes: Using a '__ref` is the most efficient way to handle data
    action: json.write
    params:
        data__ref: steps.get_todo.data.todo
        filename: data/todo1.json

For more detail on JSON response contents, see https://requests.readthedocs.io/en/latest/user/quickstart/#json-response-content

Successful Responses

Any HTTP status code returned that starts with a ‘2’ is deemed a success and will be registered as a passed step. Other status will be recorded as failed steps.

See https://developer.mozilla.org/en-US/docs/Web/HTTP/Status for a full list of HTTP status codes.

Advanced request features

Working with Cookies

If a response contains cookies, these can be accessed and used in subsequent steps. Since the response object is available in a steps’s .data property, the cookies appear in .data.cookies.

Example: Dataflow retrieving cookies and using them

steps:
  - name: login
    notes: Accessing a site that returns cookies after login
    action: requests.get
    params:
        url: "https://example.com/login"
        auth:
            username: "myuser"
            password: "mypass"

  - name: get_resource
    action: requests.get
    params:
        url: "https://example.com/resource"
        cookies__ref: steps.login.data.cookies

For more background on how requests work with cookies, see https://requests.readthedocs.io/en/latest/user/quickstart/#cookies

If cookies need to be passed to other dataflows, they can be saved to a file using the pickle adapter, and then loaded by the other dataflows.

Example: Dataflow pickling cookies

steps:
  - name: login
    notes: Accessing a site that returns cookies after login
    action: requests.get
    params:
        url: "https://example.com/login"
        auth:
            username: "myuser"
            password: "mypass"

  - name: pickle_cookies
    action: pickle.dump
    params:
        filename: "cookies.pickle"
        data__ref: steps.login.data.cookies

Example: Dataflow using pickled cookies

steps:
  - name: unpickle_cookies
    action: pickle.load
    params:
        filename: "cookies.pickle"

  - name: get_resource
    action: requests.get
    params:
        url: "https://example.com/resource"
        cookies__ref: steps.unpickle_cookies.data

class RvrdataRequests

Data adapter for HTTP/HTTPS Requests.

get()

Send GET HTTP/HTTPS request.

Parameters:
url (str) – URL endpoint
params (dict) – dictionary of key/value pairs for URL query string.
kwargs (Any) – Pass-through of keyword arguments to requests.

Example: Dataflow to retrieve a web page or call an API

steps:
  - name: get_todo
    action: requests.get
    params:
        url: "https://example.com?key=value"
    cache:
      - source: json
        key: todo

For APIs, a common pattern is for APIs to return JSON. To see how to handle this in a dataflow, see JSON Response.

Get Metrics

The following metrics are included in the step results:

rvrdata.dataflow.action.requests.get.fail_count: Number of failed API requests (int).
rvrdata.dataflow.action.requests.get.request_count: Number of API requests (int).

One row is added to the metrics each time the action is run.

post()

Send POST HTTP/HTTPS request.

Parameters:
url (str) – URL endpoint
data (Any) – Data to be sent in the body of the request
kwargs (Any) – Pass-through of keyword arguments to requests.

Example: Dataflow to POST data to an API

steps:
  - name: get_data
    notes: Get from file
    action: json.read
    params:
        filename: "data/example.json"

  - name: post_data
    notes: Send JSON in body of request
    action: requests.post
    params:
        url: "https://www.domain.com"
        data__ref: steps.get_data.data

Post Metrics

The following metrics are included in the step results:

rvrdata.dataflow.action.requests.post.fail_count: Number of failed API requests (int).
rvrdata.dataflow.action.requests.post.request_count: Number of API requests (int).

One row is added to the metrics each time the action is run.

put()

Send PUT HTTP/HTTPS request.

Parameters:
url (str) – URL endpoint
data (Any) – Data to be sent in the body of the request
kwargs (Any) – Pass-through of keyword arguments to requests.

Example: Dataflow to call API with a PUT

steps:
  - name: put_data
    notes: Update data using PUT
    action: requests.put
    params:
        url: "http://example.com"
        data: {
            "key1": "value1", "key2": "value2"
        }

Put Metrics

The following metrics are included in the step results:

rvrdata.dataflow.action.requests.put.fail_count: Number of failed API requests (int).
rvrdata.dataflow.action.requests.put.request_count: Number of API requests (int).

One row is added to the metrics each time the action is run.

patch()

Send PATCH HTTP/HTTPS request.

Parameters:
url (str) – URL endpoint
data (Any) – Data to be sent in the body of the request
kwargs (Any) – Pass-through of keyword arguments to requests.

Example: Dataflow to call API with a PATCH

steps:
  - name: patch_data
    notes: Update data using PATCH
    action: requests.patch
    params:
        url: "http://example.com"
        data:
            "key1": "value1"
            "key2": "value2"

Patch Metrics

The following metrics are included in the step results:

rvrdata.dataflow.action.requests.patch.fail_count: Number of failed API requests (int).
rvrdata.dataflow.action.requests.patch.request_count: Number of API requests (int).

One row is added to the metrics each time the action is run.

delete()

Send DELETE HTTP/HTTPS request.

Parameters:
url (str) – URL endpoint
kwargs (Any) – Pass-through of keyword arguments to requests.

Example: Dataflow to call API with a DELETE

steps:
  - name: delete_data
    action: requests.delete
    params:
        url: "http://example.com"

Delete Metrics

The following metrics are included in the step results:

rvrdata.dataflow.action.requests.delete.fail_count: Number of failed API requests (int).
rvrdata.dataflow.action.requests.delete.request_count: Number of API requests (int).

One row is added to the metrics each time the action is run.

force_pass()

Fake successful action that does nothing.

See force_pass in stdout adapter.

rvrdata.adapters.requests

Making Requests

Structuring requests

Responses

JSON Response

Successful Responses

Advanced request features

Working with Cookies

Sharing cookies between dataflows

class RvrdataRequests

get()

Get Metrics

post()

Post Metrics

put()

Put Metrics

patch()

Patch Metrics

delete()

Delete Metrics

force_pass()