Skip to content

rvrdata.adapters.json

Data adapter for reading and writing JSON files.

class RvrdataJson

Data adapter for JSON files.

read()

Read JSON file into memory.

  • Parameters: filename (str) – Path to file.

The action json.read in a dataflow step reads the contents of a JSON file into the .data property of the step.

If the filename is not correct or the file is not accessible due to permissions, this action will fail.

Example: Dataflow reading a JSON file

steps:
  - name: get_json
    notes: Load JSON file contents into 'steps.get_json.data'.
    action: json.read
    params:
        filename: "data/example.json"

This leverages json.load from the python json package. Any keywords can be passed through to the function under params.

Read Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.json.read.size: Size of data in bytes (int).
  • rvrdata.dataflow.action.json.read.file_count: Number of files read (int).

One row is added to the metrics each time the action is run.

write()

Write data to a JSON file.

  • Parameters:
  • filename (str) – Path to JSON file.
  • data (Any) – Data to be saved in the file.
  • key (str) – Key for the JSON root element. This is useful for converting lists into proper JSON.
  • mode (str) – Mode used to open the file - defaults to ‘w’.

Example: Dataflow writing JSON file with referenced array of data

steps:
  - name: save_array_from_prv_step
    notes: Write data from context reference to file.
    action: json.write
    params:
        filename: "data/example.json"
        key: "records"
        data__ref: steps.some_previous_step.data

Writing a file automatically creates parent directories if they are not present.

This leverages json.dump from the python json package. Any keywords can be passed through to the function under params.

Write Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.json.write.size: Size of data in bytes (int).
  • rvrdata.dataflow.action.json.write.file_count: Number of files read (int).

One row is added to the metrics each time the action is run.

create()

Create JSON file only if it is not already present.

This action works similar to a POST action in a REST API. It is intended for situations that need to avoid overwriting existing data. This will fail if there is an existing file in the path specified.

Syntax is the same as json.write.

See json.write metrics for create metrics.

delete()

Delete file.

This works the same as text.delete.

Delete Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.json.delete.file_count: Number of files deleted (int).

One row is added to the metrics each time the action is run.

update()

Update JSON file with supplied data.

  • Parameters:
  • filename (str) – Path to text file.
  • data (Any) – Data used to update the JSON.

This works similar to doing a PATCH in a REST API. Partial data can be supplied and the keys will be either added if new, or updated if they already exist.

The operation works as follows:

  • read a existing file into a dictionary in memory,
  • update the dictionary with the supplied data, and
  • write the updated dictionary back to the file path.

Example: Dataflow to update JSON data

steps:
  - name: update_file
    notes: Updates YAML file with new/changed data.
    action: json.update
    params:
        filename: "data/example.json"
        data:
            key1: value
            key2: value

See json.write metrics for update metrics.

force_pass()

Fake successful action that does nothing.

See force_pass in stdout adapter.