Skip to content

rvrdata.adapters.pickle

Adapter for dumping and loading pickled data objects.

For more information about pickle, see https://docs.python.org/3/library/pickle.html

New in rvrdata v0.9.3

class RvrdataPickle

Adapter to pickle data objects.

load()

Read pickled object from file into memory.

  • Parameters: filename (str) – Path to file.

The action ‘pickle.load’ in a dataflow step reads the pickled object from a file into the .data property of the step.

If the filename is not correct or the file is not accessible due to permissions, this action will fail.

Example: Dataflow reading a pickled object

steps:
  - name: get_pickle
    notes: Load file contents into 'steps.get_pickle.data'.
    action: pickle.load
    params:
        filename: "data/mydata.pickle"

This leverages the built-in pickle module of python. For more information on how the pickle.load() function works, see https://docs.python.org/3/library/pickle.html#pickle.load

Load Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.pickle.load.size: Size of data in bytes (int).
  • rvrdata.dataflow.action.pickle.load.file_count: Number of files read (int).

One row is added to the metrics each time the action is run.

dump()

Write pickled data to a file.

  • Parameters:
  • filename (str) – Path to text file.
  • data (Any) – Data to be saved as the file contents.

Example: Dataflow to write object from referenced data to file

steps:
  - name: save_pickled_data
    notes: Write data from context reference to file.
    action: pickle.dump
    params:
        filename: "data/myobj.pickle"
        data__ref: steps.some_previous_step.data
  • Returns: success = True if file is written or False if the write fails.

Writing a file automatically creates parent directories if they are not present.

By default, an existing file with the same name will be overwritten.

For more information on options for dumping pickled objects, see https://docs.python.org/3/library/pickle.html#pickle.dump

Dump Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.pickle.dump.size: Size of data in bytes (int).
  • rvrdata.dataflow.action.pickle.dump.file_count: Number of files read (int).

One row is added to the metrics each time the action is run.

delete()

Delete file.

  • Parameters:
  • filename (str) – Path to file.
  • missing_ok (bool) – Option to pass if file missing (defaults to False).

Example: Dataflow to delete file

steps:
  - name: delete_pickle
    action: pickle.delete
    params:
        filename: "data/file_to_be_deleted"
        missing_ok: True

For more information, see pathlib.Path.unlink

Delete Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.pickle.delete.file_count: Number of files deleted (int).

One row is added to the metrics each time the action is run.

force_pass()

Fake successful action that does nothing.

See force_pass in stdout adapter.