Skip to content

rvrdata.adapters.csv

Data adapter for reading and writing comma-separated value files.

class RvrdataCsv

Data adapter for comma-separated value (.csv) files.

read()

Read comma-separated value file into memory.

  • Parameters:
  • filename (str) – Path to file.
  • kwargs (Any) – Pass-through of keyword arguments to csv.DictReader().
  • Returns: success=True if the file is read.

The action csv.read in a dataflow step reads the contents of a csv file into the .data property of the step.

If the filename is not correct or the file is not accessible due to permissions, this action will fail.

Example: Dataflow reading csv file

steps:
  - name: get_csv
    notes: Load csv file contents into 'steps.get_csv.data'.
    action: csv.read
    params:
        filename: "data/simple_list.csv"

Example: Dataflow reading csv and loop items:

steps:
  - name: get_csv
    action: csv.read
    params:
        filename: "data/simple_list.csv"

  - name: loop_items
    notes: Loop over csv rows.
    pause: 0.01
    params:
        output: "{{ item.col1 }}"
    loop__ref: steps.get_csv.data

See csv.DictReader for more information about how reading csv file behaves. Any keywords can be passed through to the function under params.

Read Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.csv.read.size: Size of csv data in bytes (int).
  • rvrdata.dataflow.action.csv.read.line_count: Number of records read excluding header row (int).
  • rvrdata.dataflow.action.csv.read.file_count: Number of files read (int).

One row is added to the metrics each time the action is run.

write()

Write data to a comma-separated value file.

  • Parameters:
  • filename (str) – Path to csv file.
  • data (list) – Array of dictionaries of data
  • data_type (str) – Data type for records (e.g. dict, list, etc.), if not supplied this is derived from the first element of the data. dict uses csv.DictWriter. All other data types use csv.writer.
  • fieldnames (list) – Array of field names to be used as the header. If none is supplied, it is derived from the first element in the data.
  • extrasaction (str) – Action to take for extra fields (defaults to ‘raise’)
  • Returns: success=True if file is written.

Example: Dataflow writing csv file

steps:
  - name: save_csv
    action: csv.write
    params:
        filename: "data/simple_list.csv"
        data: [
            {'col1': 'item1', 'col2': 'value1'},
            {'col1': 'item2', 'col2': 'value2'},
            {'col1': 'item3', 'col2': 'value3'},
        ]

Example: Dataflow writing csv file with referenced data

steps:
  - name: save_csv
    action: csv.write
    params:
        filename: "data/output.csv"
        data__ref: steps.some_previous_step.data

It should be noted that the data supplied will be treated as a table. As such, each row should have a consistent format for the fields. The default behavior will generate an error if a key is found not in fieldnames (supplied or derived from the first record).

Set extrasaction: “ignore” to ignore extra fields.

This leverages the python csv package. Check out the docs for how the writers behave and the available options. There are many features to control how the data output is formatted. Any keywords can be passed through to the function under params.

Write Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.csv.write.size: Size of csv data in bytes (int).
  • rvrdata.dataflow.action.csv.write.line_count: Number of records written (int).
  • rvrdata.dataflow.action.csv.write.file_count: Number of files written (int).

One row is added to the metrics each time the action is run.

create()

Create new comma-separated value file only if it is not already present.

This action works similar to a POST action in a REST API. It is intended for situations that need to avoid overwriting existing data. This will fail if there is an existing file in the path specified.

Syntax is the same as csv.write.

See csv.write metrics for create metrics.

delete()

Delete file.

This works the same as text.delete.

Delete Metrics

The following metrics are included in the step results:

  • rvrdata.dataflow.action.csv.delete.file_count: Number of files deleted (int).

One row is added to the metrics each time the action is run.

update()

Update comma-separated value file with supplied data.

  • Parameters:
  • filename (str) – Path to csv file
  • data (Any) – Data to be appended

Note: This is to be deprecated and replaced by an append function.

See json.write metrics for update metrics.

force_pass()

Fake successful action that does nothing.

See force_pass in stdout adapter.