Local file
Writes each batch into output_dir. Nothing is uploaded. Use this to inspect what a pipeline produces, or to feed another local process.
Before you begin
- A directory Skipprd can create and write.
Configure
python
from skippr import Config, DataSinkFile, Pipeline
cfg = Config.discover()
out = cfg.data_sink("out", DataSinkFile(output_dir="./out", format="parquet"))
cfg.pipeline("sample", Pipeline(data_source=cfg.get_data_source("sample"), data_sink=out))
cfg.save()bash
skipprd connect data-sink file \
--pipeline sample \
--name out \
--output-dir ./out \
--format parquetyaml
data_sinks:
out:
File:
output_dir: ./out
format: parquetOptions
| Key | Type | Required/Default | Description |
|---|---|---|---|
output_dir | path | Required | Directory for files |
format | string | Not set | File format (parquet, json, …) |
A retry overwrites the same path.
How data lands
Files appear under output_dir/<namespace>/. This destination is not a warehouse. For local SQL, use DuckDB.
Troubleshooting
| Symptom | Fix |
|---|---|
| Permission denied | Choose a writable output_dir |
| Disk full | Free space, or point DATA_DIR and output_dir at a larger volume |