S3

Use the S3 source to load files from a bucket prefix. JSON (including gzip), CSV, and Parquet are supported. On restart, objects Skipprd has already committed are skipped.

Skippr, your data department

Speak to an expert. Upfront insights in days, not months.

Book an intro call →

20 minutes · No sales pitch

Before you begin

• A bucket and prefix the caller can ListBucket and GetObject.
• AWS credentials in the environment or an instance role.

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-east-1"

Public sample data used in the quickstarts is s3://skippr-public-sample-data/bike-hire/. Any AWS account can read it.

Configure

Python

from skippr import Config, DataSourceS3, LocalStorage, Pipeline

cfg = Config().workspace("bikehire").storage(LocalStorage())
src = cfg.data_source(
    "sample",
    DataSourceS3(s3_bucket="skippr-public-sample-data", s3_prefix="bike-hire"),
)
cfg.pipeline("bikehire", Pipeline(data_source=src))
cfg.save()

CLI

skipprd --workspace bikehire --storage-mode local connect data-source s3 \
  --pipeline bikehire \
  --name sample \
  --s3-bucket skippr-public-sample-data \
  --s3-prefix bike-hire

YAML

data_sources:
  sample:
    S3:
      s3_bucket: skippr-public-sample-data
      s3_prefix: bike-hire

skipprd discover --pipeline bikehire
skipprd sync --pipeline bikehire --once --log

Options

Key	Type	Required/Default	Description
s3_bucket	string	Required	Bucket name
s3_prefix	string	Not set	Key prefix to list
format	string	Inferred	json, csv, or parquet when inference is wrong
batch_size_seconds	integer	Not set	Max seconds per read batch
batch_size_bytes	integer	Not set	Max bytes per read batch
endpoint_url	URL	Not set	S3-compatible endpoint
region	string	Not set	Region if it is not inferred
s3_prefix_ordered_depth	integer	0	How many prefix levels to walk in order
s3_delimiter	string	/	Listing delimiter

Environment variables

Variable	Description
AWS_ACCESS_KEY_ID	Access key
AWS_SECRET_ACCESS_KEY	Secret key
AWS_DEFAULT_REGION	Region

What gets synced

Each object under the prefix becomes records in a namespace derived from the key. Progress is the object key plus last-modified time. New objects and updated objects are read; deleted objects are not retracted.

Troubleshooting

Symptom	Fix
Access Denied	Grant s3:GetObject and s3:ListBucket
NoSuchBucket	Check the name and region
No files found	Check s3_prefix matches the keys (no leading s3://)
Wrong parse	Set format explicitly

Next steps

• Quickstart: S3 to Athena
• S3 destination