S3

The S3 destination writes files to a bucket prefix. It does not register a catalog. Query the objects with Athena only after you add a table definition yourself, or use the Athena destination instead.

Before you begin

A bucket and prefix Skipprd can PutObject to.
AWS credentials in the environment or an instance role.

bash

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-east-1"

Configure

PythonCLIYAML

python

from skippr import Config, DataSinkS3, Pipeline

cfg = Config.discover()
raw = cfg.data_sink("raw", DataSinkS3(s3_bucket="acme-landing", s3_prefix="skipprd/events"))
cfg.pipeline("events", Pipeline(data_source=cfg.get_data_source("sample"), data_sink=raw))
cfg.save()

bash

skipprd connect data-sink s3 \
  --pipeline events \
  --name raw \
  --s3-bucket acme-landing \
  --s3-prefix skipprd/events

yaml

data_sinks:
  raw:
    S3:
      s3_bucket: acme-landing
      s3_prefix: skipprd/events

Options

Key	Type	Required/Default	Description
`s3_bucket`	string	Required	Destination bucket
`s3_prefix`	string	Required	Key prefix
`format`	string	Not set	Object format when you override the default
`endpoint_url`	URL	Not set	S3-compatible endpoint

A retry overwrites the same key, so the final objects hold each row once.

How data lands

Files are written under s3://<bucket>/<prefix>/<namespace>/. There is no Glue database. Use this for a raw zone, deadletters, or another engine that lists the prefix.

Troubleshooting

Symptom	Fix
`AccessDenied`	Grant `s3:PutObject` and `s3:ListBucket`
`NoSuchBucket`	Check the bucket name and region
Wrong endpoint	Set `endpoint_url` for MinIO or another S3 API

Next steps

Athena — same files plus a Glue table
S3 source

S3 ​

Before you begin ​

Configure ​

Options ​

How data lands ​

Troubleshooting ​

Next steps ​