Skippr Docs

Appearance

How to Configure S3 as a Source Connector ​

May 2026

S3 source setup is easiest when you narrow the bucket prefix and verify IAM access before trying to ingest every object format at once.

Short Answer ​

Configure S3 as a source by setting source.kind to s3, then filling in s3_bucket and, when useful, s3_prefix to limit which objects Skippr scans. If you want the incoming events namespaced by a field in the file content, add transform.namespace_fields so downstream tables stay organized when multiple record types share one bucket.

A clean first setup is one bucket, one narrow prefix such as raw/ or data/, and credentials supplied through the standard AWS variables or a role. Because Skippr auto-detects JSON, CSV, Parquet, Avro, and related formats, the fastest way to validate the connector is usually to point it at a well-known prefix with a small number of representative files.

Why Teams Struggle with This ​

S3 connectors usually fail at the IAM or prefix boundary. Teams know the bucket exists, but the runner cannot list it, or the prefix is slightly off and the connector appears idle even though the credentials are valid.

  • The IAM principal needs at least s3:ListBucket on the bucket and s3:GetObject on the objects it should read.
  • The bucket name and region have to match the actual S3 location the runner can access.
  • A wrong s3_prefix often looks like an empty source rather than a clear error.
  • Namespace fields are useful, but you should validate the raw files first before adding more transform logic.

How Skippr Handles It ​

Skippr keeps S3 setup grounded in the pieces operators already understand: bucket, prefix, AWS credentials, and optional namespacing. That makes the first connector easy to reason about, especially when the source files are produced by existing applications or batch exports rather than by Skippr itself.

The first successful read gives you a clean signal that the source is ready. Skippr can list the bucket, open the target files, detect their format, and emit records under a namespace that still points clearly back to the original object layout.

  • S3 source configuration centered on s3_bucket and optional s3_prefix.
  • Authentication through the standard AWS credential chain or attached IAM roles.
  • Support for common file formats including JSON, CSV, Parquet, and Avro.
  • Optional namespace-field transforms when one bucket contains multiple event families.

What the First Useful Version Looks Like ​

Keep the first run scoped to a small prefix with known files so you can distinguish a parsing issue from a bucket-permission issue quickly.

If the source will expand later, document the bucket and prefix ownership now so future teams know which application or export process is responsible for the objects Skippr ingests.