Skippr Docs

Appearance

Common S3 Source Setup Mistakes ​

May 2026

S3 source setup usually breaks when teams underestimate how much the bucket, prefix, and IAM policy define the ingest shape.

Short Answer ​

The most common S3 source mistakes are scanning the wrong prefix, granting only object reads without bucket listing, and ignoring how namespace fields affect the landed shape. Skippr expects an s3_bucket, an optional s3_prefix, and optional transform.namespace_fields, then auto-detects file formats such as JSON, CSV, Parquet, and Avro under that source scope.

That setup can fail in subtle ways. A team may grant s3:GetObject on object keys but forget s3:ListBucket, which means Skippr cannot enumerate what to ingest. Another team may point at raw/ when the real exports live under raw/orders/, then spend time debugging an apparently empty source that is actually just scanning the wrong prefix.

Why Teams Struggle with This ​

S3 source setup is mostly about being precise with storage boundaries. The bucket name, prefix, IAM policy, and namespace choice collectively define what Skippr can see and how the incoming files are organized once they land.

  • Missing s3:ListBucket is a common mistake because teams focus only on object-read permissions.
  • An overly broad or overly narrow prefix changes the ingest scope far more than many teams realize.
  • Region settings and bucket location can matter when credentials are valid but requests still fail.
  • Namespace fields are easy to skip, but they often determine whether mixed event files land in an interpretable structure.

How Skippr Handles It ​

Skippr keeps the S3 connector direct and reviewable: bucket, prefix, and optional namespace fields. The docs also spell out the minimum IAM policy shape, which helps teams stop treating S3 as a magical filesystem and start treating it as an explicitly permissioned source.

Because file formats are auto-detected, setup conversations can focus on storage scope and event organization instead of writing custom readers too early. That makes the first useful version of an S3 pipeline much easier to validate.

  • Clear S3 source configuration through bucket and prefix rather than hidden discovery rules.
  • Documented IAM requirements for both s3:GetObject and s3:ListBucket.
  • Automatic detection of common file formats such as JSON, CSV, Parquet, and Avro.
  • Optional namespace fields to shape multi-event ingestion more intentionally.

What the First Useful Version Looks Like ​

A useful first version is one bucket, one narrow prefix that definitely contains known files, and an IAM policy validated against both listing and reading behavior.

Once the read scope is stable, decide whether namespace fields should split the incoming data by event type or other dimensions, because that choice changes how raw tables are consumed later.