Skippr Docs

Appearance

When to Use S3 as a Source Connector ​

June 2026

The S3 source connector is best when the upstream handoff is already file-based and the bucket plus prefix are the cleanest integration boundary.

Short Answer ​

Use the S3 source connector when the upstream system already delivers files to S3 and the bucket path is the real contract. It is a strong fit for batch-oriented integrations, partner drops, application exports, and lake-style raw zones where the team wants Skippr to ingest supported formats directly instead of standing up another file mover.

A practical example is an application or upstream service writing JSONL or Parquet files under raw/ in one bucket. Skippr can read from that bucket and prefix, detect the file format, and use namespace fields such as event_type to split incoming records into clearer downstream datasets.

Why Teams Struggle with This ​

S3 works best when the file layout is already disciplined. If the bucket mixes unrelated exports, inconsistent schemas, and ad hoc prefixes, the connector reflects that mess instead of fixing it.

  • The IAM principal needs both s3:ListBucket and s3:GetObject on the relevant bucket and objects.
  • The bucket name and region have to line up, or even simple access checks can fail before file discovery begins.
  • Namespace fields are useful, but they only help when the incoming records already include stable fields worth namespacing on.
  • S3 is a weak fit when the real need is low-latency streaming from a live system rather than file-oriented ingestion.

How Skippr Handles It ​

Skippr makes S3 practical because the connector embraces the file handoff directly: bucket, prefix, and optional namespace fields. It also auto-detects common formats such as JSON, CSV, Parquet, and Avro, which makes the first useful version faster to validate.

That means you can start with the bucket path teams already understand. No one has to invent a second ingestion surface just to move files that are already landing in S3.

  • Reads directly from an S3 bucket and optional prefix.
  • Uses standard AWS credentials and IAM policies.
  • Auto-detects JSON, CSV, TSV, Parquet, and Avro files.
  • Supports namespace fields to split incoming events into clearer downstream groupings.

What the First Useful Version Looks Like ​

The first useful version is one bucket, one prefix, and one representative file family that proves schema consistency and access permissions.

S3 is a bad fit when the upstream file contract is unstable, when IAM access is fragmented, or when the better integration boundary is a database or stream rather than a bucket full of exports.