Skippr Docs

Appearance

How to Configure S3 Bucket Access ​

August 2026

S3 access is correct when the runner can list the intended bucket prefix and read the objects behind it with the same AWS identity that production uses.

Short Answer ​

To configure S3 bucket access, grant s3:ListBucket on the bucket itself and s3:GetObject on the objects inside it, then make sure the runner uses a valid AWS credential path through access keys, an IAM role, or SSO-backed AWS credentials. In Skippr, the source contract is small on purpose: s3_bucket, optional s3_prefix, and optional transform.namespace_fields, so the first real question is whether the runtime identity can list and read the exact keys behind that boundary.

A minimal IAM policy looks like the docs example: one resource for arn:aws:s3:::my-data-bucket and another for arn:aws:s3:::my-data-bucket/*. If your prefix is raw/events/, test that exact path rather than the bucket in general. Being able to see raw/ is not the same as being able to read the objects in raw/events/2026/08/. The first useful check is one known object under the same prefix the source config will scan.

Why Teams Struggle with This ​

S3 permissions often fail because teams grant only part of the contract. Listing a bucket, reading objects, choosing the right region, and selecting the right prefix are separate pieces that can each break the first run.

  • Granting s3:GetObject without s3:ListBucket makes prefix-based discovery fail even when direct object reads would work.
  • Granting access to the bucket but not to bucket/* leaves the runner able to list and unable to open the actual files.
  • A wrong s3_prefix can make a healthy setup look empty. raw/ and raw/events/ are different data contracts.
  • Testing with local AWS credentials while production uses an IAM role can hide the fact that the deployed runner does not have the same permissions.

How Skippr Handles It ​

Skippr keeps the S3 source honest because it uses the normal AWS credential chain and exposes the real source boundary directly. You can decide whether the problem is bucket access, prefix selection, or file shape without reverse-engineering a larger abstraction.

That is especially useful when you are trying to keep the first version tight. One bucket, one prefix, and one predictable file format is easier to validate than a broad bucket scan that mixes unrelated keys and authentication paths.

  • Uses standard AWS credentials rather than a separate connector-specific auth system.
  • Supports access-key, IAM-role, or SSO-based credential resolution.
  • Keeps s3_bucket and s3_prefix explicit so the source boundary is reviewable.
  • Auto-detects common file formats such as JSON, CSV, Parquet, and Avro after access is working.

What the First Useful Version Looks Like ​

The first useful version is one bucket, one narrow prefix, and a handful of known files you can inspect by hand. If the runner can list that prefix and open one object that should obviously exist, the permissions story is grounded in reality.

Once that path is stable, widen the prefix or add namespace fields to shape the downstream layout. Do not start with the broadest possible bucket path. S3 is simpler when the first contract is small and deliberate.