How S3 Defines the Raw Data Contract

July 2026

For S3, raw data is whatever files appear under the chosen bucket and prefix, which means path layout, file format, and namespace fields are the real contract.

Short Answer

The raw data contract for S3 is the set of files that appear under the selected bucket and prefix, plus the file formats Skippr detects there. If the source is s3_bucket: my-bucket and s3_prefix: raw/orders/, raw data means the objects under that path and the rows or records those files contain when Skippr reads JSON, CSV, TSV, Parquet, or Avro.

That makes the bucket layout part of the source contract. A partner drop under raw/orders/2026/07/ can be a solid raw layer if the files stay consistently Parquet with the same order fields. If the prefix starts mixing JSONL event files, one-off CSV exports, and unrelated folders, the raw contract is no longer one thing analysts can reason about.

Why Teams Struggle with This

S3 looks simple because it is just files, but file-based sources are only stable when naming, prefix discipline, and format conventions are treated as part of the data contract instead of as implementation trivia.

• The s3_bucket and s3_prefix values define which objects belong to the raw layer.
• Format detection only helps when the files under that prefix are actually consistent enough to be read as one feed.
• Namespace fields can split incoming events usefully, but only if the records already contain a stable field such as event_type.
• IAM and region setup decide whether Skippr can read the files, but they do not rescue a bucket path that mixes unrelated data contracts.

How Skippr Handles It

Skippr matches S3 well because the connector surface is literal: bucket, prefix, and optional namespace fields. That makes it easier to talk about the path as the boundary instead of pretending object storage behaves like a database table.

Auto-detection of JSON, CSV, TSV, Parquet, and Avro is especially useful during evaluation because you can confirm quickly whether the file family under one prefix is coherent enough to support a durable raw layer.

• Reads directly from one S3 bucket and optional prefix.
• Auto-detects JSON, CSV, TSV, Parquet, and Avro files.
• Supports transform.namespace_fields to split records by stable fields.
• Uses standard AWS credentials and IAM policies instead of a separate file-ingestion secret model.

What the First Useful Version Looks Like

The first useful version is one prefix with one representative file family, such as daily Parquet order exports. That is enough to judge whether the path naming and file structure are stable enough for downstream consumers.

If the bucket currently mixes unrelated feeds, separate them before calling S3 the raw boundary. File-based contracts work best when one prefix means one feed and one format story.