Skippr Docs

Appearance

How to Define Delta Lake Ingestion Scope Clearly ​

July 2026

A Delta Lake boundary is not a bucket in general. It is one table URI, optionally one fixed version, and only the rows left after any filter you put in front of the reader.

Short Answer ​

Define Delta Lake ingestion scope with the exact table_uri first, then decide whether version or filter should narrow that scope further. The docs treat those three fields as the whole read contract. table_uri names the Delta table, version pins a specific table snapshot, and filter trims rows from that snapshot.

That is more precise than saying "read from S3" or "read from ADLS." A config that points at s3://finance-lake/curated/invoices is promising one Delta table, not a whole bucket. If it also sets version: 5, the source boundary becomes that table at version 5. If it adds filter: invoice_date >= date_sub(current_date, 30), the boundary becomes a filtered slice of that versioned table.

Why Teams Struggle with This ​

Delta Lake scope drifts when teams talk about storage credentials and forget that the real handoff is a table path plus Delta log semantics. The object store matters for access, but the ingestion contract lives at the table URI and in any version or filter choices layered on top.

  • Calling a bucket or container the source hides which Delta table the connector is actually opening.
  • Pinning version can be useful for controlled reprocessing, but it also freezes the extract to an older table state until someone moves it deliberately.
  • A filter changes the source boundary just as much as a WHERE clause in SQL would, so it should be reviewed as part of the contract.
  • Successful object-store access does not prove the Delta log and the selected table version are the right ones.

How Skippr Handles It ​

Skippr keeps Delta scope explicit because the connector does not invent another layer above the table. You give it table_uri, optional storage_options, optional version, and optional filter, and those fields map directly to what the runtime will try to open.

That is especially helpful when the upstream team already treats a Delta table as the handoff boundary. You can keep the storage credential story separate and still make the table path and row-shaping choices visible in the config the team reviews.

  • Reads one Delta table from a table_uri on S3, ADLS, or local storage.
  • Lets version define a fixed table snapshot when controlled replay matters.
  • Treats filter as an explicit narrowing of the source dataset.
  • Keeps the boundary centered on the Delta log and table path rather than on generic object-store access.

What the First Useful Version Looks Like ​

The first useful version is one Delta table URI, one successful read of the transaction log and files behind it, and no extra filter unless the team already agrees that the filtered subset is the public handoff.

If reprocessing is part of the reason you chose Delta, decide whether version pinning belongs in the first release. If not, leave version out and let the table itself be the boundary.