Skippr Docs

Appearance

When to Use Delta Lake as a Source Connector ​

May 2026

The Delta Lake source connector is best when the data contract is already a Delta table and you want to extract that table directly instead of rebuilding it from lower-level files.

Short Answer ​

Use the Delta Lake source connector when the table itself is already the integration boundary. It is a strong fit when a Delta table on S3, ADLS, or local storage contains the cleaned dataset you want, and the team would rather read that table directly than reconstruct it from raw files or upstream application systems.

It is especially useful when versioning matters. If finance needs version 5 of a Delta table for a controlled backfill, or if you only want a filtered subset of a large table, Skippr can read the exact table URI, apply storage options for the backing object store, and keep the extraction tied to the Delta log rather than to a fragile folder convention.

Why Teams Struggle with This ​

Delta Lake is a good source only when the team truly treats the table and its transaction log as the source of truth. If the real ownership still lives upstream and the Delta table is just a transient byproduct, the extraction boundary tends to drift.

  • The runner needs read access to both the Delta transaction log and the underlying data files, not just to the directory name.
  • Choosing a specific version creates a stable extraction point, but it also means the team has to be deliberate about when to move forward.
  • A wrong table_uri or incomplete storage credentials usually fails before any meaningful data can be read.
  • If the main need is low-latency operational CDC from a database, Delta Lake as a source is often one layer too far downstream.

How Skippr Handles It ​

Skippr works well here because the connector is deliberately direct: table URI, storage options, optional version, optional filter. That is enough to make Delta Lake a reliable handoff point when your integration contract is already encoded in the table layout and transaction log.

The result is cleaner than treating object storage as an opaque file dump. Skippr is reading a Delta table, not guessing from filenames or partition folders alone.

  • Reads Delta tables from s3://, ADLS, or local paths.
  • Supports backend-specific storage_options for object store access.
  • Can pin to a table version for controlled reprocessing.
  • Can apply an optional filter to limit the extracted slice.

What the First Useful Version Looks Like ​

The first useful version is one table URI, one storage credential path, and one extraction that proves the Delta log is readable and the resulting schema is what downstream consumers expect.

It is a bad fit when no one owns the Delta table as a durable interface, when storage credentials are inconsistent, or when the team really needs to integrate from the upstream database rather than from a lakehouse artifact.