Appearance

How to Configure Delta Lake as a Source Connector ​

May 2026

Delta Lake source setup is mostly about the table URI and storage credentials, with version pinning helpful when you want a repeatable first read.

Short Answer ​

Configure Delta Lake as a source by setting source.kind to delta_lake, then providing the table_uri that points to the Delta table location. That URI can be s3://, abfss://, or a local filesystem path, and any backend-specific credentials or region settings belong under storage_options.

For a stable first run, it helps to pin an explicit version and keep any filter narrow. If the table lives at s3://my-bucket/path/to/table, setting AWS_REGION in storage_options and optionally reading version 5 lets you confirm that Skippr can open the Delta log, resolve the schema, and scan the exact snapshot you expect.

Why Teams Struggle with This ​

Delta Lake issues usually come from storage access, not from the connector syntax itself. Teams point at a valid-looking path but forget the runner must read both the transaction log and the underlying parquet files with whatever credentials that object store requires.

  • The table_uri must point at an actual Delta table root, not just a nearby bucket prefix.
  • storage_options needs the backend-specific keys the runner requires, such as AWS region or credentials.
  • A versioned first read is often easier to validate than reading an actively changing latest snapshot.
  • The runner needs read access to the Delta log and all referenced data files.

How Skippr Handles It ​

Skippr exposes Delta Lake in a practical way: one table URI, optional storage options, and optional version or filter controls. That keeps the source configuration focused on what matters most for a first extract, which is whether the engine can open the Delta metadata and turn it into a clean table read.

The first successful setup should feel deterministic. You should be able to tell which exact table location Skippr scanned, whether it read a pinned version or the latest snapshot, and whether the object-store credentials are sufficient for repeated runs.

  • Single-URI Delta source configuration for S3, ADLS, or local tables.
  • Backend credentials supplied through storage_options with environment interpolation when needed.
  • Optional version support for repeatable first-pass validation.
  • Optional filter support when you want a smaller initial extract.

What the First Useful Version Looks Like ​

If the table is large or changes frequently, pin a version for the first run so row counts and schema checks stay repeatable while you validate the connector.

For cloud-backed tables, confirm the same runner environment that executes Skippr also has access to the referenced storage backend, not just your local shell.