Skippr Docs

Appearance

When Databricks Is the Right Final-State Destination ​

July 2026

Databricks is the right final-state destination when governed Delta tables are part of the design and the team wants current-state CDC to land inside that same lakehouse surface.

Short Answer ​

Databricks is the right final-state destination when the team wants Delta-backed tables under Unity Catalog governance and a CDC contract built around MERGE on those tables. The destination docs spell out the important parts: workspace_url, token, optional warehouse_id, catalog, and schema, while the CDC docs explain that Skippr applies final-state reconciliation with order-token guards and tombstones on top of Unity Catalog MERGE.

That makes Databricks a strong fit when the warehouse should live inside the same governed lakehouse surface where notebooks, SQL, and platform teams already work. A data platform team can land raw or bronze tables into a governed catalog and keep them current without splitting the CDC logic away from the rest of the Delta table lifecycle.

Why Teams Struggle with This ​

Databricks is a weak fit when the organization has access to the platform but not to the specific catalog, schema, or SQL warehouse permissions the connector needs. It is also heavier than necessary if the team simply wants a small relational landing zone and not the broader lakehouse operating model.

  • The token must be allowed to write to the chosen catalog and schema.
  • If warehouse_id is used, that SQL warehouse must be reachable and authorized for COPY INTO or query execution.
  • Databricks is most compelling when governed Delta tables are a feature, not merely a brand preference.
  • If analysts do not actually work in Databricks, the surrounding platform overhead can outweigh the destination benefits.

How Skippr Handles It ​

Skippr aligns well with Databricks because the load and reconciliation surfaces are both explicit. The connector uses the Files API for Parquet uploads, supports optional COPY INTO, and then relies on Unity Catalog MERGE with _skippr_order_token and tombstones to keep final state correct.

That means the warehouse contract stays coherent. One governed catalog owns the landing area, and one documented merge policy decides whether an incoming mutation is newer, stale, or blocked by a newer delete.

  • Uploads data as Parquet through the Databricks Files API.
  • Supports optional COPY INTO when a SQL warehouse is part of the design.
  • Uses Unity Catalog MERGE for exactly-once final-state CDC.
  • Automatically creates order-token columns and tombstone tables.

What the First Useful Version Looks Like ​

The first useful version is one workspace, one token, one catalog and schema, and one CDC-managed table that downstream users can query without extra replay repair logic.

If the team would have to adopt Databricks mainly to host a single simple pipeline, a lighter destination is often the better final-state fit.