Skippr Docs

Appearance

When to Use Databricks as a Warehouse Destination ​

May 2026

Databricks is the right destination when Delta tables, Unity Catalog governance, and MERGE-based CDC semantics are central to how the team runs analytics.

Short Answer ​

Use Databricks as a warehouse destination when you want Delta-backed tables under Unity Catalog, clear catalog and schema governance, and a warehouse that fits notebook, SQL, and lakehouse-style transformation workflows. It is a strong fit when a personal access token can write to the target catalog and schema, and when the team wants MERGE-based final-state CDC without building that reconciliation logic itself.

A good example is a data platform team landing raw product and finance tables into Unity Catalog before modeling them in notebooks or SQL. Skippr can upload Parquet files through the Files API, use an optional SQL warehouse for COPY INTO, and keep CDC-managed tables current through Unity Catalog MERGE with order tokens and tombstones.

Why Teams Struggle with This ​

Databricks becomes a bad fit when the organization has workspace access but not the specific catalog, schema, or SQL warehouse privileges needed for loading. The setup looks simple on paper, but warehouse permissions often decide whether the integration feels smooth or blocked.

  • The token must be allowed to write to the selected catalog and schema before the first load succeeds.
  • If you use warehouse_id, that SQL warehouse must be reachable and authorized for COPY INTO or query execution.
  • Databricks is a warehouse-first choice, so it is less appealing when the team only wants a lightweight local evaluation surface.
  • If downstream consumers are not already centered on Databricks, the platform overhead can outweigh the connector benefits.

How Skippr Handles It ​

Skippr matches Databricks well because the destination surface is narrow: workspace URL, token, optional warehouse ID, catalog, and schema. CDC tables use MERGE with automatic _skippr_order_token columns and tombstone tables, so the Delta tables stay in final state without extra hand-written reconciliation logic.

That gives teams an integration they can explain operationally. One token owns the write path, one catalog owns the landing area, and one MERGE contract decides whether an incoming mutation is newer or stale.

  • Uploads Parquet files through the Databricks Files API.
  • Optionally uses a SQL warehouse for COPY INTO workflows.
  • Supports exactly-once final-state CDC with Unity Catalog MERGE.
  • Automatically creates order-token columns and tombstone tables.

What the First Useful Version Looks Like ​

The first useful version is one workspace, one catalog such as main, one schema such as default, and one source feeding a small raw table that downstream teams can query immediately.

Databricks is a bad fit when the platform is available but not actually where analysts or model authors work, or when the token cannot create and update objects in the chosen catalog and schema.