Skippr Docs

Appearance

Postgres CDC to Databricks Final State Guide ​

June 2026

Postgres to Databricks works when WAL changes and Delta MERGE semantics are treated as one consistent reconciliation path.

Short Answer ​

Postgres CDC to Databricks works by reading row-level changes from PostgreSQL logical replication and applying them with Unity Catalog MERGE on Delta tables. PostgreSQL contributes inserts, updates with full after images, deletes, and committed LSN ordering. Databricks contributes transactional MERGE behavior, _skippr_order_token STRING columns, and tombstone tables so the final table keeps only the newest valid state for each key.

A simple example is invoices.id 442 changing paid_at from null to a timestamp. PostgreSQL emits the update through the replication slot, Skippr stores the committed LSN, and Databricks MERGE updates the row only when the incoming token is newer than the stored token for id 442. If the row is deleted later, the tombstone protects the table from an older replayed insert.

Why Teams Struggle with This ​

This integration only stays reliable when both the replication setup and the warehouse permissions are in place. PostgreSQL needs logical replication enabled, and Databricks needs a token that can write to the selected catalog and schema before MERGE can keep the table in final state.

  • PostgreSQL CDC requires wal_level = logical plus a reusable replication slot.
  • The replication user must have REPLICATION rights or superuser access.
  • Databricks final-state correctness depends on MERGE with token comparison, not on the order batches happen to reach Delta tables.
  • Deletes rely on tombstones so removed keys do not return during replays.

How Skippr Handles It ​

Skippr gives this integration a consistent ordering story. It resumes PostgreSQL from the stored committed LSN, reuses the existing replication slot, and applies the resulting mutations in Databricks with MERGE and automatic tombstone creation. That makes the warehouse table converge on the same order the source committed.

Because the path is explicit, downstream model authors can trust the input contract more easily. The integration has already decided how updates, deletes, and replays should behave before dbt or BI queries depend on it.

  • PostgreSQL WAL logical replication with durable LSN resumes.
  • Databricks MERGE on Delta tables with _skippr_order_token guards.
  • Automatic tombstone tables for delete protection.
  • A direct path from committed source rows to current-state analytics rows.

What the First Useful Version Looks Like ​

Use this integration when Databricks is the analytical home for data that still originates in PostgreSQL.

The right review question is whether every committed PostgreSQL row change has one clear target key and one clear winning-token rule in Databricks.