Skippr Docs

Appearance

Kafka CDC to Databricks Final-State Guide ​

June 2026

Kafka to Databricks CDC works well when Debezium messages are translated directly into Delta-table row rules.

Short Answer ​

Kafka CDC to Databricks means parsing Debezium envelopes from a topic and reconciling them into Delta tables. Skippr reads op, before, and after fields, uses the key fields as row identity, and applies Unity Catalog MERGE so the Delta target reflects the latest valid mutation per business key.

Databricks is especially destination-aware here because Delta tables give you ACID MERGE behavior. A replayed older update from the topic should not overwrite a newer customer record that is already present, and the _skippr_order_token guard gives the pipeline that exact rule.

Why Teams Struggle with This ​

Teams sometimes assume Kafka already solved the hard part because offsets are durable. Offsets solve resume. They do not solve final-state reconciliation.

  • Debezium key fields need to be preserved as the warehouse business key.
  • Before and after payloads are useful only if the target table has a clear newer-wins rule.
  • Delete messages need tombstone-aware handling in the destination.
  • A Delta table filled from Kafka can still drift if row identity is guessed rather than defined.

How Skippr Handles It ​

Skippr keeps the Kafka side straightforward with a stable group_id derived from the project, so the pipeline resumes from committed offsets after restart. That matches the documented Kafka CDC behavior instead of layering on a second resume mechanism.

On Databricks, Skippr adds _skippr_order_token and tombstone tables, then uses Unity Catalog MERGE on Delta tables. That creates a clean handoff from message semantics into warehouse semantics.

  • Debezium envelope parsing turns topic messages into CDC mutations.
  • Kafka consumer group commits provide durable restart behavior.
  • Delta MERGE updates only when the incoming mutation is newer.
  • Tombstone protection keeps delete behavior explicit.

What the First Useful Version Looks Like ​

This integration is a strong match when Kafka already carries the operational CDC stream and Databricks is where the team wants trustworthy current-state tables.

The key design choice is to make Delta MERGE the place where message history becomes warehouse truth.