Skippr Docs

Appearance

Kafka CDC to BigQuery Final-State Guide ​

June 2026

Kafka to BigQuery CDC works when Debezium envelopes are reconciled into rows instead of treated as the final warehouse artifact.

Short Answer ​

Kafka CDC to BigQuery starts with Debezium-formatted messages. Skippr reads a topic, parses the standard Debezium envelope with op, before, and after fields, and turns create, update, and delete messages into warehouse mutations that BigQuery can MERGE into the target table.

What changes for BigQuery is atomic reconciliation. A Debezium update with op = u should not leave two customer versions behind in the warehouse. BigQuery MERGE lets the pipeline update the row keyed by the Debezium key fields, compare order tokens, and keep the target table aligned to the latest valid state.

Why Teams Struggle with This ​

A Kafka topic can look healthy while the warehouse is still wrong. Transport and final-state correctness are not the same thing.

  • Teams sometimes land Debezium envelopes as raw JSON and postpone row reconciliation until much later.
  • Consumer-group offsets provide resume, but they do not define the warehouse business key.
  • Delete messages need tombstone-aware handling, not just a count of delete events.
  • Replay safety depends on comparing order tokens during MERGE, not just on offset commits.

How Skippr Handles It ​

Skippr uses a stable Kafka consumer group derived from the project name, so the pipeline resumes from committed offsets after restart. That gives Kafka CDC a durable read position without custom offset bookkeeping.

Once the message is parsed, BigQuery gets the documented CDC treatment: _skippr_order_token, tombstone tables, and MERGE DML with newer-wins guards. The warehouse table becomes the product, not the topic itself.

  • Debezium op = c, u, and d messages map to CDC mutation kinds.
  • Kafka consumer group offsets provide durable resume behavior.
  • BigQuery MERGE reconciles messages into one current row per business key.
  • Tombstones stop stale inserts from reviving deleted rows.

What the First Useful Version Looks Like ​

This integration is a strong fit when Kafka is already the CDC transport layer and BigQuery is where analysts expect stable relational tables.

The useful mental shift is to treat the topic as delivery and the BigQuery table as truth.