Skippr Docs

Appearance

Kafka CDC to Redshift Final-State Guide ​

June 2026

Kafka to Redshift CDC depends on turning topic messages into a staged MERGE workflow that respects row identity and ordering.

Short Answer ​

Kafka CDC to Redshift starts with Debezium-formatted topic messages and ends with a Redshift MERGE. Skippr consumes create, update, and delete events from Kafka, stages them through the Redshift loading path, and uses MERGE with order-token guards so the target row reflects the newest valid message for each business key.

What changes for Redshift is the staging workflow. Because Redshift loads through S3 COPY into staging before reconciliation, the pipeline needs both reliable topic consumption and a disciplined warehouse merge phase. Getting data into staging is only half the job; MERGE is what gives you a usable final-state table.

Why Teams Struggle with This ​

Teams often debug the Kafka side first because it is louder. The quieter failure is usually downstream: the Redshift merge key or delete handling is underspecified.

  • Debezium key fields must survive the trip into Redshift staging unchanged.
  • Consumer offsets solve resume but do not solve stale-write protection.
  • Delete events need tombstone tracking or replayed creates can revive removed rows.
  • S3 staging can be configured correctly while the final-state table is still wrong.

How Skippr Handles It ​

Skippr follows the documented Kafka pattern with a stable consumer group and durable offset commits, so restart behavior stays predictable.

For Redshift, Skippr stages the batch through the S3 COPY workflow, adds _skippr_order_token and tombstone tables, and then reconciles through MERGE. That makes the source-to-destination contract explicit at each stage.

  • Debezium messages are parsed into mutation kinds and key-aware rows.
  • Kafka consumer groups provide durable topic resume.
  • Redshift staging tables bridge COPY loading and final-state MERGE.
  • Deletes and stale replays are guarded with tombstones and order tokens.

What the First Useful Version Looks Like ​

Choose this setup when Kafka is the CDC transport and Redshift is where reporting tables need to live.

The pipeline works well when the team treats staging as part of correctness, not as a temporary detail to ignore.