Skippr Docs

Appearance

DynamoDB CDC to Databricks Final-State Guide ​

June 2026

DynamoDB to Databricks CDC works best when stream records and Delta MERGE rules are designed as one path.

Short Answer ​

DynamoDB CDC maps well to Databricks when you think in item mutations on the source side and Delta MERGE on the destination side. DynamoDB Streams gives Skippr INSERT, MODIFY, and REMOVE events from the table, and Databricks applies those changes into Unity Catalog managed Delta tables with order-token checks.

The destination-specific advantage is ACID reconciliation on Delta tables. If an order item is modified several times in quick succession, Databricks MERGE lets the pipeline update the target row only when the incoming _skippr_order_token is newer, which is the behavior you want after retries or delayed batches.

Why Teams Struggle with This ​

The hard part here is not reading DynamoDB Streams. The hard part is preserving item identity and mutation order all the way into the Delta table that downstream models will trust.

  • Teams sometimes treat DynamoDB stream output as a log to archive instead of a set of row mutations to reconcile.
  • Composite DynamoDB keys need to stay intact when defining the target business key.
  • Delete handling becomes messy if REMOVE events are not paired with tombstone protection.
  • A live Databricks table is not automatically a correct final-state table unless the pipeline encodes ordering rules.

How Skippr Handles It ​

Skippr reads DynamoDB Streams from shard iterators, tracks the committed sequence number per shard, and resumes with the stored position after a restart. That keeps the source contract durable without inventing a parallel offset system.

On Databricks, Skippr creates _skippr_order_token columns and tombstone tables automatically, then uses Unity Catalog MERGE against Delta tables. That means bronze loading and final-state reconciliation stay in one documented warehouse path.

  • DynamoDB item changes are captured with stream semantics the docs describe.
  • Delta MERGE updates rows only when the incoming mutation is newer.
  • Tombstones protect against replayed inserts after deletes.
  • The pipeline ends in Unity Catalog tables that are easier to model with dbt.

What the First Useful Version Looks Like ​

This integration is attractive when the source workload already lives on AWS but the analytics platform lives on Databricks.

The key is to let Delta tables represent the current item state, while the stream and tombstones preserve how that state was reached.