Skippr Docs

Appearance

DynamoDB CDC to BigQuery Final-State Guide ​

May 2026

DynamoDB to BigQuery CDC works when stream records are treated as warehouse mutations, not just events to append.

Short Answer ​

To move DynamoDB CDC into BigQuery final-state tables, start from what DynamoDB Streams actually emits: item-level INSERT, MODIFY, and REMOVE records, with full item state available when the stream uses NEW_AND_OLD_IMAGES. Skippr turns those records into CDC mutations and BigQuery reconciles them with MERGE DML instead of leaving analysts to decode stream history.

The destination-specific change is that BigQuery gives you atomic MERGE behavior. If an item like account_id = 17 changes from plan = free to plan = pro, the useful outcome is one BigQuery row keyed to that account ending in plan = pro, plus a tombstone record when a REMOVE arrives so an older replay cannot resurrect the deleted item.

Why Teams Struggle with This ​

Teams usually get the stream running before they think through the warehouse contract. That is how a clean DynamoDB event feed turns into a confusing BigQuery table.

  • DynamoDB Streams must be enabled with NEW_AND_OLD_IMAGES or the warehouse will not have the full item context the pipeline expects.
  • A partition key and sort key pair often defines row identity, but teams sometimes only carry one part of that key into the warehouse.
  • REMOVE events need delete handling, not just an appended audit row.
  • Replays and out-of-order retries stay dangerous if the destination does not compare order tokens during upsert.

How Skippr Handles It ​

Skippr keeps the DynamoDB side explicit. It reads stream shards, stores the last processed sequence number per shard, and resumes from AT_SEQUENCE_NUMBER on restart. That gives the pipeline a durable position instead of a best-effort poller.

On the BigQuery side, Skippr adds _skippr_order_token and a companion tombstone table automatically, then applies MERGE DML with order-token guards. The table people query is designed to converge on final state, not on whichever record happened to load last.

  • DynamoDB INSERT, MODIFY, and REMOVE records map cleanly into CDC mutation kinds.
  • Per-shard sequence numbers provide a durable resume point for the stream reader.
  • BigQuery MERGE applies newer mutations while rejecting stale writes.
  • Tombstone tables prevent ghost resurrections after deletes.

What the First Useful Version Looks Like ​

This integration is a good fit when the source of truth lives in DynamoDB but the reporting surface lives in BigQuery.

The important design choice is not just turning on the stream. It is deciding that the BigQuery table should represent the latest item state after retries, deletes, and restarts.