Skippr Docs

Appearance

How BigQuery Handles CDC Replay Safety ​

May 2026

BigQuery replay safety comes from guarded MERGE behavior, not from hoping retries never happen.

Short Answer ​

BigQuery handles CDC replay safety by applying changes with atomic MERGE DML and comparing the incoming _skippr_order_token to the token already stored on the row. If an older mutation is replayed after a newer one, the older write loses and the final-state row stays correct.

Deletes are protected the same way. Skippr writes the delete intent into a companion tombstone table and only allows a later insert for that business key if its order token is newer than the delete. That is what stops a replayed old insert from bringing a deleted row back.

Why Teams Struggle with This ​

This matters because retries are normal in warehouse pipelines. A load can be retried after a network hiccup, a job can be restarted after partial progress, or a backfill can overlap with data that already landed. Without an ordering rule inside BigQuery, the warehouse can accept stale state even when the transport layer behaved correctly.

  • A replayed update should not overwrite a newer row just because it arrived later.
  • A replayed insert should not resurrect a key that was already deleted more recently.
  • A business key is required because reconciliation happens per logical row, not per batch file.
  • Atomic MERGE behavior matters because the table being queried is the thing analysts will trust.

How Skippr Handles It ​

Skippr makes BigQuery replay safety concrete instead of implied. CDC-managed tables get a STRING _skippr_order_token column automatically, and deletes are tracked in skippr_tombstones so replay handling stays visible in the destination design.

That gives teams a clearer review surface. A concrete example: if order 42 is updated to status = shipped and an older status = packed event is replayed later, the BigQuery MERGE guard keeps shipped as the winner because the newer order token is already on the row.

  • BigQuery uses MERGE DML with order-token comparison in the matched branch.
  • The destination creates tombstone tables automatically for delete safety.
  • BigQuery MERGE is documented as atomic and consistent for final-state application in Skippr.
  • The warehouse contract stays close to the generated dbt starting point.

What the First Useful Version Looks Like ​

BigQuery replay safety is really a warehouse-state rule: newer wins, older loses, deleted keys stay deleted unless a newer insert arrives.

If that rule is clear, retries become an operational detail instead of a data-trust problem.