Skippr Docs

Appearance

MongoDB CDC to BigQuery Final-State Guide ​

June 2026

MongoDB to BigQuery CDC works when document mutations are translated into guarded row state instead of raw event history.

Short Answer ​

MongoDB CDC to BigQuery starts with change streams. Skippr opens a change stream on the database, receives insert, update, and delete operations, and uses fullDocument: updateLookup so update events carry the full document after image. BigQuery then reconciles those document mutations into one current row per document key.

The destination-specific advantage is BigQuery MERGE atomicity. If a document with _id = 42 changes from status = pending to status = approved, the pipeline should end with one BigQuery row for _id = 42 at status = approved, plus tombstone protection if that document is deleted later and an older event is replayed.

Why Teams Struggle with This ​

The source and destination represent data differently: MongoDB mutates documents, but BigQuery readers want queryable rows. The integration fails when that translation is left implicit.

  • MongoDB change streams require a replica set or sharded cluster, which is easy to forget in development.
  • Resume tokens matter because a restarted stream needs a durable continuation point.
  • Document identity usually comes from _id and must stay intact in the warehouse.
  • Delete handling needs tombstones so older document versions cannot reappear.

How Skippr Handles It ​

Skippr uses MongoDB change streams directly, stores the resume token after each committed batch, and resumes with resume_after on restart. That keeps the source side aligned to MongoDB change-stream semantics instead of relying on polling.

On BigQuery, Skippr adds _skippr_order_token and tombstone tables and applies MERGE DML with newer-wins guards. The result is a warehouse table that answers "what is the document now?" rather than "what events happened?"

  • MongoDB insert, update, and delete operations map to CDC mutation kinds.
  • fullDocument updateLookup gives update events a full after image.
  • Resume tokens provide durable restart behavior.
  • BigQuery MERGE produces a stable final-state row per document key.

What the First Useful Version Looks Like ​

This integration is useful when MongoDB is the operational system but BigQuery is where teams want relational analytics.

The important design move is to treat _id and resume tokens as part of the warehouse contract, not just as source implementation details.