Skippr Docs

Appearance

MongoDB CDC to Databricks Final-State Guide ​

June 2026

MongoDB to Databricks CDC works best when change-stream document mutations are mapped directly into Delta-table row rules.

Short Answer ​

MongoDB CDC to Databricks begins with change streams backed by the oplog and ends with Unity Catalog MERGE on Delta tables. Skippr reads insert, update, and delete operations, uses fullDocument updateLookup to carry the post-update document, and reconciles those changes into one current row per document key.

Databricks is destination-aware here because Delta MERGE gives you ACID final-state behavior. If a user profile document is updated twice and then the pipeline retries an older batch, the _skippr_order_token guard keeps the Delta table anchored to the newest valid document version rather than to the last batch processed.

Why Teams Struggle with This ​

The main mistake in this setup is to land MongoDB documents in Delta and assume the warehouse semantics will sort themselves out. They will not unless the merge contract is explicit.

  • MongoDB change streams need a replica set or sharded cluster before CDC can even start.
  • Resume tokens must be stored and reused on restart.
  • Document identity should stay tied to _id or another explicit business key.
  • Delete operations need tombstones so replayed older documents cannot reappear.

How Skippr Handles It ​

Skippr uses MongoDB change streams natively, stores the resume token after each committed batch, and resumes with resume_after. That keeps the source side durable without a polling loop.

On Databricks, Skippr adds _skippr_order_token and tombstone tables and applies Unity Catalog MERGE against Delta tables. The pipeline turns mutable document history into a table that downstream models can treat as present-state truth.

  • Change streams provide real-time document mutation capture.
  • fullDocument updateLookup carries full after images for updates.
  • Resume-token restarts keep the CDC reader on a durable path.
  • Delta MERGE plus tombstones enforces newer-wins final state.

What the First Useful Version Looks Like ​

This integration is a strong fit when MongoDB powers the application and Databricks is the warehouse platform for broader modeling.

The design win is making document mutation semantics explicit before the first Delta table becomes an input to dbt.