The Ultimate Guide to Change Data Capture

June 2026

A practical guide to change data capture: source logs, resume positions, business keys, replay safety, final-state tables, rollout strategy, and operational checklists.

Start Here: What Change Data Capture Actually Is

Change data capture, usually shortened to CDC, is the practice of reading inserts, updates, and deletes from a system as they happen instead of repeatedly reloading whole tables. The source of truth is usually a native change surface such as the PostgreSQL WAL, the MySQL binlog, MongoDB change streams, DynamoDB Streams, or Debezium-formatted messages in Kafka.

The short version is simple: CDC tells you what changed, what ordering scope the source exposes, and where to resume after a restart. That is what makes it useful for warehouses, downstream projections, and continuously updated operational analytics.

A concrete example makes the difference obvious. Imagine an orders table where order 8472 is inserted, then updated from pending to paid, then later cancelled. A nightly full reload will eventually show the final row state. CDC gives you the sequence of mutations, which means you can keep downstream systems current without rereading every row in the table.

That does not mean every pipeline should use CDC. The right question is not “can this source emit changes?” It is “does this workload benefit from continuous, ordered mutation delivery enough to justify the extra operational complexity?”

• CDC is usually the right fit when you need low-latency warehouse freshness, durable incremental syncs, or correct handling of updates and deletes without full-table reloads.
• CDC is often overkill when a source changes slowly, batch windows are acceptable, and a daily snapshot already solves the business problem.
• CDC is most valuable when the source is large enough that repeated full extracts become expensive, slow, or disruptive.
• CDC becomes dangerous when a team treats it like a magical faster sync instead of a real replication-style system with state, ordering, and operational constraints.

The Mental Model That Makes CDC Click

Most CDC confusion comes from missing the mental model. Teams jump from “turn on logical replication” or “enable Debezium” straight into tool configuration without understanding the four things a healthy CDC system must preserve.

• Mutation fidelity: can the source represent inserts, updates, and deletes accurately enough for downstream use?
• Resume state: if the consumer stops, can it restart from a durable position instead of guessing where it left off?
• Ordering: can downstream systems apply changes in a way that converges to the same final state after retries and restarts?
• Identity: do you have stable business keys or row identities so a change can be matched to the correct downstream row?

If you remember only one thing, remember this: CDC is not just transport. It is a contract between a source change log and a downstream apply model. Breaking that contract usually looks like one of four failure patterns: duplicate rows, resurrected deletes, out-of-order updates, or pipelines that cannot resume cleanly.

Use this row-level example. If order 8472 moves through pending -> paid -> cancelled, the pipeline needs to know all three mutations happened, their order, and the key that ties them to one downstream record. If it loses the resume position after the second mutation, or replays them in the wrong order, the destination can drift from the real source even though every individual write looked valid.

Once you see CDC as a stateful mutation system rather than a faster connector, the rest of the design choices become much easier to reason about.

CDC vs Batch Snapshots vs Event Tracking

Before you build CDC, compare it with the two alternatives teams most often confuse with it: batch snapshots and application event tracking.

PatternWhat it capturesBest forWeaknessBatch snapshotCurrent table state at extract timeSimple, low-frequency ingestionExpensive to rerun at scale, weak on fast updates and deletesCDCOrdered inserts, updates, and deletes from a change logLow-latency warehouse sync and final-state tablesMore operational state, retention, and ordering complexityApplication eventsBusiness events emitted by app codeDomain workflows, audits, and product telemetryMay not fully describe the actual database row state

A batch snapshot answers “what does the table look like right now?” CDC answers “what changed, within the source ordering guarantees, and how do I keep another system in sync?” Product events answer “what business action happened?” Those are related questions, but they are not interchangeable.

• Choose batch when simplicity matters more than freshness.
• Choose CDC when you need continuous updates with correct handling of updates and deletes.
• Choose domain events when the business action matters more than the exact row mutation surface.
• Use both CDC and events when you need an operational replica for analytics and a domain-level activity stream for product reasoning.

How Different Sources Expose Change Data

The phrase “CDC” hides the fact that each source exposes changes differently. That matters because resume behavior, permissions, failure modes, and payload shape all depend on the source mechanism.

SourceNative CDC surfaceDurable resume stateWhat to validate firstPostgreSQLWAL logical replication via pgoutputLSN and replication slotwal_level=logical, slot capacity, replication roleMySQLBinlog replicationBinlog file and positionbinlog_format=ROW, full row image, replication grantsMongoDBChange streams backed by oplogResume tokenReplica set or sharded cluster, collection access, change-stream supportDynamoDBDynamoDB StreamsShard sequence numbersStream enabled, correct view type, IAM stream permissionsKafkaDebezium-formatted change eventsConsumer group offsetsTopic content, group behavior, broker auth, Debezium envelope shape

That table is the first hero move in CDC design: stop saying “we are using CDC” as if it were a single feature. Ask instead: which log or stream are we consuming, what is the resume token, and what exactly counts as a mutation in that source?

For PostgreSQL, the key terms are WAL, logical replication, replication slot, and publication. For MySQL, think row-based binlog and binlog position. For MongoDB, think change stream resume token. For Kafka, think Debezium envelope and consumer group offset. Once those nouns are clear, setup and troubleshooting become more concrete.

• PostgreSQL: watch slot retention pressure and remember that update and delete fidelity may depend on replica identity configuration for certain table shapes.
• MySQL: row-based binlog and full row image are the safe defaults for downstream apply, while GTID or file-position resume choices affect recovery behavior.
• MongoDB: change streams require a replica set or sharded deployment, and the oplog window can close underneath a slow consumer.
• DynamoDB: stream retention is short, so lag is less forgiving than database-log CDC and the selected stream view type matters immediately.
• Kafka: Kafka is often the transport layer for CDC messages rather than the source of truth itself, so envelope quality and consumer offsets matter more than pretending the topic invented the mutation.

What a Good CDC Architecture Looks Like End to End

A good CDC pipeline is not just “source log to warehouse.” It has clear boundaries at each stage so you can observe, replay, and reason about the system.

• Choose the source boundary. Decide which tables, collections, streams, or topics are actually in scope for the first rollout.
• Validate the source mechanism. Prove the source can emit ordered changes and that the runner can resume from a durable position.
• Land raw change data. Keep a replayable raw zone or bronze layer so you can inspect what really arrived.
• Apply changes into final-state tables. Use a deterministic apply model keyed on business identity and ordering metadata.
• Model from final-state tables. Treat gold metrics and marts as a separate semantic contract, not as direct consumers of noisy raw change events.
• Monitor lag, errors, and retention. CDC only stays healthy when the operational state is visible.

A strong first architecture is boring in the best way. One source. One small set of tables. One warehouse destination. One well-understood apply path. One rollout checklist. Teams get in trouble when they try to onboard ten sources and five destinations before they have proven a single restart-safe CDC contract end to end.

For teams using Skippr, this is where the product angle becomes useful. The local runner owns the ingest execution path, captures source-native changes, and then applies them into supported warehouse destinations with replay-safe semantics. The reason that matters is not marketing. It is that fewer boundaries means fewer places for ordering and restart logic to go missing.

How Initial Snapshot, Backfill, and Cutover Actually Work

This is the part many CDC explainers skip. Real systems already contain data before you turn on the live stream. So the practical job is not just “read new mutations.” It is “create a correct starting state, then hand off to continuous changes without gaps or double-apply.”

• Pick a checkpoint. Record the source high-water mark you will hand off from: an LSN, binlog position, resume token boundary, or equivalent.
• Take the initial snapshot. Copy the existing rows that are already in scope for the warehouse.
• Start reading live changes from the checkpoint. Do not wait until the snapshot finishes to decide where the stream starts.
• Apply snapshot and live mutations with clear precedence rules. A late-arriving snapshot row must not overwrite a newer streamed mutation.
• Validate counts and representative rows. Pick rows that changed during the backfill window and prove the final-state table matches the source.
• Cut downstream consumers over only after convergence is proven. Dashboards and marts should switch to the CDC-backed table after the handoff is verified, not before.

Using the orders example, imagine your snapshot copies order 8472 while its status is still pending. Ten seconds later, the live stream emits an update to paid. If the snapshot row lands after the streamed update and blindly overwrites it, the warehouse goes backwards in time. That is why the handoff needs a checkpoint plus precedence rules, not just good intentions.

Re-bootstrap strategy matters too. If lag breaches the source retention window and the consumer can no longer resume, the pipeline needs a documented reset path: take a new snapshot, re-establish the stream boundary, and rebuild final-state tables without pretending the old resume token still works.

How CDC Lands in a Warehouse

The warehouse side is where many “working” CDC pipelines quietly fail. Ingesting change events is not the same thing as maintaining correct final-state tables.

The practical pattern is usually:

• Raw or bronze change data: append-only landing that preserves payload, metadata, and mutation order.
• Final-state tables: warehouse tables that reflect the latest known row state after replay-safe application.
• Silver and gold models: analytics tables built on top of the stable final-state contract, not directly on top of every raw mutation. LayerWhat it storesWhy it existsRaw CDCMutation payloads, order metadata, delete markers, resume-safe historyReplay, inspection, and debuggingFinal stateLatest row state keyed by business identityStable downstream serving layerModeled martsBusiness metrics, dimensions, and consumption-ready tablesBI, product analytics, and AI consumers

For one concrete end-to-end example, a raw landing row for order 8472 might look like this:

{ op: "u", order_id: 8472, status: "paid", source_lsn: 98123340, committed_at: "2026-05-14T12:11:03Z" } { op: "d", order_id: 8472, source_lsn: 98123398, committed_at: "2026-05-14T12:14:19Z" }

The final-state apply rule is then straightforward but strict: match on order_id, ignore any mutation older than the newest applied source position for that row, update the row on c or u, and treat d as a tombstone or delete action according to the destination pattern. That is the warehouse-side contract in one paragraph.

This is also where warehouse-specific semantics matter. BigQuery and Databricks lean on MERGE. ClickHouse may rely on ReplacingMergeTree. Snowflake benefits from order-token guards and tombstone handling. PostgreSQL-style warehouse targets often need staging-table apply logic. The warehouse is not a passive sink. It is part of the CDC contract.

Business Keys, Deletes, Replays, and Why Teams Get Burned

If you want the part that most often decides whether CDC works, learn this section well. CDC pipelines usually do not fail because “the connector is down.” They fail because the row-identity and replay model was weak from the beginning.

• Business keys: the downstream system needs a stable identity such as order_id or customer_id. Without that, updates and deletes cannot safely target the correct row.
• Deletes: a delete is not just “missing from the latest batch.” It is an explicit mutation that the destination must apply or remember.
• Replays: retries happen. Restarts happen. Your apply logic must converge to the same final table even when a committed batch is replayed.
• Ordering: if paid arrives after cancelled due to an out-of-order replay and you apply it blindly, the warehouse becomes wrong while every individual event still looked valid.

Use the orders example again. If row 8472 is deleted and your pipeline only overwrites current values, that delete can be silently lost. If a stale update is replayed after the delete, the row can be resurrected. That is why tombstones, ordering metadata, and replay-safe apply rules are not “advanced extras.” They are the difference between a warehouse replica and a misleading approximation.

A simple warehouse guardrail is to store the newest applied source token alongside the row. If a retry tries to apply an older mutation, you ignore it. If the newest mutation is a delete, you keep the tombstone semantics intact instead of letting an earlier update recreate the row.

Exactly Once, At Least Once, and the Truth You Need

People often argue about exactly-once as if it were one checkbox. It is more useful to separate delivery behavior from final-state correctness.

• At-least-once delivery means a change may be delivered more than once, so downstream apply logic must be idempotent or replay-safe.
• Exactly-once transport is a stronger claim about the transport layer, but it is not enough on its own if the warehouse apply logic is weak.
• Exactly-once final state means retries and restarts still converge to the correct downstream row state. That is usually the claim users actually want.

This is where source order metadata and destination apply semantics meet. A replay-safe system usually needs some mix of stable keys, source order metadata, dedupe or guard logic, and explicit delete handling. If any one of those is missing, retries can become corruption rather than recovery.

Be careful with the word “order.” Many systems only give strong ordering within a stream, partition, shard, table, or transaction scope. That is still useful, but it means your downstream apply logic has to respect the source boundary rather than assuming a single universal order across the whole data platform.

Skippr’s CDC docs are useful here because they frame the guarantee at the warehouse outcome level. Eligible destinations apply mutations with order-token guards and tombstone protection so restarts and replays do not produce duplicate or resurrected rows. That is a more practical promise than hand-waving about “real-time sync.”

A Practical Rollout Plan: From First Table to Production

The fastest safe path to production is smaller than most teams think. Start with one table that changes often enough to matter, but is still easy to reason about by hand.

• Pick one table or stream with a clear business key. Do not start with the hardest or widest object in the source.
• Validate the source mechanism in isolation. Confirm the source can emit inserts, updates, and deletes and that the runtime can resume from a durable position.
• Design the snapshot and cutover path. Record the starting checkpoint and decide how snapshot rows lose to newer streamed mutations.
• Land raw changes first. Inspect what the pipeline actually receives before designing fancy marts.
• Build one final-state table. Prove that updates, deletes, retries, and restarts converge correctly.
• Add a small dbt or SQL model on top. This forces the semantic contract discussion early instead of leaving it vague.
• Monitor lag and retention. If the consumer falls behind far enough, many source systems will stop being able to resume cleanly.
• Only then widen the source scope. More tables are easy to imagine and expensive to debug if the first one was not stable.

A good rollout question is: “If this pipeline restarts after 36 hours, what exact state lets it continue safely?” If the team cannot answer that in one sentence, the design is not mature enough yet.

Common Failure Modes and What They Usually Mean

CDC failure patterns are repetitive. That is good news, because repetitive failures are easier to prevent once you name them clearly.

• Slot or retention pressure: the consumer is too far behind and the source cannot keep the necessary history forever.
• Weak source permissions: the system can connect, but it cannot open the real CDC surface such as a replication slot, stream, or topic.
• Bad row identity: the source changes arrive, but downstream updates cannot match the correct row.
• Delete blindness: inserts and updates look fine, but deleted rows remain in the warehouse forever.
• Out-of-order apply: retries replay stale mutations after newer ones and corrupt the final table.
• Schema drift shock: the source adds or changes fields and downstream consumers were built as if shape never evolves.
• Consumer lag hidden by shallow monitoring: dashboards say the job is “up,” but the actual CDC stream is hours behind.

A strong operating posture means each of those has a first check and an owner. For PostgreSQL, that might mean watching WAL retention and slot lag. For Kafka, it means consumer group lag and envelope quality. For DynamoDB, it means stream configuration and shard progress. For warehouse sinks, it means apply errors, duplicate protection, and delete handling.

Schema Evolution and Recovery Playbooks

Schema evolution is where many CDC systems stop feeling elegant. New columns appear. Types widen. Nested payloads change shape. Sometimes a consumer falls behind so far that the source retention window is gone. A real CDC operating model needs answers for those cases before they happen.

• DDL changes: know whether a new column should flow straight through to raw landing, wait for warehouse DDL, or trigger a controlled schema-update path.
• Type changes: widening from INT to BIGINT is different from changing a scalar field into nested JSON. Treat them differently.
• Retention loss: if the source can no longer serve the missing log window, stop pretending resume is possible and run the documented re-bootstrap path.
• Duplicate replay: reprocess the affected raw changes through the same guarded apply rules instead of writing ad hoc one-off fixes directly into the destination.
• Delete mismatch: compare source row counts and a sample of known deletes before declaring recovery complete.

A compact recovery runbook is better than a long philosophy document. Name the symptom, the first diagnostic check, the safe decision point, and the reset path. That is how CDC becomes operable rather than clever.

When CDC Is the Wrong Tool

A good CDC guide is not just about knowing how to use CDC. It is also about knowing when to stop reaching for it.

• Use batch snapshots when the source is small, changes slowly, and daily or hourly freshness is acceptable.
• Use domain events when business actions matter more than reconstructing exact table state.
• Avoid CDC-first designs when the source has no stable row identity and no one is willing to define one.
• Avoid CDC when the team cannot yet monitor lag, retention, and retries. An unmonitored CDC system is a slow-moving incident.
• Avoid CDC when the business only needs monthly reporting from a handful of tables. Simpler pipelines win more often than teams admit.

The goal is not to make CDC look universal. The goal is to use it where its strengths matter: low-latency state propagation, durable incremental syncs, correct delete handling, and operationally legible downstream replicas.

How Skippr Fits the CDC Problem in Practice

Skippr’s value in CDC is not “we support source X.” The useful value is that the system keeps more of the hard path in one execution model: local runner, source-native CDC capture, deterministic landing, replay-safe warehouse apply where supported, and generated dbt project scaffolding for the downstream modeling contract.

That matters for teams trying to reduce boundary count. Instead of wiring together one connector product, one scheduler, one warehouse apply layer, and a separately bootstrapped modeling repo, the path is tighter. Your data stays within your system and in your destination, while the runner executes the ingestion and modeling bootstrap flow locally.

• CDC-capable sources include PostgreSQL, MySQL, MongoDB, DynamoDB, and Kafka.
• Supported warehouse destinations apply CDC into final-state tables with source-aligned metadata and replay safety semantics documented in CDC destinations and CDC guarantees.
• The dbt project is generated locally so teams can keep extending bronze, silver, and gold layers in ordinary Git-backed analytics workflows.
• Troubleshooting stays grounded in real system behavior because the docs expose the actual source and destination contracts instead of abstracting them away.

If you want the practical next step, pair this guide with the existing posts on CDC and exactly-once final state, PostgreSQL logical replication, and Kafka Debezium events in a warehouse.

Your Practical CDC Checklist

If you want one section to keep open while you build, use this one.

• Define the business problem first: freshness, delete handling, or scale pressure on batch reloads.
• Pick one source with a native CDC mechanism and one warehouse destination with a replay-safe apply model.
• Validate the source CDC surface, resume state, permissions, and retention assumptions.
• Choose stable business keys before you worry about dashboards or marts.
• Land raw changes, then prove one final-state table converges correctly after retries and restarts.
• Monitor lag, apply failures, and delete behavior before widening the rollout.
• Only model gold metrics after the final-state contract is stable.
• Document the restart story so on-call engineers know exactly what “resume safely” means for this pipeline.

That sequence mirrors the real maturity curve. First understand the nouns. Then prove one small system. Then widen the blast radius only after the first contract is stable.