Change Data Capture (CDC): Real-Time Replication with Exactly-Once Guarantees
April 2026
How Skippr derives the CDC contract, carries mutation fidelity through the WAL, distinguishes exact final-state sinks from CDC-encoded landing, and ties checkpoint authority to durable ownership.
CDC Is a Contract, Not Just Log Tailing
Change Data Capture (CDC) is easy to describe loosely and hard to implement precisely. Reading PostgreSQL WAL, MySQL binlog, MongoDB change streams, DynamoDB Streams, or Kafka/Debezium envelopes is only the first step. A credible CDC system must preserve mutation fidelity, prove what it durably owns across crashes, and apply those mutations into the sink with enough ordering information to reject stale writes.
That is why the useful question is not just “can you read the source log?” It is “what guarantee does the runtime derive from this source, this sink, and this namespace contract?” Skippr treats CDC as an explicit compatibility and ownership problem, not as a generic stream-processing slogan.
Freshness is the outcome people notice first. Correctness is the property they notice later, when a replay, failover, or delete-heavy workload exposes the difference between append semantics and true final-state reconciliation.
CDC contract pathStep 1Source-native logRelational WAL, binlog, change stream, or keyed event log provides mutations plus source ordering.Step 2Namespace contractSkippr derives the strongest supported guarantee from source capability, sink capability, and business keys.Step 3Committed WAL segmentRow payload and row-aligned CDC metadata become replayable only after the segment is published.Step 4Sink applyEligible sinks use order-token and tombstone rules; CDC-encoded sinks land mutation metadata faithfully.Step 5Recovered final stateAfter crashes or retries, the destination converges either to final-state tables or to faithful CDC landing, depending on the contract.
Source Classes and Order Models Drive the Ceiling
Internally, Skippr classifies CDC-capable sources by what kind of ordering and restart proof they can provide. Relational logs such as PostgreSQL and MySQL provide a global total order tied to log-native checkpoints. MongoDB and DynamoDB still qualify for exact final-state CDC, but their order is modeled as stable per business key rather than a single global sequence. Kafka is treated as a stream log with stable key-scoped ordering. Event sources that only provide message identity, or incremental scanners that only expose cursors or object paths, do not earn the same guarantee ceiling.
That order model is not a cosmetic detail. Exact final-state sinks need to know whether two mutations are meaningfully comparable. If the source cannot produce a comparable order token for final-state reconciliation, the runtime should refuse to claim more than CDC-encoded landing.
Source shapeWhat Skippr expectsTypical examplesSnapshot + logAnchored bootstrap, log-native checkpoint, exact mutation replayPostgres, MySQL, MongoDB, DynamoDBLog streamStream-native checkpoint and stable per-key orderingKafkaIdentity-only or incrementalEnough identity to land events, but not enough comparable order for final-state reconciliationMessage buses and scanner-style sources
Skippr Derives the Strongest Guarantee at Startup
Skippr does not let the operator simply declare “this pipeline is exact-once.” The runtime derives the strongest supported guarantee from three inputs: the source capability, the sink capability, and the namespace contract. The namespace contract includes the configured business key columns, because final-state reconciliation has no meaning without keys that identify which row should win.
The strongest possible outcome is exactly-once final state. That requires a log-class source, a supported order model, and a sink that can manage Skippr-owned bookkeeping columns, maintain tombstone state, and compare order tokens. If any of those prerequisites are missing, but the pair can still carry CDC payloads faithfully, Skippr falls back to CDC-encoded delivery instead of pretending the stronger guarantee still holds.
This is a design choice worth highlighting: the effective guarantee is derived, not operator-invented. Business keys are mandatory for final-state mode, and unsupported order models are rejected from that path rather than being waved through as “probably fine.”
Derived modeRequired conditionsExactly-once final stateLog-class source, comparable order token, non-empty business keys, and a sink that can apply upsert/delete-if-newer semantics with tombstonesCDC-encoded landingSource can produce CDC metadata and sink can land that metadata faithfully, but cannot prove transactional final-state reconciliation
What a CDC Row Carries Through the WAL
Inside Skippr's WAL, CDC is represented as more than a row payload. Each CDC partition carries a row-aligned sidecar with three pieces of metadata: mutation kind (snapshot, insert, update, delete), event identity, and a comparable order token. That metadata stays aligned one-to-one with the Arrow row order in the same partition.
The separation between event identity and order token is important. Event identity answers “is this the same source event?” Order token answers “which mutation is newer for this key?” Those are related but not identical questions. A strong CDC system needs both concepts, because intake deduplication and final-state reconciliation are different jobs.
Because the CDC sidecar is persisted in the same committed segment as the row payload, restart recovery has access to the same mutation semantics that the live runtime saw before the crash. The WAL is not just a row buffer. It is the carrier of CDC meaning.
For the lower-level WAL durability mechanics behind this, read Exactly-Once Delivery.
Checkpoint Authority and Restart Semantics
Skippr also distinguishes between authoritative and advisory progress. Authoritative checkpoints mean the runtime has durably taken ownership of the corresponding CDC events in the committed WAL. Advisory checkpoints are operational hints that may help a source resume efficiently, but they are not allowed to become the sole proof of ownership.
That split matters during runtime plugin execution. When a source emits a checkpoint update marked as WAL ownership and there are still unflushed batches, Skippr flushes the segments first and only then stores the checkpoint envelope. The checkpoint is therefore tied to durable segment ownership, not to a hope that a later flush will succeed.
On restart, recovery reindexes committed segments and reloads stored checkpoint envelopes. That combination is what makes CDC restart behavior believable. The runtime is not just remembering a source token in isolation. It is pairing that token with the committed WAL window that proves the events were durably staged.
How Eligible Sinks Converge Final State
For eligible warehouse sinks, each CDC row resolves into one of two actions: upsert if newer or delete if newer. Skippr stores a _skippr_order_token with the live row and maintains a companion tombstone table for deletes. A replayed update only wins if its order token is newer than the current live row and any competing tombstone. A delete only wins if its token is newer than the current live row, and then it leaves behind a tombstone token so older inserts cannot resurrect the row later.
That is the implementation reason the guarantee is called exactly-once final state rather than “exactly one delivery.” The system is optimizing for the final table answer per business key after retries and restarts, not for proving that a network call physically occurred once.
Today that full contract is available on the warehouse-class sinks that can manage Skippr columns, compare order tokens, and maintain tombstone state: Postgres, Snowflake, BigQuery, Redshift, Databricks, MotherDuck, ClickHouse, and Synapse.
CDC-Encoded Landing Is Deliberately a Different Guarantee
Not every sink should pretend it can do final-state reconciliation. Object stores and file-style targets such as S3, GCS, Azure Blob, File, SFTP, AMQP, plus query layers such as Athena, can still land CDC faithfully, but they do so by augmenting the payload with explicit metadata columns like _skippr_mutation and _skippr_order_token rather than claiming sink-native final-state merge semantics.
That fallback is not a downgrade in honesty. It is a better product boundary. The destination still receives faithful CDC intent, but Skippr stops short of claiming that the destination now has transactional or deterministic final-state convergence when it does not.
What This Design Does and Does Not Claim
What Skippr does claim is precise. If the source has a compatible order model, the sink is CDC-final-state eligible, and the namespace defines business keys, the runtime can derive an exact final-state contract and enforce it through WAL-backed recovery plus sink-side newer/older rules. If those conditions are not met, the runtime drops to CDC-encoded landing instead of blurring the distinction.
What Skippr does not claim is that every stream of row-shaped events is now magically exact-once. Sources with unsupported ordering do not qualify. Empty business keys fail the stronger contract. CDC metadata without a namespace contract can fall back to append behavior. Event identity is not the same thing as winner selection. Those boundaries are what make the stronger guarantees believable.
Getting Started with CDC
Enabling CDC in Skippr starts with an explicit source flag and an explicit namespace contract:
`source: kind: postgres host: localhost database: mydb cdc_enabled: true
warehouse: kind: snowflake database: ANALYTICS schema: RAW
cdc: business_key_columns: - id`
Set cdc_enabled: true on your source, define the business keys that establish row identity, and run skippr run. From there the runtime derives the strongest supported mode for the source/sink pair, persists checkpoints with the right authority semantics, and carries CDC metadata through the WAL into the sink path.
For source-specific prerequisites and destination behavior, see the CDC docs, guarantees, and destinations. If you want the broader category framing around this workflow, read What Is an AI Data Agent?.