CDC Configuration
Install
See the Install guide for the full setup, including Windows PowerShell.
curl -fsSL https://install.skippr.io/install.sh | shClick to copyInstalling Skippr means accepting the Skippr EULA.
CDC is configured with a source cdc_mode plus a pipeline cdc: block in skippr.yml. The cdc.default contract applies to tables discovered at runtime, and cdc.namespaces can override it for tables that use different business keys.
This page describes configuration. For the semantic contract, see CDC Guarantees.
The cdc: block
pipelines:
my_pipeline:
data_source: data_sources.source
data_sink: data_sinks.warehouse
cdc:
default:
business_key_columns:
- id
namespaces:
public.orders:
business_key_columns:
- order_id| Field | Required | Description |
|---|---|---|
default.business_key_columns | Yes for final-state CDC | Default columns that uniquely identify a row for dynamically discovered namespaces/tables. |
namespaces.<name>.business_key_columns | No | Override business keys for one namespace/table when it differs from the default. |
namespaces.<name>.null_key_policy | No | Controls how null business keys are handled for that namespace. Final-state sinks reject null keys unless explicitly documented otherwise. |
Precedence is deterministic: namespace/table override first, then cdc.default. This keeps Skippr able to discover namespaces on the fly while still supporting per-table contracts.
Source cdc_mode
CDC-capable sources support the same mode field:
| Value | Behavior |
|---|---|
snapshot | Run a bounded snapshot only. This is the default. |
snapshot_then_cdc | Run an initial full snapshot, store a source checkpoint, then continue from the native CDC stream. Later runs resume from checkpoints and skip the snapshot. |
cdc_only | Skip the initial snapshot and consume only CDC events from the native stream. |
Kafka has no source snapshot API, so use cdc_mode: cdc_only when consuming Debezium-style Kafka CDC events.
Automatic guarantee inference
Skippr automatically determines the most complete CDC contract your source and destination pair supports. You do not need to specify a guarantee level manually; the system derives it at startup and applies it throughout the run.
Exactly-once final state
When both the source and sink support full CDC reconciliation, Skippr uses exactly-once final-state semantics:
- Inserts, updates, and deletes are applied via MERGE with order-token guards
- Stale writes are rejected
- Deletes are tracked in tombstone tables to prevent ghost resurrections
cdc.default.business_key_columnsor a namespace override is required -- Skippr will error at startup if final-state CDC has no business key contract
This guarantee is about the final table state after replay and restart. Internal retries can still happen inside the pipeline.
CDC-encoded
When the destination cannot perform full final-state reconciliation but can faithfully land CDC payloads, Skippr writes events with their mutation metadata (_skippr_mutation_kind, _skippr_order_token) as an append-only change log. Downstream consumers can process this log independently.
Validation at startup
Skippr performs the following checks before starting a CDC pipeline:
- The source connector must support the selected
cdc_mode - The destination connector must be capable of accepting CDC payloads
- If the pair supports exactly-once final state, the resolved business key columns must be non-empty
- Column names in the resolved business key contract must exist in the source schema/table
If any check fails, the pipeline exits with a descriptive error message before any data is read.
Source/destination compatibility
All CDC-capable sources work with all warehouse destinations:
| Source | Destinations |
|---|---|
| PostgreSQL | Snowflake, BigQuery, PostgreSQL, Redshift, ClickHouse, Databricks, Synapse, MotherDuck |
| MySQL | Snowflake, BigQuery, PostgreSQL, Redshift, ClickHouse, Databricks, Synapse, MotherDuck |
| MongoDB | Snowflake, BigQuery, PostgreSQL, Redshift, ClickHouse, Databricks, Synapse, MotherDuck |
| DynamoDB | Snowflake, BigQuery, PostgreSQL, Redshift, ClickHouse, Databricks, Synapse, MotherDuck |
| Kafka (Debezium) | Snowflake, BigQuery, PostgreSQL, Redshift, ClickHouse, Databricks, Synapse, MotherDuck |
Non-CDC sources (e.g. S3, SFTP, HTTP) use snapshot mode only. Skippr validates source/destination compatibility at startup and returns a clear error if the combination is unsupported.
Complete example
PostgreSQL CDC to Snowflake (exactly-once final state is inferred automatically):
skippr:
workspace: pg_cdc_to_snowflake
pipelines:
pg_cdc_to_snowflake:
data_source: data_sources.source
data_sink: data_sinks.warehouse
cdc:
default:
business_key_columns:
- id
data_sources:
source:
Postgres:
host: db.example.com
port: 5432
user: replicator
password: "${POSTGRES_PASSWORD}"
database: production
cdc_mode: snapshot_then_cdc
data_sinks:
warehouse:
Snowflake:
database: ANALYTICS
schema: RAW
warehouse: COMPUTE_WH
role: SKIPPR_ROLEMySQL CDC to BigQuery:
skippr:
workspace: mysql_cdc_to_bq
pipelines:
mysql_cdc_to_bq:
data_source: data_sources.source
data_sink: data_sinks.warehouse
cdc:
default:
business_key_columns:
- id
namespaces:
ecommerce.orders:
business_key_columns:
- order_id
data_sources:
source:
Mysql:
connection_string: "mysql://replicator:${MYSQL_PASSWORD}@host:3306/ecommerce"
cdc_mode: snapshot_then_cdc
data_sinks:
warehouse:
Bigquery:
project: my-gcp-project
dataset: raw
location: USKafka Debezium CDC to Redshift:
skippr:
workspace: kafka_cdc_to_redshift
pipelines:
kafka_cdc_to_redshift:
data_source: data_sources.source
data_sink: data_sinks.warehouse
cdc:
default:
business_key_columns:
- id
data_sources:
source:
Kafka:
brokers: "kafka.example.com:9092"
topic: dbserver1.public.orders
cdc_mode: cdc_only
data_sinks:
warehouse:
Redshift:
cluster_identifier: my-cluster
database: analytics
db_user: admin
region: us-east-1Install
See the Install guide for the full setup, including Windows PowerShell.
curl -fsSL https://install.skippr.io/install.sh | shClick to copyInstalling Skippr means accepting the Skippr EULA.