Appearance

Introducing In-Stream SQL Migrations ​

October 2026 · Paul Hudson

In my last post about the COPY pipeline, I mentioned an ongoing frustration with a client's analytics lake: old Hive tables in AWS Glue where ALTER DDL statements essentially are not supported, and how we were migrating them to Iceberg.

I promised a future post on it. Here it is.

The problem wasn't just migrating legacy tables out of Glue. The deeper problem—the one that data engineering has quietly ignored for fifteen years—is what happens when a schema changes on a table that is actively being written to by a streaming engine.

The four bad choices ​

If you run an event-driven system or CDC from Postgres into a data lake or warehouse, schema drift is inevitable. Upstream developers add a field. They rename a column. They change an integer into a string because a third-party vendor started sending alphanumeric IDs.

What does the data platform do about that?

The industry currently offers four equally unpalatable options.

The first is the legacy catalog way: AWS Glue and Hive metastores. ALTER TABLE is essentially a fiction. Renaming a column or changing a type either fails silently, splits your partitions, or requires you to run an offline CTAS job that rewrites three terabytes of Parquet to change an eight-character header.

The second is the modern open lakehouse way: Apache Iceberg and Delta Lake. To their immense credit, Iceberg's metadata architecture actually supports in-place schema evolution without rewriting data files. You can run an ALTER TABLE statement in Athena, Trino, Snowflake, or Spark, and the metadata updates atomically.

Except your streaming ingest engine didn't get the memo.

While Athena is executing your ALTER TABLE, your Flink job, Debezium connector, Kafka consumer, or Python daemon is still running in the background. It is holding an in-memory schema that reflects five minutes ago. It has half-written batches buffered in memory or staged in a write-ahead log. When the next batch flushes to the lake, the streaming committer attempts to write files whose column names or types conflict with the newly updated Iceberg snapshot. Ingestion crashes. Your on-call engineer gets paged at 2:00 AM to reconcile the mismatch and rebuild the checkpoint.

The third option is the dbt cop-out. The engineering team agrees on an unwritten rule: never alter the lake tables. Leave the bronze landing layer messy. When a column is renamed or types diverge, let the landing table accumulate dead columns, and write a four-hundred-line staging model with COALESCE(new_field, old_field) and nested TRY_CAST gymnastics. Now every query against your lake pays a compute penalty in staging, your raw storage is littered with orphaned columns nobody understands, and your semantic definitions drift away from the real world.

The fourth option is the operational nightmare: stop the world. You schedule a maintenance window on a Tuesday night. You pause ingest. You wait for every buffered queue and compaction worker to drain. You execute your DDL against the lake. You update the ingestion configuration. You restart the engine. And you pray that your PostgreSQL replication slot didn't blow past its disk threshold while the pipeline was stopped.

Discovery is not migration ​

We built Skippr with automatic in-stream schema evolution from day one. When a source starts sending a new field, Skippr samples it, infers the nested structure, and adds it to the Iceberg table in-stream without dropping a batch. When a source sends a conflicting type—a string where an integer used to be—Skippr creates a sibling column (like price_string next to price) so the pipeline stays alive and data is never dropped.

That solves 2:00 AM survival. But it doesn't solve human curation.

At some point, a data engineer needs to clean up the table. You want to rename region to region_code. You want to move a flat column into a nested record like detail.region. You want to drop an obsolete tracking field. And once the upstream service fixes its bug, you want to fold price_string back into price.

You cannot leave that to automated inference. That requires deliberate, operator-directed DDL.

And it has to happen without stopping ingestion of new data or interrupting real-time in-stream client SQL queries.

How skipprd migrate works ​

With Skippr 16.0.0, we added skipprd migrate. Schema migrations live in plain SQL files in migrations/ next to your skippr.yml, formatted exactly the way application developers have managed relational databases for twenty years:

sql
-- migrations/20261024_cleanup_rides.sql
-- migrate:up
ALTER TABLE bikehire.rides RENAME COLUMN region TO detail.region;
ALTER TABLE bikehire.rides MERGE COLUMN price_string INTO price;
ALTER TABLE bikehire.rides ALTER COLUMN trip_id TYPE BIGINT;
ALTER TABLE bikehire.rides DROP COLUMN legacy_junk;

-- migrate:down
ALTER TABLE bikehire.rides RENAME COLUMN detail.region TO region;

If you work in Python, you can author those files with the typed Migration class:

python
from skippr import Migration

Migration(
    "20261024_cleanup_rides",
    up="ALTER TABLE bikehire.rides RENAME COLUMN region TO detail.region;",
    down="ALTER TABLE bikehire.rides RENAME COLUMN detail.region TO region;",
).write("migrations")

Skippr cli reads the static SQL file, parses the operations, and applies them:

bash
skipprd migrate --pipeline bikehire

Or programmatically in Python:

python
s.migrate()

The pipeline does not stop. You do not need to disable ingest. Skippr pauses incoming records for a fraction of a second at an atomic batch boundary, mutates the schema across every layer of the system, and resumes ingestion.

The mechanism: what happens under the hood ​

Applying an ALTER TABLE to a live, continuous ingestion engine requires synchronising four different layers that normally operate independently: in-memory metadata, uncompacted write-ahead logs, the destination catalog, and the durable state store.

Here is what happens when skipprd migrate executes:

1. The schema barrier and idle flush ​

Skippr raises a schema barrier so incoming CDC batches do not advance past a clean boundary. It then waits until any active WAL compaction jobs and pending sink writes finish their current flush. The pipeline is not killed or restarted; it is momentarily suspended at an exact commit boundary.

2. Destination capability check ​

Skippr checks that the destination can execute native ALTER in-stream. Iceberg destinations (SkipprLake, Athena Iceberg, DuckDB) and modern analytical warehouses (Snowflake, Postgres, BigQuery, Redshift, MotherDuck) support this. If someone attempts to run a migration against a destination that cannot—such as AWS Glue/Hive or ClickHouse—Skippr fails closed immediately. It leaves the table and the WAL untouched.

3. Canonicalisation and field IDs ​

Apache Iceberg does not identify columns by string names alone; it assigns stable numeric field ids. That is how Iceberg allows schema evolution without rewriting parquet files.

Skippr resolves your SQL against stored pipeline metadata and keeps the correct field id. When you run:

sql
ALTER TABLE bikehire.rides RENAME COLUMN region TO detail.region;

Skippr creates the nested detail struct in metadata, maps the flattened Iceberg column name to detail_region, and preserves the original field id. Existing Parquet data files in S3 or local disk remain readable without rewriting them.

4. Live WAL schema rewrite ​

This is where naive implementations fail. Batches captured thirty seconds ago are still sitting in uncompacted WAL segments on disk or S3 waiting for the next snapshot window. If you update the Iceberg catalog and then flush those old WAL files, the committer will attempt to write stale column schemas into a mutated Iceberg table.

Skippr opens the live uncompacted WAL segments and rewrites batch schemas by field id before the committer touches them. The in-flight data is adapted to the new schema before it ever reaches the destination.

5. Journaled destination apply ​

A schema migration cannot be a best-effort dual write. If the machine loses power halfway through, you cannot leave the local WAL rewritten while the Iceberg catalog is untouched.

Skippr journals every migration in the Skippr Store across three strict steps: update metadata and live WAL, run native DDL at the destination, then persist and seal the migration in pipeline state. On a clustered setup that journal lives in your configured state store alongside the rest of the pipeline.

If the process is killed at any point, the next run inspects the journal. A finished migration is a no-op. An incomplete one resumes from the step it reached. If someone edits a migration file after it has been applied, the checksum mismatch fails closed.

6. Release and resume ​

The schema barrier drops. Ingestion resumes with the new schema. Because Skippr maintains a dual query path, skipprd query immediately sees the updated column structure across both committed Iceberg tables and newly arrived WAL rows.

The safety boundaries ​

Because this runs against production tables, several operations fail closed by design:

  • Rollbacks (down) are restricted to Rename and Promote. If you drop a column or merge a sibling column, that operation is fundamentally destructive. If a migration file containing DROP or MERGE includes a -- migrate:down section, Skippr refuses to parse it.
  • MERGE COLUMN src INTO dst does not rewrite history. When an upstream type conflict is resolved and you fold price_string back into price, Skippr drops the sibling and retargets all future incoming values to dst. It does not backfill historical Parquet files that only ever had price_string. Historical files are immutable. Backfilling belongs in an explicit dbt model, not an in-stream migration.
  • Type widening strictly obeys Iceberg promotion rules. Integer widens to bigint, float widens to double, millisecond timestamp widens to microsecond timestamp. There is no ALTER COLUMN col TYPE string—to widen to string, you merge into a string sibling.

Try it in Skippr 16.0.0 ​

With Skippr 16.0.0, data lake tables finally have the same migration workflow that software engineers take for granted in application databases. Migrations sit in version control, you test them in CI, and you deploy them against live streaming pipelines without maintenance windows or dropped CDC batches.

Install the release:

bash
brew install skipprd/tap/skipprd

Or update the Python package:

bash
pip install skippr==16.0.0

Read the full CLI syntax in skipprd migrate and the supported DDL matrix in ALTER operations.