Why we built a distributed Write-Ahead Log data ingestion engine in Rust instead of Java
22 October 2026 · Paul Hudson
If you tell a serious data team you need continuous ingest from Postgres into Iceberg, the architecture that appears on the whiteboard is almost always Java.
Debezium reads the WAL. Kafka keeps the log. Flink (or Spark Structured Streaming) applies it. A catalog service owns Iceberg commits. Three to five JVM clusters, a checkpoint store, a schema registry, and a platform team to keep the classpath quiet.
That stack works. It is also how a "sync some tables" job becomes a department.
We built skipprd the other way: one Rust binary, a write-ahead log that can live on disk or on S3, and Iceberg/Parquet written on the same machine that captured the CDC. This is why.
The default stack is a distributed operating system
Apache Flink is a great stream processor. It is a poor ingest product.
To run Flink in anger you take on JobManagers, TaskManagers, slot sharing, restart strategies, checkpoint barriers, savepoints, RocksDB state backends, JVM flags, and a deployment story that looks like Kubernetes even when the workload is "tail the logical replication slot and write Parquet."
Debezium plus Kafka plus Flink plus an Iceberg sink is four products agreeing on a payload. Each one is allowed to buffer. Each one has a different idea of "committed." Exactly-once becomes "exactly-once inside Flink, plus at-least-once from Debezium, plus whatever the Iceberg committer does when two files land in the same snapshot."
The operational tell is always the same. Someone asks "can we add a source?" and the answer is a change request against the cluster, not a line in a config file.
We did not want a cluster. We wanted a commit protocol.
Ingest is not stream processing
Stream processing asks: given an unbounded collection of events, compute a result. Windows, joins, watermarks, state.
CDC ingest asks something smaller and stricter:
- Read a change from a source that may never send it again (a logical slot, a binlog, an HTTP body).
- Make that change durable before you acknowledge it.
- Apply it to a destination that has its own idea of transactions (Iceberg snapshot, Snowflake COPY, Postgres INSERT).
- Survive SIGKILL, a lost disk, and a retry that must not duplicate a row the destination already has.
- Do that while the source schema grows a column at 2am.
That is a write-ahead log with an apply loop. It is closer to Postgres than to Flink. Treating it as a Flink job imports a distributed OS to get a fsync.
DuckDB is the other magnet. It is an excellent query engine. It is a poor continuous writer. Transactional write-locks, micro-inserts, and live schema drift are exactly the load that makes an analytical file format unhappy if you pretend the query engine is also your ingest pipeline. Skipprd owns that write path so DuckDB can query cleanly — local, in-process, against Iceberg/Parquet that already committed.
What a WAL actually is here
A buffer and a WAL look identical on the happy path. Records go in. A file comes out. The destination gets a batch.
They diverge the first time the process dies.
- If you acknowledged the source and the batch was only in memory, you lost data.
- If you spilled files "so it is safer" without a commit record, you have leftovers you cannot identify, complete, or replay.
- If you retry from the source without a ledger, you duplicate.
Skipprd writes each batch to its write-ahead log before the destination, then applies, then reclaims. A crash retries from the log. Whether a retry can duplicate a row depends on the destination — Iceberg snapshot commits and destinations with merge keys can honour exactly-once; some sinks cannot, and we refuse to pretend otherwise. The engine will not start a pipeline the destination cannot honour.
The log can sit on local disk. It can also sit on S3. That second option is the one that surprised me.
Why the log belongs on object storage
The compute in this product is allowed to be cheap and mortal. A laptop. A VM. A container that exists for one hourly job. A Lambda.
If the WAL is a file on that host, the host is the ledger. Lose the disk, lose the committed set you have not applied yet. That is fine for a demo. It is not fine for CDC from a slot you cannot rewind for free.
S3 (and R2, and any store with per-key overwrite and read-after-write) gives you a ledger that outlives the process. The protocol is deliberately boring: a segment object and a commit object. The commit object is the cut line. Left of it, the batch is not owned. Right of it, the runner may ACK the source, die, and come back on another host. Apply is replay-safe work after that.
We did not put a Kafka cluster in front of the writer to get a durable log. Kafka is a product. We needed a commit bit.
The protocol is a pair of objects and a reclaim-after-apply loop, not a new message bus. The decision is the product: the log is the engine's ledger, and the Iceberg snapshot is a later, replay-safe apply.
Why Rust, not Java
Three reasons, none of them "we like rewriting things."
One binary. brew install skipprd/tap/skipprd is the product. No JVM, no checkpoint directory, no TaskManager heap, no Flink distro pinned to a Scala line. Connectors download on first use so the engine stays small. Operators already know how to run a binary. They do not want to learn another cluster.
The write path is CPU-heavy and memory-honest. CDC ingest is decode, type, fan out nested structs, encode Parquet, commit Iceberg metadata. That is a lot of allocations next to a lot of small writes. Rust lets us take the multi-threaded Parquet write without a GC pause in the middle of a snapshot, and without pretending the JVM's allocator is free because "the cluster has headroom."
Fearless concurrency is a shipping constraint. Local CDC, WAL append, and Iceberg writer run together on one box. Data races in that loop are data-loss bugs. We would rather the type system argue with us than a customer argue with a duplicated order.
Java would have gotten us to a prototype faster if the prototype was "call Flink." It would not have gotten us to a process you can run next to the database it is reading.
The anti-Flink line on the homepage is not branding. It is the install story. No JVM cluster. One ultra-lightweight Rust binary.
What we refused to wrap
The tempting product is a thin scheduler around Debezium, Kafka Connect, and an Iceberg sink. You can ship that as "software" and still be a services firm with YAML.
We refused it because the failure domains stay yours. When the slot falls behind, when the sink compact stalls, when schema drift hits a nested field Kafka Connect mapped as a string, the customer is still operating four systems. Skipprd has one recovery procedure and chaos tests that SIGKILL the process during ingest and assert the destination still matches.
Fivetran is the other default: send us the data, we will bill you by volume, and the copy lives in our plane. That is a reasonable product for teams who want a vendor. It is the opposite of "an open Iceberg lake. You own it." Skipprd runs in your VPC. The files are yours. There is no volume-based surprise because we are not metering your rows through our network.
What the engine actually does
You describe a pipeline in skippr.yml. skipprd:
- samples the source and infers a schema, including nested objects and arrays
- evolves that schema when the source grows a field
- captures CDC including updates and deletes
- commits through the WAL
- writes Iceberg/Parquet (or Snowflake, BigQuery, Postgres, DuckDB — same ingest, different sink)
- lets you
skipprd querythe lake table plus rows that just arrived
Agents and notebooks then read those tables. Not a scrape of the public web, and not yesterday's warehouse batch copied into a prompt. Live transactional state, local DuckDB, private cloud.
If you want to argue with this
Install it. The binary does not need us on the call.
brew install skipprd/tap/skipprd
skipprd --versionThen Developer Docs for connect, discover, sync, and query. If you want the first pipelines landed in your VPC by the person who wrote the WAL, that is Enterprise Jumpstart — optional, on top of software that already runs.