Why Skippr’s write-ahead log lives in S3

August 2026

I confess, I set out to write a buffer for the EL bit of ELT and somewhat accidentally found I'd written a write-ahead log... on S3. Fashonable - ay. Let me take you through what a WAL on S3 actually is, teh challenges (there's many) and where it's approriate.

Briefly, skipprd is a single binary ELT tool, you can put it on a laptop, a VM, or a Lambda. skipprd ingests data from a souce, discovers and evolves schemas and outputs (typically) parquet onto object store, updating Hive/Iceerg, etc metadata. Simple enough right? Sort of.

What I want to do is walk through the thing properly. What a WAL normally means in a database, how Skippr’s started life as something more modest, where a buffer stops being a buffer, and then the S3 design in enough detail that you could argue with it. There will be illustration briefs along the way for figures we will build later as live UI; they are notes for those pictures, not the pictures themselves.

What people normally mean by a WAL

If you have spent time around databases, write-ahead log probably already means something quite specific, and it is probably not Kafka. Before I talk about Skippr I want that meaning on the table, because otherwise it is easy to hear “WAL” and picture a message bus, or a file we fsync because someone said we should.

If a tree falls in the forrest, and a WAL wasn't around to hear it... it did NOT happen.

The original idea is almost annoyingly small. You are about to change a structure (e.g. the INSERT, UPDATE, etc the client requested) that is expensive or awkward to make crash-safe in place — a B-tree page, a bunch of heap slots, an in-memory table that has not been flushed yet — and you do not want a crash in the middle of that change to leave you with a structure you cannot trust. So you write a description of the change to a sequential log first, you make that log durable, and only then do you apply the change to the structure.

• If you die after the log write and before the structure apply, recovery can read the log and finish the job.
• If you die before the log write, the change simply never happened.

If a tree falls in the forrest, and a WAL was't arround to hear it... it did NOT happen.

The log is written ahead of the structure. It's a Write Ahead Log (WAL), and it is your boundry of garunteed consistent durablilty, capiche?

B-trees

Postgres and InnoDB are the examples most people have in their heads. The store is a B-tree (or a heap plus indexes that are themselves trees). Pages are the unit of I/O, the boxes you're stuffing data into to save and pack away. If you write those pages and crash halfway, you can easily have a tree that is not a tree: a parent that points at a have written page, an index entry with no heap tuple, a heap tuple with no index entry... you get the idea. In practice, you can't just retry the page write because you don't know which writes landedand where.

LSM trees

RocksDB, LevelDB, and Cassandra (and a family of descendants) do something that looks different on the surface and is the same idea underneath (commence twitter nerd fight).

Incoming writes go into a memtable, which is an in-memory sorted structure. That is a buffer in the ordinary sense: it is there so you can absorb a lot of writes

Periodically the memtable is frozen and flushed to an immutable sorted file on disk, an SST. Later, Reads merge the memtable with the SST files which requires some periodic shuffling of data for longer term read perofrmance.

We'll not get into the weeds of it, but folks get their knickers in a twist debating B-trees vs LSM trees, which is only possible when each side failse to acknowladge the dsicussion hindges on a read/write latency trade-off. You just need to understand your workload and pick your tool.

The WAL is not the product

So we must first write a record describing the mutation — enough redo information to (re-)apply it again (a -> b) — and persist that record or group of records to druable store (disk, S3, magnetic tape, stone tablet), before we consider the transaction committed and can return a success response (ACK) to the client.

Skippr WAL, is basically an incarnation of LSM tree. Which makes sense, because Skippr does the EL in ELT, it's high write thoughput. Hence, I naively though I was building a buffer... and was, untill I started picking my way though consistency in distributed systems. Which we'll now discuss.

Where a buffer stops and a WAL starts

A buffer and a WAL can look identical in the happy path. Records go in, a file comes out, the destination gets a batch.

The problems start when you decide the buffer has to garuntee data consistency for a SIGKILL. If the process dies, whatever was in memory is gone. If you had already ACKed the source, you have data loss. If you had not ACKed the source, you will re-read, which is annoying but survivable for some sources (e.g. Kafka, S3, MQTT) and disastrous for others (HTTP, TCP, stdin).

Even worse, IMHO, if you had spilled some files to disk “so it is a bit safer” without a commit protocol, you now have leftover files with absolutly no idea what they contain or how complete they are, surely the definition of not consitent: were they applied? where are they? is this one complete? should this one exist at all?

Skippr’s apply target is its Data and Schema Sink plugins — Snowflake, Postgres, Iceberg, a bucket of Parquet, Kafka, TCP, whatever the pipeline is for — it is an abolute hard constraint, that messages are delivered exactly once (so long as the sink supports it).

Therefore the Skippr WAL must have one recovery procedure that is safe in every eventuality. To test this, we run chaos mode (SKIPPR_CHAOS_MODE=yes) which injects random SIGKILL during ingest and assert exaclty-once delivery garuntees.

Illustration 0 — Where a buffer stops

• Later UI: four-row comparison table as cards, plus a small three-column “classic WAL” strip above it.
• Layout / labels: Strip of three: (A) B-tree — WAL ahead of pages, tree is the store. (B) LSM — WAL ahead of memtable, SST flush drops WAL. (C) Skippr ingest — WAL ahead of the sink, slice-complete drops the segment. Four rows under Buffer vs WAL: State, Commit, Durability, Consistency under all failures. Buffer column: leftover files or nothing; flush when convenient; lost on SIGKILL; undefined. WAL column: committed set; all-or-nothing boundary; persist visible before ACK; one recovery procedure for every crash window.
• Implementation to encode: the exam is the product; Skippr’s reclaim-after-apply matches the LSM lifecycle with the sink as the SST.
• Do not show: a fake history of Skippr logos, or “we always knew.”
• Caption: I started with the left column. The product is the right column.

Once it is a WAL, the disk is an identity problem

A local-disk WAL is a completely respectable design. Postgres has been doing it for decades. Our WAL_STORAGE=disk path still exists, it is the default, and for a long-lived host with a persistent DATA_DIR it is the simple thing. The trouble starts when you notice that the disk has become part of the process’s identity.

If the WAL is on the volume attached to this VM, then this VM is no longer cattle. You can snapshot the volume, you can attach it to a replacement instance if you are careful, you can run a network filesystem and inherit a different set of problems. What you cannot do, trivially, is let the orchestrator throw the instance away and start another one somewhere else with an empty disk. The new process has no committed set. If you also kept offsets on that disk, it does not even know where the source had got to, except by hoping the source will rewind. If you ACKed the source against a WAL that just vanished, you have a hole. The machine was the ledger.

That is fine for a beefy always-on box. It is not fine for Lambda, for spot, for autoscaling groups that replace nodes because they feel like it, or for the sort of “run this pipeline once an hour in a container and go away” job that a lot of ELT actually is. Once I had a WAL, I wanted the compute to be disposable. The log had to live somewhere that outlasts the process without requiring me to babysit a volume.

S3 is the obvious place if you are already an AWS-shaped data pipeline. The lake is there. The metadata is there. The IAM story is there. Object storage is also, famously, not a POSIX disk. There is no fsync of a directory. There is no rename that is a commit on all implementations in the way people pretend. A PutObject is atomic for that key. Two keys are two atomic operations, not a transaction. Multipart upload can leave you with an unfinished upload. Listing is not a snapshot of the world. If you want a WAL on S3 you have to invent a visibility protocol that makes those facts into a commit, rather than pretending S3 is a disk with higher latency.

That is the motivation for WAL_STORAGE=s3. Not “S3 is trendy.” The WAL had become the product’s recovery contract, and the disk it lived on had become an identity, and I did not want ingest compute to have an identity.

Illustration 1 — Three ways teams fake an ingest WAL

• Later UI: three cards. Shared footer row: “After SIGKILL, is the batch owned?”
• Layout / labels: (A) Kafka cluster — brokers + topic partitions; ownership = committed offset on the log. (B) Local disk WAL + sticky VM — DATA_DIR fused to the machine; ownership = fsync on that volume. (C) S3 WAL — disposable runner; two objects {id}.seg + {id}.seg.commit; ownership = both keys visible.
• Implementation to encode: compute in (C) has no identity; restart on another host is valid. (A) and (B) still have a place, but they are not this design.
• Do not show: Cloud metal, Firecracker, vendor logos.

What the industry actually does

Skippr didn't invent ingest-time durability and it's worth considering why a S3 WAL isn't simply a refusal to run Kafka or some other cluster of disks.

Kafka as the ingest log. This is the serious version. You stand up a cluster, producers write, consumers apply to the warehouse, offsets live in the consumer group. After a crash, the consumer resumes from a committed offset. Ownership is the log’s. It works. It also means you are now operating a distributed log in order to copy Postgres into Snowflake, which is a lot of machinery for an ELT runner, and a lot of cost if the thing you wanted was “run this pipeline.” Kafka’s produce latency can be low; its operational surface is not. We refused to operate a broker just to ingest.

Local fsync WAL. Database-shaped, as above. Fast, well understood, sticky compute. Our disk mode.

Flink / Spark checkpoints. These are snapshots of operator state, periodically, to durable storage, often S3. They are not a live ingest WAL. Between checkpoints, the job’s memory is the truth. After a crash you rewind to the last checkpoint and replay sources from whatever offsets that checkpoint held. That is a legitimate model for stream processing. It is not “every source batch is owned when we ACK it.” The checkpoint interval is the durability interval. People blur this because the checkpoint files live on S3, which looks a bit like a WAL.

SaaS at-least-once plus sink upsert. Fivetran, Airbyte, and friends typically push again on failure and hope the destination can take a duplicate, often via a primary key merge. That can converge. It is not a WAL. There is no committed ingest set on the runner’s side that you can recover independently of the warehouse. If you needed to know what you owned before the sink applied, you do not have it.

Skippr’s S3 WAL is trying to be the first kind of contract — a committed ingest set that survives the process — without the broker, and without fusing that set to a volume. The destination still has to be replay-safe, because compaction can run twice. Exactly-once in our sense is WAL ownership plus sink apply rules, not “the PUT happened once.” I'll write about the CDC side of those apply rules separately; they are carried as context inside the log and facinating.

What we are trying to buy

For an ELT ingestion log, throughput matters more than produce latency. It's implicit in the requirement or large batches of output data, typically Parquet.

WarpStream showed in production, that you can treat object storage as the log by streaming batched collections into S3 rather than fsyncing every record to a broker disk. I'm greatful to them for launching right at the time I needed the confidence that this was not a silly idea.

By accepting tens/hundreds of milliseconds of ACK delay, you can fill a few megabytes, pay one multipart upload and one tiny commit PUT, and get throughput and a predicatable bill.

Skippr supports a Clustered Disk WAL mode: local segments, synchronous replicas, a primary leases, gossip protocols, consus over qourum of nodes, stale and joining node catchup. It's complex, but tried and tested and lighting fast on NVME's. There, I said it.

In the recent twitter hype-cycle (that I'm nakedly taking advatage of) around S3 WAL's, Per-record S3 PUT latency is the wrong hill to die on. Or even the incedible avaliblitity of S3. It's value is the throughput and bone headed simplicity of the co-ordination it offers.

Illustration 2 — Ingest cut line

• Later UI: horizontal pipeline as connected cards; a vertical “durable boundary” rule through the WAL card.
• Layout / labels: runtime source plugin → host ingest buffer → live accumulator (SEGMENT_LIVE) → coalescing WAL writer → S3 {.seg,.commit} → compaction (slice jobs) → sink plugin. ACK arrow returns from the commit card to the source, not from the sink. Left of S3 is not owned; right of the commit object is owned.
• Implementation to encode: plugins do not open the offsets DB; host is the only writer of durable progress.
• Do not show: Kafka, warehouse as the ledger.

Design principles

Skippr WAL really deals with three contrains.

• WAL-first: nothing is ingested, in the sense the rest of the system is allowed to depend on, until a segment is committed.
• Tiger Style commits: all-or-nothing segments, checksummed, fail closed, no ACK of a maybe.
• WarpStream-shaped S3 batching: throughput over latency for an ingest log.
• Arrow as the record currency: RecordBatches in the live accumulator, Arrow IPC in each PART, not JSON lines we parse again at compact time.

I'll briefly mentions Skipprs offsets database (in implementation of the very interesting research database SledDB). Skippr Offsets are a cache of WAL-visible progress, they provide Data Sources like S3 with a "Has been seen and ingested" semaphore for each object. Some data sources need us to track a offsets, some dont. These offsets are materialised into the Offsets DB during runtime as we commit the WAL. And, also during startup/recovery where we index the WAL files/objects and byte range read all the commited offsets.

Internals auto-tune under fixed ceilings; we did not grow a forest of FOO_BATCH_SIZE knobs for memory. Fail closed on persist error. Compaction is replay-safe via stable compaction_id. One writer per pipeline on this path. Pack many namespaces into one segment so we pay S3 per filled object. Reclaim the segment when every slice has been applied, so the WAL does not become a second lake.

The ingest path itself is straightforward to narrate and easy to get wrong in the details. Runtime source plugins read the outside world and send batches to the host. The host appends those batches into one live accumulator. A single coalescing WAL writer decides when that accumulator becomes a snapshot, streams it to S3 as a SEGF object, and publishes a sibling commit marker. Only then do we ACK the source. Compaction later reads slices of committed segments, groups them by destination work, and applies them through sink plugins. When every slice of a segment is done, we delete the objects. Plugins do not open the offsets database. The host is the only writer of durable progress.

Consistency: S3 is not a transaction

S3 will atomically put or get a single key. It will not atomically put two keys. That is the whole problem of a commit protocol on object storage, and it is why “just PutObject the segment” is not a WAL.

We write the segment body first, as a multipart upload with 8 MiB parts, retries with exponential backoff and jitter, cap of five attempts per step. The object key looks like {prefix}/p_year=…/p_month=…/p_day=/{snapshot_id}.seg. Completing the multipart upload makes that key visible. At that moment we still do not own the batch. A crash now leaves an orphan .seg with no commit marker. Recovery lists objects under the prefix, and it only indexes keys that have a sibling {same}.seg.commit. The orphan is ignored. It can be swept later. It is not in the committed set.

Then we PutObject the commit marker, a small blob whose header carries the part count, the total bytes, and the SHA-256 we just computed over the body. That PUT is the commit. After it is visible, both keys exist, and recovery will find the pair. The source may now be ACKed. Compaction may apply. The process may die.

Tiger Style, in this setting, is that pairing. Nothing before the commit object is durable in the sense we are willing to ACK. Nothing after it is “maybe owned.” Deterministic segment commits, checksummed, fail closed. We did not get a two-object transaction from S3. We got per-object atomicity and we built a protocol on top of it that recovery can evaluate without reading our mind.

List-after-write is not treated as truth on its own. If listing is delayed or incomplete, we might not see a commit yet; the next recovery will. If listing shows a .seg without a .commit, we do not invent a commit. The intersection is the consistency trick.

There is one live accumulator per process, not one per namespace. Concurrent source namespaces merge into that structure. They do not each open a writer, because that would be the naive PUT-per-table design I am about to complain about. Offsets carried in a snapshot are max-merged per offset key: if two batches in the same snapshot advanced the same partition, the committed offset is the later one. The offsets database — sled on disk, or DynamoDB, or Cloud tables depending on how you configured it — is a materialized view of that WAL-visible progress. If a DynamoDB row is missing after a committed write, the next run rebuilds it from S3. That is the opposite of treating the offset store as the log.

Compaction uses a stable compaction_id so that if we apply twice, the sink can see it is the same work. Exactly-once is not “S3 PUT ran once.” S3 PUTs can be retried. Exactly-once is: we own a definite set, we apply it with identifiers and sink rules that make replay safe. For CDC that includes order tokens and tombstones; that story lives in the CDC exactly-once post.

Illustration 3 — Consistency protocol

• Later UI: two-column timeline (success vs crash). SVG sequence, not a flowchart soup.
• Layout / labels: Success: CreateMPU → UploadPart (8 MiB, retry/jitter) → CompleteMPU {snapshot}.seg → PutObject {snapshot}.seg.commit (header: part count, total bytes, sha256) → owned. Crash row 1: die after MPU complete, before commit PUT → orphan .seg → recovery ignores. Crash row 2: die after commit PUT, before sink apply → recovery indexes, compaction replays.
• Implementation to encode: S3 atomicity is per object; the pair is our transaction.
• Do not show: a single PutObject as “the commit,” or TigerBeetle’s replica diagram.
• Caption: Tiger Style: the commit is the unit; a half-written .seg is not a commit.

Durability: ACK after the commit object

A source batch is durable only after that commit object is visible. The ACK back to the source waits on persist. We do not ACK from the sink. Landing in Snowflake is a later, replay-safe step. If we ACKed from the sink, a crash between WAL and sink would look like a hole even though we owned the data, or we would have to treat the warehouse as the ledger, which is what I did not want.

The body carries a footer magic FOOT, a part count, and a SHA-256 of the bytes we streamed. The commit header repeats enough of that to bind the marker to the body we think we wrote. If you cannot checksum it, you should not own it.

Multipart parts retry. Persist error does not ACK. Live bytes are restored into the accumulator so we have not dropped the batch on the floor because S3 blinked. That is fail closed. Chaos mode still SIGKILLs us between WAL and sink, on purpose, so that recovery plus replay is a tested path rather than a comment in a design doc.

We recommend SKIPPR_WAL_S3_BUCKET as a dedicated bucket, falling back to SKIPPR_S3_BUCKET if you have not split them. The ledger for the pipeline should not be mixed with customer lake objects if you can avoid it. Lifecycle policies, permissions, and “why is there a .seg.commit in my analytics prefix” are all worse when you mix them.

On a disk WAL the equivalent of “visible” is Direct I/O plus fdatasync, and a failure poisons that file handle rather than retrying on a descriptor you no longer trust. I mention it only to say the discipline is the same even though the medium is not. S3 WAL does not fsync a volume. It waits for two objects.

PUT latency, and why we batch

The naive S3 WAL, the one you write on a whiteboard in twenty minutes, is: each source flush, or worse each namespace, becomes its own PutObject. Every table in a discover-heavy pipeline gets its own tiny object, its own RTT, its own request charge. Latency becomes “S3 PUT time times table count.” Cost becomes the same. You also end up with a recovery listing that is enormous and a compaction scheduler that is drowning in files of a few kilobytes.

WarpStream’s public work is what made me comfortable not doing that. Batched, streamed collections into S3 are how you get log throughput without a broker disk. For ELT ingest the trade is even clearer than for a Kafka-compatible stream. Nobody’s dashboard is waiting on a 5 ms ingest ACK so they can confirm a row hit the WAL. They are waiting on the warehouse table being right. We will wait hundreds of milliseconds to fill a segment. We will not pay a PUT per table to shave ACK time.

The writer is a single task with a queue whose capacity is derived from ingest threads, clamped between 1 and 32. Source batches arrive as commit units and are appended into the live accumulator. Pending ACKs wait. Flush happens when live bytes reach the target — WAL_BYTES_PER_FILE, auto-derived from BUFFER_THRESHOLD_BYTES and clamped to 4–64 MiB — or when a delay budget fires. The delay is adaptive: at least 100 ms, at most 500 ms, typically around four times the exponential moving average of persist latency, and also capped by how long we think it will take to fill the rest of the segment at the current ingest rate. If the segment is already at target, we flush immediately and the delay collapses toward the 100 ms floor. Cold start uses 250 ms because we do not have a persist EMA yet.

That ACK is “owned in the WAL,” not “landed in the destination.” Pending ACKs share the next flush, which is the whole point of coalescing: many source units, one MPU, one commit PUT. You pay S3 per committed segment, not per table.

WAL_MAX_DELAY_SECONDS is a coarse ceiling, default 60 seconds, so a quiet pipeline still commits. Runtime ingest ACK latency is auto-tuned from observed throughput and persist latency; that coarse ceiling is not the knob I expect anyone to live in.

Illustration 4 — Cost: PUT per table vs fill one segment

• Later UI: split comparison, two stacked card columns.
• Layout / labels: Left (naive): six namespace chips, each firing its own PutObject. Annotation: six RTTs, six request charges, six tiny objects. Latency and cost explode with table count. Right (Skippr): six chips flowing into one SEGMENT_LIVE tank with a fill gauge (current bytes / WAL_BYTES_PER_FILE, 4–64 MiB derived from the buffer threshold). Flush fires at full or at the time budget. One MPU + one tiny commit PUT.
• Implementation to encode: PartitionKey is sink_ref, namespace, partition, time, schema_fingerprint; same key extends batches; distinct keys sit side by side; offsets max-merge.
• Do not show: mixing namespaces into one sink apply.

Illustration 5 — PUT latency hidden by coalescing

• Later UI: timing strip (SVG). Dual traces: persist duration vs ACK delay.
• Layout / labels: persist EMA; ACK delay = clamp(100 ms, min(4× persist EMA, estimated fill time), 500 ms); immediate flush if live bytes ≥ target (delay collapses to 100 ms).
• Implementation to encode: writer queue capacity from ingest threads (clamp 1–32); pending ACKs share the next flush.
• Do not show: a tuning dashboard of env knobs, or WarpStream’s Kafka-on-S3 topology.
• Callouts: source ACK is “owned in WAL,” not “landed in Snowflake.” ELT ingest log: throughput over latency; we batch into S3 the way WarpStream gave the industry confidence to, without being a Kafka protocol.

Coalescing namespaces into one live segment

The live accumulator is one GlobalSegment behind a mutex. Every ingest unit is keyed by a PartitionKey: sink reference, namespace, partition, optional time bucket, and a schema fingerprint. If two units share a key, we extend the Arrow RecordBatch list for that key and max-merge offsets. If they differ, they sit side by side in the same forthcoming .seg. CDC row metadata, when present, stays aligned with the batches of its key.

That is how a quiet namespace hitchhikes. Suppose you are ingesting a busy events stream and a trickle of pageviews and a small orders snapshot. If each waited to fill 10 MiB on its own, orders would sit in memory for a long time or flush tiny objects forever. Because they share the live segment, events fills the byte target, orders and pageviews ride along, and we write one object. Compaction does not then smash them into one sink write. Compaction reads by slice, groups work that actually shares a destination transaction, and applies orders as orders. Packing is a durability and cost choice. It is not a mixing of apply.

The published segment is a concatenation of PARTs, one per partition key, plus an optional CDC sidecar on each PART, plus a header and a footer. I want to walk the bytes, because “Arrow WAL” is otherwise a slogan.

Illustration 6 — Packed segment, many namespaces

• Later UI: one wide “tape” card (the object) with labelled bands; a legend card to the side.
• Layout / labels: bands in order: SEGF + version 3 + created_at → bincode offsets blob → repeating PART (Arrow IPC stream + optional CDC sidecar aligned 1:1 with rows) → FOOT (parts_count + sha256). Each PART labelled with a different namespace (orders, events, pageviews) and slice_ordinal, start, len. Small namespaces occupy narrow bands next to a fat one.
• Implementation to encode: one object, many partition keys; compaction addresses a PART by byte range, not by scanning the tape.
• Do not show: Parquet as the WAL format (Parquet is compaction/sink).

The on-disk (on-object) format is version 3. Magic SEGF, a version word, a created-at timestamp, then a length-prefixed bincode blob of the offsets this snapshot is committing. Then, for each partition, a PART: the Arrow IPC stream for those RecordBatches, and if this partition is CDC, a sidecar of per-row mutation kind, event identity, and order token, aligned one-to-one with Arrow row order. Each PART records its slice_ordinal (stable index in the segment), byte start and len for the Arrow stream, and the byte range of the sidecar if any. Then FOOT, the part count, and the SHA-256.

Parquet is not the WAL format. Parquet shows up when compaction produces destination files. If we stored Parquet in the WAL we would be encoding for the sink too early, and schema evolution plus CDC sidecars would fight the file format. Arrow IPC is the currency we already had in memory.

Reading slices, not the whole log

Once a segment is committed, compaction does not replay it as a single stream from byte zero. The index is the API. A compaction job is given one or more slices — ordinals into that index — that share a sink and namespace transaction. For S3, we fetch the segment body once, or hit an LRU that already has it, and then we slice [start, start+len) in process and decode Arrow IPC from that window only. We do not currently issue HTTP Range GETs for those windows. Range would be a reasonable later optimisation; it is not what the code does today, and I am not going to draw it as if it did. Disk compaction seeks the same ranges with Direct I/O.

The LRU sits in the same memory envelope as ingest admission, which I will describe with the trade-offs. While a compaction job is in flight we pin the segment ids it needs so the LRU cannot evict them under the job. If the partition is CDC, the sidecar’s row count must equal the Arrow row count or we refuse the slice. Silent drift between payload and mutation metadata is how you get a “WAL” that no longer means what you ACKed.

Grouped compaction may prefetch several windows that belong to the same work item so the sink sees a stream rather than a trickle of tiny applies. Eager mode loads small groups entirely into memory; larger groups stream. Either way, the addressing is ordinal plus byte range, not “read the log until you recognise this table.”

Illustration 7 — Compaction reads a window, not the log

• Later UI: object tape on top; a highlighted [start, start+len) window; below, a compaction job card holding a pin.
• Layout / labels: GetObject (or LRU hit) of the whole .seg into the body cache → decode Arrow IPC from the window only → grouped job may prefetch several windows that share sink/namespace.
• Implementation to encode: slice_ordinal addresses the completion bitmap; CDC sidecar row count must equal Arrow rows or the slice is refused; pin prevents LRU eviction while the job is in flight.
• Do not show: HTTP Range GET as the current path (today: fetch body, then slice in process).

Closing when done

A committed segment that lived forever would be a data lake with a worse query path. The LSM lesson applies: once apply is durable, the WAL records that protected it should go away.

After a successful apply, compaction marks those ordinals complete in a per-segment bitmap. The on-disk (local) sidecar is a small SCBL file: magic, version, bit count, bits, SHA-256 of the preceding bytes. Bits are OR-durable. Concurrent compactors serialize per segment so two jobs cannot clobber each other’s bits. We migrated off per-slice tombstone files; the bitmap is the ledger of “this ordinal is done.”

When every indexed slice is complete, the segment is closed. For S3 that means DeleteObject on the .seg and on the .seg.commit, drop the body cache entry, remove the completion state. For disk it means deleting the file the same way. If the bitmap is unreadable we refuse to delete, because deleting a segment whose completion we cannot prove is how you punch a hole in the owned set.

Incomplete segments stay. Recovery reindexes them. Compactors continue from the bits that are already set, not from scratch. That is the difference between a WAL and a folder of files you reprocess entirely because you cannot remember what you did.

Durability here is temporary by design. We own the batch until it is applied, then we reclaim. If you need a long-term copy, that is the destination, or a lake snapshot, not the ingest log.

Illustration 8 — Bitmap then delete

• Later UI: the same tape, with a bit-row under each PART (SCBL completion bitmap). Three static frames are fine: 2/5 bits, 5/5 bits, objects gone.
• Layout / labels: after all bits set, DeleteObject .seg and .seg.commit; drop body cache; remove bitmap.
• Implementation to encode: bits OR-durable; concurrent compactors serialize per segment; refuse delete if the bitmap is unreadable.
• Do not show: WAL as an infinite archive / data lake.
• Caption: Durability is temporary — owned until applied, then reclaimed.

Recovery

Startup on the S3 path does not trust the local disk, because the local disk may be empty. That is the point of disposable compute. wal_recover_s3 lists objects under the hive prefix, partitions them into .seg keys and .seg.commit keys, and only processes segment keys whose commit sibling exists. For each pair we download the body far enough to parse metadata — we need the index and the offsets blob — mark those offsets durable in the host’s offset view, and register the segment in the cache without retaining the body. Bodies come back later if compaction needs them, through the LRU. Keeping every recovered segment hot in RAM would make recovery itself a memory incident.

If DynamoDB offset rows are missing, this rebuild fills them. Cold start on an empty volume is supported. Uncommitted multipart uploads and orphan .seg objects are not owned. Compaction then looks at the completion bitmaps and continues. We do not replay from the warehouse. The warehouse is not the ledger.

Hive-style keys (p_year=, p_month=, p_day=) keep listings from being a single unbounded prefix of the entire history of the pipeline, which matters once you have been running for months and have not yet reclaimed everything. Recovery cost is still listing plus a metadata read per committed segment that still exists. That is a real trade-off against disk WAL, where recovery is a directory scan of a volume you already have mounted. If you have not been reclaiming, you will feel it. Close-when-done is not optional hygiene. It is how recovery stays bounded.

Illustration 9 — Intersect, then rebuild

• Later UI: three stacked cards: List → Intersect → Rebuild.
• Layout / labels: List: hive prefix {tenant}/{workspace}/{pipeline}/…/p_year=/p_month=/p_day/ showing mixed .seg and .seg.commit keys, plus one orphan .seg. Intersect: two-set filter; only pairs proceed; orphan greyed. Rebuild: parse SEGF metadata → materialize offsets (max) → register segment cache without body → compaction resumes from existing bitmap, not from empty.
• Implementation to encode: DynamoDB offset miss is fine; S3 WAL is source of truth. Empty local disk is a supported cold start.
• Do not show: “replay from the warehouse.”

Illustration 10 — Three layers, one source of truth

• Later UI: three stacked cards with a source-of-truth badge only on the WAL card.
• Layout / labels: WAL (source of truth): committed {.seg,.commit} pair. Offsets DB (view): 24-byte layout, sled or DynamoDB, rebuilt from WAL. Sink (apply): replay-safe compaction_id; warehouse is not the ledger. Arrows: WAL → offsets (materialize); WAL → compaction → sink. Sink never arrows into WAL.
• Implementation to encode: source plugins do not write offsets.
• Do not show: source plugins writing offsets.

Availability, and why we are not master/master

S3’s availability is the availability of the store. Ingest availability on this path is one writer process. I want that said plainly, because “the WAL is in S3” is sometimes heard as “this is a multi-AZ active-active log.” It is not.

If the process dies, another machine can start, recover from committed pairs, and continue. The compute is disposable; the log remains. There is no peer quorum, no lease, no hot standby in WAL_STORAGE=s3. We start no cluster services. Two processes writing the same pipeline at once are not a supported HA topology. They are a mess.

Why not master/master. Two writers coalescing into one live segment would race on partition merge, on offset max, on CDC sidecar alignment, and on compaction_id identity. S3 has no compare-and-swap that spans a .seg and a .seg.commit as a pair. You can do conditional puts on one key. You cannot transactionally say “create these two keys iff nobody else did.” A shared object-store log without a sequencer is dual-write fiction. Kafka’s trick is the broker: there is a leader for the partition, there is an ISR, there is a committed offset that means something. We refused to operate that broker. Pretending S3 PUT order is a total order across two processes is how you get a log that cannot decide which write happened.

Master/master, done honestly, would mean partitioned logs per writer plus a merge and epoch story, or a consensus sequencer in front of object storage. That is a different product. I am not going to draw a consensus ring as if S3 WAL had one.

Clustered mode is the honest alternative we did build, and it is still not multi-master. One exclusive ActivePrimary per pipeline, lease-fenced, two durable disk copies, failover by promotion. The architecture write-up says it does not allow concurrent writers for one pipeline, and I agree with that sentence. Automatic continued writes after a failure need three live processes: the promoted replica plus a new synchronous replica. Two nodes preserve data but cannot restore quorum after either is lost. Availability of ingest, on that path, is failover. It is not dual-active.

Future availability, said out loud and without a fake roadmap: higher ingest availability is that clustered path. A future S3-mode standby would still be failover — a second process that does not write until the first is gone — not two live writers. Multi-AZ storage is already S3. Multi-active ingest is the thing we are not doing.

Illustration 11 — Why one writer

• Later UI: three cards, last two valid, first marked invalid.
• Layout / labels: Invalid: two runner cards both MPU/PUT the same {id}.seg / .commit. Crossed arrows, “no CAS on the pair,” races on live merge / offset max / CDC alignment / compaction_id. S3 WAL (valid): one writer; process dies; new host recovers from committed pairs. Storage HA is not ingest multi-master. Clustered (valid, different product): ActivePrimary + lease + two disk copies; promotion is failover; still one writer. Small future callout: standby on S3 WAL would still be failover; multi-AZ storage is already S3.
• Implementation to encode: availability of ingest is failover, not dual-active.
• Do not show: a consensus ring drawn as if S3 WAL had one.

Trade-offs

S3 persist latency is worse than a local fdatasync. That is why we coalesce and why ACK delay tracks persist EMA. If you need sub-ten-millisecond ingest ACK, this is the wrong log; disk WAL on a quiet box is closer, and a broker is closer still. For ELT, I have not found that to be the constraint.

Recovery listing costs money and time in proportion to how many committed segments still exist. Reclaim is the mitigation. If compaction is stuck, the WAL grows, recovery slows, and you will notice. That coupling is real.

We hold Arrow batches in the live accumulator until persist succeeds. Memory is the other side of batching. The envelope is one pool: 70% of the OS memory hint — MemAvailable on Linux, total RAM on macOS — split between ingest admission and the S3 body LRU, with floors and ceilings in code (admission 256 MiB to 64 GiB, LRU 32 MiB to 16 GiB, combined not more than the 70%). There is no operator slider. Compaction pins sit outside eviction so a job cannot lose its body because ingest got busy. If the machine is tiny and cannot satisfy both minima, the split degrades inside the pool rather than inventing a knob.

Illustration 12 — One pool, no knobs

• Later UI: a single bar (70% of MemAvailable) split into admission (ingest live bytes) and S3 body LRU. Floor/ceiling annotations: admission 256 MiB–64 GiB, LRU 32 MiB–16 GiB, combined ≤ 70%.
• Layout / labels: autotune from /proc/meminfo MemAvailable (Linux) or HW_MEMSIZE (macOS).
• Implementation to encode: no FOO_BATCH_SIZE env; compaction pins sit outside eviction.
• Do not show: operator sliders.

Reclaim depends on sink progress. If the destination is down, segments stay, which is correct — we still own the data — and also means S3 fills up. A buffer that dropped data when the sink was sad would look cheaper. It would not be a WAL.

S3 listing consistency is handled by the commit pair, not by hoping the list is complete in the millisecond after PUT. A commit you cannot see yet is a commit you will see on the next recovery. An orphan you can see is still an orphan.

We do not checksum-compare the commit header against the body on every recovery in a way I want to oversell as a full end-to-end audit of every byte; we parse metadata, we trust the footer we wrote, and compaction re-reads bodies when it applies. Defence in depth could go further. I would rather be honest about what recovery indexes than claim a verification story we do not run on the hot path.

What I am not claiming

This is not Kafka. This is not WarpStream. This is not TigerBeetle. This is not a multi-master log. This is not the Cloud managed elt clustered-disk WAL. It is an ingest write-ahead log for the installable runner, stored as Arrow-backed segments in S3, with a two-object commit, a coalescing writer, slice-oriented compaction, and reclaim when apply is done.

I started with a buffer because buffering is what you do when you want a pipeline to be efficient. I kept going until the buffer had a committed set, a commit protocol, durability that survived the process, and a recovery procedure I was willing to run under SIGKILL. At that point it was a WAL, the disk had become an identity, and S3 was the place a disposable runner could still tell the truth.

If you want the CDC apply rules that sit on top of this log, I wrote that up in Change Data Capture (CDC): Real-Time Replication with Exactly-Once Guarantees. If you want to run the runner, start at Install. The shorter engineer-facing summary is on ELT for engineers.