Appearance

Book a call

DST in skipprd ​

November 2026

Yep, I have been reading TigerBeetle's code again.

If you build storage or ingestion software in 2026 and you do not have at least mild VOPR envy, you are probably not paying attention. TigerBeetle's simulator runs their real consensus and storage engine inside a single-threaded process where clocks, disks, and network links are all deterministic fakes, ages a cluster by months in a few minutes, and replays any bug from a u64 seed and a git commit. They even put it in a browser at sim.tigerbeetle.com.

We have a module in skipprd called src/sqlrt/dst/. It runs on every precommit hook and in a dedicated linux_dst job on every release tag. I am very proud of what it caught. I also just audited every file in that directory, and if a TigerBeetle engineer opened it today, they would politely point out at least four places where we are still bluffing.

Here is how we ended up building it, what it actually does under the hood, and what we need to fix next.

Why our test matrix stopped working ​

We learned the hard way that hand-written test matrices have a ceiling.

Before we shipped in-stream schema evolution to beta clients, I wrote what I thought was a comprehensive matrix of tests: new columns, widened integers, nested structs, arrays. Then real production traffic arrived from a client's field-device telemetry archive and walked straight around the matrix. One object had a nested record, the next had a mixed-type array, the third had a leaf field that changed from an integer to a string halfway through the hour, and a Parquet batch arrived with Decimal128(15, 2) that needed rescaling to (38, 9) without dropping the stream.

A test matrix is only a list of the bugs you were imaginative enough to invent before lunch. For a system where events arrive in ugly combinations, testing individual scenarios is not enough. You have to test properties across long, awkward sequences of events.

And schema evolution was not even the hardest state machine in skipprd. The hardest state machine is the Streamhouse read and commit boundary: Iceberg ∪ WAL.

What TigerBeetle's VOPR actually does ​

Before comparing what we built, it is worth being precise about what TigerBeetle does, because people throw the letters "DST" around as if adding proptest to a cargo workspace makes you FoundationDB.

In TigerBeetle's VOPR (the Viewstamped Operation Replicator), each node in the cluster is a simulated replica VM living inside one OS process (src/testing/cluster.zig). There is no Docker, no QEMU, and no wall-clock sleep. Every replica gets its own simulated storage grid (which can corrupt sectors, misdirect reads and writes, or drop flushes), its own simulated network path (packet loss, reordering, asymmetric partitions), and its own simulated clock (TimeSim) with configurable linear drift.

Look at how they model a VM pause (replica_pause): when VOPR pauses a replica to simulate a live hypervisor migration or a GC stall, the replica stops stepping its storage and stops processing ticks, but its simulated clock keeps ticking. When the replica unpauses, time has moved on around it. Leases, heartbeats, and views have advanced while the machine was frozen.

On top of those replica VMs, TigerBeetle runs protocol-aware checkers. In VSR, replicas move through numbered views (view_normal, view_change), and storage moves through checkpoints and LSM compactions. The simulator does not just wait until the end of a run and run a SELECT to see if the numbers look plausible. It maintains a time-versioned view of the cluster's history at every tick, peeking inside each replica's journal, grid blocks, and prepare headers to assert that no committed state was lost and that caught-up replicas converge to byte-for-byte identical storage. Because the checker knows the invariants of every transition, VOPR can run hundreds of thousands of random seeds without a human writing down the expected end state for each seed.

What skipprd actually runs today ​

skipprd is not a five-replica consensus ledger. It is a single-binary Streamhouse engine with one writer per pipeline. Our nightmare is not split-brain view change across five NVMe nodes. Our nightmare is the seam where a live Write-Ahead Log meets an Apache Iceberg table:

  1. A CDC source streams inserts, updates, and deletes into WAL segments ({id}.seg plus a 60-byte {id}.seg.commit marker with the SHA-256 digest of the body).
  2. A background compaction worker groups committed WAL slices, merges them into Iceberg Parquet files, commits a new Iceberg snapshot S, and records the compacted slice refs (segment_id, start) in the Iceberg table properties under skippr.wal-slices.
  3. Once every slice in a WAL segment is marked done in its SliceBitmap and sits below the stream processor hold cursor, reclaim deletes the segment from the catalog and object store.
  4. While all of that is happening, a user or a downstream service runs skipprd query.

If skipprd query reads Iceberg snapshot S after compaction committed, and also reads the WAL segment before reclaim deleted it, you double-count rows or resurrect a deleted key. If reclaim deletes a WAL segment between query planning (GetFlightInfo) and stream execution (DoGet), the query reads a hole.

To test that seam, we built src/sqlrt/dst/mod.rs. It defines a World and a closed 12-variant Op enum:

rust
pub enum Op {
    IngestLake(UserRow),
    IngestWal(WalEvent),
    CompactThru(u64),
    Failpoint(FailpointName),
    Reopen,
    HoldPin,
    ReleasePin,
    ReleaseCompaction,
    DiskPause,
    DiskResume,
    DeadletterSkip,
    EvolveAddColumn(String),
}

When current_table(&world) computes the visible table at any step in a schedule, it converts the lake rows and live WAL events into Arrow RecordBatches, attaches the _skippr_cdc_* sidecar columns, and calls the exact production merge functions from src/sqlrt/cdc_reconcile.rs: collapse_wal_batches followed by anti_join_lake. Latest order_token wins per business key; equal tokens break ties on (seq, ordinal, row_i); deletes suppress the key from the lake anti-join.

We did not write a second toy merge in the test harness. That part is genuine Tiger Style.

We also built our own miniature version of a time-versioned view around Op::HoldPin and Op::ReleasePin. In production (src/sqlrt/read_boundary.rs), a query pins a triple (S, E, W) at plan time:

  • S: the Iceberg snapshot (LakeSnapshot::Empty or LakeSnapshot::At(snapshot_id)).
  • E: the excluded set of WalSliceRef(segment_id, start) already compacted into S.
  • W: the live WAL slices in MemoryWalCatalog or FjallSkipprStore, filtered to W \ E.

In the DST World, HoldPin captures that boundary while subsequent CompactThru and IngestWal operations keep mutating the world underneath it:

text
IngestWal(id=1, Insert, seq=1)
IngestWal(id=2, Insert, seq=2)
HoldPin                          # freezes pin at (S=Empty, compacted_thru=0, W={seq 1, seq 2})
CompactThru(2)                   # writes Iceberg snapshot S=1, advances live compacted_thru to 2
IngestWal(id=3, Insert, seq=3)   # lands in live WAL after the pin
# current_table(&world) under HoldPin -> {1, 2}
ReleasePin
# current_table(&world) after ReleasePin -> {1, 2, 3}

Around that World, we have 57 named scripts across ten required groups (a_ through j_: append ingest, CDC merge, catalog recovery, COPY wire frames, schema evolution, in-stream ALTER, stream transforms and aggregates, query pins, offsets, and client telemetry shapes). A test called fuzz_corpus_has_every_named_script serialises every named script into our binary DST1 wire format (capped fail-closed at MAX_SCHEDULE_OPS = 1024) and writes it straight into fuzz/corpus/world_schedule/ so cargo fuzz always starts from the real scenario catalogue. Crash windows use a closed six-variant FailpointName enum (BeforeCatalogCommit, AfterCatalogCommit, AfterOffsetsPublished, BeforeCompactionSink, AfterCompactionSink, HoldBeforeCompactionSink), where unknown failpoint names fail closed.

Red-teaming our own DST: four places where we are still bluffing ​

If this were a marketing post, I would stop at the paragraph above and declare victory.

But if you read src/sqlrt/dst/ with a cold eye, here is what is actually going on.

1. Only the merge is production code; World::apply is a shadow model ​

The comment at the top of src/sqlrt/dst/mod.rs says: The current table is collapse_wal_batches + anti_join_lake. There is no second merge oracle.

That is true for the read path. It is not true for the write path.

When apply(world, Op::IngestWal(event)) or Op::CompactThru(seq) runs, it does not call PipelineWalHandle::persist_segment or the real compaction coordinator. It runs commit_ingest and commit_compaction inside dst/mod.rs, a 250-line simplified model that constructs SegmentRow and CompactionRow and commits them to MemoryWalCatalog.

Meanwhile, of the roughly 150 tests sitting under src/sqlrt/dst/, only the 57 scripts in scripts.rs plus proptest_schedules.rs and regressions/ actually run through World. The other files in that folder (store.rs, processor.rs, inventory.rs, catalog.rs, evolution.rs) test the real FjallSkipprStore, leftover sled migration, and stream processor state, but they do it as ordinary unit tests that spin up tempfile::tempdir() and tokio::runtime::Builder::new_current_thread(). Putting a test in a folder called dst/ and prefixing its name with c_ does not make it deterministic simulation. We currently have a pure in-memory World that models the catalog, and a pile of real-disk unit tests next door.

2. World fakes the (S, E, W) query pin by cloning Vec<WalEvent> ​

This is the biggest technical gap between dst/mod.rs and production.

In production (src/sqlrt/read_boundary.rs and src/buffer/wal_reader.rs), (S, E, W) is a query-local list of WalSliceRef(segment_id, start) pointers, not a reclaim lease. Holding a query pin does not block compaction from reclaiming a segment once all of its slices are compacted into a newer Iceberg snapshot. If a slow Flight client calls GetFlightInfo (which pins (S, E, W)), pauses while compaction compacts and reclaims one of those segments, and then calls DoGet, WalCatalogReader::pinned_slice fails closed with PinReadError::Lost(WalPinLost). DataFusion catches WalPinLost and re-pins a fresh (S', E', W') up to QUERY_PIN_RETRY_CEILING = 3 times (src/sqlrt/wal_table.rs).

Now look at how dst/mod.rs implements Op::HoldPin:

rust
pub struct QueryPin {
    pub lake: LakeSnapshot,
    pub compacted_thru: u64,
    pub wal: Vec<WalEvent>,
}

HoldPin clones the WalEvent structs by value into QueryPin.wal! Even if a segment were deleted from world.catalog and world.payloads, the held pin in World would never notice, because it is reading its own private Vec<WalEvent>. And because World tracks compaction as a single scalar watermark (compacted_thru: u64) instead of Iceberg's E: HashSet<WalSliceRef>, World cannot simulate out-of-order slice compaction or partial-segment SliceBitmap reclaim at all. We had to test WalPinLost in a separate one-off unit test (b_reclaim_between_getflight_and_doget_wal_pin_lost in query_pin.rs) because World literally cannot express it.

3. "No second merge oracle" left our fuzzer without an invariant checker ​

Refusing to write a second toy merge function was the right call for the 57 named scripts, where each script carries a hand-worked expect map.

Where it bit us is proptest_schedules.rs and fuzz/fuzz_targets/world_schedule.rs. Here is the entire fuzz target for world_schedule:

rust
fuzz_target!(|data: &[u8]| {
    let world = run(&schedule_from_bytes(data));
    let _ = current_table(&world);
});

And in proptest_schedules.rs, we generate 1 to 12 random Ops (for 32 cases in precommit) and assert that run(&ops) produces the same current_table when you run it twice.

Of course it produces the same table when you run it twice. It is a synchronous single-threaded function over a BTreeMap! Without a reference oracle or step-by-step protocol invariants, random schedule generation only checks that current_table does not panic or underflow. TigerBeetle's VOPR can run 450,000 random seeds because its StateChecker checks protocol invariants on every tick without needing a human to pre-calculate the final ledger balances.

4. We have no simulated clock, no VM pause, and a Uuid::new_v4() in World::new() ​

The small insult first: World::new() calls NodeId::generate() for its CatalogFence, which calls Uuid::new_v4(). It does not break current_table today because the UUID is not part of the row key, but calling OS entropy inside the constructor of a deterministic simulator is a sin we should be mildly embarrassed about.

More importantly, Op::DiskPause in World just sets world.disk_paused = true, which causes Op::IngestWal to return; early. That is not a TigerBeetle VM pause. There is no simulated clock in World, so pausing the writer does not advance lease epochs, expire Flight query tickets, or trigger time-threshold compaction (buffer_threshold_seconds) while an S3 PUT is hanging in mid-air. Nor does Op model the two-object S3 commit boundary ({id}.seg landed, {id}.seg.commit lost or delayed) where real object-store recovery bugs actually live.

How we are fixing it ​

The point of auditing your own engineering in public is to give yourself nowhere to hide. We do not need a five-node VSR simulator in skipprd, and I have no intention of turning an ingestion and query engine into a cosplay consensus project. We do need our single-writer Streamhouse simulator to stop pulling its punches:

  1. Make QueryPin in World hold Vec<WalSliceRef> and E: BTreeSet<WalSliceRef>, not cloned rows. Add Op::ReclaimThrough and split query execution into Op::PlanPin and Op::ExecPin. When reclaim deletes a segment between plan and execute, World should return WalPinLost and exercise the exact 3-attempt re-pin loop that production runs.
  2. Add a transition invariant checker that runs after every Op in proptest and fuzz. We still do not want a second implementation of Arrow anti-join, and we do not need one to check invariants on every step:
    • Every business key has at most one row in current_table(&world), and its value equals the highest (order_token, seq, ordinal, row_i) among admitted WAL events and un-suppressed lake rows (unless that winner is MutationKind::Delete, in which case the key must be absent).
    • If a pin is held and none of its WalSliceRefs have been reclaimed, current_table(&world) under the pin must be bit-identical before and after any subsequent IngestWal or CompactThru.
    • world.offsets never advances across an uncommitted catalog write (BeforeCatalogCommit) and never regresses on Reopen.
    • CompactionState::Acked in MemoryWalCatalog is terminal.
  3. Replace Uuid::new_v4() with a seeded World::with_seed(u64) and inject a logical tick. Give World a deterministic PRNG and a monotonic tick counter. Replace DiskPause with a writer pause where the clock ticks forward while an in-flight .seg body is sitting uncommitted on simulated S3 without its .seg.commit marker.
  4. Drive the real PipelineWalHandle state transitions from Op. Collapse the split between the 57 World scripts and the tempfile unit tests in src/sqlrt/dst/ so catalog commit, slice completion bitmaps, stream-processor reclaim holds (allows_reclaim), and recovery run through the same schedule runner over MemoryWalCatalog and simulated object storage.

Skippr is not done till everything is done. That includes the tests we brag about.

If you want to kick the tyres on the engine as it stands today, start at Install, or read how the CDC merge rules and the Iceberg ∪ WAL architecture fit together in production.