Spark Structured Streaming alternative — Spark Structured Streaming vs Skippr

Apache Spark Structured Streaming is Spark's streaming DataFrame API. Current Spark is 4.3.0 (30 Oct 2026). You read a source, transform, and writeStream. Iceberg uses DataSourceV2: format("iceberg"), output modes append (add rows each micro-batch) and complete (replace the table each micro-batch). The destination table must exist before the query starts. Iceberg 1.12.0 (29 Sep 2026) ships runtime jars for Spark 3.5 (Scala 2.12 and 2.13), Spark 4.0, and Spark 4.1. Spark 3.4 support was removed in that Iceberg line. Streaming reads only consume append snapshots; overwrite and delete snapshots throw unless you set streaming-skip-overwrite-snapshots or streaming-skip-delete-snapshots. Iceberg does not support Spark's experimental continuous processing, because that path has no commit interface. Spark 4.1 added Real-time Mode for sub-second Structured Streaming (update output mode, checkpoint required); that is Spark's engine, not Iceberg's sink. Spark 4.2 added a SQL CHANGES clause for CDC on Spark tables, which is not Postgres WAL capture. There is no licence fee. You pay the cluster: drivers, executors, checkpoints, catalogs. This is not a Streamhouse. There is no Iceberg ∪ WAL.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No Spark session. ELT into a warehouse is one path, not the product.

Feature	Skippr	Spark Structured Streaming
Deploy	Single Rust binary, your VPC or on-prem	Spark cluster (YARN, K8s, Databricks, EMR, …)
Schema	In-stream evolution, sibling columns	Spark schema plus Iceberg table DDL. Create the table first
Storage	Open Iceberg on Parquet you own	Iceberg via writeStream.format("iceberg"). Catalog is yours
CDC	Log-based; fails closed if the dest cannot honour	A source you plug in (Kafka, files, Spark CHANGES). Not Postgres WAL
Query	skipprd query on Iceberg ∪ WAL; history in any Iceberg engine	Spark SQL on the table after each micro-batch commit
Pricing	Free self-hosted engine	Apache-2.0. Pay executors. Cloud Spark is DBU / EMR / HDInsight.

Talk through skipprd now

Paul Bot knows the docs. Google Meet as soon as you join, any time of day.

Chat now →

Any time · 20 minutes · Google Meet now

What Spark Structured Streaming is good at

A Spark estate that already transforms streams and needs those DataFrames in Iceberg, with the same catalog the batch jobs use. That is the product. Micro-batch exactly-once plus Iceberg's snapshot commits is why lake teams still write writeStream. If the job is "this Spark query into this table," that is a real product.

Where the job is wider

Spark is a compute cluster. It is not a Streamhouse. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight rows. Postgres CDC is a connector or a Kafka topic in front of Spark, not Structured Streaming itself. Iceberg streaming reads skip or fail on overwrite/delete snapshots. You must create the table before the sink. Real-time Mode is a Spark trigger, not live SQL on object-store WAL.

The cluster is the ops. Checkpoints, executors, catalog jars, and a runtime that must match Iceberg (3.5 / 4.0 / 4.1 on 1.12.0). Skippr will not replace a Spark job that already joins streams. Skippr also will not run a Spark driver. Flink CDC is a separate page.

If the job is "Spark Structured Streaming we already staff," Spark is the narrower tool. If the job is live SQL on Iceberg you run, the cluster was never that.

Getting started

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

YAML

skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1

CLI

skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema

Python

from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")

export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose

Choose Spark Structured Streaming if you already run Spark, and writing those streaming DataFrames to Iceberg is the job.

Choose Skippr if you want one binary, live SQL on Iceberg ∪ WAL, and files that stay yours. Skippr will not run a Spark cluster.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.