Appearance

Book a call

Apache Beam alternative — Apache Beam vs Skippr ​

Apache Beam is Apache-2.0 portable batch and stream pipelines (Java, Python, Go, SQL, YAML). You write a pipeline; a runner executes it: DirectRunner, Dataflow, Flink, Spark, and others. Latest released SDK is 2.77.0 (7 Oct 2026). 2.78.0 is still RC1. Iceberg is beam-sdks-java-io-iceberg / Managed.read("iceberg") / Managed.write("iceberg") / YAML WriteToIceberg. Tables can be created on first insert. IcebergAddFiles (batch) registers existing Parquet into a table, with schema-evolution options from 2.77. 2.76 added an Iceberg batch and streaming changelog source. CDC from databases is DebeziumIO (optional withFullMetadata for op/source fields in 2.78 RC). There is no licence fee. You pay the runner: Dataflow jobs, a Flink/Spark cluster, or the machines under DirectRunner. This is not Iceberg ∪ WAL you query in one binary.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No runner. ELT into a warehouse is one path, not the product.

FeatureSkipprApache Beam
DeploySingle Rust binary, your VPC or on-premSDK plus a runner (Dataflow, Flink, Spark, Direct, …)
SchemaIn-stream evolution, sibling columnsBeam schema / Row. Iceberg DDL on write
StorageOpen Iceberg on Parquet you ownIcebergIO / AddFiles. Catalog is yours
CDCLog-based; fails closed if the dest cannot honourDebeziumIO or Iceberg changelog source. Not Beam itself
Queryskipprd query on Iceberg ∪ WAL; history in any Iceberg engineAfter the sink commits. Beam SQL is another pipeline
PricingFree self-hosted engineApache-2.0. Pay Dataflow / Flink / Spark.

What Apache Beam is good at ​

One pipeline graph that can run on more than one engine, with Iceberg as a first-party IO instead of a homegrown Parquet writer. That is the product. Managed Iceberg write plus AddFiles is why some Dataflow estates skip a second Spark job. If the job is "this Beam pipeline into this table," that is a real product.

Where the job is wider ​

Beam is an SDK. It is not a Streamhouse. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight lake rows. Postgres CDC is DebeziumIO or a Kafka source, not Pipeline.run(). IcebergAddFiles is batch. The runner is still a cluster.

Dataflow units, Flink TaskManagers, or Spark executors are the ops. Skippr will not replace a Beam job that already fans out to BigQuery and Iceberg. Skippr also will not run Dataflow. Flink and Spark Structured Streaming are separate pages.

If the job is "Beam we already staff," Beam is the narrower tool. If the job is live SQL on Iceberg you run, the runner was never that.

Getting started ​

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

yaml
skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
bash
skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema
python
from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")
bash
export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose ​

Choose Beam if you want a portable pipeline SDK, and IcebergIO on a runner you already pay is enough.

Choose Skippr if you want one binary, live SQL on Iceberg ∪ WAL, and files that stay yours. Skippr will not run Dataflow.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.