Appearance

Book a call

Google Cloud Dataflow alternative — Google Cloud Dataflow vs Skippr ​

Google Cloud Dataflow is Apache Beam as a service. You submit a batch or streaming pipeline. Workers are VMs (default streaming worker is 4 vCPU, 15 GB). Streaming Engine moves shuffle and state off the VM. Resource-based billing meters Streaming Engine Compute Units: 0.089 USD per unit-hour in us-central1 (1-year CUD 0.0712, 3-year 0.0534). Batch Data Compute Units list 0.06 USD. Usage is per second, shown as hours. Persistent Disk, Shuffle, and GPU are extra lines. Iceberg I/O exists: Kafka-to-Iceberg, Iceberg-to-Lakehouse CDC (Beam SDK 2.77+), and a Managed I/O Iceberg CDC read that is append-only snapshots only (full CDC not yet). Streaming Iceberg CDC polls source commits (default 60 seconds, minimum 1 second). Automatic schema evolution is not supported on the Lakehouse CDC path. This is not a Streamhouse. There is no Iceberg ∪ WAL.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No Compute Unit. ELT into a warehouse is one path, not the product.

FeatureSkipprGoogle Cloud Dataflow
DeploySingle Rust binary, your VPC or on-premBeam job on Dataflow workers
SchemaIn-stream evolution, sibling columnsPipeline coder. Lakehouse CDC does not auto-evolve
StorageOpen Iceberg on Parquet you ownSink you wired. Iceberg I/O and Lakehouse CDC exist
CDCLog-based; fails closed if the dest cannot honourIceberg change-log poll, or Kafka. Not a Postgres WAL tail
Queryskipprd query on Iceberg ∪ WAL; history in any Iceberg engineBigQuery / Lakehouse after the job
PricingFree self-hosted enginePer-second workers plus Streaming Engine CUs (0.089 USD in us-central1).

What Google Cloud Dataflow is good at ​

A Beam pipeline that already joins Pub/Sub with a BigQuery sink and a side input. That is the product. Streaming Engine is why some estates keep shuffle off the VM. Iceberg I/O is a real dest if you already write Beam. If the job is "this Beam graph in GCP," that is a real product.

Where the job is wider ​

Dataflow is a job runtime. It is not a Streamhouse. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight rows. The job ends or the streaming workers stay up; then you query BigQuery.

CU math is the ops. A streaming job bills every second it is up. Iceberg CDC here is table-to-table, not a database log. Managed Iceberg CDC is append-only. Skippr will not run a Beam pipeline. Skippr also will not replace a Dataflow job that already drains Pub/Sub.

If the job is "Beam we already staff," Dataflow is the narrower tool. If the job is live SQL on Iceberg you run from Postgres WAL, Dataflow was never that.

Getting started ​

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

yaml
skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
bash
skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema
python
from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")
bash
export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose ​

Choose Google Cloud Dataflow if you already write Beam and need managed workers, and BigQuery after the job is enough.

Choose Skippr if you want one binary, live SQL, and Iceberg files that stay yours. Skippr will not run a Dataflow job.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.