Cloud Data Fusion alternative — Cloud Data Fusion vs Skippr

Cloud Data Fusion is managed CDAP. You draw a pipeline. An instance stays up for design. Editions are Developer (0.35 USD an hour, two users, two pipelines), Basic (first 120 hours a month free, then 1.80 USD an hour), and Enterprise (4.20 USD an hour, RBAC, regional HA). Instances bill by the minute from create to delete. Delete loses pipeline data. Execution is a second bill: Managed Service for Apache Spark clusters plus Cloud Storage, networking, and BigQuery. Replication is log CDC from MySQL (binlog), SQL Server (CDC tables, 2016+), and Oracle (LogMiner, versions Datastream supports) into BigQuery only. Google recommends BigQuery flat-rate for replication jobs. There is no Iceberg replication target. This is not a Streamhouse. There is no Iceberg ∪ WAL.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No instance-hour. ELT into a warehouse is one path, not the product.

Feature	Skippr	Cloud Data Fusion
Deploy	Single Rust binary, your VPC or on-prem	Always-on CDAP instance plus Spark per run
Schema	In-stream evolution, sibling columns	Pipeline plugins. Replication assesses schema first
Storage	Open Iceberg on Parquet you own	BigQuery or the sink you wired. Replication is BQ only
CDC	Log-based; fails closed if the dest cannot honour	Replication: MySQL, SQL Server, Oracle to BigQuery
Query	skipprd query on Iceberg ∪ WAL; history in any Iceberg engine	BigQuery after the job
Pricing	Free self-hosted engine	Instance-hours (0.35 / 1.80 / 4.20 USD) plus Spark.

Talk through skipprd now

Paul Bot knows the docs. Google Meet as soon as you join, any time of day.

Chat now →

Any time · 20 minutes · Google Meet now

What Cloud Data Fusion is good at

A visual pipeline that already talks 150-plus CDAP plugins and lands in BigQuery. That is the product. Replication is why some estates skip Datastream for SQL Server CDC tables. If the job is "CDAP in this GCP project," that is a real product.

Where the job is wider

Data Fusion is a design studio plus Spark. It is not a Streamhouse. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight rows. The instance stays billed while you are not running jobs.

Two meters is the ops. A Basic instance up all month is the 1.80 USD line after the 120 free hours, then Spark on every run. Replication is BigQuery, not Iceberg. Skippr will not run a CDAP pipeline. Skippr also will not replace Oracle LogMiner coverage Datastream already lists.

If the job is "GCP visual ETL we already staff," Data Fusion is the narrower tool. If the job is live SQL on Iceberg you run, Data Fusion was never that.

Getting started

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

YAML

skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1

CLI

skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema

Python

from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")

export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose

Choose Cloud Data Fusion if you want CDAP pipelines into BigQuery, and the warehouse after Spark is enough.

Choose Skippr if you want one binary, live SQL, and Iceberg files that stay yours. Skippr will not run a Data Fusion pipeline.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.