Lakeflow alternative — Databricks Lakeflow vs Skippr

Databricks Lakeflow is the ingest and pipeline brand on a Databricks workspace. Lakeflow Connect is the managed connector layer: Salesforce and Workday on serverless, plus database CDC (SQL Server GA, PostgreSQL and MySQL Public Preview, Oracle LogMiner in Beta). A database connector is three pieces. A Unity Catalog connection holds credentials. An ingestion gateway runs on classic compute, continuously, so the source log is not truncated. A staging volume holds extracts for 30 days. An ingestion pipeline on serverless apply writes streaming tables, which are Delta tables with incremental processing. Auto Loader is the file path inside Lakeflow pipelines or a job. Zerobus Ingest is the push path at 0.050 USD a GB. Every workspace gets 100 free DBUs a day on managed SaaS and database connectors (Databricks's planning figure is up to 100 million records). Auto Loader, Kafka, community connectors, and Zerobus are outside that allowance. Premium list in us-east-1 is 0.35 USD a DBU for serverless jobs, pipelines, and Connect, 0.15 USD a DBU for classic Jobs, 0.20 to 0.36 USD a DBU for classic Declarative Pipelines (Core / Pro / Advanced). The gateway bills even when the apply pipeline is idle. This is not a Streamhouse. There is no Iceberg ∪ WAL.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No DBU. ELT into a warehouse is one path, not the product.

Feature	Skippr	Databricks Lakeflow
Deploy	Single Rust binary, your VPC or on-prem	Connectors and pipelines in a Databricks workspace
Schema	In-stream evolution, sibling columns	Connector mapping into Delta. Auto full refresh can wipe history
Storage	Open Iceberg on Parquet you own	Unity Catalog streaming tables (Delta). Iceberg is a different table type
CDC	Log-based; fails closed if the dest cannot honour	Gateway tails the log. Apply is a second serverless pipeline
Query	skipprd query on Iceberg ∪ WAL; history in any Iceberg engine	Databricks SQL / Spark after the streaming table commits
Pricing	Free self-hosted engine	DBUs. Serverless 0.35 USD. Classic Jobs 0.15 USD. Gateway is always on.

Talk through skipprd now

Paul Bot knows the docs. Google Meet as soon as you join, any time of day.

Chat now →

Any time · 20 minutes · Google Meet now

What Databricks Lakeflow is good at

A Salesforce or SQL Server log that already needs to land in Unity Catalog without you writing Structured Streaming. That is the product. The 100 DBU daily allowance is why some teams start Connect before they buy a second ELT seat. Auto Loader is the honest file watcher when the source is already S3. If the job is "this connector into this workspace," that is a real product.

Where the job is wider

Lakeflow is managed ingest plus Spark pipelines. It is not a Streamhouse. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight rows. Dest streaming tables are Delta. Unity Catalog can host Iceberg, but the managed database connectors do not write it.

DBU math is the ops. The gateway is classic compute that does not sleep. A serverless-only workspace cannot run it. Postgres CDC is still Public Preview. Auto full refresh on a schema event erases history tracking. Skippr will not match Workday or Salesforce the way Connect does. Skippr also will not replace a Lakeflow pipeline that already writes Delta in that workspace.

If the job is "Databricks ingest we already staff," Lakeflow is the narrower tool. If the job is live SQL on Iceberg you run, Lakeflow was never that.

Getting started

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

YAML

skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1

CLI

skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema

Python

from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")

export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose

Choose Databricks Lakeflow if you want managed connectors into Delta streaming tables, and the workspace after apply is enough.

Choose Skippr if you want one binary, live SQL, and Iceberg files that stay yours. Skippr will not run a Lakeflow pipeline.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.