Apache Gobblin alternative — Apache Gobblin vs Skippr

Apache Gobblin is a distributed ingest framework. LinkedIn built it for Hadoop, donated it, and it became a TLP in January 2021. Jobs pull Kafka, JDBC, files, and copies into HDFS or object storage. FastIngest is their streaming path: write ORC, publish Iceberg metadata, cut Kafka-to-lake latency from about 45 minutes to about 5. Gobblin-as-a-Service (GaaS) plus Temporal is how LinkedIn schedules the rest. The ASF board still marks the project Ongoing in 2026 (Iceberg partition fixes, GaaS latency, manifest copy). Commit volume is a handful a quarter. There is no public SaaS price. You run it. Wikimedia and others have already asked whether to teach it S3 or replace it.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No Hadoop, no GaaS, no Temporal. ELT into a warehouse is one path, not the product.

Feature	Skippr	Apache Gobblin
Deploy	Single Rust binary, your VPC or on-prem	JVM jobs on a cluster. GaaS and Temporal if you want their scheduler
Schema	In-stream evolution, sibling columns	Dataset descriptors and Iceberg metadata you publish
Storage	Open Iceberg on Parquet you own	HDFS or object store. FastIngest writes ORC plus Iceberg catalogs
CDC	Log-based; fails closed if the dest cannot honour	Kafka topics, often CDC already landed on the bus. JDBC pulls exist
Query	skipprd query on Iceberg ∪ WAL; history in any Iceberg engine	Presto / Spark after the commit. No engine in Gobblin
Pricing	Free self-hosted engine	Apache-2.0, pay the cluster. See their docs.

Talk through skipprd now

Paul Bot knows the docs. Google Meet as soon as you join, any time of day.

Chat now →

Any time · 20 minutes · Google Meet now

What Apache Gobblin is good at

This is how LinkedIn still lands thousands of Kafka topics. FastIngest's honest win was dropping the Avro-to-ORC conversion hop and using Iceberg so readers see isolated snapshots instead of a Hive partition that is still being written. Lifecycle, copy, and retention were always part of the design, not a later platform. If you already run Hadoop-shaped infra and need that copy-and-organize job, that is a real framework.

Where the job is wider

Gobblin is ingest into a lake you already query elsewhere. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight rows. FastIngest still needs a dedicated cluster allocation. Inline Iceberg commits at high task count create small manifests and commit conflicts; LinkedIn's own write-up moved metadata publish off the data path for that reason. Equality-delete CDC at their scale needs compaction you own. A 2026 Iceberg full-table corruption fix was still an open draft PR.

The project is Ongoing, not busy. A quiet PMC is not a dunk. It is why Wikimedia treated Gobblin as a candidate to replace, not to carry forever.

Skippr will not give you GaaS, Temporal flow groups, or LinkedIn's 10,000-topic bus. Skippr also will not organize an HDFS dataset lifecycle the way Gobblin does. If the job is "LinkedIn-shaped Kafka to Iceberg on a JVM cluster," Gobblin is the narrower tool. If the job is live SQL on Iceberg you run, Gobblin was never that.

Getting started

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

YAML

skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1

CLI

skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema

Python

from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")

export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose

Choose Apache Gobblin if you already run JVM lake ingest at Hadoop scale and want FastIngest-style Kafka to Iceberg.

Choose Skippr if you want one binary, live SQL, and Iceberg files that stay yours. Skippr will not replace Gobblin-as-a-Service.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.