Luigi alternative — Luigi vs Skippr

Luigi is an Apache-2.0 batch workflow library from Spotify. You declare Task classes with requires, output, and run; a central scheduler coordinates workers. Latest on PyPI is 3.8.1 (7 May 2026). Python 3.10 through 3.14. There is no SaaS SKU and no first-party Iceberg writer. CDC is a task you author. Targets are files, S3, or a database you wrap. This is not Iceberg ∪ WAL you query in one binary.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No central scheduler process for ingest. ELT into a warehouse is one path, not the product.

Feature	Skippr	Luigi
Deploy	Single Rust binary, your VPC or on-prem	Scheduler + workers you run. No managed cloud
Schema	In-stream evolution, sibling columns	Target paths. No Iceberg schema of its own
Storage	Open Iceberg on Parquet you own	Whatever `Target` you implement
CDC	Log-based; fails closed if the dest cannot honour	A task you author. Luigi is the schedule
Query	`skipprd query` on Iceberg ∪ WAL; history in any Iceberg engine	After `run` finishes. Visualiser shows the DAG
Pricing	Free self-hosted engine	Apache-2.0. You pay the machines.

What Luigi is good at

Python tasks with explicit outputs, so a batch graph does not rerun work that already landed. That is the product. The central scheduler and the visualiser are why some shops still run Luigi next to cron. If the job is "this task after that target exists," that is a real product.

Where the job is wider

Luigi is a batch scheduler. It is not a Streamhouse. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight lake rows. Iceberg still starts in PyIceberg or Spark inside run. Postgres CDC still starts in Debezium or skipprd. Skippr will not replace luigid or your existing Task graph.

The scheduler process and worker hosts are the ops. Skippr will not run Luigi. Airflow, Prefect, and Dagster are separate pages.

If the job is "Luigi we already staff," Luigi is the narrower tool. If the job is live SQL on Iceberg you run, the task graph was never that.

Getting started

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

YAMLCLIPython

yaml

skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1

bash

skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema

python

from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")

bash

export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose

Choose Luigi if you want a Python batch scheduler, and Iceberg or CDC as tasks you already write is enough.

Choose Skippr if you want one binary, live SQL on Iceberg ∪ WAL, and files that stay yours. Skippr will not run Luigi.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.

Luigi alternative — Luigi vs Skippr ​

What Luigi is good at ​

Where the job is wider ​

Getting started ​

Which to choose ​

Luigi alternative — Luigi vs Skippr

What Luigi is good at

Where the job is wider

Getting started

Which to choose