Appearance

Book a call

Amazon Data Firehose alternative — Amazon Data Firehose vs Skippr ​

Amazon Data Firehose is a managed delivery truck for streaming records. You point a stream at a destination. Sources are Direct PUT, Kinesis Data Streams, and Amazon MSK. Destinations include S3, Redshift, OpenSearch, Splunk, HTTP endpoints, and Apache Iceberg tables (self-managed on S3, or Amazon S3 Tables). One stream can route inserts, updates, and deletes into more than one Iceberg table. Lake Formation can gate table and column access. Iceberg dest writes Parquet and Iceberg v2 only. Self-managed tables leave compaction and snapshot expiry to you; S3 Tables and Glue can compact. A November 2024 preview that tailed PostgreSQL and MySQL into Iceberg ended on 30 September 2025. AWS says that path cannot be tried. Iceberg delivery from a Kinesis source is 0.045 USD per GB ingested, with no 5 KB rounding. Extra Lambda bytes bill at the same Iceberg rate. This is not a Streamhouse. There is no Iceberg ∪ WAL.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No per-GB delivery line. ELT into a warehouse is one path, not the product.

FeatureSkipprAmazon Data Firehose
DeploySingle Rust binary, your VPC or on-premDelivery stream in an AWS account
SchemaIn-stream evolution, sibling columnsRecord mapping. Iceberg dest can apply I/U/D
StorageOpen Iceberg on Parquet you ownS3, Redshift, or Iceberg v2 on S3 / S3 Tables
CDCLog-based; fails closed if the dest cannot honourStream records. Database-to-Iceberg preview ended 30 Sep 2025
Queryskipprd query on Iceberg ∪ WAL; history in any Iceberg engineAthena after delivery. Compaction is yours or S3 Tables
PricingFree self-hosted enginePer GB ingested. Iceberg from Kinesis is 0.045 USD/GB.

What Amazon Data Firehose is good at ​

A Kinesis or MSK stream that already needs to land Parquet or Iceberg without a Flink job. That is the product. AppendOnly on the CreateDeliveryStream API is why some estates let Firehose scale inserts. If the job is "this stream into that table," that is a real product.

Where the job is wider ​

Firehose is a delivery service. It is not a Streamhouse. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight rows. You still open Athena after the put.

GB math is the ops. Iceberg dest skips the 5 KB pad that S3 dests use; Lambda growth still bills. The database CDC preview is gone. Skippr will not replace a Firehose that already drains MSK into S3 Tables. Skippr also will not ingest Kinesis as a first-class source today.

If the job is "stream delivery we already staff," Firehose is the narrower tool. If the job is live SQL on Iceberg you run from a database log, Firehose was never that.

Getting started ​

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

yaml
skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
bash
skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema
python
from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")
bash
export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose ​

Choose Amazon Data Firehose if you already have a Kinesis or MSK stream and need managed delivery into S3 or Iceberg.

Choose Skippr if you want one binary, live SQL, and Iceberg files that stay yours. Skippr will not run a Firehose stream.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.