Appearance

Book a call

BigQuery Data Transfer Service alternative — BigQuery DTS vs Skippr ​

BigQuery Data Transfer Service is scheduled ingest into BigQuery. You configure a transfer. Google sources (Ads, Analytics 4, YouTube, Campaign Manager, and peers except Google Play), object storage (Cloud Storage, S3, Azure Blob), and migration connectors (Redshift, Teradata, Snowflake Preview) have no orchestration fee. Paid GA connectors (Salesforce, ServiceNow, SFMC, Facebook Ads, MySQL, PostgreSQL, Oracle) bill slot-hours. Google's planning figure is up to 20 slot-hours per hour of run time, about 1.20 USD an hour in us-central1. Preview connectors (Stripe, PayPal, HubSpot, Klaviyo, SQL Server) stay free until GA. Google Play is 25 USD per package name a month. Loads into native tables are free. Merges into native tables bill as BigQuery jobs. Iceberg managed-table dest is Preview: S3, Azure Blob, and Cloud Storage only, append-only, no backfill. Load and merge into Lakehouse Iceberg use Iceberg rates. This is not a Streamhouse. There is no Iceberg ∪ WAL.

Skippr is an open Streamhouse you run as one Rust binary. Write skippr.yml (or the same thing in Python), discover, sync. You query live state over Iceberg ∪ WAL, and the history is ordinary Iceberg in your bucket. No slot-hour. ELT into a warehouse is one path, not the product.

FeatureSkipprBigQuery Data Transfer Service
DeploySingle Rust binary, your VPC or on-premTransfer config in a GCP project
SchemaIn-stream evolution, sibling columnsConnector mapping into BQ
StorageOpen Iceberg on Parquet you ownNative BQ, or Iceberg managed tables (Preview, append-only)
CDCLog-based; fails closed if the dest cannot honourScheduled extract. Not a log tail
Queryskipprd query on Iceberg ∪ WAL; history in any Iceberg engineBigQuery after the transfer
PricingFree self-hosted engineFree Google/object/migration connectors. Paid GA is slot-hours.

What BigQuery Data Transfer Service is good at ​

A nightly Ads or Analytics 4 pull that already lands in the same project as the warehouse. That is the product. Free orchestration on Google and object-store sources is why some teams never buy a second ELT seat. Iceberg managed-table ingest from S3 is a real Preview dest. If the job is "this report into BigQuery," that is a real product.

Where the job is wider ​

DTS is a loader. It is not a Streamhouse. There is no Iceberg ∪ WAL. There is no skipprd query on in-flight rows. Iceberg dest is append-only Preview. There is no reverse path out of BigQuery.

Slot math is the ops. A Salesforce hour is up to 20 slot-hours on Google's guide, plus merge query cost. Skippr will not match the Ads and YouTube connectors. Skippr also will not replace a DTS that already loads Google sources for free.

If the job is "BQ scheduled load we already staff," DTS is the narrower tool. If the job is live SQL on Iceberg you run, DTS was never that.

Getting started ​

Install skipprd from the install guide, or pip install skippr. The three tabs write the same skippr.yml. Then discover, sync, and query.

yaml
skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db
    data_sink: data_sinks.lake

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]
      cdc_mode: snapshot_then_cdc

data_sinks:
  lake:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    AthenaIceberg:
      warehouse: s3://my-iceberg-warehouse/
      glue_database_name: analytics
      athena_workgroup_name: primary
      athena_results_s3_bucket: my-athena-results
      region: us-east-1
bash
skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers \
  --cdc-mode snapshot_then_cdc

skipprd connect data-sink athenaiceberg \
  --pipeline orders \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --glue-database-name analytics \
  --athena-workgroup-name primary \
  --athena-results-s3-bucket my-athena-results \
  --region us-east-1

skipprd connect schema-sink athenaiceberg \
  --pipeline orders \
  --name lake_schema
python
from skippr import (
    Config,
    DataSinkAthenaIceberg,
    DataSourcePostgres,
    DataSourcePostgresSourceCdcMode,
    EnvRef,
    LocalStorage,
    Pipeline,
)

cfg = Config().workspace("analytics").storage(LocalStorage())

app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
        cdc_mode=DataSourcePostgresSourceCdcMode.SNAPSHOTTHENCDC,
    ),
)

lake = cfg.data_sink(
    "lake",
    DataSinkAthenaIceberg(
        warehouse="s3://my-iceberg-warehouse/",
        glue_database_name="analytics",
        athena_workgroup_name="primary",
        athena_results_s3_bucket="my-athena-results",
        region="us-east-1",
    ),
    schema_sink="lake_schema",
)

cfg.pipeline("orders", Pipeline(data_source=app_db, data_sink=lake))
cfg.save("skippr.yml")
bash
export APP_DB_PASSWORD='your-postgres-password'
skipprd discover --pipeline orders
skipprd sync --pipeline orders --once
skipprd query "SELECT count(*) FROM analytics.orders"

Which to choose ​

Choose BigQuery Data Transfer Service if you want scheduled loads into BigQuery, and the warehouse after the transfer is enough.

Choose Skippr if you want one binary, live SQL, and Iceberg files that stay yours. Skippr will not run a DTS transfer.

Install from Install. Connector and query reference live in Developer Docs. For optional onboarding in your VPC, Enterprise Jumpstart or Contact.