SkipprLake

SkipprLake is Skipprd's Iceberg destination. Files go to a warehouse URI you name. Catalog pointers live in a DynamoDB table you name (catalog_table). That table may be the same one you use as the state store.

Pair it with the SkipprLake schema sink.

Skippr, your data department

Speak to an expert. Upfront insights in days, not months.

Book an intro call →

20 minutes · No sales pitch

Before you begin

• A warehouse URI: s3://bucket/path/ or file:///absolute/path/ for local files.
• A catalog table name, and credentials for that backend.
• Unique (catalog_table, table_namespace) per SkipprLake destination. Two pipelines must not share that pair.

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-east-1"

Configure

Python

from skippr import Config, DataSinkSkipprLake, Pipeline

cfg = Config.discover()
lake = cfg.data_sink(
    "lake",
    DataSinkSkipprLake(
        warehouse="s3://my-iceberg-warehouse/",
        catalog_table="my-iceberg-catalog",
        region="us-east-1",
        table_namespace="bronze",
    ),
    schema_sink="lake_schema",
)
cfg.pipeline("reports", Pipeline(data_source=cfg.get_data_source("sample"), data_sink=lake))
cfg.save()

CLI

skipprd connect data-sink skipprlake \
  --pipeline reports \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --catalog-table my-iceberg-catalog \
  --region us-east-1 \
  --table-namespace bronze

skipprd connect schema-sink skipprlake \
  --pipeline reports \
  --name lake_schema

YAML

data_sinks:
  lake:
    SkipprLake:
      warehouse: s3://my-iceberg-warehouse/
      catalog_table: my-iceberg-catalog
      region: us-east-1
      table_namespace: bronze
    schema_sink: schema_sinks.lake_schema

schema_sinks:
  lake_schema:
    SkipprLake:
      warehouse: s3://my-iceberg-warehouse/
      catalog_table: my-iceberg-catalog
      region: us-east-1
      table_namespace: bronze

Local files:

object_store:
  type: file
# warehouse must be file:///...

Cloudflare R2 (or another path-style S3 store):

object_store:
  type: r2
  endpoint: ${OBJECTS_S3_ENDPOINT}
  region: auto
  access_key_id: ${OBJECTS_ACCESS_KEY_ID}
  secret_access_key: ${OBJECTS_SECRET_ACCESS_KEY}
  path_style: true

Options

Key	Type	Required/Default	Description
warehouse	URI	Required	Iceberg root (s3://… or file:///…)
catalog_table	string	Required	Catalog pointer table. May be the SkipprStore table.
region	string	Not set	Region for the catalog backend
object_store	object	s3	File store: file, s3, or r2
table_namespace	string	default	Iceberg namespace

Table files live at {warehouse}/{table_namespace}/{table_name}.

How data lands

Supports append, merge_by_key, replace_partition, and replace_table. Skipprd expires old snapshots (recent commits are kept), rewrites small files, and folds delete files. Query results stay the same; do not use old snapshots as time travel.

skipprd query unions the Iceberg table with in-flight WAL rows.

Troubleshooting

Symptom	Fix
Catalog permission errors	Grant read/write on catalog_table
Two pipelines fight over tables	Give each destination its own table_namespace
file:// warehouse fails	Set object_store.type: file and use an absolute file:/// URI

Next steps

• ALTER operations — rename, merge, drop, and widen columns after the table exists
• SkipprLake schema sink
• Datalake
• How sources land