Appearance

SkipprLake schema ​

Pair this with the SkipprLake destination. Use the same warehouse, catalog_table, and table_namespace on both.

Before you begin ​

  • The SkipprLake destination already in skippr.yml.
  • Write access to catalog_table.

Configure ​

python
from skippr import Config, DataSinkSkipprLake, Pipeline

cfg = Config.discover()
lake = cfg.data_sink(
    "lake",
    DataSinkSkipprLake(
        warehouse="s3://my-iceberg-warehouse/",
        catalog_table="my-iceberg-catalog",
        region="us-east-1",
        table_namespace="bronze",
    ),
    schema_sink="lake_schema",
)
cfg.pipeline("reports", Pipeline(data_source=cfg.get_data_source("sample"), data_sink=lake))
cfg.save()
bash
skipprd connect data-sink skipprlake \
  --pipeline reports \
  --name lake \
  --warehouse s3://my-iceberg-warehouse/ \
  --catalog-table my-iceberg-catalog \
  --region us-east-1 \
  --table-namespace bronze

skipprd connect schema-sink skipprlake \
  --pipeline reports \
  --name lake_schema
yaml
schema_sinks:
  lake_schema:
    SkipprLake:
      warehouse: s3://my-iceberg-warehouse/
      catalog_table: my-iceberg-catalog
      region: us-east-1
      table_namespace: bronze

data_sinks:
  lake:
    SkipprLake:
      warehouse: s3://my-iceberg-warehouse/
      catalog_table: my-iceberg-catalog
      region: us-east-1
      table_namespace: bronze
    schema_sink: schema_sinks.lake_schema

Options ​

KeyTypeRequired/DefaultDescription
warehouseURIRequiredSame Iceberg root as the destination
catalog_tablestringRequiredSame catalog table
regionstringNot setCatalog region
object_storeobjectNot setSame file-store override as the destination
table_namespacestringNot setSame namespace

Troubleshooting ​

SymptomFix
Tables missing from skipprd queryConfirm this sink is referenced and table_namespace matches
Catalog permission errorsGrant read/write on catalog_table

Next steps ​