Datalake
A datalake stores tables as files in object storage so many tools can read them. You do not copy the data into one vendor's warehouse first.
Skipprd is not a warehouse. It lands pipelines as Apache Iceberg tables on Parquet. Athena, Snowflake, DuckDB, and skipprd query can all read those tables.
Use a lake destination when you want one copy of the data and several query engines. Use a warehouse destination (Snowflake, BigQuery, PostgreSQL) when that warehouse is the system of record.
How a pipeline lands in the lake
- Skipprd reads a batch from the source and commits it to the write-ahead log.
- It writes the batch into Iceberg files in your bucket or warehouse path.
- The catalog (Glue, SkipprLake, or a local DuckDB catalog) points at the new snapshot.
You query the Iceberg name (namespace.table). You do not open Parquet files by hand.
Destinations that write Iceberg
| Destination | Catalog | Typical use |
|---|---|---|
| SkipprLake | A table you name (catalog_table) | Skippr-managed lake, query with skipprd query |
| Athena Iceberg | AWS Glue | SQL in Athena on Iceberg tables |
| DuckDB | Files on disk (file://) | Local analytics and CI |
Hive Athena (the Athena destination) writes Hive tables on S3, not Iceberg. skipprd query can still read in-flight WAL rows for that pipeline, but the durable table is the Glue/Hive one.
What Skipprd maintains
Iceberg destinations keep their own tables. They expire old snapshots (they keep recent commits, not unbounded history), compact small files, and drop delete files so metadata stays small. Older snapshots are not a time-travel product promise — query the current table.
Query
skipprd query --pipeline bikehire "SELECT * FROM rides LIMIT 20"For Iceberg destinations this is the lake table plus any rows still in the WAL. For the query command itself, see skipprd query.