Data cleansing
Wrong types, new fields, and structural changes do not stop ingestion. Skippr evolves the schema. Records are dead-lettered only when they are genuinely unparseable.
What happens on drift
- A field changes type: Skippr adds a new column. Existing data stays untouched.
- New fields appear: they are added automatically.
- Nested structures reorganize: maps can become records, arrays can gain element types.
Only truncated JSON, binary garbage, or irrecoverable encoding is quarantined. Dead letters are stored as queryable Parquet with the original record, error, run ID, and timestamp. Nothing is silently dropped.
Pipeline
- Parse — compare each record to the current schema.
- Evolve — add columns and fields so data keeps flowing.
- Dead-letter — quarantine unparseable records.
- Report — you can see what evolved and what was quarantined.