Appearance

Sources ​

A source is where a pipeline reads data: a database, files in object storage, a stream, or a SaaS API. You declare sources under data_sources: in skippr.yml, give each one a name, and point a pipeline at it.

Choose a source ​

KindConnectors
DatabasesClickHouse, Delta Lake, DynamoDB, MongoDB, MotherDuck, MySQL, PostgreSQL, Redshift, SQL Server
Files and object storesLocal file, S3, SFTP
StreamingAMQP, EventBridge, Kafka, Kinesis, MQTT, SNS, SQS, WebSocket
HTTP and networkHTTP client, HTTP server, OTLP, PCAP, Socket, StatsD, Stdin
Marketing and adsAdRoll Ads, Apple Search Ads, Bing Webmaster Tools, Google Ads, Google Analytics (GA4), Google Search Console, LinkedIn Ads, Meta Ads, Meta Instagram Ads, X Ads
Commerce, finance, and CRMHubSpot CRM, Revolut Business, Shopify Admin, Stripe, SumUp, Xero Accounting

Each connector page lists the fields you can set, what Skipprd reads, and how to troubleshoot it.

Add a source to a pipeline ​

This example reads two Postgres tables into a pipeline called orders.

  1. Write the source entry and the pipeline that uses it:

    python
    from skippr import Config, DataSourcePostgres, EnvRef, LocalStorage, Pipeline

cfg = Config().workspace("analytics").storage(LocalStorage())
app_db = cfg.data_source(
    "app_db",
    DataSourcePostgres(
        host="db.internal",
        user="skippr_reader",
        password=EnvRef("APP_DB_PASSWORD"),
        database="app",
        tables=["orders", "customers"],
    ),
)
cfg.pipeline("orders", Pipeline(data_source=app_db))
cfg.save("skippr.yml")
    bash
    skipprd --workspace analytics --storage-mode local connect data-source postgres \
  --pipeline orders \
  --name app_db \
  --host db.internal \
  --user skippr_reader \
  --password '${APP_DB_PASSWORD}' \
  --database app \
  --tables orders \
  --tables customers
    yaml
    skippr:
  workspace: analytics
  skipprd_el_storage_mode: local

pipelines:
  orders:
    data_source: data_sources.app_db

data_sources:
  app_db:
    Postgres:
      host: db.internal
      user: skippr_reader
      password: ${APP_DB_PASSWORD}
      database: app
      tables: [orders, customers]

    app_db is a logical name. Postgres is the connector type; write it with the casing shown on the connector page. skipprd connect creates the orders pipeline if it does not exist and sets its data_source.

  2. Provide the secret:

    bash
    export APP_DB_PASSWORD='your-postgres-password'

  3. Discover the source's schema:

    python
    import skippr

session = skippr.Session(skippr.Config.discover().get_pipeline("orders"))
session.discover()
    bash
    skipprd discover --pipeline orders

Check it worked ​

Show the schema discover saved for the pipeline:

bash
skipprd metadata show --pipeline orders

You should see the orders and customers tables with their columns. If they are missing or the command fails, see the Troubleshooting section of the connector page and Troubleshooting.

The pipeline has no destination yet, so a sync keeps the data in the write-ahead log, where skipprd query can read it. Add a destination next.

Credentials ​

  • Put every secret in skippr.yml as a ${NAME} reference. Secret fields reject plaintext. The skippr.yml reference covers the rules and .env files.
  • AWS sources (S3, DynamoDB, Kinesis, SQS, SNS, EventBridge) use the standard AWS credential chain: environment variables, a shared profile, or the instance or task role.
  • Give Skipprd a read-only account where the source supports one.

One source, several pipelines ​

Several pipelines can reference the same data_sources entry. Each pipeline keeps its own progress, so each reads the source independently.

Next steps ​