Appearance

MotherDuck ​

The MotherDuck source reads tables, or the result of one SQL query, from MotherDuck over HTTPS with your access token. It is a one-time full load: each table is copied once, and later sync runs do not load new or changed rows. Use it to move MotherDuck data into another warehouse or lake.

Before you begin ​

  • Network access. The machine running Skipprd must reach https://api.motherduck.com over HTTPS.
  • An access token. Create a MotherDuck access token for a user that can read the database. Skipprd only runs the SELECT statements you configure.

Configure ​

Store the token in an environment variable and reference it as ${MOTHERDUCK_TOKEN}. Skipprd also reads .env and .env.local next to skippr.yml.

bash
export MOTHERDUCK_TOKEN="your-access-token"
python
from skippr import Config, DataSourceMotherduck, EnvRef, Pipeline

cfg = Config.discover()
src = cfg.data_source(
    "motherduck",
    DataSourceMotherduck(
        motherduck_token=EnvRef("MOTHERDUCK_TOKEN"),
        database="analytics",
        tables=["users", "events"],
    ),
)
pipe = cfg.pipeline("motherduck", Pipeline(data_source=src))
cfg.save()
bash
skipprd connect data-source motherduck \
  --pipeline motherduck \
  --name motherduck \
  --motherduck-token '${MOTHERDUCK_TOKEN}' \
  --database analytics \
  --tables users \
  --tables events
yaml
pipelines:
  motherduck:
    data_source: data_sources.motherduck

data_sources:
  motherduck:
    Motherduck:
      motherduck_token: ${MOTHERDUCK_TOKEN}
      database: analytics
      tables: ["users", "events"]

To load the result of a query instead, set query and omit tables:

yaml
data_sources:
  motherduck:
    Motherduck:
      motherduck_token: ${MOTHERDUCK_TOKEN}
      database: analytics
      query: "SELECT country, count(*) AS users FROM users GROUP BY country"

Add a destination to the pipeline (see the connector catalog), then discover the schema and run one sync pass:

python
from skippr import Session

session = Session(pipe)
session.discover()
session.sync(once=True)
bash
skipprd discover --pipeline motherduck --log
skipprd sync --pipeline motherduck --once --log

Options ​

Set either tables or query. If you set both, Skipprd runs query and ignores tables.

KeyTypeDefaultDescription
motherduck_token${ENV} referenceRequiredMotherDuck access token. Must be an environment reference such as ${MOTHERDUCK_TOKEN}.
databasestringnoneDatabase to run queries in. Also used in destination table names.
tableslist of stringsnoneTables to read. Each entry is used as written in SELECT * FROM <entry>.
querystringnoneOne SQL SELECT to run instead of tables.
batch_size_rowsinteger10000Rows per batch handed to Skipprd.
formatstringjsonLeave unset. Rows are always sent as JSON.
batch_size_bytesintegernoneNot used by this source.
batch_size_secondsintegernoneNot used by this source.

What gets synced ​

Tables. Each entry in tables, or the single result of query. Skipprd reads the whole result of each query before writing it.

Table names. Each table lands in a destination table named motherduck_<database>_<table>: users in analytics becomes motherduck_analytics_users. Without database, the middle part is motherduck. A query lands in motherduck_<database>_query. Skipprd lowercases names and replaces characters other than letters, digits, and _ with _.

Types. Skipprd infers column types from the returned JSON values when you run skipprd discover. See Schema discovery and evolution.

Incremental behaviour. Skipprd records each table (or the query) as loaded once its rows are durably written. Later sync runs still send the query to MotherDuck, but its rows are not loaded again. To reload, run the source under a new pipeline name, which starts with fresh progress.

Deletes. Not captured.

Troubleshooting ​

SymptomCauseFix
MotherDuck API HTTP 401 or 403The token is wrong, expired, or not exported.Check that MOTHERDUCK_TOKEN is exported in the shell that runs Skipprd and that the token is still valid.
MotherDuck API HTTP 4xx naming a missing table or catalogdatabase or a table name is wrong.Fix database or the entry in tables.
MotherDuck request: ...Skipprd cannot reach the MotherDuck API.Allow outbound HTTPS to api.motherduck.com.
MotherDuck: must specify either 'tables' or 'query'Neither is set.Add tables or query.
New rows never arriveEach table loads once.Run the source under a new pipeline name to take a fresh copy.

Next steps ​