Skippr Docs

Appearance

When to Use BigQuery as a Warehouse Destination ​

May 2026

BigQuery is the right warehouse destination when the team wants Google-managed operations, dataset-level setup, and final-state CDC tables instead of a hand-built reconciliation layer.

Short Answer ​

Use BigQuery as a warehouse destination when you want a fully managed warehouse with a simple project-and-dataset boundary, job-based loading, and strong support for final-state CDC reconciliation. The connector is a good fit when creating or updating datasets is acceptable, the service account can run jobs, and the useful warehouse interface is one current row per key rather than an append-only mutation log.

A strong example is a SaaS team landing product events and billing tables into one dataset such as raw_data in US. Skippr can write through BigQuery jobs, keep current-state CDC tables reconciled with MERGE, and use _skippr_order_token columns plus tombstone protection so a replayed older change does not overwrite the current row.

Why Teams Struggle with This ​

BigQuery is not hard to start, but it does require the warehouse boundary to be explicit. The service account has to create jobs and update tables, the dataset location has to match the plan, and CDC correctness depends on MERGE semantics rather than on append-only loading.

  • The service account needs dataset creation, table write, and job creation permissions before the first useful pipeline can land cleanly.
  • Dataset location matters early because teams often notice regional constraints only after the first raw dataset is already in use.
  • BigQuery is the right fit for final-state reconciliation when you want one current row per key, not when the main goal is preserving every mutation as the primary warehouse interface.
  • If your team cannot use a JSON service account key file through GOOGLE_APPLICATION_CREDENTIALS, setup friction shows up immediately.

How Skippr Handles It ​

Skippr keeps the BigQuery setup narrow. The warehouse config only needs project, dataset, and optional location, and CDC destinations use MERGE with automatic _skippr_order_token columns plus tombstone tables so replayed older changes do not win later.

That makes the first production-style integration easier to reason about. You can explain the warehouse in terms of datasets, jobs, and final-state tables instead of inventing a separate deduplication policy in downstream SQL.

  • Simple project, dataset, and location configuration for the warehouse.
  • Authentication through a GCP service account key file referenced by GOOGLE_APPLICATION_CREDENTIALS.
  • Exactly-once final-state CDC reconciliation through BigQuery MERGE.
  • Automatic order-token columns and tombstone tables for anti-resurrection protection.

What the First Useful Version Looks Like ​

The first useful version is one dataset such as raw_data, one service account with BigQuery Data Editor and BigQuery Job User, and one source landing into raw tables that downstream models can trust as current state.

BigQuery is a bad fit when the team does not control GCP IAM, cannot create datasets or jobs, or mainly wants an event archive rather than warehouse tables that stay reconciled.