Skippr Docs

Appearance

How to Manage BigQuery Destination Credentials in Skippr ​

July 2026

Keep BigQuery destination credentials safe by storing only the path to a service account key file in GOOGLE_APPLICATION_CREDENTIALS, leaving project and dataset visible, and granting the service account just the BigQuery roles the docs require.

Short Answer ​

Manage BigQuery destination credentials in Skippr by keeping project, dataset, and location in config and authenticating through GOOGLE_APPLICATION_CREDENTIALS, which points to a JSON service account key file. The safe part is not hiding the dataset name. The safe part is keeping the key file itself out of the repo and out of skippr.yaml.

That matches the documented connector model exactly. BigQuery auth is file based, and the service account still needs the BigQuery Data Editor and BigQuery Job User roles so Skippr can create datasets, write tables, and run jobs. When CDC is enabled, those same warehouse credentials support BigQuery MERGE statements plus the automatic _skippr_order_token and tombstone tables described in the CDC destination docs.

Why Teams Struggle with This ​

BigQuery setups usually break because teams protect the wrong thing. They hide the project and dataset while leaving the service account JSON in a shared folder or inside the repo. The JSON key is the secret. The dataset name is operational context.

  • Checking the service account JSON into source control is a direct credential leak.
  • Pointing GOOGLE_APPLICATION_CREDENTIALS at the wrong file produces an auth failure that looks mysterious until you verify the path.
  • If the service account lacks bigquery.datasets.create, bigquery.tables.updateData, or bigquery.jobs.create, the destination can authenticate and still fail the first useful load.
  • Choosing the wrong dataset location can look like a connector problem even though the credential is valid.

How Skippr Handles It ​

Skippr keeps BigQuery credential handling narrow. The warehouse config names the target project and dataset, while the runtime reads one service account key file from the standard Google environment variable. That is simple to explain and simple to rotate.

It also lines up with BigQuery CDC behavior. The same service account that can land raw tables and run jobs can also execute the documented MERGE-based final-state reconciliation, including order-token guards and tombstone protection.

  • Uses GOOGLE_APPLICATION_CREDENTIALS for the service account key-file path.
  • Keeps project, dataset, and location visible and reviewable in config.
  • Requires concrete BigQuery roles instead of a vague platform-admin credential.
  • Supports CDC final-state reconciliation through BigQuery MERGE, order tokens, and tombstone tables.

What the First Useful Version Looks Like ​

The first useful version is one GCP project, one dataset such as raw_data, one location, and one service account with the documented BigQuery roles. That proves both auth and warehouse behavior before you add more sources.

If the team later rotates the key, only the file behind GOOGLE_APPLICATION_CREDENTIALS should change. The visible warehouse config can stay stable and understandable.