BigQuery
BigQuery integration details

Data anomalies monitoring as dbt tests

Supports Bigquery since version 0.3.4

Lineage

BigQuery query source

The query history in BigQuery is pulled from the information schema, by accessing a view that contains jobs metadata by project.
The retention of this view is 180 days, meaning that queries older than that will not be part of the lineage.
Note that access to the relevant view requires permissions.
If these are not granted, Elementary will create partial lineage only based on accessible queries, or will return an error if there is no access at all.

BigQuery connection profile

The profiles.yml file to connect to BigQuery should contain a profile in the following format:
1
###########################
2
## Template profiles.yml ##
3
###########################
4
5
## BIGQUERY ##
6
## By default, the tool expects the profile name 'elementary' ##
7
## This profile will be used for elementary's outputs. We ##
8
## reccomend creating a dedicated database, or at least schema ##
9
10
elementary:
11
outputs:
12
default:
13
type: bigquery
14
15
## Service account auth, other options require other configs ##
16
method: service-account
17
keyfile: [full path to your keyfile]
18
19
project: [project id]
20
dataset: [dataset name]
21
threads: 1
22
## Location is mandatory, can be one of US or EU, or a regional location
23
location: [dataset location]
24
priority: interactive
Copied!
We support the same format and connection methods as dbt. Please refer to dbt's documentation of BigQuery for further details.
Note that although in dbt the 'location' parameter is optional, in Elementary it is mandatory. Here are details about BigQuery locations.

BigQuery permissions

The queries are pulled from the information schema view of jobs by project.
This requires bigquery.jobs.listAll permission for the project, that is available by default only to the Project Owner and BigQuery Admin roles.
There are two options for creating an account with the required permissions:
  1. 1.
    Create an account and grant it Project Owner or BigQuery Admin role.
  2. 2.
    Recommended: create an account and grant it BigQuery Data Editor + BigQuery User + custom role that has bigquery.jobs.listAll permission.
Here is a guide for creating a service account for both options.
Have a question