BigQuery
BigQuery integration details

BigQuery query history

The query history in BigQuery is pulled from the information schema, by accessing a view that contains jobs metadata by project.
The retention of this view is 180 days, meaning that queries older than that will not be part of the lineage.
Note that access to the relevant view requires permissions.
If these are not granted, Elementary will create partial lineage only based on accessible queries, or will return an error if there is no access at all.

BigQuery connection profile

The profiles.yml file to connect to BigQuery should contain a profile in the following format:
1
# example BigQuery profiles.yml file
2
3
## profile name, replace 'my_bigquery_profile' with a name of your choice ##
4
my_bigquery_profile:
5
outputs:
6
default:
7
type: bigquery
8
9
## Service account auth, other options require other configs ##
10
method: service-account
11
keyfile: [full path to your keyfile]
12
13
project: [project id]
14
## Dataset is used as filter by default, you can use the 'ignore schema'
15
## option to see cross dataset lineage (only in the same location)
16
dataset: [dataset name]
17
threads: 1
18
## Location is mandatory, can be one of US or EU, or a regional location
19
location: [dataset location]
20
priority: interactive
Copied!
We support the same format and connection methods as dbt. Please refer to dbt's documentation of BigQuery for further details.
Note that although in dbt the 'location' parameter is optional, in Elementary it is mandatory. Here are details about BigQuery locations.

BigQuery permissions

The queries are pulled from the information schema view of jobs by project.
This requires bigquery.jobs.listAll permission for the project, that is available by default only to the Project Owner and BigQuery Admin roles.
There are two options for creating an account with the required permissions:
    1.
    Create an account and grant it Project Owner or BigQuery Admin role.
    2.
    Recommended: create an account and grant it BigQuery Data Editor + BigQuery User + custom role that has bigquery.jobs.listAll permission.
Here is a guide for creating a service account for both options.
Last modified 7d ago