Supports Bigquery since version 0.3.4
After installing Elementary’s dbt package upon running
you can generate Elementary’s profile for usage with
edr by running the following command within your project:
dbt run-operation elementary.generate_elementary_cli_profile
The command will print to the terminal a partially filled template of the profile that’s needed for
edr to work.
Fill in the missing authentication fields
in the output and copy it to
profiles.yml file to connect to BigQuery should contain a profile in the following format:
## BIGQUERY ## ## By default, edr expects the profile name 'elementary'. ## ## Configure the database and schema of elementary models. ## ## Check where 'elementary_test_results' is to find it. ## elementary: outputs: default: type: bigquery ## Service account auth, other options require other configs ## method: service-account keyfile: [full path to your keyfile] project: [project id] dataset: [dataset name] # elementary dataset, usually [dataset name]_elementary threads: 4 location: [dataset location] priority: interactive
We support the same format and connection methods as dbt. Please refer to dbt’s documentation of BigQuery for further details.
For Elementary to be able to access the jobs by project in the information schema, there are two options:
In the Cloud Console, go to: IAM & Admin > Roles
Click on ‘CREATE ROLE’
Give the role a title, description, etc.
Click on ’+ ADD PERMISSIONS’
Using the filter, find and add the permissions bigquery.jobs.listall and bigquery.jobs.create, then click ‘ADD’:
In the Cloud Console, go to: IAM & Admin > Service Accounts
Click on ‘CREATE SERVICE ACCOUNT’
The last step is optional so skip it if you don’t need to manage this service account with another service account, and press done.
Press on the dots icon to the right of your screen for your new service account and select ‘Manage keys’:
This will automatically generate and download a JSON file with your private key information for this service account. This JSON file provides the credentials to programmatically connect and work with your BigQuery environment.
Add the full path of this JSON file to your connection profile under ‘keyfile’.
We are available on Slack, reach out for any kind of help!