You will connect Elementary Cloud to Bigquery for syncing the Elementary schema (created by the Elementary dbt package).
Create service account:
In the Cloud Console, go to: IAM & Admin > Service Accounts
Click on ‘CREATE SERVICE ACCOUNT’
- Fill in the service account name (‘elementary’) and account description (‘Elementary Data’) and click ‘CREATE AND CONTINUE’:
Now we need to configure the relevant permissions for this new service account. Select the following role: BigQuery Job User (you will need to grant read access to specific datasets later).
The last step is optional, skip it and press done.
- Press on the dots icon to the right of your screen for your new service account and select ‘Manage keys’:
- Press on ‘ADD KEY’ and select ‘Create new key’:
- Use the ‘JSON’ option radio button and press ‘CREATE’:
- This will automatically generate and download a JSON file with your private key information for this service account. This JSON file provides the credentials to programmatically connect and work with your BigQuery environment.
Grant service user access to specific datasets
In order for a service user to work with Elementary cloud, it requires the following permissions:
- Role “BigQuery Data Viewer” on your Elementary dataset.
- Role “BigQuery Metadata Viewer” on your dbt datasets.
To grant a role on a specific dataset, follow these steps:
Go to your project in BigQuery console
In the “Explorer” tab, find your desired dataset.
Click on the three dots icon next to the dataset name, then Share.
- Click the “ADD PRINCIPAL” button on the top right corner.
- Fill out the form:
- In the “New principals” textbox, write the email address of your user.
- In the “Select a role” dropdown menu, choose the desired role (BigQuery Data Viewer for your Elementary dataset, BigQuery Metadata Viewer for your dbt dataset).
- Click “Save”.
Make sure to grant the correct access to your Elementary dataset and your dbt dataset.
Permissions and security
Elementary cloud doesn’t need permissions to your tables and schemas. It is recommended to create a read only user for the elementary schema only. For more details, refer to security and privacy.
Fill the connection form
Provide the following fields:
- Service account file: The service account file you generated for Elementary. For more information, refer to Create BigQuery service account.
- Project: The name of your BigQuery project.
- Elementary dataset: The name of your Elementary dataset. Usually
- Location: Use this field to configure the location of BigQuery datasets as per the BigQuery documentation.
Uploading columns data
In order to let Elementary Cloud access metadata about the columns in your data warehouse, you need to run the following command:
dbt run-operation elementary.upload_dbt_columns
You may choose to run this command in one of the following ways:
- Manually whenever there is a schema change in your dataset - Not recommended.
- As part of your
dbtpipeline - Recommended if you have one main pipeline.
- As an independent pipeline - Recommended if you have a lot of small pipelines running in different times.
Add the Elementary IP to allowlist
Elementary IP for allowlist: