Before you can start using the alerts, make sure to install the dbt package, configure a profile and install the CLI.
This is required for the alerts to work.
In order to connect, Elementary needs a connection profile in a file named profiles.yml.
This profile will be used by the CLI, to connect to the DWH and find the dbt package tables.
The easiest way to generate the profile is to run the following command within the dbt project where you deployed the elementary dbt package (works in dbt cloud as well):
Ensure that you fill in your password and any other missing fields after you
paste the profile in your localprofiles.yml file.
Copy the output, fill in the missing fields and add the profile to your profiles.yml.
Here is a demonstration:
Elementary profile details and format
Path: HOME_DIR/.dbt/profiles.yml
Profile name: elementary
Schema name: The schema of elementary models, default is <your_dbt_project_schema>_elementary
## SNOWFLAKE #### By default, edr expects the profile name 'elementary'. #### Configure the database and schema of elementary models. #### Check where 'elementary_test_results' is to find it. ##elementary:outputs:default:type: snowflake
account:[account id]## User/password auth ##user:[username]password:[password]role:[user role]database:[database name]warehouse:[warehouse name]schema:[schema name]_elementary
threads:4
Install Elementary CLI
To install the monitor module run:
pip install elementary-data
Run one of the following commands based on your platform (no need to run all):
Go to the “OAauth & Permissions” page for your newly-created app, and add the following scopes under “Bot Token Scopes”:
channels:join - Join public channels in a workspace
channels:read - View basic information about public channels in a workspace
chat:write - Send messages as <app>
files:write - Upload, edit, and delete files as <app>
users:read - View people in a workspace
users:read.email - View email addresses of people in a workspace
groups:read - View basic information about private channels that your slack app has been added to
2. Install app at your Workspace
At the “OAuth & Permissions” page, press on “Install to Workspace” in order to generate Slack token:
Create a webhook
Go to the ‘Incoming Webhooks’ page for your newly-created app and toggle ‘Activate Incoming Webhooks’ to turn it on. Then click on ‘Add New Webhook to Workspace’:
Select the channel that the notifications will be posted to:
Lastly, pass the token / webhook to the CLI as a param or in the config.yml file:
Token:
Use the Slack token when you execute edr monitor using the option -st, --slack-token and -ch, --slack-channel-name:
By default, alerts are sent on failed tests, errored models and errored snapshots.
You can choose to enable / disable alert types by adding a var to your dbt_project.yml.
Below are the available vars and their default config:
dbt_project.yml
vars:# Alerts configuration vars ## All set to false by default #disable_model_alerts:falsedisable_test_alerts:falsedisable_warn_alerts:falsedisable_skipped_model_alerts:truedisable_skipped_test_alerts:true
If you want to tag a model owner in a slack alert:
Use ’@’ and the email prefix of the slack user.
For example, if we want to tag a user named Jessica with an email jessica.jones@marvel.com in our Slack workspace, simply add the email prefix (with lower case) jessica.jones as follows to your model schema.yml / properties.yml:
Elementary allows you to customize alerts to distribute the right information to the right people. This way you can ensure your alerts are valuable and to avoid alert fatigue.
By default Elementary uses the Slack channel that was configured in the Slack integration.
Elementary supports configuring custom slack channels that are configured on your models / sources / tests and snapshots.
If you configure a custom slack channel for a model, all the test alerts that belong to this model will be sent to this custom slack channel.
If you configure a custom slack channel for both a model and a test, the test channel will override the model channel.
If you configure a custom slack channel and you decide to group your alerts by table into a single message, it will be sent to the model channel (even if a differnt channel was configured on the test level)
To set it up, simply add the relevant channel to your models in the properties.yml:
Currently this feature is supported only by test alerts!
Elementary supports the following alert fields:
table: Displays the table name of the test
column: Displays the column name of the test
description: Displays the description of the test
owners: Displays the owners of the model on which the test is running
tags: Displays the dbt tags of the test/model
subscribers: Displays the subscribers of the test/model
result_message: Displays the returned message from the test result
test_parameters: Displays the parameters that were provided to the test
test_query: Displays the query of the test
test_results_sample: Displays a sample of the test results
By default, all of the fields are shown in the alerts.
Elementary supports configuring alert fields on your dbt project / models and tests.
If you configure alert fields on your dbt project, all the test alerts of all of your tests will display only the configured alert fields.
If you configure alert fields for a model, all the test alerts that belong to this model will display only the configured alert fields.
If you configure alert fields for both a model and a test, the test configured alert fields will override the model configured alert fields (same as for the dbt project configured alert fields).
To set it up globaly for your project, add the desired alert fields to your models and tests in the dbt_project.yml file:
Elementary support configuring suppression interval for alerts.
By default, the suppression interval for all of the alerts is set to 0.
Elementary won’t send any alert that is generated within suppression interval.
alert_suppression_interval can accept values of 0-24 (including unrounded numbers) - this number represents the number of hours for which alerts will be skipped.
To set it up globaly for your project, add the alert suppression interval to your models and tests in the dbt_project.yml file:
By default, Elementary sends a single alert to notify on each failure. When using single alerts, the alert will include extensive information for fast triage.
Elementary also supports grouping alerts by table. In this case, a single Slack notification will be generated containing all test warnings/failures/errors as well as the errors associated with the model. The created notification will contain a union of the relevant owners, tags and subscribers. Due to their nature, grouped alerts will contain less information on each issue. As always, you can use our (detailed report) for easy triage.
To group alerts by table:
edr monitor --group-by table
Grouping can also be configured through the yml files. To set it up globaly for your project, add the configuration to your models in the dbt_project.yml file:
dbt_project.yml
models:marketing_bi:+meta:alerts_config:# alerts on models in marekting_bi should be grouped by table:slack_group_alerts_by: table
To set it up for a model, add the configuration to your model in the properties.yml:
properties.yml
models:-name: marketing_leads
meta:alerts_config:# all alerts on marketing_leads should group together to one slack message:slack_group_alerts_by: table
Grouping by table can be configured globally (in the dbt_project.yml) but if you wish to override it for a specific model where you want a single alert for each failure, you can add the configuration to your model in the properties.yml:
properties.yml
models:-name: marketing_leads
meta:alerts_config:# alerts on marketing_leads will not be grouped:slack_group_alerts_by: alert
To alert on source freshness, you will need to run edr run-operation upload-source-freshness right after each execution of dbt source freshness.
This operation will upload the results to a table, and the execution of edr monitor will send the actual alert.
Note that it needs to be from the same machine.
Note that this requires passing --project-dir argument.
In order to monitor continuously, use your orchestrator to execute it regularly (we recommend running it right after
your dbt job ends to monitor the latest data updates).
Read more about how to deploy Elementary in production.
If you need help or wish to consult on this, reach out to us
on Slack.