After you install the dbt package and related configuration, you can add Elementary data tests.
Elementary is the first solution that delivers data monitoring and anomaly detection as dbt tests. Elementary dbt tests are actually data monitors that collect metrics and metadata over time. On each execution, the tests analyze the new data, compare it to historical metrics, and alert on anomalies and outliers. These tests are configured and executed like any other tests in your project.
Elementary tests can only be used with the elementary models, so you must run “dbt run —select elementary” to create the Elementary package models.
Elementary tests are added to models / sources / columns of your dbt project just like the native dbt tests. You can configure specific monitors under the relevant keys (otherwise all relevant monitors will run).
Elementary tests have three levels of configurations:
More on data tests configuration here.
We recommend adding a tag to the tests so you could execute these in a dedicated run using the selection parameter
If you wish to only be warned on anomalies, configure the severity of the tests to warn.
version: 2 models: - name: < model name > config: elementary: timestamp_column: < timestamp column > tests: - elementary.table_anomalies: table_anomalies: < specific monitors, all if null > # optional - configure different freshness column than timestamp column freshness_column: < freshness_column > - elementary.all_columns_anomalies: column_anomalies: < specific monitors, all if null > - elementary.schema_changes - elementary.dimension_anomalies: dimensions: < columns or sql expressions of columns > # optional - configure a where a expression to accurate the dimension monitoring where_expression: < sql expression > - name: < model name > ## if no timestamp is configured, elementary will monitor without time filtering columns: - name: < column name > tests: - elementary.column_anomalies: column_anomalies: < specific monitors, all if null >
On each test elementary package executes the relevant monitors, and searches for anomalies by comparing to historical metrics.
At the end of the
dbt test run, all results and collected metrics are merged into the elementary models.
When a test fail, it means that an anomaly was detected on this metric and dataset. To learn more, refer to anomaly detection.
At the end of the
dbt test run, all results and collected metrics are merged into the elementary models. The relevant models are:
alerts_data_monitoring- All the detected anomalies are merged to this table.
data_monitoring_metrics- All the collected metrics are merged to this table.
metrics_anomaly_score- The anomaly score calculated for all the metrics from the last 7 days.
To get Slack notifications, refer to the Slack integration.