1. Guides
  2. Add anomaly detection tests

After you install the dbt package and related configuration, you can add Elementary data tests.

Data monitors as dbt tests

Elementary is the first solution that delivers data monitoring and anomaly detection as dbt tests. Elementary dbt tests are actually data monitors that collect metrics and metadata over time. On each execution, the tests analyze the new data, compare it to historical metrics, and alert on anomalies and outliers. These tests are configured and executed like any other tests in your project.

Elementary tests can only be used with the elementary models, so you must run “dbt run —select elementary” to create the Elementary package models.

Elementary available monitors as tests

Table (model / source) tests:

Column test:

How to configure your Elementary tests?

Elementary tests are added to models / sources / columns of your dbt project just like the native dbt tests. You can configure specific monitors under the relevant keys (otherwise all relevant monitors will run).

Elementary tests have three levels of configurations:

  1. Test arguments - Test specific arguments.
  2. Table configuration - Configure the timestamp column and details of a monitored table.
  3. Global vars - Optional configuration parameters of the operation.

More on data tests configuration here.

We recommend adding a tag to the tests so you could execute these in a dedicated run using the selection parameter --select tag:elementary. If you wish to only be warned on anomalies, configure the severity of the tests to warn.

version: 2

  - name: < model name >
        timestamp_column: < timestamp column >
      - elementary.table_anomalies:
          table_anomalies: < specific monitors, all if null >
          # optional - configure different freshness column than timestamp column
          freshness_column: < freshness_column >
      - elementary.all_columns_anomalies:
          column_anomalies: < specific monitors, all if null >
      - elementary.schema_changes
      - elementary.dimension_anomalies:
          dimensions: < columns or sql expressions of columns >
          # optional - configure a where a expression to accurate the dimension monitoring
          where_expression: < sql expression >

  - name: < model name >
    ## if no timestamp is configured, elementary will monitor without time filtering
      - name: < column name >
          - elementary.column_anomalies:
              column_anomalies: < specific monitors, all if null >

What happens on each test?

On each test elementary package executes the relevant monitors, and searches for anomalies by comparing to historical metrics. At the end of the dbt test run, all results and collected metrics are merged into the elementary models.

What does it mean when a test fails?

When a test fail, it means that an anomaly was detected on this metric and dataset. To learn more, refer to anomaly detection.

Where are Elementary tests results stored?

At the end of the dbt test run, all results and collected metrics are merged into the elementary models. The relevant models are:

  • alerts_data_monitoring - All the detected anomalies are merged to this table.
  • data_monitoring_metrics - All the collected metrics are merged to this table.
  • metrics_anomaly_score - The anomaly score calculated for all the metrics from the last 7 days.

Alerts on detected anomalies (failed tests)

To get Slack notifications, refer to the Slack integration.