Elementary anomaly detection tests
Elementary dbt package includes anomaly detection tests, implemented as dbt tests. The tests are configured and executed like any other tests in your project.
Core concepts
In data anomaly detection tests Elementary monitors a specific metric (like row count, null rate, average value, etc.) and compare recent values to historical values. This is done to detect significant changes and deviations, that are probably data reliability issues.
Anomaly detection tests core concepts
Anomaly
A value in the detection set that is an outlier comparing to the expected range calculated based on the training set.
Monitored data set
The data set we run the data monitor against, and includes the training set values and detection set values.
Data monitors
When we use anomaly detection tests we can monitor different metrics to detect problems - freshness, volume, nullness, uniqueness, distribution, etc. Each different metric we collect is a ‘data monitor’.
Training set
The set of values used as a reference point to calculate the expected range.
Detection set
The set of values that are compared to the expected range. If a value in the detection set is an outlier to the expected range, it will be flagged as an anomaly.
Expected range
Based of the values in the training test, we calculate an expected range for the monitor. Each data point in the detection period will be compared to the expected range calculated based on it’s training set.
Training period
The period of time for which the training set is collected. As data changed over time, we don’t consider the entire history of the metric, just a recent period.
Detection period
The values in the detection period will be compared to the expected range calculated using the training set. If a data point is outside the expected range and is part of the detection period, it is flagged as an anomaly.
Time bucket
To calculate how data changes over time and detect issues, we split the data into consistent time buckets. For example, if we use daily time bucket and monitor for row count anomalies, we will count new rows per day.
Detection algorithm
Read about it in data anomaly detection.
Tests configuration and core concepts
Not all data sets are the same, so Elementary offers configuration options for the anomaly detection tests.
To detect data issues with high accuracy, it is important to leverage the configuration options.
Configuration params related directly to the test’s core concepts:
Data monitors
Expected range
Detection period and detection set
Training period and training set
Time buckets
Monitored data set