Skip to main content

Webinar: Deep dive into anomaly detection

Elementary data anomaly detection tests monitor a specific metric (like row count, null rate, average value, etc.) and compare recent values to historical values. This is done to detect significant changes and deviations, that are probably data reliability issues.

​
What happens on each test?

Upon running a test, your data is split into time buckets based on the time_bucket field and is limited by the training_period var. The test then compares a certain metric (e.g. row count) of the buckets that are within the detection period (detection_period) to the row count of all the previous time buckets within the training_period period. If there were any anomalies in the detection period, the test will fail. On each test elementary package executes the relevant monitors, and searches for anomalies by comparing to historical metrics. To learn more, refer to core concepts.

​
How the three time settings relate

time_bucket, training_period and detection_period all take a period and a count, which makes them look like alternatives. They are not. One sets how wide each data point is, and the other two set how far back you look.
  • time_bucket is how wide each data point is. One day by default.
  • training_period is how far back data is collected. 14 days by default.
  • detection_period is how much of the newest end can be flagged. 2 days by default.
Your number of data points is the training period divided by the bucket size. On the defaults that is 14 days of daily buckets, so 14 points, with the newest 2 eligible to fail and the older 12 serving only as baseline. Wider buckets give you fewer data points, without changing how far back you look:
That last row is a real risk. Widening the time bucket without also raising the training period leaves too few points to detect anything. If you move to weekly buckets, raise training_period to match.
The two periods are nested, not sequential. training_period is the whole window, and detection_period is carved out of the end of it, so training_period: 14 days with detection_period: 2 days is 14 days of data in total and not 16. Raising detection_period shrinks your baseline instead of extending the window.

​
What does it mean when a test fails?

When a test fail, it means that an anomaly was detected on this metric and dataset. To learn more, refer to anomaly detection method.

​
Core concepts

​
Anomaly

A value in the detection set that is an outlier comparing to the expected range calculated based on the training set.

​
Monitored data set

The data set we run the data monitor against, and includes the training set values and detection set values.

​
Data monitors

When we use anomaly detection tests we can monitor different metrics to detect problems - freshness, volume, nullness, uniqueness, distribution, etc. Each different metric we collect is a ‘data monitor’.

​
Training set

The set of values used as a reference point to calculate the expected range.

​
Detection set

The set of values that are compared to the expected range. If a value in the detection set is an outlier to the expected range, it will be flagged as an anomaly.

​
Expected range

Based of the values in the training test, we calculate an expected range for the monitor. Each data point in the detection period will be compared to the expected range calculated based on its training set.

​
Training period

The period of time for which the training set is collected. As data changed over time, we don’t consider the entire history of the metric, just a recent period.

​
Detection period

The values in the detection period will be compared to the expected range calculated using the training set. If a data point is outside the expected range and is part of the detection period, it is flagged as an anomaly.

​
Time bucket

To calculate how data changes over time and detect issues, we split the data into consistent time buckets. For example, if we use daily time bucket and monitor for row count anomalies, we will count new rows per day.

​
Detection algorithm

Read about it in data anomaly detection method.

​
Tests configuration and core concepts

Not all data sets are the same, so Elementary offers configuration options for the anomaly detection tests.
To detect data issues with high accuracy, it is important to leverage the configuration options. Configuration params related directly to the test’s core concepts: Data monitors Expected range Detection period and detection set Training period and training set Time buckets Monitored data set