The purpose of the Freshness monitor is to alert when a data asset hasn’t been updated in a period of time that exceeds the update SLA of that table. Freshness monitors are by default created for all sources in your dbt project. They can be created for additional tables upon request. The freshness monitoring has 2 operation modes, it’s possible to choose the desired one from the Test Overview side panel:
Automated freshness result

​
Anomaly detection based

The default operation mode of the freshness monitors. It learns the update frequency of your tables and consistently checks if the table is currently fresh based on our model’s forecast. By default we use 21 days of training data to understand the intervals between table updates. The only condition that determines the status of the monitor is the time that has passed since the last update, which is compared to the model’s prediction. The model takes into account seasonality, and supports cases such as tables that update on weekdays and not weekends.

​
SLA based

Sometimes you might want to monitor a table based on a fixed SLA, in order to have full control over when the monitor will alert. This mode is based on a fixed SLA that you define for each table. The monitor will alert if the table hasn’t been updated in the defined SLA period.

​
Understand the monitor result

Automated freshness result
The test result is a timeline of updates. The right end of the timeline, marked with a black triangle ▽, is the timestamp of the test result, which is near real time (can be considered as “now”). Each update to the table is presented as a line in the timeline. Hovering on the gaps between updates will show the updates time and time gap. To understand the test result, focus on the gap between the last update and now (▽):
  • Green - The gap between latest update and now is still within the expected range.
  • Yellow / Red - The gap between latest update and now is above the expected range, a dotted line will show what was the expected gap limit. The color represents if this is a warning or failure.
Use the Anomaly settings and result feedback buttons to impact the monitor.

​
Anomaly settings

You can choose between 2 detection methods for the Freshness monitor- Automatic and Manual.
  • Automatic - Elementary uses machine learning models to detect anomalies in the data freshness. This is the default setting. You can change the sensitivity level to Low, Medium, or High. For each level, you will see a simulation of the change impact on the latest result, and you can use theSimulate Configuration button to examine the change impact.
  • Manual - You can set the SLA breach threshold for the freshness monitor manually. This is useful for assets that are updated regularly at the same time every day, hour or week.
Freshness monitor configuration
  • Severity - Should a failure be considered a warning or a failure. Default is warning.
  • Test metadata - Add metadata such as tags and owner to the test.