Automated Freshness & Volume Monitors
ELEMENTARY CLOUD
Out-of-the-box ML-powered monitoring for freshness and volume issues on all production tables. The automated monitors feature provides broad coverage and detection of critical pipeline issues, without any configuration effort.
These monitors track updates to tables, and will detect data delays, incomplete updates, and significant volume changes. Additionally, there will be no increase in compute costs as the monitors leverage only warehouse metadata (e.g. information schema, query history).
Once your environment is set up, we automatically collect metadata from your warehouse, which our ML models run on. The models are operational when the initial backfill is completed, there is no “loading / training period” - Elementary will collect enough historical data after setup to train the models.
Monitors are created for all models and sources in your dbt project, and their results are displayed in the application in the same way as package tests.
Benefits of automated monitors
- Zero configuration - Our machine learning models learn data behavior, eliminating the need for manual configuration.
- Out-of-the-box coverage - Rather than manually configuring a test for each model, Elementary automatically creates monitors for every model and source in your dbt project once you set up your environment.
- Metadata only, minimal cost - The monitors rely on data warehouse metadata, and don’t consume compute resources.
How it works?
The monitors collect metadata, and the anomaly detection model adjusts based on updates frequency, seasonality and trends.
As soon as you connect Elementary Cloud Platform to your data warehouse, a backfill process will begin to collect historical metadata. Within an average of a few hours, your automated monitors will be operational. By default, Elementary collects at least 21 days of historical metadata.
You can fine tune the configuration and provide feedback to adjust the detection to your needs.
Automated Monitors
Automated Freshness
Monitors updates to tables and how frequently a table is updated, and fails if there is an unexpected delay.
Automated Volume
Monitors how many rows were added or removed to a table on each update, and fails if there is an unexpected drop or spike in rows.
Alerts on Failures
By default, automated monitors failures don’t create alerts.
To activate alerts on automated monitors, navigate to Setup > Alert Rules.
- To alert on all automated monitors failures - Change the default rule (#1) alert categories to include automated monitors.
- To alert on specific datasets - Change / Create alert rules for these specific datasets, and include automated monitors in their alert categories.