ELEMENTARY CLOUD Out-of-the-box ML-powered monitoring for freshness and volume issues on all production tables. The automated monitors feature provides broad coverage and detection of critical pipeline issues, without any configuration effort. These monitors track updates to tables, and will detect data delays, incomplete updates, and significant volume changes. Additionally, there will be no increase in compute costs as the monitors leverage only warehouse metadata (e.g. information schema, query history). Once your environment is set up, we automatically collect metadata from your warehouse, which our ML models run on. The models are operational when the initial backfill is completed, there is no “loading / training period” - Elementary will collect enough historical data after setup to train the models. Monitors are automatically created on all of your sources, and additional ones can be added, edited and removed manually. Their results are displayed in the application in the same way as package tests.

​
Benefits of automated monitors

  1. Zero configuration - Our machine learning models learn data behavior, eliminating the need for manual configuration.
  2. Out-of-the-box coverage - Rather than manually configuring a test for each model, Elementary automatically creates monitors for every source in your dbt project once you set up your environment.
  3. Metadata only, minimal cost - The monitors rely on data warehouse metadata, and don’t consume compute resources.

​
How it works?

The monitors collect metadata, and the anomaly detection model adjusts based on updates frequency, seasonality and trends. As soon as you connect Elementary Cloud Platform to your data warehouse, a backfill process will begin to collect historical metadata. Within an average of a few hours, your automated monitors will be operational. By default, Elementary collects at least 21 days of historical metadata. The automated monitors are created for all the sources in your dbt project. If you would like a different configuration of which tables to create the monitors on, or adding monitors for all models, you can reach out to us. You can fine tune the configuration and provide feedback to adjust the detection to your needs.
As views are stateless, automated volume and freshness monitors only apply on tables.

​
Automated Monitors

Automated Freshness

Monitors updates to tables and how frequently a table is updated, and fails if there is an unexpected delay.

Automated Volume

Monitors how many rows were added or removed to a table on each update, and fails if there is an unexpected drop or spike in rows.

​
Alerts on Failures

By default, automated monitors failures don’t create alerts. To activate alerts on automated monitors, navigate to Setup > Alert Rules.
  • To alert on all automated monitors failures - Change the default rule (#1) alert categories to include automated monitors.
  • To alert on specific datasets - Change / Create alert rules for these specific datasets, and include automated monitors in their alert categories.
Alert categories in alert rules