Anomaly detection tests configuration

timestamp_column: [column_name]

It is recommended to configure a timestamp column (if there is one). The best column for this will have an ‘updated at’ timestamp for each row (date type also works). Elementary will use this column to filter the table when running monitors. ( timestamp_column can be defined for the whole table as well).

If undefined, default is null (no time buckets).

time_bucket:
  period: < time period > # supported periods: day, hour, week
  count: < number of periods >

This field controls the cadence in which we monitor metrics during the training period (the days_back parameter). By default, the time bucket is daily (period=day, count=1) - meaning we calculate a metric value per day and use the resulting values for anomaly calculation (for example, daily volume / row count).

Depending on the nature of your data, it may make sense to modify this parameter. For example, if you wish to have hourly cadence for a volume anomalies test, it may make sense to set the time bucket to period=hour and count=1.

Note - you should only decrease the size of the time buckets if this value is expected to be stable during the period of choosing (for example - if row count tends to be stable on a daily level, but changes dramatically between different hours, avoid setting an hourly time bucket)

sensitivity: [int]

Threshold for the anomaly sensitivity, the global default is 3. If defined for a test, the test param is used instead of the global one. A higher number means less sensitivity (less anomalies), and a lower number means more sensitivity (more anomalies).

backfill_days: [int]

The minimal amount of buckets for running time buckets monitors. If these buckets were already calculated, Elementary will overwrite them. The reason behind it is to monitor recent backfills of data, if there were any. This configuration should be changed according to your data delays.

If undefined, default is 2 buckets.

where_expression: [sql expression]

Filter the tested data using a valid sql expression. Example usage: exlude_regexp: "user_name = 'test'"

column_anomalies: [list]

Select which monitors to activate as part of the test. Example usage: column_anomalies: [null_count, missing_count, average]

Available monitors:

event_timestamp_column: [column name] update_timestamp_column: [column name]

This test compliments the freshness_anomalies test and is primarily intended for data that is updated in a continuous / streaming fashion.

The test can work in a couple of modes:

• If only an event_timestamp_column is supplied, the test measures over time the difference between the current timestamp (“now”) and the most recent event timestamp.
• If both an event_timestamp_column and an update_timestamp_column are provided, the test will measure over time the difference between these two columns.

exclude_regexp: [regex]

Param for the all_columns_anomalies test only, which enables to exclude a column from the tests based on regular expression match. Example usage: exlude_regexp: '.*SDC$'

exclude_prefix: [string]

Param for the all_columns_anomalies test only, which enables to exclude a column from the tests based on prefix match. Example usage: exlude_prefix: 'dbt_'

days_back: [int]

The maximum timeframe for running time buckets monitors, as well as to calculate a baseline for anomaly detection. This limit is applied in three cases: - On the first run of elementary or on full refresh (first run for all tables). - For each new table that is added to the monitoring (first run for table). - On each new metric that is added to the configuration (first run of new metric). - If there was a gap bigger than this limit since the last run. If undefined, default is 14 days back.

backfill_days: [int]

The minimal amount of buckets for running time buckets monitors. If these buckets were already calculated, Elementary will overwrite them. The reason behind it is to monitor recent backfills of data, if there were any. This configuration should be changed according to your data delays. If undefined, default is 2 buckets.

anomaly_sensitivity: [int]

This threshold determines the sensitivity level for anomaly detection. Elementary uses statistical analysis to grade deviations of recent metrics from historical metrics. Increasing this threshold makes the detection less sensitive for anomalies, and reducing makes it more sensitive (smaller deviations will be flagged as anomalies). If undefined, default threshold is 3.