elementary.column_anomalies

Executes column level monitors and anomaly detection on the column. Specific monitors are detailed in the table below and can be configured using the columns_anomalies configuration.

The test checks the data type of the column and only executes monitors that are relevant to it.

Default monitors by type:

Data quality metricColumn Type
null_countany
null_percentany
min_lengthstring
max_lengthstring
average_lengthstring
missing_countstring
missing_percentstring
minnumeric
maxnumeric
zero_countnumeric
zero_percentnumeric
standard_deviationnumeric
variancenumeric

Opt-in monitors by type:

Data quality metricColumn Type
sumnumeric

Test configuration

No mandatory configuration, however it is highly recommended to configure a timestamp_column.

tests:   — elementary.column_anomalies:     column_anomalies: column monitors list     timestamp_column: column name     where_expression: sql expression     anomaly_sensitivity: int     anomaly_direction: [both | spike | drop]     days_back: int     backfill_days: int     min_training_set_size: int     time_bucket:       period: [hour | day | week | month]       count: int     seasonality: day_of_week

models:
  - name: < model name >
    config:
      elementary:
        timestamp_column: < timestamp column >
    columns:
      - name: < column name >
        tests:
          - elementary.column_anomalies:
              column_anomalies: < specific monitors, all if null >
              where_expression: < sql expression >
              time_bucket: # Daily by default
                period: < time period >
                count: < number of periods >

  - name: < model name >
    ## if no timestamp is configured, elementary will monitor without time filtering
    columns:
      - name: < column name >
        tests:
          - elementary.column_anomalies:
              column_anomalies: < specific monitors, all if null >
              where_expression: < sql expression >