Skip to main content

Generate your anomaly test with Elementary AI

Let our Slack chatbot create the anomaly test you need.
elementary.column_anomalies Executes column level monitors and anomaly detection on the column. Specific monitors are detailed in the table below and can be configured using the columns_anomalies configuration. The test checks the data type of the column and only executes monitors that are relevant to it. Default monitors by type:
Data quality metricColumn Type
null_countany
null_percentany
min_lengthstring
max_lengthstring
average_lengthstring
missing_countstring
missing_percentstring
minnumeric
maxnumeric
averagenumeric
zero_countnumeric
zero_percentnumeric
standard_deviationnumeric
variancenumeric
Opt-in monitors by type:
Data quality metricColumn Type
sumnumeric

​
Test configuration

No mandatory configuration, however it is highly recommended to configure a timestamp_column. 
data_tests:
  — elementary.column_anomalies:
    arguments:
      column_anomalies: column monitors list
      dimensions: sql expression
      timestamp_column: column name
      where_expression: sql expression
      anomaly_sensitivity: int
      anomaly_direction: [both | spike | drop]
      detection_period:
        period: [hour | day | week | month]
        count: int
      training_period:
        period: [hour | day | week | month]
        count: int
      time_bucket:
        period: [hour | day | week | month]
        count: int
      seasonality: day_of_week
      detection_delay:
        period: [hour | day | week | month]
        count: int
      ignore_small_changes:
        spike_failure_percent_threshold: int
        drop_failure_percent_threshold: int
      anomaly_exclude_metrics: [SQL expression]
models:
  - name: < model name >
    config:
      elementary:
        timestamp_column: < timestamp column >
    columns:
      - name: < column name >
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies: < specific monitors, all if null >
                where_expression: < sql expression >
                time_bucket: # Daily by default
                  period: < time period >
                  count: < number of periods >

  - name: < model name >
    ## if no timestamp is configured, elementary will monitor without time filtering
    columns:
      - name: < column name >
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies: < specific monitors, all if null >
                where_expression: < sql expression >


models:
  - name: login_events
    config:
      elementary:
        timestamp_column: 'loaded_at'
    columns:
      - name: user_name
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies:
                  - missing_count
                  - min_length
                where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
                time_bucket:
                  period: day
                  count: 1
              config:
                tags: ['elementary']

  - name: users
    ## if no timestamp is configured, elementary will monitor without time filtering
    data_tests:
        elementary.volume_anomalies
          config:
            tags: ['elementary']
    columns:
      - name: user_id
        data_tests:
          - elementary.column_anomalies:
              config:
                tags: ['elementary']
              arguments:
                timestamp_column: 'updated_at'
                where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
                time_bucket:
                  period: < time period >
                  count: < number of periods >
      - name: user_name
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies:
                  - missing_count
                  - min_length
              config:
                tags: ['elementary']