Skip to main content

Generate your anomaly test with Elementary AI

Let our Slack chatbot create the anomaly test you need.
elementary.all_columns_anomalies Executes column level monitors and anomaly detection on all the columns of the table. Specific monitors are detailed in the table below and can be configured using the columns_anomalies configuration. The test checks the data type of each column and only executes monitors that are relevant to it. You can use column_anomalies param to override the default monitors, and exclude_prefix / exclude_regexp to exclude columns from the test. Default monitors by type:
Data quality metricColumn Type
null_countany
null_percentany
min_lengthstring
max_lengthstring
average_lengthstring
missing_countstring
missing_percentstring
minnumeric
maxnumeric
averagenumeric
zero_countnumeric
zero_percentnumeric
standard_deviationnumeric
variancenumeric
Opt-in monitors by type:
Data quality metricColumn Type
sumnumeric

​
Test configuration

No mandatory configuration, however it is highly recommended to configure a timestamp_column. 
data_tests:
  — elementary.all_columns_anomalies:
    arguments:
      timestamp_column: column name
      column_anomalies: column monitors list
      dimensions: sql expression
      exclude_prefix: string
      exclude_regexp: regex
      where_expression: sql expression
      anomaly_sensitivity: int
      anomaly_direction: [both | spike | drop]
      detection_period:
        period: [hour | day | week | month]
        count: int
      training_period:
        period: [hour | day | week | month]
        count: int
      time_bucket:
        period: [hour | day | week | month]
        count: int
      seasonality: day_of_week
      detection_delay:
        period: [hour | day | week | month]
        count: int
      ignore_small_changes:
        spike_failure_percent_threshold: int
        drop_failure_percent_threshold: int
      anomaly_exclude_metrics: [SQL expression]
models:
  - name: < model name >
    config:
      elementary:
        timestamp_column: < timestamp column >
    data_tests:
      - elementary.all_columns_anomalies:
          arguments:
            column_anomalies: < specific monitors, all if null >
            where_expression: < sql expression >
            time_bucket: # Daily by default
              period: < time period >
              count: < number of periods >

models:
  - name: login_events
    config:
      elementary:
        timestamp_column: "loaded_at"
    data_tests:
      - elementary.all_columns_anomalies:
          config:
            tags: ["elementary"]
          arguments:
            where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
            time_bucket:
              period: day
              count: 1
            
            # optional - change global sensitivity
            anomaly_sensitivity: 3.5