Skip to main content

Generate your anomaly test with Elementary AI

Let our Slack chatbot create the anomaly test you need.
elementary.dimension_anomalies The test counts rows grouped by given dimensions (columns/expressions). This test practically monitors the frequency of values in the configured dimension over time, and alerts on unexpected changes in the distribution. It is best to configure it on low-cardinality fields. If timestamp_column is configured, the distribution is collected per time_bucket. If not, it counts the total rows per dimension.

​
Test configuration

Required configuration: dimensions 
data_tests:
  — elementary.dimension_anomalies:
    arguments:
      dimensions: sql expression
      timestamp_column: column name
      where_expression: sql expression
      anomaly_sensitivity: int
      anomaly_direction: [both | spike | drop]
      detection_period:
        period: [hour | day | week | month]
        count: int
      training_period:
        period: [hour | day | week | month]
        count: int
      time_bucket:
        period: [hour | day | week | month]
        count: int
      seasonality: day_of_week
      detection_delay:
        period: [hour | day | week | month]
        count: int
      ignore_small_changes:
        spike_failure_percent_threshold: int
        drop_failure_percent_threshold: int
      anomaly_exclude_metrics: [SQL expression]
      exclude_final_results: [SQL expression]
models:
  - name: < model name >
    config:
      elementary:
        timestamp_column: < timestamp column >
    data_tests:
      - elementary.dimension_anomalies:
          arguments:
            dimensions: < columns or sql expressions of columns >
            # optional - configure a where a expression to accurate the dimension monitoring
            where_expression: < sql expression >
            time_bucket: # Daily by default
              period: < time period >
              count: < number of periods >

models:
  - name: login_events
    config:
      elementary:
        timestamp_column: "loaded_at"
    data_tests:
      - elementary.dimension_anomalies:
          arguments:
            dimensions:
              - event_type
              - country_name
            where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
            time_bucket:
              period: hour
              count: 4
          config:
            # optional - use tags to run elementary tests on a dedicated run
            tags: ["elementary"]
            # optional - change severity
            severity: warn

  - name: users
    # if no timestamp is configured, elementary will monitor without time filtering
    data_tests:
      - elementary.dimension_anomalies:
          arguments:
            dimensions:
              - event_type
          config:
            tags: ["elementary"]