column_anomalies

models:
  - name: < model name >
    config:
      elementary:
        timestamp_column: < timestamp column >
    columns:
      - name: < column name >
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies: < specific monitors, all if null >
                where_expression: < sql expression >
                time_bucket: # Daily by default
                  period: < time period >
                  count: < number of periods >

  - name: < model name >
    ## if no timestamp is configured, elementary will monitor without time filtering
    columns:
      - name: < column name >
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies: < specific monitors, all if null >
                where_expression: < sql expression >

models:
  - name: login_events
    config:
      elementary:
        timestamp_column: 'loaded_at'
    columns:
      - name: user_name
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies:
                  - missing_count
                  - min_length
                where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
                time_bucket:
                  period: day
                  count: 1
              config:
                tags: ['elementary']

  - name: users
    ## if no timestamp is configured, elementary will monitor without time filtering
    data_tests:
        elementary.volume_anomalies
          config:
            tags: ['elementary']
    columns:
      - name: user_id
        data_tests:
          - elementary.column_anomalies:
              config:
                tags: ['elementary']
              arguments:
                timestamp_column: 'updated_at'
                where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
                time_bucket:
                  period: < time period >
                  count: < number of periods >
      - name: user_name
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies:
                  - missing_count
                  - min_length
              config:
                tags: ['elementary']

models:
  - name: < model name >
    config:
      elementary:
        timestamp_column: < timestamp column >
    columns:
      - name: < column name >
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies: < specific monitors, all if null >
                where_expression: < sql expression >
                time_bucket: # Daily by default
                  period: < time period >
                  count: < number of periods >

  - name: < model name >
    ## if no timestamp is configured, elementary will monitor without time filtering
    columns:
      - name: < column name >
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies: < specific monitors, all if null >
                where_expression: < sql expression >

models:
  - name: login_events
    config:
      elementary:
        timestamp_column: 'loaded_at'
    columns:
      - name: user_name
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies:
                  - missing_count
                  - min_length
                where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
                time_bucket:
                  period: day
                  count: 1
              config:
                tags: ['elementary']

  - name: users
    ## if no timestamp is configured, elementary will monitor without time filtering
    data_tests:
        elementary.volume_anomalies
          config:
            tags: ['elementary']
    columns:
      - name: user_id
        data_tests:
          - elementary.column_anomalies:
              config:
                tags: ['elementary']
              arguments:
                timestamp_column: 'updated_at'
                where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
                time_bucket:
                  period: < time period >
                  count: < number of periods >
      - name: user_name
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies:
                  - missing_count
                  - min_length
              config:
                tags: ['elementary']

Generate your anomaly test with Elementary AI

Let our Slack chatbot create the anomaly test you need.

elementary.column_anomalies Executes column level monitors and anomaly detection on the column. Specific monitors are detailed in the table below and can be configured using the columns_anomalies configuration. The test checks the data type of the column and only executes monitors that are relevant to it.

BigQuery: nested STRUCT fields. On BigQuery you can reference nested STRUCT leaf fields by their dotted path, for example user.address.city, wherever you name a column or dimension in this test. There is no need to flatten them into their own columns first, and the full dotted path is preserved in alerts.Fields under a REPEATED (array) ancestor are not supported.

Default monitors by type:

Data quality metric	Column Type
`null_count`	any
`null_percent`	any
`min_length`	string
`max_length`	string
`average_length`	string
`missing_count`	string
`missing_percent`	string
`min`	numeric
`max`	numeric
`average`	numeric
`zero_count`	numeric
`zero_percent`	numeric
`standard_deviation`	numeric
`variance`	numeric

Opt-in monitors by type:

Data quality metric	Column Type
`sum`	numeric

Test configuration

No mandatory configuration, however it is highly recommended to configure a timestamp_column.

data_tests:
  — elementary.column_anomalies:
    arguments:
      column_anomalies: column monitors list
      dimensions: sql expression
      timestamp_column: column name
      where_expression: sql expression
      anomaly_sensitivity: int
      anomaly_direction: [both | spike | drop]
      detection_period:
        period: [hour | day | week | month]
        count: int
      training_period:
        period: [hour | day | week | month]
        count: int
      time_bucket:
        period: [hour | day | week | month]
        count: int
      seasonality: day_of_week
      detection_delay:
        period: [hour | day | week | month]
        count: int
      ignore_small_changes:
        spike_failure_percent_threshold: int
        drop_failure_percent_threshold: int
      anomaly_exclude_metrics: [SQL expression]

models:
  - name: < model name >
    config:
      elementary:
        timestamp_column: < timestamp column >
    columns:
      - name: < column name >
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies: < specific monitors, all if null >
                where_expression: < sql expression >
                time_bucket: # Daily by default
                  period: < time period >
                  count: < number of periods >

  - name: < model name >
    ## if no timestamp is configured, elementary will monitor without time filtering
    columns:
      - name: < column name >
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies: < specific monitors, all if null >
                where_expression: < sql expression >

models:
  - name: login_events
    config:
      elementary:
        timestamp_column: 'loaded_at'
    columns:
      - name: user_name
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies:
                  - missing_count
                  - min_length
                where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
                time_bucket:
                  period: day
                  count: 1
              config:
                tags: ['elementary']

  - name: users
    ## if no timestamp is configured, elementary will monitor without time filtering
    data_tests:
        elementary.volume_anomalies
          config:
            tags: ['elementary']
    columns:
      - name: user_id
        data_tests:
          - elementary.column_anomalies:
              config:
                tags: ['elementary']
              arguments:
                timestamp_column: 'updated_at'
                where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
                time_bucket:
                  period: < time period >
                  count: < number of periods >
      - name: user_name
        data_tests:
          - elementary.column_anomalies:
              arguments:
                column_anomalies:
                  - missing_count
                  - min_length
              config:
                tags: ['elementary']

Elementary dbt package

Elementary Data tests

Anomaly Detection Tests

Schema Tests

AI Data Tests (Beta)

Other Tests

dbt Package Release Notes

Generate your anomaly test with Elementary AI

Test configuration

Generate your anomaly test with Elementary AI

​Test configuration

Test configuration