# AI credits
Source: https://docs.elementary-data.com/cloud/ai-agents/ai-credits
AI credits are consumed when Elementary's AI does the reasoning.
With [MCP](/cloud/mcp/overview), Elementary returns data to the connected agent. The LLM (Cursor, Claude, etc.) handles reasoning and bills to that provider.
***
## What uses AI credits
| Action | What the AI does |
| ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| [**Incident investigation**](/cloud/ai-agents/triage-resolution-agent) | Diagnoses a data issue: root cause, impacted tables, suggested fix |
| [**Data discovery**](/cloud/ai-agents/catalog-agent) | Identifies which tables or columns to use for a specific metric or use case |
| [**Test coverage**](/cloud/ai-agents/test-recommendation-agent) | Generates dbt test PRs based on configured data policies and opens them for review |
| [**CI PR review**](/cloud/features/ci) | Lineage-aware review on every dbt PR, surfacing downstream impact before merge |
***
## What doesn't use AI credits
| Feature | Why |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| **All MCP tool calls** | Elementary returns data to the connected agent. The LLM handles reasoning |
| **Slack alerts and notifications** | Pure notification delivery. No AI involved |
| **Anomaly detection and monitors** | Volume and freshness monitors run on statistical algorithms, not LLMs |
| **Lineage and catalog browsing** | All UI navigation: lineage graphs, asset catalog, test coverage pages |
| **Manual incident merging** | Selecting and merging incidents by hand is a platform feature, not AI |
| **Deterministic incident grouping** | Rule-based clustering (e.g. Fivetran fail + downstream freshness). Coming soon |
| [**Reasoning on your own LLM key (BYOK)**](/cloud/ai-agents/byok) | When you connect your own Anthropic or OpenAI key, reasoning is billed by that provider, not by Elementary |
***
## Get more from credits
Enable automatic root cause. With it on, Elementary runs one AI call per incident posted to Slack instead of multiple team members each triggering their own analysis. More efficient, and everyone sees the same context.
Use MCP for ad hoc exploration. Querying lineage, assets, or test results via MCP costs zero Elementary credits. Use it freely from Cursor, Claude, or any MCP-compatible tool.
[Bring your own LLM key (BYOK)](/cloud/ai-agents/byok). Connect your own Anthropic or OpenAI key and the agent's reasoning is billed by that provider — Elementary charges no AI credits for it.
Credits consumed and credits remaining are always visible in the Elementary dashboard.
Need a breakdown by user, action type, or time period? Contact the account team for a detailed usage report.
# Bring your own key (BYOK)
Source: https://docs.elementary-data.com/cloud/ai-agents/byok
By default, the Elementary AI Agent runs on self-hosted models through Amazon Bedrock, and reasoning is billed as [AI credits](/cloud/ai-agents/ai-credits).
With Bring Your Own Key (BYOK), you connect your own LLM provider key instead. The agent then runs on your account with that provider, and **Elementary does not bill AI credits for any reasoning that uses your key** — you pay the provider directly.
## Supported providers
| Provider | Notes |
| ------------- | ----------------------------------------------------------------------- |
| **Anthropic** | Any Claude model available to your API key |
| **OpenAI** | Chat-capable models available to your API key (`gpt-*`, `o1`/`o3`/`o4`) |
Amazon Bedrock remains the default when no key is configured.
## How it works
* The key is used for both the interactive **chat** agent and the background **runner** (incident triage, test recommendations, and other automated workflows).
* When a key is configured, Elementary's AI credit quotas are **bypassed** — usage is metered by your provider, not by Elementary.
* Token counts are still recorded for observability, but no cost is attributed on Elementary's side.
* You can curate which models are exposed in the chat model picker. Leave the selection empty to expose every model your key can access.
On BYOK accounts, the agent's requests (including the system prompt) are sent to your provider using your key, so they appear in your provider's own request logs.
## Set it up
1. Go to **Account → Agent Settings → Bring your own key**.
2. Choose a provider (Anthropic or OpenAI) and paste your API key. Keys are stored encrypted.
3. Optionally select which models to expose in the chat picker.
4. Save. New chat sessions and agent runs will use your key.
To revert to Elementary's Bedrock models and AI credits, delete the key from the same screen.
Managing BYOK requires the **Edit integrations** permission.
# Catalog Agent
Source: https://docs.elementary-data.com/cloud/ai-agents/catalog-agent
**The Catalog Agent helps data analysts, business users, and AI tools find the right data on their own so engineers don't have to be the bottleneck.** By interpreting natural language questions and surfacing the right context—including data health, lineage, and metric definitions—it enables faster, more confident access to the data people need. The Catalog Agent lays the groundwork for more accurate and effective AI-powered workflows across your organization.
### Using the Catalog Agent
Open the Catalog Agent to start a conversation. Just ask a question like **“Which table should I use to analyze user retention?”** or **“Can I trust the orders table?”** and the agent will guide you to the right assets or insights.
The agent will interpret your request using metadata, lineage, and semantic context, and suggest relevant datasets, tables, or metrics. It will also surface freshness, incident status, and test results, and explain how the data is used and whether it can be trusted.
### Key Capabilities
### Intelligent Discovery
Helps users find the right data using metadata (tags, descriptions, owners, lineage to understand dependencies and impact and semantic layers from dbt, BI tools, or external catalogs for metric-level context
### Trust and Quality Signals
For every asset, the agent provides real-time indicators of reliability:
* Freshness metrics
* Open incidents and recent test results
* A high-level trust signal based on metadata coverage and data health
### Who It’s For
The Catalog Agent is built for **data analysts**, **business users**, **AI copilots**, and **intelligent applications** that need to find and understand the right data without making the data team a bottleneck.
We take data privacy seriously. Read our [**AI Privacy Policy**](/cloud/general/ai-privacy-policy) to understand how AI features are secured.
# Context Engine
Source: https://docs.elementary-data.com/cloud/ai-agents/context-engine
The Context Engine is the knowledge layer that powers the Elementary AI Agent and the MCP Server.
It connects lineage across your entire stack — from your data warehouse and dbt models through to BI dashboards and code — and layers in everything Elementary knows about how your data is actually performing. The result is a unified, always-current picture of your stack: what exists, how it's connected, how it's running, and its health status.
## What it holds
**From your connected integrations:**
* **Data warehouse** — table metadata, schema definitions, query history, usage stats
* **dbt** — model definitions, test configurations, run results, documentation
* **BI tools** — dashboards, reports, and their upstream dependencies
* **Code repository** — recent commits, pull requests, change history
* **Orchestration** — job runs, execution history
**From Elementary's own monitoring:**
* **Test results** — pass/fail history across dbt, Python, and Cloud tests, coverage gaps
* **Test performance** — execution times and cost per test run
* **Model performance** — execution times, cost, trends over time
* **Health scores** — data quality dimensions across all assets
Lineage connects these layers. A table in your warehouse maps to a dbt model, which feeds a dashboard, which is owned by a team. When something breaks, the Context Engine already knows the blast radius.
## What it enables
Because the Context Engine holds a complete, connected view of your stack, the Elementary AI Agent can:
* Trace the root cause of an incident across models, code changes, and upstream dependencies
* Understand the downstream impact before escalating or fixing
* Identify coverage gaps and recommend tests for the right assets
* Understand which tests are redundant and remove them to speed up the pipeline
* Enrich metadata with descriptions that reflect how assets are actually used
The MCP Server exposes this same context to external AI tools — Claude, Cursor, or any MCP-compatible client — so they can query your stack with full awareness.
The more integrations you connect, the more complete the context. Lineage between dbt and BI tools, for example, only works when both are connected.
Add data sources to expand what the Context Engine can see.
Expose the Context Engine to external AI tools.
# Governance Agent
Source: https://docs.elementary-data.com/cloud/ai-agents/governance-agent
**Governing your data isn't just about internal organization, it's the key to making your data reliably usable by both humans and AI.**
The Governance Agent helps you scale and maintain trusted, well-documented data assets without manual effort. It evaluates your metadata, identifies gaps, and suggests improvements, all based on how your organization already tags, documents, and assigns ownership.
By learning from your environment’s existing conventions, the agent brings consistency, policy enforcement, and AI-readiness across your pipelines.
### **Using the Governance Agent**
You can trigger the Governance Agent from the Catalog screen, and ask it a question like **"Which tables in the marketing domain are missing ownership?"**. It will analyze the asset’s metadata and context, validate it against both best practices and your custom policies, and generate a list of actionable suggestions. When you’re ready, the agent can open a pull request with the proposed metadata changes.
### Key Capabilities
### Learns from Your Environment
The agent understands and scales your internal standards by analyzing:
* Tag structures and naming conventions
* Ownership assignments
* Folder hierarchies and asset types
* Custom metadata fields already in use
This ensures all recommendations align with your existing governance style — not just a static set of rules.
### Validates Governance Coverage
The agent identifies:
* Missing or inconsistent documentation for tables, columns, and BI assets
* Unassigned or conflicting owners
* Gaps in classification, tags, or custom metadata
* Sensitive data fields that aren’t properly flagged or controlled
### Metadata Enrichment
The agent suggests or autofills:
* Table and column descriptions
* Tags and classifications
* Ownership fields
* Custom metadata based on known structures
### Sensitive Data Enforcement
Sensitive fields are automatically detected and checked to ensure appropriate tagging is applied, and that there's no propagation to unauthorized downstream assets
### Pull Request Automation
Once suggested changes are ready, the agent will open a **pull request** for you to review and apply.
### Who It’s For
Built for **data platform teams, analytics engineers, and governance leads** who want to increase metadata coverage, enforce governance policies, and ensure their data is ready for AI, without adding manual overhead.
We take data privacy seriously. Read our [**AI Privacy Policy**](/cloud/general/ai-privacy-policy) to understand how AI features are secured.
# Connect MCP servers to the agent
Source: https://docs.elementary-data.com/cloud/ai-agents/mcp-connections
MCP connections let the Elementary AI Agent pull context from external systems through their own [MCP](/cloud/mcp/overview) servers. Their tools are added to the agent alongside Elementary's own tools, so root cause analysis and triage can reach across systems Elementary doesn't own.
This is the inbound direction — external MCP servers feeding tools **into** the Elementary agent. For exposing Elementary's own data **to** other AI tools (Cursor, Claude, etc.), see the [MCP Server](/cloud/mcp/overview).
## Available connectors
| Connector | Transport | What it adds |
| ------------- | ----------- | --------------------------------------------------- |
| **dbt Cloud** | Remote HTTP | dbt Cloud run, job, and model context for the agent |
## dbt Cloud
Connect dbt Cloud so the agent has complete context when triaging issues — it can pull dbt Cloud run and job context directly through its MCP tools.
1. Go to **Account → Agent Settings → MCP Connections**.
2. Add a **dbt Cloud** connection.
3. Paste your dbt Cloud MCP server URL and a service token. Tokens are stored encrypted.
4. Save.
Once connected, the agent's dbt Cloud tools are available in chat and in automated runs. Enable the connection from the chat integrations picker to include it in a session.
See the [dbt Cloud integration](/cloud/integrations/transformation-and-orchestration/dbt-cloud#connect-dbt-cloud-to-the-agent) for the full setup.
Managing MCP connections requires the **Edit integrations** permission.
# Elementary AI Agent
Source: https://docs.elementary-data.com/cloud/ai-agents/overview
The Elementary AI Agent automates the workflows that slow data teams down most: finding coverage gaps, triaging incidents, fixing broken pipelines, enriching metadata, and optimizing query performance.
It's a single agent backed by the [Context Engine](/cloud/ai-agents/context-engine) — a continuously updated knowledge layer built from every integration you connect. Because it has full context across your stack, it can act on incidents, recommend tests, and surface insights without requiring you to pull together the picture manually.
Every action the agent takes goes through a pull request or requires explicit confirmation. Your team stays in control of what gets merged.
## Specialized agents
The Elementary AI Agent orchestrates a set of specialized agents, each focused on a specific workflow:
Enriches metadata and enforces documentation standards across your data assets.
Investigates incidents, traces root causes, and opens pull requests with proposed fixes.
Identifies coverage gaps and recommends or generates tests for the right assets.
Surfaces data assets, their relationships, and usage context in natural language.
Detects slow or expensive models, flags redundant tests that slow down the pipeline, and suggests optimizations.
## How it works
The agent runs on self-hosted models through Amazon Bedrock. No data leaves your environment, and all AI features are opt-in by default. Agent behavior is governed by AI policies you define — reviewable, adjustable, and aligned to your governance standards.
Understand what the agent knows and how it knows it.
How AI features are secured and what data is accessed.
Want to learn more? [Our experts will be happy to chat.](https://meetings-eu1.hubspot.com/joost-boonzajer-flaes/intro-call-docs?uuid=64e4c20d-0ff3-4454-9387-c93b47afcee6)
# Performance & Cost Agent
Source: https://docs.elementary-data.com/cloud/ai-agents/performance-cost-agent
**Optimize long-running queries, eliminate bottlenecks, and keep compute costs under control without digging through logs or rewriting SQL by hand.** The Performance and Cost Agent helps you identify inefficiencies in your pipelines and resolve them quickly, so your data stays fresh, fast, and within budget.
It improves performance by analyzing query logic, historical execution patterns, and resource usage. This makes it easier to meet data SLAs and ensure products, dashboards, and AI models get the data they need, when they need it.
### Using the Performance and Cost Agent
Trigger the agent from any asset flagged as underperforming or from a query known to be slow or expensive. It reviews the SQL, surrounding metadata, and historical performance, then recommends code changes and resource configuration tweaks to improve efficiency.
You’ll see an estimate of time and cost savings before applying changes. Once approved, the agent opens a pull request with its suggested improvements and can continue iterating until performance targets are met.
### Key Capabilities
### Performance Diagnostics and Optimization
The agent evaluates:
* Query logic and structure
* Historical execution time and resource usage
* Metadata context and lineage
It uses that analysis to identify performance bottlenecks and propose:
* Code-level SQL optimizations
* Resource allocation and configuration changes
* Structural improvements that reduce load or complexity
### Cost and SLA Awareness
Each recommendation includes an estimate of:
* Time saved per run
* Cost reduction per run or per day
* Impact on data freshness and downstream data availability
This makes it easier to prioritize which performance issues are worth fixing and why.
### Automated Implementation and Iteration
Once recommendations are approved, the agent opens a pull request with the proposed changes. It can continue to iterate based on updated performance feedback until the asset meets defined SLAs or cost targets.
### Who It’s For
Built for **analytics engineers and data engineers** who need to improve pipeline speed, reduce costs, and ensure data products stay fresh and reliable without getting buried in manual query tuning.
We take data privacy seriously. Read our [**AI Privacy Policy**](/cloud/general/ai-privacy-policy) to understand how AI features are secured.
# Test Recommendation Agent
Source: https://docs.elementary-data.com/cloud/ai-agents/test-recommendation-agent
**Make sure you’re covered—not by how many tests you write, but by how many problems you catch.** The Test Recommendation Agent analyzes your data asset, metadata, lineage, and existing test suite to suggest high-impact tests that match your team’s style and priorities.
It helps you expand meaningful coverage, reduce alert fatigue, and maintain a reliable test strategy as your pipelines evolve.
### Using the Test Recommendation Agent
Open the agent from the test configuration screen and ask a question like **"What tests can ensure all of my customer data is up to date?"** or **"Which upstream tables need tests to protect my revenue dashboard?"**. The agent generates recommendations based on asset type, criticality, lineage, and your existing test configurations. You can approve, adjust, or skip suggestions as needed. Once approved, the agent opens a pull request with changes that follow your team’s current test structure.
### Key Capabilities
### Coverage Analysis and Test Recommendations
The agent analyzes assets individually and in context, taking into account lineage, metadata, SQL logic, and usage, to identify where coverage is missing or misaligned. It suggests new tests based on best practices and your team's existing style, and your own test policy configuration.
### Fatigue Reduction
The agent flags tests that are overly noisy, ineffective, or no longer relevant, and helps tune configurations to make your test suite more reliable and actionable.
### Automated Implementation
Once you're satisfied with the recommendations, the agent opens a pull request with all changes. You stay in control, but no longer have to do everything manually.
### Who It’s For
Built for **analytics engineers and data engineers** who want to improve test coverage intelligently, avoid alert fatigue, and maintain high data quality without slowing down development.
We take data privacy seriously. Read our [**AI Privacy Policy**](/cloud/general/ai-privacy-policy) to understand how AI features are secured.
# Triage & Resolution Agent
Source: https://docs.elementary-data.com/cloud/ai-agents/triage-resolution-agent
**Automates root cause analysis, clarifies the impact, and suggests actionable fixes so you spend less time triaging and more time solving what matters.** The result is a shift from manual investigation to a faster, more focused workflow, allowing you to focus on delivering business value rather than fighting fires.
Whether it's a failed test, a model issue, or an unexpected anomaly, it inspects recent code changes, execution history, lineage, and, when allowed, the data itself. It prioritizes the incidents that matter, proposes a fix, and can open a pull request for you. That way, incidents stop slowing you down, and pipelines begin to fix themselves.
## Using the Triage & Resolution agent
From the Incident Management screen click **“Investigate with AI”** to trigger the agent. It will begin analyzing the incident and surface insights. You can follow along, ask clarifying questions, and when you're ready, allow the agent to open a pull request with its suggested fix - all within your review flow.
## Key Capabilities
### Automated Root Cause Investigation
The agent dives deep into each incident to uncover why it happened. It inspects:
* **Recent code changes**, including relevant commits and pull requests.
* **Historical executions** of the pipeline or asset.
* **Upstream lineage**, to detect whether the issue originated from a dependency.
* **The data itself** (when allowed), including the affected asset and its related tables.
### Impact Clarification
It doesn’t stop at what’s broken, it tells you **what’s affected**:
* Analyzes **downstream dependencies** to understand the blast radius.
* Validates whether **data products, dashboards, metrics**, or **AI applications** are impacted.
### Incident Prioritization
Each issue is automatically ranked by the downstream dependencies impact, and the criticality of the affected asset.
This helps your team know what to focus on, and what can wait.
### Automated Remediation (Where Possible)
Once the problem is identified, the agent can:
* Recommend **concrete steps to fix** the issue.
* Automatically **open a pull request** with proposed changes.
* **Notify stakeholders** affected by the issue.
## Who It’s For
Designed for **hands-on data professionals**, including data engineers, analytics engineers, and platform teams, who need a faster, more reliable way to stay on top of incidents without getting buried in triage work.
We take data privacy seriously. Read our [**AI Privacy Policy**](/cloud/general/ai-privacy-policy) to understand how AI features are secured.
# Detection and coverage
Source: https://docs.elementary-data.com/cloud/best-practices/detection-and-coverage
In Elementary you can detect data issues by combining data validations (as dbt tests, custom SQL) and anomaly detection monitors.
As you expand your coverage, it's crucial to balance between coverage and meaningful detections. While it may seem attractive to implement extensive monitoring throughout your data infrastructure, this approach is often suboptimal. Excessive failures can lead to alerts fatigue, potentially causing them to overlook significant issues. Additionally, such approach will incur unnecessary compute costs.
In this section we will cover the available tests in Elementary, recommended tests for common use cases, and how to use the data quality dimensions framework to improve coverage.
## Supported data tests and monitors
Elementary detection includes:
* Data tests - Validate an explicit expectation, and fail if it is not met.
* Example: validate there are no null values in a column.
* Anomaly detection monitors - Track a data quality metric over time, and fail if there is an anomaly comparing to previous values and trend.
* Example: track the rate of null values in a column over time, fail if there is a spike.
### Data tests
* dbt tests - Built in dbt tests (`not_null`, `unique`, `accepted_values`, `relationship` )
* dbt packages - Any dbt package test, we recommend installing `dbt-utils` and `dbt-expectations` .
* Custom SQL tests - Custom query, will pass if no results and fail if any results are returned.
### Anomaly detection monitors
Elementary offers two types of anomaly detection monitors:
* **Automated Monitors** - Out-of-the-box volume and freshness monitors activated automatically, that query metadata only.
* **Opt-in anomaly detection tests** - Monitors that query raw data and require configuration.
### Recommendations
* Deploy the packages dbt-utils and dbt-expectations in your dbt projects, to enrich your available tests
* Refer to the [dbt test hub](https://www.elementary-data.com/dbt-test-hub) by Elementary, to explore available tests by use case
## Fine-tuning automated monitors
As soon as you connect Elementary Cloud Platform to your data warehouse, a backfill process will begin to collect historical metadata. Within an average of a few hours, your automated monitors will be operational. By default, Elementary collects at least 21 days of historical metadata.
You can fine tune the [**configuration and provide feedback**](/cloud/features/data-tests/cloud-tests-overview) to adjust the detection to your needs.
You can read here about how to interpret the result, and what are the available setting of each monitor:
* [Automated Freshness](/cloud/features/anomaly-detection/automated-freshness)
* [Automated Volume](/cloud/features/anomaly-detection/automated-volume)
## Common testing use cases
We have the following recommendations for testing different data assets:
### Data sources
To detect issues in sources updates, you should monitor volume, freshness and schema:
* Volume and freshness
* Data updates - Elementary cloud provides automated monitors for freshness and volume. **These are metadata monitors.**
* Updates freshness vs. data freshness - The automated freshness will detect delays in **updates**. \*\*\*\*However, sometimes the update will be on time, but the data itself will be outdated.
* Data freshness (advanced) - Sometimes a table can update on time, but the data itself will be outdated. If you want to validate the freshness of the raw data by relaying on the actual timestamp, you can use:
* Explicit threshold [freshness dbt tests](https://www.elementary-data.com/dbt-test-hub) such as `dbt_utils.recency` , or [dbt source freshness](https://docs.getdbt.com/docs/deploy/source-freshness).
* Elementary `event_freshness_anomalies` to detect anomalies.
* Data volume (advanced) - Although a table can be updated as expected, the data itself might still be imbalanced in terms of volume per specific segment. There are several tests available to monitor that:
* Explicit [volume expectations](https://www.elementary-data.com/dbt-test-hub) such as `expect_table_row_count_to_be_between`.
* Elementary `dimension_anomalies` , that will count rows grouped by a column or combination of columns and can detect drops or spikes in volume in specific subsets of the data.
* Schema changes
* Automated schema monitors are coming soon:
* These monitors will detect breaking changes to the schema only for columns being consumed based on lineage.
* For now, we recommend defining schema tests on the sources consumed by downstream staging models.
Some validations on the data itself should be added in the source tables, to test early in the pipeline and detect when data is arriving with an issue from the source.
* Low cardinality columns / strict set of values - If there are fields with a specific set of values you expect use `accepted_values`. If you also expect a consistency in ratio of these values, use `dimension_anomalies` and group by this column.
* Business requirements - If you are aware of expectations specific to your business, try to enforce early to detect when issues are at the source. Some examples: `expect_column_values_to_be_between`, `expect_column_values_to_be_increasing`, `expect-column-values-to-have-consistent-casing`
### Recommendations
* Add data freshness and volume validations for relevant source tables, on top of the automated monitors (advanced)
* Add schema tests for source tables
### Primary / foreign key columns in your transformation models
Tables should be covered with:
* Unique checks on primary / foreign key columns to detect unnecessary duplications during data transformations.
* Not null checks on primary / foreign key columns to detect missing values during data transformations.
For incremental tables, it’s recommended to use a `where` clause in the tests, and only validate recent data. This will prevent running the tests on large data sets which is costly and slow.
#### Recommendations
* Add `unique` and `not_null` tests to key columns
### Public tables
As these are your data products, coverage here is highly important.
* Consistency with sources (based on aggregation/primary keys)
* Volume and freshness
* Unique and not null checks on primary keys
* Schema to ensure the "API" to data consumers is not broken
* Business Metrics / KPIs
* Sum / max anomalies group by your critical dimensions / segments (For example - country, platform…)
### Data quality dimensions framework
To ensure your detection and coverage have a solid baseline, we recommend leveraging the quality dimensions framework for your critical and public assets.
The quality dimensions framework divides data validation into six common dimensions:
* **Completeness**: No missing values, empty values, nulls, etc.
* **Uniqueness**: The data is unique, with no duplicates.
* **Freshness**: The data is up to date and within the expected SLAs.
* **Validity**: The data is in the correct format and structure.
* **Accuracy**: The data adheres to our business requirements and constraints.
* **Consistency**: The data is consistent from sources to targets, and from sources to where it is consumed.
Elementary has already categorized all the existing tests in the dbt ecosystem, including all elementary anomaly detection monitors, into these quality dimensions and provides health scores per dimension automatically. It also shows if there are coverage gaps per dimension.
We highly recommend going to the relevant quality dimension, then filtering by a business domain tag to see your coverage gaps in that domain.
Example -
In this example, you can see that accuracy tests are missing for our sales domain. This means we don't know if the data in our public-facing "sales" tables adheres to our business constraints. For example, if we have an e-commerce shop where no product has a price below $100 or above $1000, we can easily add a test to validate this. Implementing validations for the main constraints in this domain will allow us to get a quality score for the accuracy level of our data.
NOTE: The `Test Coverage` page in Elementary allows adding any dbt test from the ecosystem, Elementary anomaly detection monitors, and custom SQL tests. We are working on making it easier to add tests by creating a test catalog organized by quality dimensions and common use cases.
Example for tests in each quality dimension -
* **Completeness**:
* not\_null, null count, null percent, missing values, empty values, column anomalies on null count, null percent, etc
* **Uniqueness**:
* unique, expect\_column\_values\_to\_be\_unique, expect\_column\_unique\_value\_count\_to\_be\_between, expect\_compound\_columns\_to\_be\_unique
* **Freshness**: The data is up to date and within the expected SLAs.
* Elementary automated freshness monitor, dbt source freshness, dbt\_utils.recency, expect\_grouped\_row\_values\_to\_have\_recent\_data
* **Validity**: The data is in the correct format and structure.
* expect\_column\_values\_to\_match\_regex, expect\_column\_min\_to\_be\_between, expect\_column\_max\_to\_be\_between, expect\_column\_value\_lengths\_to\_be\_between, column anomalies on min, max, string lengths
* **Accuracy**: The data adheres to our business requirements and constraints.
* expression\_is\_true, custom SQL
* **Consistency**: The data is consistent from sources to targets, and from sources to where it is consumed.
* relationship, expect\_table\_row\_count\_to\_equal\_other\_table, expect\_table\_aggregation\_to\_equal\_other\_table
# Governance for observability
Source: https://docs.elementary-data.com/cloud/best-practices/governance-for-observability
For an effective data observability process, it’s recommended to establish clear ownership, priorities and segmentation of data assets. This structure enhances governance, speeds up issue resolution, and improves data health tracking.
Segmenting assets organizes data into manageable units, making monitoring and triage easier. Ownership ensures accountability, with specific individuals responsible for quality and response to incidents.
Once you’ve set up ownership and tagging, the next step is identifying what matters most — see [Critical Assets](/cloud/features/data-governance/critical_assets) for how to mark and prioritize your most important tables.
## Introduction to tags, owners and subscribers
### Tags
As your data platform evolves and more people are maintaining it, structure and context become significantly more important. Tags are a great tool to create that context, and segment your data assets by business domains, data products, priority, etc.
In Elementary tags are automatically included in alerts, and you can create rules to distribute alerts to different channels by tag. Additionally, different views in the platform can be filtered by tag, and provide a view for a subset of your data assets.
* Tags for tables can be added in code at the model or folder level, and the `tags` key.
* It’s recommended to leverage dbt directories hierarchy to set tags to entire directories (in the dbt\_project.yml). Tags are aggregated, so if a specific model under the directory has a different tag, the model will have both tags.
```yaml theme={null}
models:
analytics:
marketing:
+tags: marketing
public:
+tags: marketing-public
```
* Tags for tests can be added in code or in the Elementary UI when adding a test.
### Owners and subscribers
The best method to reduce time to response when there is a data issue is having a clear owner that is in charge of initial triage and accountable for the asset health. In Elementary owners are automatically tagged in alerts. Additionally, different views in the platform can be filtered by owner.
A data asset or test should have only one owner, but other people might want to be notified on issues. These people can be listed as subscribers, and will be automatically tagged in alerts.
* If you use a valid Slack / MS teams user as owner / subscriber, they will be tagged in alerts.
* The owner of an asset should be the person / team that is expected to respond to an issue in that asset.
* If there are specific tests or monitors that are relevant to other people, they can be the owners of these tests.
For example: A data engineer is the owner of a model and will be notified on freshness, volume, and data validations issues. A data analyst added some custom SQL tests to validate business logic on this model, and he owns these tests.
* It’s recommended to leverage dbt directories hierarchy to set owners to entire directories (in the dbt\_project.yml). Owners are unique, so an owner that is defined on a model overrides the directory configuration. (Subscribers are aggregated).
```yaml theme={null}
models:
- name: return_on_ad_spend
config:
tags:
- marketing-public
- marketing
meta:
owner: "@analytics.engineer"
subscribers:
- "@marketing.data.analyst"
- "@another.marketing.data.analyst"
```
## Business domains & Data products
* We recommend configuring the following tags for models:
* **Business domains** - These tags should be useful to understand what is the business context of the asset, and for stakeholders to filter and view the status of assets relevant to their business unit. Relevant examples are tags such as: `product-analytics` , `marketing` , `finance` , etc.
* **Data products** - Public tables that are exposed as “data products” to data consumers. These are the most important tables within a specific domain, similar to an API for an application. Public tables are usually the interface and focal point between analytics engineers and data analysts. It's crucial for both to be aware of any data issues in these tables. Relevant examples are tags such as: `product-analytics-public` , `marketing-public` , `data-science-public` , etc.
* Another possible implementation is using 3 types of tags -
* `marketing-internal` for all internal transformations on marketing data.
* `marketing-public` for all public-facing marketing data.
* `marketing` for all marketing-related data assets.
* **Owners and subscribers -**
* Make sure to have clear ownership defined for all your public-facing tables. We also recommend adding subscribers to the relevant public tables.
* Usually, the owners of these public tables are the analytics engineering team, and the subscribers are the relevant data analysts who rely on the data from these tables.
### Recommendations
* Add business domain tags to public tables
* Define owners for public facing tables
* Add data consumers as subscribers to relevant public facing tables
## Priorities (optional)
Another useful tagging convention can be to set a tag that filters a subset of assets by their priority, so you could establish a process of response to issues with higher criticality.
Decide how many levels of priority you wish to maintain, and implement by adding a `critical` tag to your critical assets, or create a `P0`, `P1` , `P2` tags for several priority levels.
This will enable you to filter the results in Elementary by priority, and establish workflows such as sending `critical` alerts to Pagerduty, and the rest to Slack.
### Recommendations
* Add priorities / critical tags to tables / tests (Optional)
* Add owners to all top priority tables / tests (Optional)
## Data sources
Many data issues are a result of a problem in the source data, so effectively monitoring source tables is significant to your pipeline health.
Use tags to segment your source tables:
* If multiple source tables are loaded from the same source, we recommend grouping them by tags, such as: `mongo-db-replica`, `salesforce`, `prod-postgres`, etc.
* To make triage easier, you can also add tags of the ingestion system, such as: `fivetran`, `airflow` , `airbyte` , `kafka` , etc.
Ownership and subscribers:
* Usually, sources are managed by data engineers and analytics engineers are their consumers. One common way to manage this is to set data engineers as the owners and analytics engineering team members as the subscribers.
```yaml theme={null}
sources:
- name: fivetran_salesforce_sync
config:
tags:
- fivetran
- salesforce
meta:
owner: :"@data.engineer"
subscribers: "@analytics.engineer"
```
### Recommendations
* Add tags to source tables that describe the source system and / or ingestion method
* Add owners and subscribers to source tables
## Recommendations
* Add business domain tags to public tables
* Define owners for public facing tables
* Add data consumers as subscribers to relevant public facing tables
* (Optional) Add priorities / critical tags to tables / tests
* (Optional) Add owners to all top priority tables / tests
* Add tags to source tables that describe the source system and / or ingestion method
* Add owners and subscribers to source tables
# Elementary Best Practices
Source: https://docs.elementary-data.com/cloud/best-practices/introduction
The goal of this collection of guides is to help you effectively implement and use Elementary. We'll cover best practices and provide practical tips to enhance your governance, detection,
coverage, response and collaboration.
Whether you're new to Elementary or looking to optimize your current usage, these guides will help you leverage its full potential to improve your data
reliability.
# Triage & response
Source: https://docs.elementary-data.com/cloud/best-practices/triage-and-response
Maintaining high data quality is much more than adding tests - It's about creating processes.
The processes that will improve your data quality, reduce response times, and prevent repeating incidents have to do with:
* Clear ownership and response plan
* Incident management
* Effective triage and resolution
* Ending incidents with improvements, not just resolution
Elementary has tools in place to help, and this guide is meant to help get as much value as possible from Elementary in the process of handling data incidents.
## Plan the response in advance
Your response to a data incident doesn't actually start when the failure happens. An effective response starts when you add a test / monitor / dataset.
For every test or monitor you add, think about the following -
* Who should look into a failure?
* Who should be notified of the failure?
* What is the potential impact and severity of a failure?
* What information should the notification include?
* How to go about resolving the issue? What are the steps?
According to these answers, you should add configuration that will impact the alert, alert distribution, and triage:
### Recommendations
* Add a test description that details what it means if this test fails, and context on resolving it. Descriptions can be added in UI or in code.
* Each failure should have an owner, that should look into the failure. It can be the owner of the data set or an owner of a specific test.
* If others need to be notified, add subscribers.
* Use the [severity of failures](https://docs.getdbt.com/reference/resource-configs/severity) intentionally, and even leverage conditional expressions (`error_if`, `warn_if`)
* Test failures and alerts include a sample of the failed results, and the test query. You can change the test query and / or add comments to it, that can provide triage context.
```yaml theme={null}
data_tests:
- unique:
config:
error_if: ">10"
meta:
description: "More than 10 duplicate records arriving from the source, as this is a staging table"
owner: "@data.provider"
tags: "critical", "mongo-db", "raw-production-replica"
```
## Alert distribution
As far as alerts are concerned, the desired situation is that team members will only get alerts they need to do something about - Fix the issue, wait for resolution to refresh a dashboard, etc.
Alert distribution can be configured in the [Alert rules](/cloud/features/alerts-and-incidents/alert-rules) feature.
The alerts can be distributed to different channels (within Slack / MS Teams) and to different tools (Pagerduty, Ops Genie, etc).
Elementary users usually distribute alerts by:
1. Business domain tags - In teams where each domain has their own data teams, it's recommended to have a separate Slack channel for alerts on that domain's models. The domain alert rules are usually defined by tags.
2. Responsible team - For example, if there is a problem with null values in a Salesforce source, it makes sense to send the alert straight to the Salesforce team. These alert rules can be defined by model / source name, tag or owner.
3. Criticality - The most critical alerts are usually model error alerts, and handling them is critical because it blocks the pipeline. Since those issues are sometimes time sensitive, some teams choose to send them to Pager Duty or Ops Genie, or at least a dedicated Slack channel with different notification settings.
4. Low priority alerts / warnings - We generally recommend refraining from sending Slack alerts for failures that don't have a clear response plan yet. These failures can not be sent at all, or sent to a muted channel that will operate as a "feed".
Such failures can be: 1. Newley configured anomaly detection tests or explicit tests where you have low certainty about the threshold / expectation. 2. Anomaly detection tests that you consider as a safety measure, not a clear failure.
This is not to say that they are not interesting - but, they can be investigated within the Elementary UI, using the incidents page, at a time of convenience. We believe alerts are an interruption to the daily schedule and such an interruption should only occur if it's justified. To avoid getting such alerts, we recommend filtering your alert rules on "Failure" or "Error" statuses.
## Notifying stakeholders
There are several ways to notify data consumers and stakeholders about ongoing problems.
While some customers prefer to do it personally after triaging the incidents, others prefer saving this time and going with automated notifications.
For models intended for public consumption (by BI dashboards, ML models etc) we recommend setting up [subscribers](/cloud/features/alerts-and-incidents/owners-and-subscribers#subscribers). Those subscribers will be tagged in Slack on every alert that is sent on those tables. Unlike owners, there can be many subscribers to an alert.
Tagging subscribers is of course optional, and simply adding them to the relevant channels can also suffice.
You can also use [Incidents Digest](/cloud/features/alerts-and-incidents/incident-digest) to send scheduled summaries of incidents to broader teams, reducing noise while keeping stakeholders informed.
## Incident management
Elementary has an [Incidents page](/cloud/features/alerts-and-incidents/incidents) where failures automatically create incidents. The page gives a comprehensive overview of all current and previous incidents, organized by status (Open, Acknowledged, Resolved), where users can view, prioritize, assign, and resolve incidents.
### Incident management usability
* Each incident has 3 settings: Assignee, status and severity.
* These can be changed directly from the Slack notification.
* The severity is set to `high` for failures and `normal` for warnings. You can manually change to `critical` or `low` .
* You can select several incidents and make changes to the settings in bulk.
* Incidents contain one or more monitors. You can [merge related incidents](/cloud/features/alerts-and-incidents/incident-merging) to group them by root cause, reducing noise.
* The incident detail drawer provides **Lineage**, **Monitors**, and **Timeline** tabs for investigation.
### Incident management best practices
* Your goal should be to lower the time to resolution of incidents.
* Incidents should have a clear assignee.
* Use the quick view of open and unassigned incident to monitor this.
* The best implementation for this is pre-defining the assignee as owner, so they will get tagged on the failure.
* Set clear expectations with assignees.
* These can be set on severity of incidents. For example:
* Critical - Should be handled immediately.
* High - Should be resolved by end of day.
* Normal - Should be resolved by end of week.
* Low - Should be evaluated weekly, might trigger a change in coverage.
* If no one cares about an incident, this should impact coverage.
* Use [auto-merge rules](/cloud/features/alerts-and-incidents/incident-merging#auto-merge-rules) to automatically group related freshness incidents, reducing the number of incidents your team needs to triage.
## Triage incidents
When triaging incidents, there are 4 steps to go through:
1. Impact analysis - Although root cause analysis will lead you to resolve the issue, impact analysis should be done first. The reason is the impact determines the criticality of the incidents, and therefore the priority and response time.
2. Root cause analysis
3. Resolution
4. Post mortem - Quality learnings from incidents is how you improve over time, and reduce the time to resolution and frequency of future incidents.
Use the **Investigate with AI** action on any incident to trigger the [Triage & Resolution Agent](/cloud/ai-agents/triage-resolution-agent). It can perform impact analysis, root cause analysis, and even suggest [merging related incidents](/cloud/features/alerts-and-incidents/incident-merging#ai-agent-assisted-merge) by shared root cause.
### Impact analysis
The goal of doing an impact analysis is to determine the severity and urgency of the incident, and understand if you need to communicate the incident to consumers (if there isn't a relevant alert rule).
These are the questions that should be asked, and product tips on how to answer with Elementary:
* Was this a failure or just a warning?
* As long as you and your team are intentional in determining severities, this can focus you on failures first.
* Does the incident break the pipeline / create delay?
* Is the failure is a model failure, or a freshness issue?
* Do we run `dbt build` and this failure stopped the pipeline?
* Check the **Model runs** section of the dashboard to see if there are skipped models, as failures in build cause the downstream models to be skipped.
* How important is the data asset?
* Check in the catalog or node info section in the lineage if it has a tag like `critical` , `public` or a data product tag. You can also look at the description of the data asset, whether it's a table or a column.
* Does the failure impact important downstream assets? Did the issue propagate to downstream assets?
* A table might not be critical, but it's upstream from a critical one, making it part of a critical path.
* Check in the lineage if there are downstream important BI assets / public tables. You can navigate to the lineage directly from the incident detail's **Lineage** tab, or from test results by clicking `view in lineage`. If the incident is a failed column test, you can filter only the downstream lineage of the specific column by clicking on `filter column`.
* Use the lineage filters to color and highlight all the tables in a path that match your filtering criteria.
* If there are downstream critical tables, you might want to check if the issue actually propagated to it. A quick way to do it is to copy the test query, and run it on downstream assets (by changing the referenced table and column).
* What is the magnitude of the failure? How many failed results out of the total volume?
* Most tests return the number of failed results. A failure in a `unique` test for can be dramatic if it impacts many rows, but insignificant if there is just one case of duplicates.
* You can see the total number of failures as part of the test result / alert.
* On the `Test performance` page, you can compare this number to previous failures of the same tests.
## Root cause analysis
If the incident is not important we recommend resolving the incident and then removing / disabling the test.
If the incident is important we need to start the investigation process, and understand the root cause. Your failures would usually be caused by issues at the source, code changes, or an infrastructure issue.
* Is there a data issue at the source?
* Check in the lineage and see if there is coverage and failures on upstream tables, you can use the lineage filters to limit the scope to relevant failures (if `not_null` failed, filter on `not_null` tests).
* Check the test result sample. If you want to see more results copy the test SQL and run it in your DWH console.
* Sometimes an issue would be in a certain dimension, like a specific product event that stopped arriving or changed. Aggregate the test query by key dimensions in the table to understand if it's relevant to specific subset of the data.
* Check if the test is flaky in the `test performance` screen. This usually means it's a problem that happens frequently at the source data.
* Is it a code issue?
* Check recent PRs to the underlying monitored table.
* Check recent PRs merged to upstream tables.
* Are there any other related failures that happen at the same time following a recent release?
* Check metrics and test results like volume of tables to see if there is a wrong join.
* If the result is an `error` and not `fail` or `warning`, it means the test / model failed to run. This can either be caused by a timeout or issue at the DWH, or by a code change that lead to a syntax error / broken lineage.
* Look at the error message to understand if it comes from dbt or the DWH, and what is the issue.
## Post mortem - Learning from incidents
Learning from incidents we had is how we improve our coverage, response times and reliability.
Here are some common actions to take following an incident:
* Incident wasn't important - If the incident wasn't important or significant, remove the test or change the severity to warning.
* It was hard to determine the severity of the incident - Make changes to the tags and descriptions of the test / asset, to make it easier next time.
* The relevant people weren't notified - Make changes to owners and subscribers, and create the relevant alert rules.
* The result sample was not helpful - Make changes to the test query, to make it easier next time.
* Reoccurring incidents at the source - For incidents that keep happening, the most productive approach is to have a conversation with your data providers, and figure out how to improve response. You can use the `test performance` page, and past incidents on the `incidents` page to communicate stats on the previous incidents.
# Elementary OSS vs. Elementary Cloud
Source: https://docs.elementary-data.com/cloud/cloud-vs-oss
Elementary comes in two offerings: an open-source package you host yourself, and a fully managed cloud platform. Which one fits depends on your goals, scale, and team setup.
**Elementary OSS**
A self-maintained, open-source CLI that integrates with your dbt project and the Elementary dbt package. It covers a single dbt project with dbt test results, table-level lineage, test coverage, and basic alerting, all through the self-hosted Elementary observability report.
**Elementary Cloud**
A fully managed platform for data observability, quality, and governance, with no infrastructure to maintain. It covers many dbt projects end to end, from ingestion to BI and AI consumption, with column-level lineage, ML-powered detection, governance, and enterprise security built in.
All of these signals feed the context engine that powers the Elementary AI agents and MCP. The agents work in the background while you supervise, and help you work at scale: they recommend tests, triage and resolve incidents, enrich governance metadata, and optimize costly models and tests.
Through the MCP, the Elementary context is provided to other agents, like analytics and code agents, so they return more accurate results on trustworthy data.
Elementary is code-first. Tests, monitors, and configuration live in your dbt project and code repository, not locked inside a vendor UI. OSS and Cloud share the same foundation, so there is no vendor lock-in.
If you're trying to decide what would work best, feel free to talk to the team.
Below is a detailed comparison between the OSS and Cloud features:
| Feature | **OSS** | **Cloud** |
| -------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **AI agents & MCP** |
No AI
|
A [team of AI agents](/cloud/ai-agents/overview) you put to work at scale while you supervise (governance, triage, test recommendation, catalog, performance & cost)
[Context engine](/cloud/ai-agents/context-engine) across your stack
[MCP server](/cloud/mcp/overview) for your own AI tools
|
| **Coverage** |
Single dbt project and its warehouse
|
Multiple dbt projects
End to end, from [ingestion](/cloud/integrations/pipeline/fivetran) (Fivetran, SAP, and more) through the warehouse and dbt to [BI](/cloud/features/data-lineage/exposures-lineage) and [AI consumption](/cloud/mcp/overview)
|
| **Data lineage** |
Table-level lineage
|
[Column-level lineage up to BI](/cloud/features/data-lineage/column-level-lineage)
[Interactive alerts](/cloud/features/alerts-and-incidents/alerts-and-incidents-overview) and [alert rules](/cloud/features/alerts-and-incidents/alert-rules)
[Audit logs](/cloud/features/collaboration-and-communication/audit_logs/overview) and [PrivateLink](/cloud/integrations/security-and-connectivity/aws-privatelink-integration)
[SOC 2 Type II and HIPAA](/cloud/general/security-and-privacy), metadata only and no raw data access
|
### Frequently asked questions
Elementary OSS is an open-source CLI and dbt package that you host yourself. It covers a single dbt project with dbt test results, table-level lineage, test coverage, and basic alerting. Elementary Cloud is a fully managed platform that adds ML-powered anomaly detection, column-level lineage up to BI, incident management, governance, enterprise security, and a team of AI agents you put to work at scale, across multiple dbt projects, with no infrastructure to maintain.
Yes. Elementary OSS is open source and free to use. You install the dbt package and CLI in your own environment and run the observability report yourself.
Yes. You can start a free Elementary Cloud trial at [app.elementary-data.com](https://app.elementary-data.com/). If you're not sure whether OSS or Cloud fits your setup, you can also [talk to the team](https://meetings-eu1.hubspot.com/joost-boonzajer-flaes/intro-call-docs).
Yes. Cloud builds on the same open-source foundation and includes the OSS capabilities, then adds automated detection, lineage, governance, security, and AI agents on top.
Yes. Cloud uses the same Elementary dbt package, so moving from OSS to Cloud means connecting your existing dbt project rather than starting over. See the [dbt package quickstart](/cloud/onboarding/quickstart-dbt-package).
No. Elementary is code-first. Your tests, monitors, and configuration live in your dbt project and code repository, not inside a vendor UI, so you keep full ownership and can move between OSS and Cloud freely.
Elementary Cloud is SOC 2 Type II certified and HIPAA compliant. It collects metadata, logs, and aggregated metrics only, with no read access to raw data in your warehouse. It supports [RBAC](/cloud/features/roles-and-permissions), SSO, audit logs, and [PrivateLink](/cloud/integrations/security-and-connectivity/aws-privatelink-integration). See [security and privacy](/cloud/general/security-and-privacy) for details.
Choose OSS for a free, self-hosted package on a single dbt project. Choose Cloud to cover multiple projects end to end with automated detection, governance, enterprise security, and AI agents you supervise at scale, without managing infrastructure. Still unsure? [Talk to the team](https://meetings-eu1.hubspot.com/joost-boonzajer-flaes/intro-call-docs) and we'll help you decide.
### Want to know more?
Get started with a free Elementary Cloud trial.
Book some time with our team.
# Platform features
Source: https://docs.elementary-data.com/cloud/features
### Elementary AI Agent
Elementary's AI agents run continuously on your data platform — triaging incidents, recommending tests, classifying assets, and surfacing performance issues. They're powered by the Context Engine, which maps your entire stack from ingestion through transformations to BI.
What the agents do, how they work, and how to get started.
The knowledge layer that powers all agents — lineage, metadata, test results, and execution history unified.
Investigates root cause, assesses blast radius, and suggests a fix — automatically when an incident opens.
Analyzes your data and usage patterns to recommend the right tests for untested assets.
Identifies undocumented assets, assigns owners, and generates descriptions — synced back to your dbt project.
Surfaces slow models, redundant tests, and expensive queries with specific optimization recommendations.
### Data Tests
ML-powered freshness and volume anomaly detection that runs automatically — no configuration needed to get started.
All dbt, dbt-utils, dbt-expectations, and Elementary package tests appear in Elementary automatically with no extra setup.
Detect schema changes that break downstream consumers — tables, JSON fields, and BI dashboards.
Visual map of coverage gaps across your stack. Add tests from the UI or use the Test Recommendation Agent.
### Data Lineage
Full-stack lineage from ingestion sources through dbt models, Python pipelines, and BI tools — enriched with monitoring results.
Trace individual columns across transformations to pinpoint root cause and understand blast radius.
Discover and trace lineage through Snowflake Semantic Views — metrics, dimensions, and facts connected to your physical tables.
### Alerts & Incidents
Highly configurable alerts routed to any channel — Slack, PagerDuty, email, Jira. Owners tagged automatically.
Define routing logic to send the right alerts to the right people and channels.
Related failures are grouped automatically into a single incident. Manage status and assignees in one place.
Configure owners, subscribers, suppression, and alert properties directly in your dbt project YML.
### Performance & Cost
Track execution times, failure rates, and run trends for dbt models and tests over time.
Get notified when models or tests breach duration thresholds or start failing unexpectedly.
### Data Catalog & Governance
Search and explore datasets — descriptions, columns, compiled code, lineage, and health status in one place.
Mark and prioritize your most important tables for tighter monitoring and faster response.
Auto-generate column and table descriptions using AI, reviewed and synced back to your dbt project.
Set up ownership, tagging, and accountability across your data assets.
### Data Health
Live view of issue status and trends across your platform. Share with stakeholders or embed in your reporting.
Track health scores by domain, surface deterioration early, and share progress with your organization.
# Alerts destinations management
Source: https://docs.elementary-data.com/cloud/features/alerts-and-incidents/alert-destinations
The Alert Destinations tab helps you control where alerts are sent for each test, directly from the test configuration page or the test side panel.
This makes it easy to understand how alerts are routed, reduce noise, and manage settings without switching contexts.
## What You Can See
The **Alert Destinations** tab provides a detailed view of your alerting setup for each test, including:
* A list of your tests, along with their **assigned destinations**, highlighting both active configurations and any gaps in your alerting strategy
* The **current destinations** where alerts are being sent, and how many tests are being assigned for each one
* The **routing method** used for each alert: whether it’s defined by an **alert rule**, a **custom configuration**, or set directly in **code**
* The **alert status** for each test—clearly indicates whether alerts are **enabled** or **disabled**
You’ll have full visibility into how alerts for a specific test are managed.
## What You Can Do
* **Enable or disable alerts** for individual tests or in bulk
* **Override alert destinations** in bulk for faster configuration updates. After overriding, the alert type will be changed to `CUSTOM` and any other alerting logic (code/rules) will not be applied.
This feature makes it easier to manage alert delivery and ensure alerts are routed as expected, especially across different teams or test types.
# Alert rules
Source: https://docs.elementary-data.com/cloud/features/alerts-and-incidents/alert-rules
**Alert Rules** help you control where alerts are sent and when they are triggered, so you can stay focused on the incidents that matter most. In Elementary Cloud, each rule combines **filters** (what to alert on) and **destinations** (where to send the alert).
### Destinations: Where Alerts Are Sent
A destination defines where an alert notification will be delivered. You can route alerts to messaging apps (like [Slack](/cloud/integrations/alerts/slack) or [Microsoft Teams](/cloud/integrations/alerts/ms-teams)) or incident management tools (like [PagerDuty](/cloud/integrations/alerts/pagerduty) or [Opsgenie](/cloud/integrations/alerts/opsgenie)). For all [supported tools](/cloud/features/alerts-and-incidents/alerts-and-incidents-overview#supported-alert-integrations), you can also specify the exact channel, team, or escalation path for the alert. Each alert can be routed to one or multiple destinations — within the same tool or across different tools.
### Filters: What Triggers an Alert
Filters define which alerts are routed through a rule. They help you fine-tune alerting, reduce noise, and focus on critical issues.
You can filter by:
* **Assets** — Filter alerts based on model name, owner, tag, or status.
* **Test Types** — Filter by the type of test that triggered the alert, such as volume anomalies, freshness anomalies, model errors, or dbt tests.
* **Downstream Impacted Assets** — Filter alerts based on the downstream impact of an incident.
Trigger alerts only when specific downstream assets are affected. You can filter impacted assets by:
* **Criticality** — Only alert if critical assets are impacted.
* **Tags** — Filter assets by tag.
* **Owners** — Filter assets by owner.
* **Asset Type** — Model, source, seed, snapshot, or exposure.
This helps ensure that alerts are only triggered when truly important assets are impacted, reducing alert fatigue and keeping your team focused.
Downstream impact is currently determined based on table-level lineage.
### Rule order matters
The order of your rules is important. Here’s how it works:
* When an alert is generated, it's evaluated in order, from top to bottom.
* As soon as a rule matches, the alert is routed to that rule’s destination.
* No further rules are checked after the first match.
If you want alerts to be routed based on all applicable rules (not just the first match), we’re testing a Beta feature that enables this. Reach out if you’d like early access!
### Alert rules default configuration
* The channel you choose when connecting your messaging app ([Slack](/cloud/integrations/alerts/slack), [Microsoft Teams](/cloud/integrations/alerts/ms-teams), etc.)
is automatically added as a default alert rule, that sends all the failures to that channel without any filtering.
* By default, warnings do not send alerts.
To modify, deactivate or add more rules, simply navigate to the `Alert Rules` page in the menu.
# Alerts and Incidents Overview
Source: https://docs.elementary-data.com/cloud/features/alerts-and-incidents/alerts-and-incidents-overview
Alerts and incidents in Elementary are designed to shorten your time to response and time to resolution when data issues occur.
* **Alert -** Notification about an event that indicates a data issue.
* **[Incident](/cloud/features/alerts-and-incidents/incidents) -** A data issue that starts with a single event but can include multiple events grouped together. An incident includes a start time, status, severity, assignee, and end time. Incident alerts are sent when the incident is opened and when it is resolved.
Alerts provide information and context for recipients to quickly triage, prioritize and resolve issues.
For collaboration and promoting ownership, alerts include owners and tags.
You can create distribution rules to route alerts to the relevant people and channels, for faster response.
An alert would either open a new incident, or be automatically grouped and added to an ongoing incident.
From the alert itself, you can update the status and assignee of an incident. In the [incidents page](/cloud/features/alerts-and-incidents/incidents),
you will be able to track all open and historical incidents, and get metrics on the quality of your response.
## Alerts & incidents core functionality
* **Alerts customization** - Alerts should include relevant context for quick triage such as **owner**, **tags**, **description**. In Elementary, alerts can be customized to include this information.
* **Alert distribution rules** - Alerts should be sent to relevant recipients. By creating [Alert Rules](/cloud/features/alerts-and-incidents/alert-rules), alerts can be distributed to different channels and systems.
* **Incidents management** - When alerts are distributed to different channels, it can become hard to track what is open. Elementary offers a centralized [Incidents page](/cloud/features/alerts-and-incidents/incidents) to monitor what is open, and manage incident properties: **assignee**, **status** and **severity**.
* **[Incident merging](/cloud/features/alerts-and-incidents/incident-merging)** - Merge related incidents manually, let the AI agent suggest merges by root cause, or configure auto-merge rules to group incidents automatically. This reduces noise and helps your team focus on root causes instead of individual symptoms.
* **Automated resolution** - When all monitors in an incident pass again, Elementary automatically resolves the incident, keeping the state of incidents up to date in real time.
* **Mute test alerts** – Mute your test from the test configuration tab to run tests without triggering alerts, giving you more control over notifications while still monitoring data quality. This is useful when testing new data sets, refining thresholds, or adjusting test logic without unnecessary noise.
## Alert types
Elementary can be configured to send alerts on:
* Model run failures
* Failures and/or warnings of dbt tests (including Elementary dbt package and other packages)
* Failures and/or warnings Elementary Anomaly Detection monitors
* Failures and/or warning of custom SQL tests
* dbt source freshness failures
## Supported alert integrations
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
# Incidents Digest
Source: https://docs.elementary-data.com/cloud/features/alerts-and-incidents/incident-digest
**Incidents Digest** lets you send a scheduled summary of incidents to your team in addition to real-time alerts. Your team receives a consolidated digest on a daily or weekly cadence covering all relevant incidents in that period.
This is useful for reducing noise, providing management-level visibility, or keeping stakeholders informed without overwhelming them with individual alerts.
## How it works
Digest rules are configured under **Alert Rules → Incidents Digest** tab. Each rule defines:
* **When** to send the digest (cadence: daily or weekly, at a specific hour)
* **What** incidents to include (filters by tag, owner, status, model, or category)
* **Where** to send it (one or more destinations such as a Slack channel or email)
At the scheduled time, Elementary collects all incidents that match the rule's filters since the last digest was sent, and delivers them as a single message.
## Creating a digest rule
Navigate to **Alert Rules** in the left sidebar and select the **Incidents Digest** tab. Click **Create digest rule** to open the configuration drawer.
The filters, categories, and destinations are configured the same way as in [Alert Rules](/cloud/features/alerts-and-incidents/alert-rules). The one addition unique to digest rules is **cadence**.
### Cadence
Choose how often the digest should be sent:
* **Daily** — sent once a day at the hour you choose.
* **Weekly** — sent once a week on the day and at the hour you choose.
The time is in UTC.
## Managing digest rules
Once created, digest rules appear as cards in the **Incidents Digest** tab. From each card you can:
* **Edit** the rule to change any configuration.
* **Activate / Deactivate** the rule without deleting it.
* **Delete** the rule.
## Relationship with alert rules
Digest rules are independent of [Alert Rules](/cloud/features/alerts-and-incidents/alert-rules). You can have both real-time alert rules and digest rules active at the same time — they are evaluated separately. A common pattern is to use real-time alert rules for critical on-call channels and digest rules for broader team summaries.
# Incident Management
Source: https://docs.elementary-data.com/cloud/features/alerts-and-incidents/incident-management
The Incidents page is designed to help your team stay on top of open incidents and collaborate on resolving them. It provides a comprehensive overview of all current and past incidents, where users can view the status, prioritize, assign, and resolve incidents.
## Incidents list
Incidents are organized in a grouped table with three sections: **Open**, **Acknowledged**, and **Resolved**. Each group header shows a count of incidents and a breakdown by severity.
The table columns include:
* **Severity** — editable inline via dropdown
* **Incident** — title, with a "Merged" chip for incidents containing multiple monitors
* **First Event** / **Last Event** — when the incident started and when the last failure occurred
* **Monitors** — status chips showing pass/fail/warning counts
* **Assignee** — editable inline, searchable dropdown
* **Status** — editable inline
Columns can be toggled on or off via the column menu. You can sort by severity, first event, last event, monitor count, or assignee.
## Filtering
The incidents page provides several ways to scope your view:
* **Search** — full-text search across incident titles
* **Filters** — severity, assignee, owner, database/schema/table, critical asset, and monitor count
* **Quick filters** — preset combinations for common views:
* **Unassigned** — incidents with no assignee
* **High priority** — unassigned incidents with High or Critical severity
* **Critical Assets** — incidents on assets marked as critical
* **Merged** — incidents with more than one monitor (result of a merge)
* **Days back** — adjust the time window (default: 3 days)
* **Tree selection** — select a folder, asset, or test in the left navigation tree to scope the list
## Incident detail
Clicking an incident row opens a detail drawer on the right side of the screen.
### Header
The top of the drawer shows the incident title, metadata chips (assignee, status, severity), duration, and the number of failing monitors. You can also:
* Edit the title
* Add or edit a summary
* Link or create tickets (Jira, Linear, ServiceNow, etc.)
### Tabs
An interactive lineage graph showing the upstream and downstream assets related to all monitors in the incident. Use the hop controls to expand or collapse the graph.
A list of monitor cards — each showing the test name, asset, status, owners, and timestamps.
When an incident contains multiple monitors, you can select monitors and click **Detach** to [split them into separate incidents](/cloud/features/alerts-and-incidents/incident-merging#splitting-incidents-undo-a-merge).
A chronological feed of all events in the incident's history: status and severity changes, alerts sent (with destination links), monitors added or removed, tickets linked, and more.
## Bulk actions
Select multiple incidents using the checkboxes in the list to perform bulk operations via the floating action menu:
* Change **status**, **severity**, or **assignee** for all selected incidents at once
* **Merge** selected incidents into one — see [Incident merging](/cloud/features/alerts-and-incidents/incident-merging)
## Ticketing
From the incident detail drawer, you can:
* **Create a ticket** in your connected ticketing tool (Jira, Linear, ServiceNow)
* **Attach an existing ticket** by ID or URL
* **Detach a ticket** from the incident
When a ticket is linked, its status is shown in the drawer. If `close_linked_ticket_on_incident_resolve` is enabled, resolving the incident will also close the linked ticket.
## Incident settings
Click the **gear icon** on the incidents page to access incident settings. Currently this includes [auto-merge rules](/cloud/features/alerts-and-incidents/incident-merging#auto-merge-rules) for automatically grouping related incidents.
# Incident Merging
Source: https://docs.elementary-data.com/cloud/features/alerts-and-incidents/incident-merging
When multiple incidents share the same root cause, merging them into a single incident reduces noise and gives your team a unified view of the issue. Elementary supports three ways to merge incidents: manually, through the AI agent, or automatically via merge rules.
## Manual merge
To merge incidents manually:
1. In the incidents list, select two or more open or acknowledged incidents using the checkboxes.
2. Click **Merge** in the floating action menu that appears at the bottom of the screen.
3. On the merge page, choose which incident should be the **primary** — this is the one that survives with all monitors combined.
4. Edit the merged incident's title and summary if needed.
5. Click **Merge Incidents** to confirm.
After merging:
* The primary incident stays open and inherits all monitors from the merged incidents.
* The merged incidents are resolved with reason "Merged" and hidden from the main list.
* A "Merged" chip appears on the primary incident in the list view when it contains multiple monitors.
## Auto-merge rules
Auto-merge rules let Elementary automatically group related incidents as they are created, without manual intervention.
To configure auto-merge rules, click the **gear icon** on the incidents page to open the settings menu.
### Freshness issues
When enabled, freshness incidents affecting the same upstream source are automatically merged into a single incident. This is useful when a source goes stale and causes freshness failures across many downstream tables — instead of creating a separate incident for each table, they are grouped together.
The rule requires at least two freshness failures from the same source within a one-day window before merging them.
## AI agent-assisted merge
The [Triage & Resolution Agent](/cloud/ai-agents/triage-resolution-agent) can investigate multiple open incidents, identify shared root causes, and propose merging them. When the agent suggests a merge:
1. A merge plan appears in the chat with the selected incidents and a proposed primary.
2. Review the plan and click **Confirm** to proceed.
3. You are taken to the merge page with everything pre-filled, where you can make final edits before merging.
This is particularly useful when you have many open incidents and want to quickly identify which ones are related.
## Splitting incidents (undo a merge)
If monitors were incorrectly merged, you can split them back out:
1. Open the merged incident and go to the **Monitors** tab.
2. Select the monitors you want to detach using the checkboxes.
3. Click **Detach** and confirm.
Each detached monitor becomes a new open incident. If all monitors are detached, the original incident is resolved.
## How merged incidents appear
| Where | What you see |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Incidents list** | A "Merged" chip on incidents with multiple monitors. Merged child incidents are hidden — only the primary shows. |
| **Merged child incident** | An info banner at the top: "This incident has been merged into \[primary incident]" with a link to the primary. The incident is read-only. |
| **Monitors tab** | All monitors from merged incidents appear on the primary incident. |
| **Timeline** | Events showing when monitors were added from or removed to other incidents, with links. |
| **Quick filters** | Use the "Merged" quick filter to show only incidents that contain multiple monitors. |
# Incidents
Source: https://docs.elementary-data.com/cloud/features/alerts-and-incidents/incidents
Every failure or warning detected by Elementary automatically opens an incident. Incidents give your team a single place to track what's broken, who's on it, and what's happened since the issue started.
## How incidents are created
When a test or monitor fails, Elementary automatically opens a new incident for it. If [auto-merge rules](/cloud/features/alerts-and-incidents/incident-merging#auto-merge-rules) are enabled, related failures may be grouped into a single incident automatically instead of creating separate ones.
Incidents can also be [merged manually or by the AI agent](/cloud/features/alerts-and-incidents/incident-merging) after they are created.
## Incident properties
Each incident has the following properties:
* **Title** — auto-generated from the failing monitor. Can be edited manually.
* **Status** — `Open` by default. Can be changed to `Acknowledged`, then to `Resolved`. Resolving an incident closes it permanently.
* **Severity** — Low, Normal, High, or Critical. Model errors default to Critical, test failures to High, warnings to Normal.
* **Assignee** — assign to any team member. They will be notified.
* **Summary** — free-text field to capture context or notes about the incident.
* **Tickets** — link to Jira or other ticketing integrations for tracking.
Incident severity is used to prioritize response — it is not the same as dbt test severity.
## How incidents are resolved
An incident is resolved in one of two ways:
* **Automatically** — when all monitors in the incident pass again, Elementary resolves the incident.
* **Manually** — a user sets the status to `Resolved` from the incidents page or from a Slack alert action.
## Incident detail
Clicking an incident opens a detail drawer with three tabs:
* **Lineage** — interactive upstream and downstream lineage graph for all assets in the incident.
* **Monitors** — cards for each monitor in the incident, showing test name, asset, status, and owners. From here you can [split monitors into separate incidents](/cloud/features/alerts-and-incidents/incident-merging#splitting-incidents-undo-a-merge) if needed.
* **Timeline** — chronological history of all events: status changes, alerts sent, monitors added or removed, tickets linked, and more.
The detail drawer also shows the incident's metadata (assignee, severity, status), linked tickets, and a summary field.
## Triage with the AI Agent
From any incident, click **Investigate with AI** to trigger the [Triage & Resolution Agent](/cloud/ai-agents/triage-resolution-agent). It inspects code changes, execution history, and lineage to surface the root cause and propose a fix. The agent can also identify related incidents and [suggest merging them](/cloud/features/alerts-and-incidents/incident-merging#ai-agent-assisted-merge).
# Alert configuration in code
Source: https://docs.elementary-data.com/cloud/features/alerts-and-incidents/owners-and-subscribers
Most alert settings can be managed directly in the Elementary Cloud UI — no code changes needed. Code configuration is useful when you want settings version-controlled or need to apply them in bulk across many assets.
You can enrich and route alerts by adding properties to your models, sources, and tests in `.yml` files.
Elementary prioritizes configuration in the following order:
**For models / sources:**
1. Model config block
2. Model properties
3. Model path configuration under `models` key in `dbt_project.yml`
**For tests:**
1. Test properties
2. Tests path configuration under `tests` key in `dbt_project.yml`
3. Parent model configuration
## Owners
An owner is the person responsible for the model or test. Owners are tagged in alerts and shown throughout the UI.
Owners can also be added or edited directly in the [Catalog](/cloud/features/collaboration-and-communication/catalog).
Elementary enriches alerts with [owners for models or tests](https://docs.getdbt.com/reference/resource-configs/meta#designate-a-model-owner)).
* If you want the owner to be tagged on slack use '@' and the email prefix of the slack user (@jessica.jones to tag [jessica.jones@marvel.com](mailto:jessica.jones@marvel.com)).
* You can configure a single owner or a list of owners (`["@jessica.jones", "@joe.joseph"]`).
```yml model theme={null}
models:
- name: my_model_name
config:
meta:
owner: "@jessica.jones"
```
```yml test theme={null}
data_tests:
- not_null:
config:
meta:
owner: ["@jessica.jones", "@joe.joseph"]
```
```yml test/model config block theme={null}
{{ config(
tags=["Tag1","Tag2"]
meta={
"description": "This is a description",
"owner": "@jessica.jones"
}
) }}
```
```yml dbt_project.yml theme={null}
models/sources:
path:
subfolder:
+meta:
owner: "@jessica.jones"
data_tests:
path:
subfolder:
+meta:
owner: "@jessica.jones"
# table level:
sources:
- name: source_name
database: db
schema: schema
tables:
- name: orders
meta:
owner: "@jessica.jones"
```
## Subscribers
Subscribers are additional users who should be notified on alerts for a model or test, beyond the owner.
If you want additional users besides the owner to be tagged on an alert, add them as subscribers.
* If you want the subscriber to be tagged on slack use '@' and the email prefix of the slack user (@jessica.jones to tag [jessica.jones@marvel.com](mailto:jessica.jones@marvel.com)).
* You can configure a single subscriber or a list (`["@jessica.jones", "@joe.joseph"]`).
```yml model theme={null}
models:
- name: my_model_name
config:
meta:
subscribers: "@jessica.jones"
```
```yml test theme={null}
data_tests:
- not_null:
config:
meta:
subscribers: ["@jessica.jones", "@joe.joseph"]
```
```yml test/model config block theme={null}
{{ config(
meta={
"subscribers": "@jessica.jones"
}
) }}
```
```yml dbt_project.yml theme={null}
models:
path:
subfolder:
+meta:
subscribers: "@jessica.jones"
data_tests:
path:
subfolder:
+meta:
subscribers: "@jessica.jones"
```
## Description
Elementary supports configuring description for tests that are included in alerts.
It's recommended to add an explanation of what does it mean if this test fails, so alert will include this context.
```yml test theme={null}
data_tests:
- not_null:
config:
meta:
description: "This is the test description"
```
```yml test config block theme={null}
{{ config(
tags=["Tag1","Tag2"]
meta={
description: "This is the test description"
}
) }}
```
```yml dbt_project.yml theme={null}
data_tests:
path:
subfolder:
+meta:
description: "This is the test description"
```
## Tags
You can use [tags](https://docs.getdbt.com/reference/resource-configs/tags) to provide context to your alerts.
* You can tag a group or a channel in a slack alert by adding `#channel_name` as a tag.
* Tags are aggregated,so a test alert will include both the test and the parent model tags.
```yml model theme={null}
models:
- name: my_model_name
tags: ["#marketing", "#data_ops"]
```
```yml test theme={null}
data_tests:
- not_null:
tags: ["#marketing", "#data_ops"]
```
```yml test/model config block theme={null}
{{ config(
tags=["#marketing", "#data_ops"]
}
) }}
```
```yml dbt_project.yml theme={null}
models:
path:
subfolder:
tags: ["#marketing", "#data_ops"]
data_tests:
path:
subfolder:
tags: ["#marketing", "#data_ops"]
```
## Alert distribution
### Custom channel
Route alerts for a specific model or test to a dedicated Slack channel. You must always configure a default fallback channel in your integration settings.
```yml model theme={null}
models:
- name: my_model_name
config:
meta:
channel: data_ops
```
```yml test theme={null}
data_tests:
- not_null:
config:
meta:
channel: data_ops
```
```yml dbt_project.yml theme={null}
models:
path:
subfolder:
+meta:
channel: data_ops
data_tests:
path:
subfolder:
+meta:
channel: data_ops
```
### Suppression interval
Prevent repeated alerts for the same ongoing issue. Set a snooze period in hours — Elementary won't send new alerts on the same issue within that window.
```yml model theme={null}
models:
- name: my_model_name
config:
meta:
alert_suppression_interval: 24
```
```yml test theme={null}
data_tests:
- not_null:
config:
meta:
alert_suppression_interval: 12
```
```yml dbt_project.yml theme={null}
models:
path:
subfolder:
+meta:
alert_suppression_interval: 24
data_tests:
path:
subfolder:
+meta:
alert_suppression_interval: 48
```
### Group alerts by table
By default, Elementary sends one alert per failure. You can instead group all failures for a table into a single notification. Grouped alerts include a union of owners, tags, and subscribers but contain less detail per issue.
```yml model theme={null}
models:
- name: my_model_name
config:
meta:
slack_group_alerts_by: table
```
```yml test theme={null}
data_tests:
- not_null:
config:
meta:
slack_group_alerts_by: table
```
```yml dbt_project.yml theme={null}
models:
path:
subfolder:
+meta:
slack_group_alerts_by: table
data_tests:
path:
subfolder:
+meta:
slack_group_alerts_by: table
```
### Alert fields
Currently supported for test alerts only.
Control which fields appear in the alert. All fields are included by default.
Supported fields: `table`, `column`, `description`, `owners`, `tags`, `subscribers`, `result_message`, `test_parameters`, `test_query`, `test_results_sample`
```yml model theme={null}
models:
- name: my_model_name
config:
meta:
alert_fields: ["description", "owners", "tags", "subscribers"]
```
```yml test theme={null}
data_tests:
- not_null:
config:
meta:
alert_fields: ["description", "owners", "tags", "subscribers"]
```
```yml dbt_project.yml theme={null}
models:
path:
subfolder:
+meta:
alert_fields: ["description", "owners", "tags", "subscribers"]
data_tests:
path:
subfolder:
+meta:
alert_fields: ["description", "owners", "tags", "subscribers"]
```
# Cloud Freshness Test
Source: https://docs.elementary-data.com/cloud/features/anomaly-detection/automated-freshness
The purpose of the Freshness monitor is to alert when a data asset hasn't been updated in a period of time that exceeds the update SLA of that table.
Freshness monitors are by default created for all sources in your dbt project. They can be created for additional tables upon request.
The freshness monitoring has 2 operation modes, it's possible to choose the desired one from the Test Overview side panel:
### Anomaly detection based
The default operation mode of the freshness monitors.
It learns the update frequency of your tables and consistently checks if the table is **currently** fresh based on our model's forecast.
By default we use 21 days of training data to understand the intervals between table updates.
The only condition that determines the status of the monitor is the time that has passed since the last update, which is compared to the model's prediction.
The model takes into account seasonality, and supports cases such as tables that update on weekdays and not weekends.
### SLA based
Sometimes you might want to monitor a table based on a fixed SLA, in order to have full control over when the monitor will alert.
This mode is based on a fixed SLA that you define for each table.
The monitor will alert if the table hasn't been updated in the defined SLA period.
## Understand the monitor result
The test result is a timeline of updates.
The right end of the timeline, marked with a black triangle ▽, is the timestamp of the test result, which is near real time (can be considered as "now").
Each update to the table is presented as a line in the timeline.
Hovering on the gaps between updates will show the updates time and time gap.
To understand the test result, focus on the gap between the last update and now (▽):
* Green - The gap between latest update and now is still within the expected range.
* Yellow / Red - The gap between latest update and now is above the expected range, a dotted line will show what was the expected gap limit. The color represents if this is a warning or failure.
Use the `Anomaly settings` and `result feedback` buttons to impact the monitor.
### Anomaly settings
You can choose between 2 detection methods for the Freshness monitor- Automatic and Manual.
* **Automatic** - Elementary uses machine learning models to detect anomalies in the data freshness. This is the default setting. You can change the sensitivity level to *Low*, *Medium*, or *High*.
For each level, you will see a simulation of the change impact on the latest result, and you can use the`Simulate Configuration` button to examine the change impact.
* **Manual** - You can set the SLA breach threshold for the freshness monitor manually. This is useful for assets that are updated regularly at the same time every day, hour or week.
* **Severity** - Should a failure be considered a warning or a failure. Default is warning.
* **Test metadata** - Add metadata such as tags and owner to the test.
# Cloud Volume Test
Source: https://docs.elementary-data.com/cloud/features/anomaly-detection/automated-volume
The volume monitor tracks the **total row count** of a table over time, rather than individual table updates.
This means that Elementary will not consider a single update as anomalous, but rather a continuous anomalous trend occurring over a period of time.
### Understand the monitor result
The test data set is divided into two periods -
1. Training Period - The historical behavior of the table's volume, patterns, and so forth. By default it will include 21 days.
2. Detection Period - This is the period within which we look for anomalies. By default it’s set to the last 48 hours.
Data points and expected range -
* Data points within the training period are dark grey, and data points within the detection period are colored.
* The light grey area around the data points represents the model expected range. Data points outside this range are considered anomalous.
* Hovering over a data point will detail the row count timestamp, row count and expected range.
Use the `Anomaly settings` and `result feedback` buttons to impact the monitor.
### Anonmaly settings
* **Anomaly Direction** - Whether you want the monitor to fail on anomalous drops, spikes, or both. Default is both.
* **Sensitivity** - You can set the monitor's sensitivity levels to *Low*, *Medium*, or *High*. In the future, we plan to allow for more nuanced adjustments to this parameter. You can use the `Simulate Configuration` button for testing how the change will affect the monitor.
* **Detection Period** - The period in which the monitor look for anomalies. Default is the last 2 days.
- **Severity** - Should a failure be considered a warning or a failure. Default is warning.
- **Test metadata** - Add metadata such as tags and owner to the test.
# Cloud Pipeline Task Performance
Source: https://docs.elementary-data.com/cloud/features/anomaly-detection/cloud-pipeline-task-performance
The pipeline task performance monitor tracks the **execution duration** of dbt models, seeds, and snapshots on each run, and alerts when a node takes significantly longer than expected — or exceeds a fixed SLA threshold.
You create this monitor manually from the UI for the dbt models you want to track.
**Coming soon:** automated creation of pipeline task performance monitors on models, seeds, and snapshots.
## Monitoring modes
### Anomaly detection (default)
Elementary uses a z-score seasonal model to learn normal execution duration for each node. The model accounts for time-of-day and day-of-week patterns, so a model that reliably runs longer on Monday mornings will not be flagged during that window.
Elementary builds the baseline from your pipeline run history.
### Static SLA
Set a fixed duration threshold. The monitor fails whenever the observed execution time exceeds that threshold, regardless of historical norms.
Use static SLA when you have a hard operational or contractual limit (e.g., "this model must finish within 10 minutes").
## Understand the monitor result
The result shows the execution duration of the dbt node for the latest run alongside the historical baseline.
* **Anomaly detection** — data points outside the expected range (grey band) are flagged. The expected range is derived from the seasonal model trained on historical durations.
* **Static SLA** — a horizontal line marks the threshold. Any run that crosses it is a failure.
Use the **Anomaly settings** button to adjust mode, sensitivity, or training period after the monitor is created.
## Anomaly settings
| Setting | Description | Default |
| ---------------------- | ------------------------------------------------------------------------------------ | ----------- |
| `mode` | `"anomaly"` or `"threshold"` | `"anomaly"` |
| `sensitivity` | `"low"`, `"medium"`, or `"high"` (anomaly mode only) | `"medium"` |
| `training_period` | Days of history used to train the model (anomaly mode only) | `23` |
| `anomaly_direction` | `"spike"` (duration longer than expected), `"drop"`, or `"both"` (anomaly mode only) | `"spike"` |
| `excluded_time_ranges` | Time ranges excluded from training and detection (anomaly mode only) | `[]` |
| `fixed_threshold` | Max allowed execution duration in seconds (static SLA mode only) | — |
## Alerts
Failures raise alerts in the **`pipeline_performance`** alert category, routed through [alert rules](/features/alerts-and-incidents/alert-rules) to all configured integrations (Slack, PagerDuty, Teams, OpsGenie, email).
By default, alerts are not active. To enable them, go to **Setup → Alert Rules** and add **Pipeline performance** to the relevant rule's alert categories.
## Related
* [Performance Alerts](/cloud/features/performance-monitoring/performance-alerts) — the legacy dbt-test method for performance alerts (Elementary OSS); replaced in Cloud by this monitor
* [Automated monitors overview](/cloud/features/anomaly-detection/automated-monitors) — how automated monitors work across freshness, volume, and performance
# Metrics
Source: https://docs.elementary-data.com/cloud/features/anomaly-detection/metrics
In Elementary, you can monitor any metric you want on the content of your data, view the metric and set up anomaly detection tests on it!
## How does it work?
Elementary uses a type of dbt test to collect metrics on your data and sync them up to Elementary Cloud.
The metrics will be collected each time you run `dbt test`, in a way that is similar to how training data for anomaly tests is collected.
A metric can be collected with or without an anomaly detection test configured on it.
## How to set up a data content metric?
The monitored metrics are set up in the code, in a way similar to dbt tests.
No mandatory configuration, however it is highly recommended to configure a `timestamp_column`.
```yml Models theme={null}
models:
- name: < model name >
data_tests:
- elementary.collect_metrics:
arguments:
timestamp_column: < timestamp column >
time_bucket: # Daily by default
period: < time period >
count: < number of periods >
dimensions: < list of dimensions to group by >
metrics: < list of metrics >
- name: < user defined name for metric >
type: < which metric to calculate >
columns: < which columns to calculate the metric on- for column metrics >
where_expression: < sql expression >
cloud_monitored: < boolean, should Elementary automatically create anomaly tests for the collected metrics? >
```
```yml Models example theme={null}
models:
- name: login_events
data_tests:
- elementary.collect_metrics:
arguments:
timestamp_column: 'loaded_at'
time_bucket:
period: hour
count: 1
dimensions:
- country_id
- platform_id
metrics:
- name: row_count
type: row_count
- name: filtered_row_count
type: row_count
- name: null_count
type: null_count
columns: ["hello", "world"]
where_expression: "country = 'USA'"
cloud_monitored: true
```
Upon running `dbt test`, your data is split into buckets:
* The size of each bucket is configured by the `time bucket` field
* Each row in the table is assigned to a bucket based on the timestamp of the `timestamp_column` selected
* If dimensions were selected, each combination of dimensions will create a separate metric
and then we compute the metric (or metrics) of choice for each bucket.
We save the metrics in the Elementary schema and sync it to Elementary Cloud whenever you sync your environment.
The metric chart will be visible in the Metrics screen, along with all the metrics Elementary has collected.
If a test was created for the metric, it will be visible in Elementary just like any other test.
To include alerts on Metrics Tests in your alert rules, simply edit your alert rule and check the box "Metric anomalies" under "Test Categories".
# Monitor DWH Assets
Source: https://docs.elementary-data.com/cloud/features/anomaly-detection/monitor-dwh-assets
Elementary monitors all assets in your data stack — not just dbt models and sources. You can sync any schema directly from your data warehouse to extend observability to tables and views that exist outside your dbt project: tables created by ETL tools, direct SQL scripts, or other pipelines.
Synced DWH assets appear in the [lineage graph](/cloud/features/data-lineage/lineage) alongside your dbt models and sources, and support the same freshness and volume Cloud tests.
This feature is currently in **beta** and is only supported for **BigQuery**, **Snowflake**, and **Databricks**. Self-serve schema syncing will be available soon. Until then, to sync additional assets to Elementary, please contact support.
## Syncing DWH schemas
You can sync schemas directly from your data warehouse to Elementary. Once synced, these assets will appear in the side tree under the **DWH view**, making them easily accessible alongside your dbt models and sources.
This feature allows you to:
* Monitor assets that are created outside of dbt (e.g., tables created by other tools, ETL processes, or direct SQL scripts)
* Gain visibility into your entire data warehouse inventory
* Apply the same monitoring and testing capabilities to non-dbt assets
## Configuring tests on DWH assets
After syncing your DWH schemas, you can configure freshness and volume tests on these assets directly from the **test configuration menu**. The process is the same as configuring tests on dbt models:
1. Navigate to the test configuration page
2. Select the DWH asset you want to monitor
3. Choose either a **freshness** or **volume** test
4. Configure the test parameters as needed
5. Submit the test configuration
These tests will run alongside your dbt tests and provide the same level of monitoring and alerting capabilities.
## Lineage integration
DWH assets that have dependencies within your dbt projects will appear in the lineage view. Currently, **views** are fully supported in the lineage visualization, and **tables** support is coming soon.
This integration helps you:
* Understand how your dbt models depend on or are used by non-dbt assets
* Visualize the complete data flow across your warehouse
* Identify dependencies and impacts across your entire data ecosystem
# Data Quality Code Review
Source: https://docs.elementary-data.com/cloud/features/ci
Elementary posts a structured data quality comment on every pull request that touches your dbt models, giving reviewers live context on test history, incidents, downstream impact, and merge risk — without leaving the PR.
## What the review includes
* **Change analysis:** flags NULL risks, type mismatches, and join changes that could alter row counts
* **Performance & cost:** SQL anti-pattern detection (SELECT \*, cross joins, missing filters on large tables) with volume context
* **Downstream blast radius:** which models, pipelines, and dashboards depend on what's changing, including column-level impact for renamed or removed columns
* **Tests & incidents:** pass/fail history for each changed model, active data quality incidents, and coverage gaps on new columns
* **Risk summary:** a plain-language assessment of whether it's safe to merge, with prioritized recommendations
The comment is concise when everything looks clean. It expands only when there are actual issues to flag, and updates automatically on every new push.
## How it works
When a PR is opened or updated:
1. A CI job sends the repository name and branch to Elementary's API
2. Elementary fetches the diff, runs static SQL analysis, and queries live data quality context for the changed models
3. A structured Markdown comment is posted on the PR or MR
## Setup
### Prerequisites
* An Elementary Cloud account with a [code repository connected](/cloud/integrations/code-repo/connect-code-repo)
* An [Elementary API key](/cloud/integrations/code-repo/connect-code-repo)
### GitHub Actions
Create `.github/workflows/elementary-review.yml` in your dbt repository:
```yaml theme={null}
name: Elementary Data Quality Review
on:
pull_request:
paths:
- "models/**/*.sql"
- "models/**/*.yml"
- "dbt_project.yml"
jobs:
elementary-review:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
issues: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: elementary-data/elementary-ci@v1
with:
elementary-api-key: ${{ secrets.ELEMENTARY_API_KEY }}
```
Go to **Settings > Secrets and variables > Actions** in your GitHub repository and add:
| Secret | Required | Description |
| -------------------- | -------- | ------------------------------------------------------------------------------------------------------ |
| `elementary_api_key` | Yes | Your Elementary Cloud API key |
| `elementary_env_id` | No | Your Elementary environment ID. Only needed when the repository is connected to multiple environments. |
The review only runs on PRs that touch model files. Other PRs are ignored.
This works for pull requests opened from branches within the same repository. GitHub does not pass repository secrets to `pull_request` workflows triggered by forks or Dependabot.
If your repository is connected to multiple Elementary environments, add `env-id` to specify which one to use:
```yaml theme={null}
- uses: elementary-data/elementary-ci@v1
with:
elementary-api-key: ${{ secrets.ELEMENTARY_API_KEY }}
elementary-env-id: "169e9308-9a70-4200-b810-ad486cb42f3a"
```
The environment ID is the UUID in your Elementary Cloud URL. For example, in `https://app.elementary-data.com/169e9308-9a70-4200-b810-ad486cb42f3a/report/dashboard`, the environment ID is `169e9308-9a70-4200-b810-ad486cb42f3a`.
If you don't specify an `env-id` and the repository is connected to only one environment, it will be selected automatically. If connected to multiple, the review will return an error listing the available environment IDs.
### GitLab CI
```yaml theme={null}
include:
- remote: 'https://raw.githubusercontent.com/elementary-data/elementary-ci/v1/templates/mr-review.yml'
```
Go to **Settings > CI/CD > Variables** and add:
| Variable | Masked | Required | Description |
| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------- |
| `elementary_api_key` | Yes | Yes | Your Elementary Cloud API key |
| `elementary_env_id` | Yes | No | Your Elementary environment ID. Only needed when the repository is connected to multiple environments. |
| `gitlab_api_token` | Yes | No | Project Access Token with `api` scope. Only needed if you cannot enable CI/CD job token access (see below). |
To post the MR comment, the template uses one of two authentication methods:
* **`CI_JOB_TOKEN` (default):** GitLab's built-in job token, available automatically in every pipeline. Requires a project admin to enable **Settings > CI/CD > Token Access > Allow CI/CD job tokens to access this project's API**.
* **`GITLAB_API_TOKEN` (alternative):** If this variable is set, it takes priority over `CI_JOB_TOKEN`. Use a Project Access Token with `api` scope. This works without any admin settings change and is the easier option if you don't have project admin access.
## Troubleshooting
**No comment appears after the job runs**
Make sure `contents: read`, `pull-requests: write`, and `issues: write` are set under `permissions` in the workflow. An explicit `permissions` block sets any unlisted scope to `none`, so omitting a required scope causes the step to fail silently.
**The review says the repository is not connected**
Go to **Settings > Environments** in Elementary Cloud and verify your repository is connected.
**The review says the repository is connected to multiple environments**
Add `elementary-env-id` to your workflow. The error message lists the available environment IDs.
If a model has never been synced through Elementary, the comment will note that no history is available yet. Results populate automatically after the next Elementary sync.
# Logs
Source: https://docs.elementary-data.com/cloud/features/collaboration-and-communication/audit_logs/overview
The **Logs** feature allows workspace admins to track and export both user actions and system-level events across the platform. This includes user configuration changes, data synchronization events, and alert deliveries.
This feature provides visibility into workspace activity and helps support audit readiness, team coordination, operational monitoring, and troubleshooting.
## Log Types
* **[User Activity Logs](/cloud/features/collaboration-and-communication/audit_logs/user-activity-logs)** - Track changes made by users across the system
* **[System Logs](/cloud/features/collaboration-and-communication/audit_logs/system-logs)** - Track system-level events and operations
## How to Access Logs
Once enabled for your account, access the logs:
1. Click on your **account name** in the top-right corner of the UI
2. Open the dropdown menu
3. Select **Logs**
4. Choose **User Activity Logs** or **System Logs** from the log type selector
5. Select the number of days to look back
6. Click **Export** to download the log as a **CSV file**
You can open the CSV in any spreadsheet tool to review and filter the activity as needed.
## Stream Logs to External Services
In addition to exporting logs as CSV files, you can stream logs in real-time to external log management services for centralized monitoring, analysis, and long-term storage.
### Available Integrations
* **[Datadog](/cloud/integrations/log-streaming/datadog)** - Stream logs to Datadog for centralized log management, monitoring, and alerting
* **[Splunk](/cloud/integrations/log-streaming/splunk)** - Stream logs to Splunk via HTTP Event Collector (HEC) for centralized log management and analysis
* **[Google Cloud Storage (GCS)](/cloud/integrations/log-streaming/gcs)** - Stream logs to GCS buckets for long-term storage and integration with BigQuery and other Google Cloud services
Configure log streaming from the Logs page by clicking the **Connect** icon, then select your preferred destination.
### Log Scope
Each integration can be configured to receive a subset of logs:
* **All logs** — account-level events plus every environment.
* **Account-level only** — only events that aren't tied to a specific environment (e.g. user logins, role changes).
* **Specific environment** — only events from the selected environment.
Set the scope when connecting an integration, or change it later by editing the integration. Customers with separate cloud accounts per environment commonly pair one **Account-level only** destination for global audit with one **Specific environment** destination per environment.
# System Logs
Source: https://docs.elementary-data.com/cloud/features/collaboration-and-communication/audit_logs/system-logs
System logs track system-level events and operations, including data synchronization events and alert deliveries that help with monitoring system operations and troubleshooting.
## What's Included
The system logs capture system-level events and operations across your workspace, including:
* **Data synchronization events**, including:
* Beginning and end of synchronization with dbt
* Beginning and end of data warehouse (DWH) synchronization
* Beginning and end of business intelligence (BI) synchronization
* **Alert deliveries** - When alerts are sent to configured destinations
## Fields
The exported CSV file for system logs includes the following fields:
* **Timestamp** - The date and time when the event occurred (in UTC, ISO 8601 format)
* *Example:* `"2024-01-15T14:30:45.123456+00:00"`
* **Event Name** - The specific action that was performed
* *Examples:* `"dbt_data_sync_started"`, `"dbt_data_sync_completed"`, `"dwh_data_sync_started"`, `"dwh_data_sync_completed"`, `"bi_data_sync_started"`, `"bi_data_sync_completed"`, `"alerts_sent"`
* *Example:* `"dbt_data_sync_completed"` or `"alerts_sent"`
* **Success** - Whether the action completed successfully
* *Values:* `"True"` or `"False"` (as strings in CSV)
* *Example:* `"True"` or `"False"`
* **Event Content** - Additional context-specific information about the action (stored as a JSON string)
* The contents vary by action type. For example:
* For sync actions: `{"environment_id": "env_789", "environment_name": "Production"}`
* For alert delivery actions: `{"alert_count": 5, "destination": "slack"}`
* *Example:* `{"environment_id": "env_789", "environment_name": "Production"}`
* **Env ID** - The environment identifier where the action occurred
* *Example:* `"env_7890123456abcdef"`
* **Env Name** - The name of the environment where the action occurred
* *Example:* `"Production"` or `"Staging"`
**Note:** System logs do not include user information fields since they represent automated system operations rather than user-initiated actions.
# User Activity Logs
Source: https://docs.elementary-data.com/cloud/features/collaboration-and-communication/audit_logs/user-activity-logs
User activity logs track changes made by users across the system, including actions such as creating, editing, or deleting tests, monitors, metadata, and other configurations.
## What's Included
The user activity logs capture a wide range of user actions across your workspace, including:
* **User login and logout events**
* **Failed login attempts** (when applicable)
* **Creation, modification, or deletion of configuration**, including:
* Adding, editing, or deleting tests
* Adding, editing, deleting, or archiving alert rules
* Acknowledging, resolving, or assigning incidents
* Adding, editing, or deleting metadata (owners, descriptions, tags, critical assets)
* Adding, editing, or deleting integrations
* Sending team member invites
* Changing user roles
## Fields
The exported CSV file for user activity logs includes the following fields:
* **Timestamp** - The date and time when the event occurred (in UTC, ISO 8601 format)
* *Example:* `"2024-01-15T14:30:45.123456+00:00"`
* **User Email** - The email address of the user who performed the action
* *Example:* `"john.doe@example.com"`
* **User Name** - The display name of the user who performed the action
* *Example:* `"John Doe"`
* **Event Name** - The specific action that was performed
* *Examples:* `"user_login"`, `"create_test"`, `"edit_alert_rule"`, `"assign_incident"`, `"create_alert_rule"`, `"edit_metadata"`, `"delete_test"`, `"invite_team_member"`, `"change_user_role"`
* *Example:* `"user_login"` or `"create_test"`
* **Success** - Whether the action completed successfully
* *Values:* `"True"` or `"False"` (as strings in CSV)
* *Example:* `"True"` or `"False"`
* **Event Content** - Additional context-specific information about the action (stored as a JSON string)
* The contents vary by action type. For example:
* For incident actions: `{"incident_id": "inc_123", "incident_summary": "Test failure detected"}`
* For test actions: `{"test_id": "test_456", "test_name": "assert_no_null_values"}`
* For alert rule actions: `{"alert_routing_rule_id": "rule_789", "alert_routing_rule": {...}}`
* For metadata actions: `{"asset_id": "asset_123", "metadata_type": "owner"}`
* *Example:* `{"incident_id": "inc_123", "incident_summary": "Test failure detected"}`
* **Env ID** - The environment identifier where the action occurred. Empty for account-level actions like login/logout that aren't tied to a specific environment.
* *Example:* `"env_7890123456abcdef"`
* **Env Name** - The name of the environment where the action occurred. Empty for account-level actions like login/logout that aren't tied to a specific environment.
* *Example:* `"Production"` or `"Staging"`
# Data Catalog
Source: https://docs.elementary-data.com/cloud/features/collaboration-and-communication/catalog
The Elementary Cloud catalog is a dynamic workspace for managing data assets. It combines your dbt models, sources, columns, tests, and metrics with metadata, test results, and ownership in one place.
Teams use the catalog to document their assets, assign owners, monitor test and freshness status, and raise issues or questions. It’s designed for collaboration, so everyone, from data engineers to analysts and business users, can find the context they need and keep shared knowledge up to date.
From each dataset, you can navigate directly to its lineage and test results, ensuring seamless exploration of data dependencies and quality insights.
The catalog provides a unified view of the assets in your dbt project:
* Models and sources, with associated metadata
* Test results for each asset
* Source freshness information
* Column-level documentation
* Ownership and tagging
* Upstream and downstream dependencies
You can browse or search the catalog to find a specific asset or get an overview of the current state of your project.
## Key Features
### Easily Discoverable Data Assets
Quickly and easily navigate the catalog using the side tree and its search bar. Easily find tables, views, BI dashboard and more.
You can view the data assets by their location in the DWH or in the dbt project, and group them by tags, owners or path.
### Editing Metadata
You can edit metadata directly in the catalog — model and column descriptions, owners, tags, and critical asset status — individually or in bulk. For example, assign an owner to multiple models at once or apply a shared tag to a group of assets. All edits are synced back to your dbt project as a pull request, so documentation stays version-controlled.
You can also use the [Governance Agent](/cloud/ai-agents/governance-agent) to identify missing documentation, unowned assets, or inconsistent tags and generate suggested changes automatically.
Identify metadata gaps and generate descriptions, tags, and ownership assignments automatically.
#### Supported metadata fields
* **Tags** — categorize and organize assets
* **Owners** — assign responsibility
* **Descriptions** — document purpose and usage
* **Critical assets** — mark high-priority assets
#### Editing metadata for related assets
From the Dependencies tab, you can apply metadata to all upstream and downstream assets of a selected asset in a single action.
Need to display additional metadata fields from your dbt `meta` config? Reach out to our team to customize the catalog with fields that best fit your data workflows.
### Marking Critical Assets
You can mark assets as critical directly in the catalog to highlight their importance. Critical assets are prioritized in monitoring and surfaced in governance views to help teams focus on what matters most. Read more [here](/cloud/features/data-governance/critical_assets).
### Lineage & Dependency Export
View upstream and downstream dependencies for each dataset, and export the full dependency list as a CSV for further analysis or documentation.
### Table usage
See how often each table is queried in production over a rolling 30-day window, compare usage at a glance from the **Metadata** tab, and hover for percentile rank, query volume, and unique users. Usage is also available as context to Elementary’s AI agents. Read more in [Table usage](/cloud/features/table-usage).
### AI Agents for Discovery and Governance
Elementary includes two agents that use the catalog to support your team:
* The [Catalog agent](/cloud/ai-agents/catalog-agent) helps users find relevant models, columns, or metrics using natural language. This is especially helpful for new team members or business users who are less familiar with your dbt structure.
* The [Governance agent](/cloud/ai-agents/governance-agent) identifies missing documentation, unowned assets, or inconsistent tags. It surfaces these issues automatically and can suggest actions to improve coverage.
These agents run continuously and help keep the catalog useful and complete without requiring constant manual work.
# Data Health Dashboard
Source: https://docs.elementary-data.com/cloud/features/collaboration-and-communication/data-health
Once you start sharing data with downstream consumers and stakeholders one of the most important things that you want to create is trust.
Trust that the data that is being used is “healthy”. Imagine being a data analyst using a specific data asset but you constantly run into data quality issues.
You will eventually lose trust.
This is why we created **data health scores** in Elementary. It is a way to share an overview of the health of your data assets.
To measure health we use an industry standard framework of [Data Quality Dimensions](/cloud/features/collaboration-and-communication/data-quality-dimensions#data-quality-dimensions).
These dimensions help assess the reliability of data in various business contexts.
Ensuring high-quality data across these dimensions is critical for accurate analysis, informed decision-making, and operational efficiency.
To learn more, **watch the webinar** [**Measuring Data Health with Elementary**](https://www.elementary-data.com/webinar/measuring-data-health-with-elementary)
### Data Health Dashboard
The Data Health Dashboard is intended for your data consumers and stakeholders, that want to get a summary of what is happening with the data in your organization.
It gives a high-level overview that doesn't require deep technical knowledge or going into specific test results.
the dashboard presents the data health in a simple way, by giving a health score, and using a color code to indicate if this score is healthy.
Filters are available at the top of the page, making it easy to see the data health in different contexts.
The dashboard is based on the 6 [Data Quality Dimensions](/cloud/features/collaboration-and-communication/data-quality-dimensions#data-quality-dimensions):
Ensures that data is up to date and reflects the latest information.
Ensures all required data is available, without missing values.
Ensures that data represents the real-world scenario correctly.
The degree to which data remains uniform across multiple instances.
Ensures that each entity is represented only once and there are no duplicates.
Ensures that data conforms to rules or expectations, such as acceptable ranges or formats.
### How is the data health score calculated?
Each test you run in either dbt or Elementary is mapped to one of these pillars, and given a score.
The scoring method is very simple:
* If the test passes, the score is 100
* If the test is in `warn` status, the score is 50
* If the test is in `fail` status, the score is 0
The results are aggregated to give a health score for each pillar.
The total score is a weighted average of the 6 pillars, where the weight is configurable.
The thresholds for the color coding (green, yellow and red) are also configurable.
### Critical assets alerts score
To focus on what matters most, see the top right of the screen to filter on your [critical assets](/cloud/features/data-governance/critical_assets).
### Can I customize the quality dimension mapping of my tests?
Of course!
Each test you run, whether it's a generic or a custom test, can be mapped to one of the 6 quality dimensions.
The way to do so is to add `quality_dimension` to the test definition in your dbt project:
```yml test theme={null}
data_tests:
- not_null:
config:
meta:
quality_dimension: completeness
```
```yml test/model config block theme={null}
{{ config(
meta={
"quality_dimension": "completeness",
}
) }}
```
## Coming soon
* **Send a daily report** of the data health to your stakeholders
* **Compare the data health** of different domains
* **Set up alerts** for when the data health is below a certain threshold
# Data Observability Dashboard
Source: https://docs.elementary-data.com/cloud/features/collaboration-and-communication/data-observability-dashboard
Managing data systems can be a complex task, especially when there are hundreds (or even thousands) of models being orchestrated separately across multiple DAGs. These models serve different data consumers, including internal stakeholders, clients, and reverse-ETL pipelines.
Our Data Observability Dashboard provides an easy-to-use control panel for data teams to monitor the quality and performance of their data warehouse.
# Data Quality Dimensions
Source: https://docs.elementary-data.com/cloud/features/collaboration-and-communication/data-quality-dimensions
## Measuring data quality
Once you start sharing data with downstream consumers and stakeholders one of the most important things that you want to create is trust.
Trust that the data that is being used is “healthy”. Imagine being a data analyst using a specific data asset but you constantly run into data quality issues.
You will eventually lose trust.
This is why we created **data health scores** in Elementary. It is a way to share an overview of the health of your data assets.
To measure health we use an industry standard framework of [Data Quality Dimensions](/cloud/features/collaboration-and-communication/data-quality-dimensions#data-quality-dimensions).
These dimensions help assess the reliability of data in various business contexts.
Ensuring high-quality data across these dimensions is critical for accurate analysis, informed decision-making, and operational efficiency.
To learn more, **watch the webinar** [**Measuring Data Health with Elementary**](https://www.elementary-data.com/webinar/measuring-data-health-with-elementary)
## Data quality dimensions
The 6 Data Quality Dimensions are:
Ensures that data is up to date and reflects the latest information.
Ensures all required data is available, without missing values.
Ensures that data represents the real-world scenario correctly.
The degree to which data remains uniform across multiple instances.
Ensures that each entity is represented only once and there are no duplicates.
Ensures that data conforms to rules or expectations, such as acceptable ranges or formats.
## Data quality dimensions example
To help understand different aspects of data quality, let's explore these concepts using a familiar example - the IMDb movie database.
IMDb is a comprehensive database of movies, TV shows, cast members, ratings, and more.
Through this example, we'll see how different data quality issues could affect user experience and data reliability.
#### Freshness
* **Definition**: Ensures that data is up to date and reflects the latest information.
* **Example**: Consider The Godfather's IMDb rating. If the rating hasn't been updated since 2000, despite users continuing to submit reviews every year, the displayed rating would be stale. This outdated information could mislead users about the current audience sentiment toward the movie.
#### Completeness
* **Definition**: Ensures all required fields are filled in, without missing values.
* **Example**: Imagine the IMDb record for Pulp Fiction missing key cast members, such as Uma Thurman. This incomplete data would provide users with an inadequate picture of the movie's legendary cast, significantly reducing the dataset's usefulness.
#### Uniqueness
* **Definition**: Ensures that each entity is represented only once in the system.
* **Example**: Consider having two separate records for The Matrix with the same primary key but different details - one showing a release year of 1999, another showing 1998. This duplication creates confusion about the correct information and could cause problems in downstream processes, like reporting or website display.
#### Consistency
* **Definition**: Ensures data remains uniform across multiple datasets and sources.
* **Example**: If IMDb's Top 250 Movies page displays 254 movies due to a backend error, while the Ratings Summary page correctly shows 250 movies, this inconsistency would confuse users and diminish trust in the platform's data.
#### Validity
* **Definition**: Ensures that data conforms to rules or expectations, such as acceptable ranges or formats.
* **Example**: If a movie's runtime is listed as 1500 minutes when the longest movie ever made was 873 minutes, this would be an invalid value. The runtime clearly doesn't conform to expected movie length ranges and would be considered invalid data.
#### Accuracy
* **Definition**: Ensures that data represents the real-world scenario correctly.
* **Example**: If an IMDb record listed Leonardo DiCaprio as the director of Inception instead of Christopher Nolan, this would be inaccurate. While DiCaprio starred in the movie, he didn't direct it - this kind of error misrepresents the real-world facts.
## Implementation in Elementary
In Elementary, all the dbt tests and Elementary monitors are automatically attributed to the relevant data quality dimension.
Based on the results of tests and monitors, a data health score is calculated for each dimension, and a total score for the data set.
The data quality scores are presented in a [data health dashboard](/cloud/features/collaboration-and-communication/data-health#data-health-dashboard), data catalog integrations, and more.
To learn more, **watch the webinar** [**Measuring Data Health with Elementary**](https://www.elementary-data.com/webinar/measuring-data-health-with-elementary)
# Saved Views
Source: https://docs.elementary-data.com/cloud/features/collaboration-and-communication/saved-views
Saved Views in Elementary allow you to create and save filter groups that persist across all screens in your workspace. Instead of manually applying the same filters every time you navigate to different pages, you can save your preferred filter combinations and have them automatically applied wherever you go.
## Why it matters
* **Consistency** — Your filter preferences stick with you across all screens, from the Catalog to Data Health to Lineage views.
* **Efficiency** — No need to recreate complex filter combinations repeatedly. Set them once and they're always available.
* **Personalization** — Each user can create their own saved views tailored to their specific needs and workflows.
* **Focus** — Quickly switch between different perspectives on your data (e.g., "Marketing Assets", "Production Critical", "My Team's Datasets").
## How it works
Saved Views are built from **filter groups** that you configure in the side tree. The side tree appears on various screens throughout Elementary (Catalog, Data Health, Lineage, etc.) and allows you to filter assets by:
* **Tags** — Filter by asset tags (e.g., `#marketing`, `#production`)
* **Owners** — Filter by asset owners (e.g., `@data-team`, `@analytics`)
* **Paths** — Filter by dbt project paths or data warehouse locations
* **Other metadata** — Filter by any asset property or metadata field
Once you've configured your desired filter groups in the side tree, you can save this combination as a Saved View with a descriptive name. The saved view will then be available to you across all screens in your environment.
## Creating a Saved View
1. **Configure your filters** in the side tree on any screen (Catalog, Data Health, Lineage, etc.)
* Select the tree view type (e.g., Tags, Owners, Paths)
* Apply your desired filter groups
2. **Save the view** — Click the save icon or "Save View" option in the side tree
3. **Name your view** — Give it a clear, descriptive name (e.g., "Marketing Production Assets", "Critical Data Health View")
4. **Confirm** — Your saved view is now created and will persist across all screens
## Using Saved Views
Once created, your saved views appear in the side tree dropdown or menu. You can:
* **Select a saved view** — Click on any saved view to apply its filter groups to the current screen
* **Switch between views** — Quickly toggle between different saved views to see different perspectives
* **Edit a saved view** — Modify the filter groups and update the saved view
* **Delete a saved view** — Remove saved views you no longer need
Saved views are **user-specific** and **environment-specific**, meaning:
* Each user has their own set of saved views
* Saved views are scoped to the environment where they were created
* You can have different saved views for different environments (e.g., Production vs. Staging)
# Configuration as Code
Source: https://docs.elementary-data.com/cloud/features/config-as-code
At Elementary, we believe that **code should be the single source of truth.** This is the only way to scale data reliability while keeping everyone aligned: engineers, analysts, and business stakeholders all see the exact same information, directly in their natural workflows.
Elementary has access to your [code repository](https://docs.elementary-data.com/cloud/integrations/code-repo/connect-code-repo), so every configuration lives in version control. Changes are always tracked, reviewed, and deployed through your existing CI/CD processes—no new workflow required.
### One Experience, Two Interfaces
Every action in Elementary is both:
* **Simple, structured, and clear through the UI** – allowing users to create, edit, or bulk-change tests and metadata with higher efficiency and comfort.
* **Backed by code** – so every change is versioned, reviewed, and integrated seamlessly into your pipeline.
This keeps your CI process natural, while adding an extra layer of protection: every change can be peer-reviewed through a PR before it goes live.
### How Branches & PRs Work
When you make changes through the Elementary UI (such as adding tests, editing assets, or managing configurations), Elementary will:
1. Open a **new feature branch** with the prefix `elementary-` in your connected repository.
2. Commit the changes (e.g., new tests, metadata updates, asset modifications).
3. Automatically create a **pull request** for review with the title format: `{Your Name} via Elementary: {Change Description}`.
From there, your team can review, approve, and merge using the exact same process you already follow for any code change. The system supports various types of changes including test management, asset management, and dbt package upgrades.
### Built for All Users
* **Non-technical users** can define rules for business metrics (e.g., *Revenue is never negative*) without writing SQL or relying on engineering help.
* **Technical users** can manage advanced data tests, metadata, and structure at scale, with full transparency and control.
### AI-Powered Configurations
Elementary’s [AI Agents](/cloud/ai-agents/overview) work the same way:
* You can ask them to recommend tests, optimize queries, or update metadata.
* Once you’re satisfied with the iteration, the agent will automatically open a PR on your behalf—integrating seamlessly into your workflow.
### Works with MCP
Our [MCP server](/cloud/mcp/overview) makes working in code even more powerful, bridging between your local dev setup, IDE, and collaborative workflows.
### Repo Integrations
Elementary supports native [repository integrations](https://docs.elementary-data.com/cloud/features/integrations) with GitHub, GitLab, and others, so your workflows stay consistent and secure.
# AI-Powered Description Generation
Source: https://docs.elementary-data.com/cloud/features/data-governance/ai-descriptions
Elementary now offers an AI-powered description generation feature that helps you create clear, informative descriptions for your data assets without manual effort.
### Why AI-Generated Descriptions Matter
* **Improved Discoverability** - Well-described assets are easier to find and understand
* **Consistent Documentation** - Maintain a uniform description style across your data platform
* **Time Savings** - Generate quality descriptions in seconds instead of hours
* **Enhanced Data Context** - Help data consumers quickly understand asset purpose and content
* **Reduced Documentation Gaps** - Easily fill in missing descriptions across multiple assets
### How to Generate AI Descriptions
You can generate AI descriptions for individual assets or in bulk:
1. In the Assets Table, select the assets that need descriptions
2. Click the "Description" button
3. Click "Generate with AI" for all selected assets or click the "Generate with AI" button for a single asset
4. Review the generated descriptions for each asset (please give us a feedback for the AI to improve the quality of the descriptions)
5. Make any necessary adjustments
6. Click "Submit Pull Request" to submit your changes
# Critical Assets
Source: https://docs.elementary-data.com/cloud/features/data-governance/critical_assets
### **What is a Critical Asset?**
A critical asset is any data asset (such as a model, exposure, or report) that plays a crucial role in your company's data ecosystem. Issues affecting these assets can have a significant impact on business operations, dashboards, and decision-making.
Marking an asset as **critical** ensures it receives higher priority in monitoring and alerting, helping you quickly identify and respond to issues that may impact it.
Critical assets work best when ownership and tagging are already in place. See [Governance best practices](/cloud/best-practices/governance-for-observability) for how to set that foundation.
## What Should Be Set as a Critical Asset?
You should mark an asset as **critical** if:
* It directly impacts key **business reports, dashboards, or decision-making tools**.
* It serves as an essential **upstream dependency** for other important data models.
* It is frequently used by **multiple teams or stakeholders**.
* Its failure or inaccuracy could cause **significant business or operational risks**
## Why Should I Define My Critical Assets?
Defining your **critical assets** helps you:
* **Quickly identify and respond to issues**– Get notified when upstream problems may impact your critical assets, ensuring faster resolution and minimal disruption.
* **Prioritize issue resolution**– Focus on addressing incidents that have the greatest impact on business operations, dashboards, and decision-making.
* **Improve data reliability**– Ensure key stakeholders have access to accurate and up-to-date data by monitoring critical assets more effectively.
* **Enhance observability**– Gain better visibility into the health of your most important assets through prioritized monitoring and alerting.
## How to Set a Critical Asset?
You can mark an asset as **critical** directly in the UI:
* **From the Catalog Page** – Navigate to the asset in the catalog and click the **diamond icon** to **"Set as Critical Asset."**
* **From the Lineage View** – Right-click on the node representing the asset and select **"Set as Critical Asset"** from the list.
Once an asset is marked as critical, **alerts will now highlight any issues that may impact this asset or its upstream dependencies, ensuring prioritization.**
## Where Can You See Critical Assets?
Once an asset is marked as **critical**, you will be able to:
* **Identify it in the UI**, where it will be visually highlighted.
* **Receive alerts** when upstream issues may impact the critical asset.
* **Filter incidents** by their impact on critical assets.
* **Track health scores of critical data assets** over time through dashboard monitoring.
By carefully selecting which assets to mark as critical, you can quickly detect and prioritize issues that impact your most important data, reducing disruptions, improving reliability, and keeping key stakeholders informed.
# Column-Level Lineage
Source: https://docs.elementary-data.com/cloud/features/data-lineage/column-level-lineage
The table nodes in Elementary lineage can be expanded to show the columns. When you
select a column, the lineage of that specific column will be highlighted.
Column-level lineage is useful for answering questions such as:
* Which downstream columns are actually impacted by a data quality issue?
* Can we deprecate or rename a column?
* Will changing this column impact a dashboard?
### Filter and highlight columns path
To help navigate graphs with large amount of columns per table, use the `...` menu to the right of the column:
* **Filter**: Will show a graph of only the selected column and its dependencies.
* **Highlight**: Will highlight only the selected column and its dependencies.
### Column-level lineage generation
Elementary parses SQL queries to determine the dependencies between columns.
Note that the lineage is only of the columns that directly contribute data to the column.
For example, for the query:
```sql theme={null}
create or replace table db.schema.users as
select
user_name,
count(distinct login_time) as total_logins
from db.schema.login_events
where user_type != 'test_user'
```
The direct dependency of `total_logins` is `login_events.login_time`.
The column `login_events.user_type` filter the data of `total_logins`, but it is an indirect dependency and will not show in lineage.
If you want a different approach in your Elementary Cloud instance - contact us.
# Lineage to Downstream Dashboards and Tools
Source: https://docs.elementary-data.com/cloud/features/data-lineage/exposures-lineage
Some of your data is used downstream in dashboards, applications, data science pipelines, reverse ETLs, etc.
These downstream data consumers are called *exposures*.
The Elementary lineage graph presents downstream exposures of two origins:
1. Elementary automated BI integrations
2. Exposures configured in your dbt project. Read about [how to configure exposures](https://docs.getdbt.com/docs/build/exposures) in code.
```yaml theme={null}
exposures:
- name: returns_dashboard
label: Returned Orders # Pretty name, will be used if provided
type: dashboard
url: https://your.bi.tool/dashboards/2
description: >
Returned orders over time
platform: Tableau # Elementary field
path: Tableau # For the UI folders tree, elementary field
depends_on:
- ref('returned_orders')
owner:
name: Callum McData
email: data@jaffleshop.com
tags: ["finance"]
meta:
referenced_columns: # Used for exposure validation test
- column_name: "order_id"
data_type: "numeric"
```
### Automated lineage to the BI
Elementary will automatically and continuously extend the column-level lineage to the dashboard level of your data visualization tool.
If your BI tool has its own modeling layers (e.g. data sources in Tableu, Explores in Looker), these will also be included in the graph and column lineage.
### Supported BI tools:
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
>
Click for details
}
>
Click for details
}
>
Click for details
# End-to-End Data Lineage
Source: https://docs.elementary-data.com/cloud/features/data-lineage/lineage
Lineage is the foundation of the [Context Engine](/cloud/ai-agents/context-engine) — it's how Elementary maps your entire data stack, from ingestion sources through transformations to BI and downstream consumers. Every asset, test result, and health signal is connected through this graph, which is what allows the AI Agent and MCP to reason across your stack.
Elementary builds column-level lineage automatically from your data warehouse metadata and integrations with dbt, BI tools, Python pipelines, and other sources. The graph is updated frequently so it reflects your current state.
## What's included in the lineage graph
Elementary connects assets across your full stack:
* **Ingestion sources** — tables from your data warehouse, including those loaded by tools like Fivetran, and [non-dbt assets synced directly from your warehouse](/cloud/features/anomaly-detection/monitor-dwh-assets)
* **dbt models and sources** — resolved from your dbt project metadata
* **Python SDK assets** — tables and datasets produced by Python pipelines instrumented with the [Elementary Python SDK](/cloud/python-sdk/overview)
* **BI and downstream tools** — dashboards and exposures from Looker, Tableau, and other connected BI tools
* **Semantic views** — [Snowflake Semantic Views](/cloud/integrations/semantic-layer/snowflake) connected to their underlying physical tables
## What you can do with it
* **Trace root causes** — follow failures upstream to find where an issue originated
* **Assess impact** — see which downstream assets and dashboards are affected by a failure or change
* **Prioritize triage** — surface issues affecting critical or high-traffic assets first
* **Detect unused datasets** — identify assets with no downstream consumers to reduce cost
* **Plan changes safely** — understand what breaks before you make a change
## Node info and test results
Select any node in the graph to see:
* **Test results** — latest test results for that asset
* **Node info** — description, owner, tags, and job info if collected
## Job info in lineage
You can [configure Elementary to collect job information](/cloud/guides/collect-job-data) to display in the lineage node info panel. Job names can also be used to filter the graph.
# Cloud Tests
Source: https://docs.elementary-data.com/cloud/features/data-tests/cloud-tests-overview
Cloud tests run in Elementary Cloud, outside your dbt pipeline. They don't require configuration in your dbt project — Elementary activates them automatically once your data warehouse is connected.
There are three Cloud tests:
| Test | What it detects |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [Cloud freshness test](/cloud/features/anomaly-detection/automated-freshness) | Tables that haven't been updated within their expected window |
| [Cloud volume test](/cloud/features/anomaly-detection/automated-volume) | Unexpected drops or spikes in row count |
| [Data contract test](/cloud/features/data-tests/data-contract-test) | Schema mismatches against a defined contract — column names, types, nullability |
Cloud tests use warehouse metadata only — not the data itself — so they add no compute cost and run continuously without pipeline dependencies.
## Adding cloud tests
Cloud freshness and volume tests are activated automatically on all your sources. You can add additional tests from the UI or via the [Test Recommendation Agent](/cloud/ai-agents/test-recommendation-agent):
1. Navigate to the **Test Configuration** page, or select assets in the Catalog
2. Click **Add Test** and choose a **Table Test**
3. Select one or more tables to test
4. Filter on **Elementary Cloud** and choose your test — freshness, volume, or data contract
5. Configure the test parameters and add metadata if needed
6. Submit — no PR required
## Editing cloud tests
Open any test result and use **Anomaly settings** to tune the detection. For freshness and volume tests, you can:
* Adjust **sensitivity** (Low / Medium / High) to reduce noise or increase detection precision
* Switch between **automatic** (ML-based) and **manual** (SLA threshold) modes for freshness
* Change the **detection period** — the window in which the monitor looks for anomalies
* Set **anomaly direction** — fail on drops, spikes, or both
### Simulate configuration
After changing any setting, use **Simulate Configuration** to preview how the change would have affected recent results before saving.
### Exclude time ranges
Training periods sometimes include one-time anomalies — a manual script run, a known incident — that skew the model and cause `NO_DATA` results. You can exclude specific time ranges so the model learns from clean data only.
1. Click **Exclude** in the top-right of the chart, or go to the **Test Configurations** tab
2. Mark the section of the chart you want to exclude
3. Confirm — the excluded range appears under **Excluded time ranges** and can be edited or removed
After excluding, use **Simulate Configuration** to verify the impact.
## Removing cloud tests
* From the **Test Configuration** page: select one or more tests and use the delete option at the bottom
* From a test result: click `...` → **Delete test**
## Supported settings
#### All cloud tests
* **Severity** - Should a failure be considered a warning or a failure. Default is warning.
* **Test metadata** - Add metadata such as tags and owner to the test.
#### Volume test
* **Anomaly Direction** - Whether you want the monitor to fail on anomalous drops, spikes, or both. Default is both.
* **Sensitivity** - You can set the monitor's sensitivity levels to *Low*, *Medium*, or *High*. In the future, we plan to allow for more nuanced adjustments to this parameter. You can use the `Simulate Configuration` button for testing how the change will affect the monitor.
* **Detection Period** - The period in which the monitor look for anomalies. Default is the last 2 days.
#### Freshness test
You can choose between 2 detection methods for the Freshness monitor- Automatic and Manual.
* **Automatic** - Elementary uses machine learning models to detect anomalies in the data freshness. This is the default setting. You can change the sensitivity level to *Low*, *Medium*, or *High*.
For each level, you will see a simulation of the change impact on the latest result, and you can use the`Simulate Configuration` button to examine the change impact.
* **Manual** - You can set the SLA breach threshold for the freshness monitor manually. This is useful for assets that are updated regularly at the same time every day, hour or week.
## Result feedback
Use the **Result feedback** button on any test result to mark it as a true or false positive. Feedback improves the accuracy of the ML models over time.
For false positives, select a reason:
* **Insignificant change** — the anomaly isn't impactful enough to act on; consider relaxing sensitivity
* **Expected outlier** — the value should be within the expected range; triggers model retraining
* **Business anomaly** — expected due to an intentional change or event; excludes the result from training
* **Not an interesting table** — stop monitoring this table; deletes the monitor
* **Other** — add a comment to describe the case
## Metrics
Collect and monitor custom metrics using the [`elementary.collect_metrics`](/cloud/features/anomaly-detection/metrics) dbt test. Metrics are synced to Elementary Cloud and can have anomaly detection applied automatically.
For dbt tests and custom SQL tests, see the [dbt Package and Tests](/data-tests/introduction) tab.
# Custom SQL Tests
Source: https://docs.elementary-data.com/cloud/features/data-tests/custom-sql-tests
Custom SQL queries enable you to create tailored tests that align with your specific business needs.
These tests can be executed against any of the tables in your connected data warehouse.
### When to use custom SQL tests?
A custom SQL test is easier to write than a new generic dbt test, but it can't be leveraged across different data sets.
On the other hand, writing custom SQL tests enables testing complex custom calculation logic, relationships between many tables and more.
This is why most Elementary users write custom SQL tests when the behavior to be tested is complex, specific and doesn’t exist in any out of the box test.
A common use case is for data analysts to add validations as custom SQL tests.
As non-technical users are often not familiar with dbt,
Elementary has an interface for adding custom SQL test that converts it to a pull request adding a singular dbt test.
### Adding a custom SQL test
1. In `Test Configuration` choose `New test` → `custom query test`.
2. Add your query, it can be a regular SQL query on any tables in your environment. The query should only return results if something is wrong, meaning the test will pass on no results and fail on any results. Please be sure to use the full name of the table, including the db and schema. The query will then be validated and formatted, table names will be replaced with dbt model references. This can take a few seconds.
3. Add your test configuration:
* Test name (should be a valid file name- this will use as the name of your Singular test).
* Description (recommended).
* Location - this will be the directory in your dbt project where the test will be stored.
* Severity - Failure or Warning.
* Tested table (optional) - Adding the table the test is checking will link this test to that table, showing it in the table’s lineage, catalog page and more.
* Tags and Owners (Optional)
* Some users use dbt tags to help with scheduling. Creating a “daily”, “hourly”, and having scheduled jobs for those tags can help determine the test’s schedule at the time of creation.
* It is recommended to add a tag that will be used to later route the alert on failure to the right recipient.
4. Review & Submit. In this stage you’ll be able to see the translated test query and configuration.
5. Clicking “Submit” will go on to create a pull request.
6. After the pull requests is merged to production, tests should run on a `dbt test` job.
### Custom SQL test results
The results of all custom SQL tests can be found under a `tests` folder in the test results sidebar.
Additionally, if you configured a `tested table`, `tag` or `owner`, the test result will be visible under the relevant path.
### Alerts on custom SQL tests
It's recommended to use tags and owners create an [alert rule](/cloud/features/alerts-and-incidents/alert-rules) that will route these tests to the relevant recipient.
### Scheduling custom SQL tests
It's a common practice not to want these tests to run as part of your main job, or at the same frequency.
We recommend to use a tag for all these tests, use dbt tags as selectors to exclude from the main job, and run a dedicated test job that includes these tests only in the required frequency.
# Data contract test
Source: https://docs.elementary-data.com/cloud/features/data-tests/data-contract-test
Data contract tests validate that a table's schema matches an expected baseline — column names, data types, and nullability. They run as [cloud tests](/cloud/features/data-tests/cloud-tests-overview) in Elementary Cloud, using warehouse metadata only (no queries on table data).
Use them to enforce schema stability on production tables and sources, catch breaking upstream changes early, and control how strictly new columns are treated.
## How it works
On each run, Elementary compares the table's current schema in your data warehouse to the test's **baseline**. A baseline is a list of columns, each with a name, data type, and nullable flag.
The test detects four types of schema changes:
| Change | Description |
| ------------------- | ---------------------------------------------------------- |
| Column added | A column exists in the warehouse but not in the baseline |
| Column removed | A column in the baseline no longer exists in the warehouse |
| Type changed | A column's data type differs from the baseline |
| Nullability changed | A column's nullable flag differs from the baseline |
When changes are found, each one is evaluated against the test's **enforcement level**. The test fails only if at least one change meets that level. Results show every detected change, including downstream dependency impact per column (tables, BI assets, and whether critical downstream assets are affected).
Data contract tests use warehouse metadata collected during Elementary sync. They do not scan table rows.
## Cloud test behavior
Like other cloud tests (automated freshness and volume monitors, anomaly detection monitors), data contract tests:
* Are created and configured in the Elementary UI — no pull request or dbt run is required to add them
* Run on Elementary's schedule as part of cloud test execution
* Appear alongside dbt tests and other monitors in test results, incidents, and coverage views
Unlike most other cloud tests, data contract tests **can run on views, ephemeral models, and materialized views**, not only tables and sources. Other cloud tests require table or source materialization because they rely on table statistics; data contract tests only read column metadata.
The selected table must exist in your warehouse metadata. If Elementary cannot find the table in synced metadata, the test cannot be created.
## Baseline modes
When you create a test, choose how the baseline is defined:
### Complete
Elementary snapshots the table's full current schema from warehouse metadata and uses it as the baseline. This is the default and works well when the current schema is the contract you want to lock in.
### Manual
You define the expected columns yourself — name, data type, and nullable flag for each. Use this when the contract is known upfront and may differ from the current warehouse schema, or when you only care about a subset of columns.
In manual mode, at least one column with both a name and data type is required before you can submit the test.
## Enforcement levels
Enforcement level controls which schema changes cause the test to fail:
| Level | Fails when |
| ----------------------------------- | -------------------------------------------------------------------------------- |
| **All changes** | Any deviation from the baseline, including new columns |
| **Breaking changes only** | Column removals, type changes, or nullability changes. New columns are tolerated |
| **Downstream impact only** | A changed column has downstream column-level lineage dependencies |
| **Critical downstream impact only** | A changed column has downstream dependencies on assets marked as critical |
For downstream-based levels, a change with no downstream dependencies does not fail the test, even if the schema changed.
## Edit configuration
After creation, open the test from test results or an incident and go to the **Configuration** tab.
You can update:
* **Baseline columns** — edit expected columns manually, or click **Capture current schema** to replace the baseline with the table's current warehouse schema
* **Enforcement level** — change how strictly changes are treated
Save changes in the UI. Elementary re-runs the test with the updated configuration.
Use **Capture current schema** after an intentional schema change to accept the new schema as the new contract without recreating the test.
## Test results
When a run detects schema changes, the execution details list each change with:
* Change type (added, removed, type changed, nullability changed)
* Current and previous type or nullability where relevant
* Downstream impact: number of downstream table and BI assets, downstream columns, and whether any critical downstream assets are affected
A passing run with tolerated changes (for example, new columns under **Breaking changes only**) may still list detected changes in the results while reporting success.
## Alerts
Data contract test failures map to the **Data contract** alert category in [alert rules](/cloud/features/alerts-and-incidents/alert-rules). Include this category in a rule's test-type filters to receive alerts on data contract failures.
By default, the catch-all alert rule may not include every cloud test category. Review your alert rules if you want data contract failures to notify your team.
## Comparison with dbt schema tests
Elementary also offers dbt-based schema validation tests such as [`elementary.schema_changes`](/data-tests/schema-tests/schema-changes) and [`schema_changes_from_baseline`](/data-tests/schema-tests/schema-changes-from-baseline). These run in your dbt pipeline and are managed in code.
| | Data contract (cloud test) | dbt schema tests |
| ---------------------------- | ---------------------------- | ----------------------------------- |
| Where it runs | Elementary Cloud | dbt pipeline |
| Configuration | UI (no PR required) | YAML in your dbt project |
| Works on views | Yes | Depends on test and materialization |
| Baseline updates | Capture current schema in UI | Code / macros |
| Downstream impact in results | Yes | No |
Choose data contract tests when you want UI-managed schema enforcement without changing dbt code. Choose dbt schema tests when schema validation should live in version-controlled project code and run with every dbt job.
# dbt, Packages and Elementary Tests
Source: https://docs.elementary-data.com/cloud/features/data-tests/dbt-tests
Elementary Cloud is built on top of the [Elementary dbt package](/data-tests/introduction). Any tests you've already configured in your dbt project appear automatically in Elementary — results, history, and alerts included — with no additional setup.
dbt tests are very powerful. The ease of use, simplicity, and usefulness in the dev process is unmatched.
When you adopt any observability tool, you will still use dbt tests. This is why in Elementary, dbt tests are first class citizens.
There are several benefits to this approach:
* **Single interface for all observability** - Prevent the distribution of monitoring between different tools. All configuration is in code, all the results are in one interface.
* **Avoid duplicate work and vendor lock in** - The tests you implemented already are effective in Elementary, as well as additional configuration. The future tests you add will remain in your code if you decide to offboard.
* **Control of schedule and cost** - You have control of configuration and scheduling, tests can be executed when data is actually loaded and validation is needed.
* **Prevent bad data from propagating** - As tests are in pipeline, you can leverage `dbt build` and fail the pipeline on critical test failures.
* **Rich ecosystem** - The community of dbt users develops and supports various testing use cases.
### dbt Test Hub
To help you find the test that is right for your use case, we created the [dbt Test Hub](https://www.elementary-data.com/dbt-test-hub).
It's a searchable catalog of all the tests supported in Elementary, with their descriptions and example use cases.
The tests are also segmented to use cases, so you can easily find the different options for addressing your detection use case.
### Supported dbt tests and packages
Elementary collects and monitors the results of all dbt tests.
The following packages are supported in the tests configuration wizard:
* [dbt expectations](https://github.com/calogica/dbt-expectations) - A dbt package inspired by the Great Expectations package for Python. The intent is to allow dbt users to deploy GE-like tests in their data warehouse directly from dbt.
* [dbt utils](https://github.com/dbt-labs/dbt-utils) - A package by dbt labs that offers useful generic tests.
Note that you need to import these packages to your dbt project to use them.
### Elementary dbt package tests
The Elementary dbt package also provides tests for detection of data quality issues.
Elementary data tests are configured and executed like native tests in your dbt project.
* Volume
* Freshness
* Event freshness
* Column anomalies
* Dimensions
* Schema changes
* Baseline schema
* JSON schema
* Exposure schema
* Python tests
# Schema Validation Tests
Source: https://docs.elementary-data.com/cloud/features/data-tests/schema-validation-test
The Elementary dbt package includes the following schema validation tests:
Fails on changes in schema: deleted or added columns, or change of data type
of a column.
Fails if the table schema is different in columns names or column types than a
configured baseline (can be generated with a macro).
Monitors a JSON type column and fails if there are JSON events that don't
match a configured JSON schema (can be generated with a macro).
Monitors changes in your models' columns that break schema for downstream
exposures, such as BI dashboards.
# Managing Test Coverage
Source: https://docs.elementary-data.com/cloud/features/data-tests/test-coverage-screen
There are two ways to manage test coverage in Elementary:
* **Test Coverage screen** — a visual overview of coverage across all your assets and dimensions, with the ability to add missing tests directly from the UI
* **[Test Recommendation Agent](/cloud/ai-agents/test-recommendation-agent)** — analyzes your pipeline, lineage, and existing tests to suggest what’s missing and where, then adds tests on your behalf
Use the screen to get a high-level picture and act on gaps in bulk. Use the agent when you want coverage recommendations driven by context — patterns, lineage, and asset criticality.
The **Test Coverage screen** in the Elementary Cloud [catalog](/cloud/features/collaboration-and-communication/catalog) gives you a full picture of your data quality coverage across assets and dimensions.
## What It Shows
Each asset is evaluated across **seven data quality dimensions**:
* **Freshness**
* **Completeness**
* **Uniqueness**
* **Validity**
* **Accuracy**
* **Consistency**
* **Other**
For each asset, you’ll see:
* **Which dimensions are covered** by existing tests
* **Where coverage is missing**
* [A **coverage score** between **0–100%**](/cloud/features/data-tests/test-coverage-screen#how-coverage-calculation-works).
* **Links to test results**
## What You Can Do from This Screen
From the Test Coverage screen, you can:
* Filters by asset properties, test name, critical assets, and coverage ranges.
* **Select multiple assets and seamlessly add your missing tests with just a few clicks**
* **Jump directly to any asset in the catalog to review its details**
* Gain insight into gaps by grouping assets across dimensions like domain, pipeline, tag, or owner—making weak spots easy to identify. (Coming soon)
* Export the results to CSV
## How Coverage Calculation Works
The **test coverage score** is calculated based on **7 dimensions of data quality**.
The first **6 core dimensions** each contribute **15%** to the total score. The **7th dimension**, labeled **“Other,”** accounts for the remaining **10%** and is primarily used for **business logic tests** that don’t align directly with any specific core dimension.
### Upcoming Features
* **Customizable Weights:**
You’ll soon be able to tailor the weighting of each dimension to align with your organization’s unique priorities and testing standards.
* **Custom Coverage Rules:**
Define your own coverage criteria to better identify tables that do not meet your internal standards. This will make it easier to spot gaps and maintain consistent data quality practices.
## Test Recommendation AI Agent
Alongside the Test Coverage screen, you can also use our [**Test Recommendation Agent**](https://docs.elementary-data.com/cloud/ai-agents/test-recommendation-agent) to help you improve test coverage.
Together, the coverage screen and the agent give you visibility and guidance to focus your efforts where they’ll have the most impact.
# GPG Commit Signing
Source: https://docs.elementary-data.com/cloud/features/gpg-signed-commits
## Overview
GPG commit signing provides cryptographic verification that commits were made by you, ensuring the integrity and authenticity of your code changes. When enabled, Elementary will automatically sign commits with your GPG key, providing an additional layer of security and trust.
**Enhanced Security:** GPG-signed commits provide cryptographic proof of
authorship and commit integrity.
## How it works
Elementary's GPG commit signing feature allows you to:
1. **Generate GPG keys** directly in the Elementary interface
2. **Automatically sign commits** when creating pull requests or making changes
3. **Manage your keys** with options to view, revoke, or generate new keys
4. **Fallback gracefully** to unsigned commits if GPG signing fails
### Key benefits
* **Cryptographic verification** of commit authorship
* **Tamper detection** - any modification to signed commits will be detected
* **Enhanced security** for your code repository
* **Seamless integration** with your existing workflow
## Setting up GPG commit signing
1. Navigate to **User Settings** > **GPG Keys** in your Elementary account
2. Click **"Add GPG Key"** to generate a new key
3. Configure your key settings
4. Click **"Generate Key"** to create your GPG key
5. Copy the public key
6. Go to **GitHub** → **Settings** → **SSH and GPG keys**
7. Click **"New GPG key"** and paste the public key
You can only have one active GPG key at a time. To use a different key, you'll
need to revoke the current one first.
### View your GPG key details
Once generated, you can **view the public key** - useful for adding to GitHub or other Git hosting services.
### Revoke a GPG key
If you need to revoke your current GPG key:
1. Go to **User Settings** > **GPG Keys**
2. Find your active key in the table
3. Click the **trash icon** to revoke the key
4. Confirm the revocation
**Permanent action:** Revoking a GPG key is permanent and cannot be undone.
You'll need to generate a new key to continue using GPG signing.
## How commits are signed
When GPG commit signing is enabled, Elementary will:
1. Automatically detect if you have an active GPG key
2. Use your key to sign commits when creating pull requests or making changes
3. Fall back gracefully to unsigned commits if signing fails
### Commit signature verification
Signed commits will show a "Verified" badge in GitHub and other Git hosting services, indicating that:
* The commit was signed with a valid GPG key
* The signature matches the commit content
* The key belongs to the commit author
## Best practices
### Key management
* Use strong key lengths (4096 bits recommended)
* Set expiration dates (90 days recommended)
* Revoke compromised keys immediately
### Troubleshooting
If commits aren't being signed:
1. Check you have an active GPG key in your Elementary settings
2. Verify the key hasn't expired
3. Make sure you added the public key to GitHub
# Elementary integrations
Source: https://docs.elementary-data.com/cloud/features/integrations
### Data warehouses
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
### Transformation and orchestration
}
/>
}
/>
}
/>
}
/>
}
/>
### Data visualization
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
>
Click for details
}
>
Click for details
}
>
Click for details
### Reverse ETL
}
>
Click for details
}
>
Click for details
### Code repositories
}
/>
}
/>
}
/>
}
/>
### Alerts & incidents
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
### Log Streaming
}
/>
}
/>
}
/>
### Governance
}
>
Click for details
### Iceberg catalog
}
>
Click for details
# Multiple Environments
Source: https://docs.elementary-data.com/cloud/features/multi-env
Elementary supports two approaches for managing multiple environments:
## Separate Environments
An environment in Elementary is a combination of a dbt project and a target.
For example: If you have a single dbt project with three targets, prod, staging and dev, you can create 3 environments in Elementary and monitor these environments separately.
If you have several dbt projects and even different data warehouses, Elementary enables monitoring the data quality of all these environments in a single interface. Simply click on "Create environment" and set up all integration connections. Each environment is managed independently with its own monitoring, alerts, and configuration.
## Multi-Environment Support
Elementary now supports connecting multiple dbt projects to a single Elementary environment. This gives teams the flexibility to organize their dbt code by layers, domains, or separate repositories while still getting one unified view of data health, lineage, and reliability.
### How it Works
You can connect several dbt projects, each with its own Elementary schema and code repository, into the same Elementary environment. Once connected, Elementary combines lineage and context across all projects so you have a complete picture of how data flows through your entire stack.
### Key capabilities
• **Unified lineage across projects**
Elementary identifies assets by their full relation name, so lineage can span multiple dbt projects with complete accuracy.
• **No changes to your dbt structure**
Each dbt project keeps its own models, sources, repository, and Elementary schema. Elementary handles the unification at the environment level.
• **Automatic linking between sources and models**
If the same relation appears as a model in one project and a source in another, Elementary displays two nodes in lineage with a clear link between them. This makes dependencies explicit and helps you trace issues further upstream.
### When to Use Multi-Environment Support
Multi-environment support is designed for teams that organize dbt work in multiple projects, such as:
• Layered architectures (bronze, silver, gold)
• Domain-based modeling (finance, marketing, operations and others)
• Multiple dbt projects built on the same raw data
• Pipelines stored in separate repositories
This feature lets you maintain the separation between projects while still using Elementary as one consistent reliability control plane.
### Setup and Configuration
Each dbt project added to the environment must use a different Elementary schema, the Elementary dbt package must be installed in each project.
Multi-environment support is done in the environment settings, directly in the UI.
After configuring the DWH connection, click "Continue" and you will get to the environment content section, where you can add additional dbt projects to the same environment.
# Performance Alerts
Source: https://docs.elementary-data.com/cloud/features/performance-monitoring/performance-alerts
**In Elementary Cloud, pipeline task performance is monitored natively — no dbt tests required.** Elementary automatically creates a [Cloud pipeline task performance](/cloud/features/anomaly-detection/cloud-pipeline-task-performance) monitor for every model, seed, and snapshot, with anomaly detection or a static SLA. Use that monitor instead of the dbt-test setup below.
The dbt-test approach on this page still works, but it is the legacy method (and the only option in Elementary OSS). It will not appear as a native performance monitor in Cloud and produces alerts with less context.
Monitoring the performance of your dbt models is crucial for maintaining an efficient data pipeline. Elementary provides capabilities to set up alerts for long-running queries, helping you identify performance bottlenecks and optimize your data pipeline.
There are two main approaches to creating alerts for long-running model queries:
1. **Static Threshold Alerts**: Define specific time thresholds that, when exceeded, trigger an alert
2. **Anomaly Detection Alerts**: Use Elementary's anomaly detection to identify unusual increases in query execution time
## Static Threshold Alerts
You can define tests that fail when model execution times exceed predefined thresholds. This approach is straightforward and ideal when you have clear performance requirements.
### Implementation Steps
1. Create a singular test SQL file in your dbt project (e.g., `tests/test_models_run_under_30m.sql`):
```sql theme={null}
{{ config(
tags=["model_performance"],
meta={
"description": "This test will fail on any models running over 30 minutes within the last 24 hours"
}
) }}
select name, package_name, status, generated_at, execution_time
from {{ ref('elementary', 'model_run_results') }}
where CAST(generated_at AS timestamp) >= TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 day)
AND execution_time >= 30 * 60
order by execution_time desc
```
In this example:
* The test monitors model runs over the past 24 hours
* It fails if any model takes longer than 30 minutes to run (1800 seconds)
* The test is tagged with "model\_performance" for easy identification
* Results are ordered by execution time in descending order
When this test fails, Elementary will generate an alert based on your alert configurations. The test results will also be visible in the Elementary UI, showing the 5 worst-performing model runs.
## Anomaly Detection Alerts
Instead of using fixed thresholds, you can leverage Elementary's anomaly detection to identify unusual increases in execution time. This approach is more dynamic and can adapt to your evolving data pipeline.
### Implementation Steps
1. Define a source on the `model_run_results` view in your `schema.yml` file (or another YAML file):
```yaml theme={null}
sources:
- name: elementary_models
schema: "your_elementary_schema" # Replace with your Elementary schema name
tables:
- name: model_run_results
columns:
- name: execution_time
data_tests:
- elementary.column_anomalies:
config:
severity: warn
tags: ["model_performance"]
arguments:
column_anomalies:
- max
dimensions: ["package_name", "name"]
timestamp_column: generated_at
anomaly_direction: spike
ignore_small_changes:
spike_failure_percent_threshold: 10
```
In this configuration:
* Elementary monitors the `execution_time` column for anomalies
* Dimensions are set to `package_name` and `name` to analyze each model individually
* The test only detects spikes in execution time (`anomaly_direction: spike`)
* Small changes under 10% are ignored (`spike_failure_percent_threshold: 10`)
* The severity is set to "warn" but can be adjusted as needed
This test will detect when a model's execution time increases significantly compared to its historical performance, triggering an alert when the increase exceeds the normal baseline.
## Choosing the Right Approach
Both methods have their strengths:
* **Static Threshold**: Simple to implement and understand. Ideal when you have clear performance requirements (e.g., "models must run in under 30 minutes").
* **Anomaly Detection**: More adaptive to your specific environment. Better at detecting relative changes in performance rather than absolute thresholds. Useful when normal execution times vary across different models.
You can implement both approaches simultaneously for comprehensive performance monitoring.
## Viewing Performance Alerts
Performance alerts appear in your regular Elementary alert channels (Slack, email, etc.) based on your alert configuration.
You can also view performance test results in the Elementary UI under the Tests tab, filtered by the "model\_performance" tag that we added to both test types.
# Performance Monitoring
Source: https://docs.elementary-data.com/cloud/features/performance-monitoring/performance-monitoring
Monitoring the performance of your data pipeline is critical for maintaining data quality, reliability, and operational efficiency.
Proactively monitoring performance issues enables to detect bottlenecks and opportunities for optimization, prevent data delays, and avoid unnecessary costs.
There are two ways to identify and act on performance issues in Elementary:
* **Performance pages** — explore execution times, trends, and fail rates for models and tests across your pipeline
* **[Performance & Cost Agent](/cloud/ai-agents/performance-cost-agent)** — analyzes patterns across your pipeline, surfaces inefficiencies, suggests query improvements, and flags redundant tests that are slowing things down
Use the pages to monitor and spot outliers. Use the agent to investigate and get actionable recommendations.
Elementary monitors and logs the execution times of your dbt models and tests.
## Models performance
Navigate to the `Model Duration` tab.
The table displays the latest execution time, median execution time, and execution time trend for each model. You can sort the table by these metrics and explore the execution times over time for the models with the longest durations
It is also useful to use the navigation bar to filter the results, and see run times per tag/owner/folder.
## Tests performance
Navigate to the `Test Execution History` tab.
On the table you can see the median execution time and fail rate per test.
You can sort the table by this time column, and detect tests that are compute heavy.
It is also useful to use the navigation bar to filter the results, and see run times per tag/owner/folder.
## Optimize with the AI Agent
Once you've identified slow or expensive models and tests, use the [Performance & Cost Agent](/cloud/ai-agents/performance-cost-agent) to go deeper. It analyzes query patterns, surfaces inefficiencies, suggests improved SQL, and identifies redundant tests that slow down your pipeline — so you can act on what the performance data is showing.
# Roles & Permissions Management
Source: https://docs.elementary-data.com/cloud/features/roles-and-permissions
Roles & Permissions management in Elementary lets you control **who can see or do what** inside your account.
Instead of everyone having the same access, you can assign **roles** to users or groups so each person sees only what's relevant to them, and has only the permissions they need.
## Why it matters
* **Security & least privilege** — Users only get access to data and actions required for their role.
* **Team/group separation** — You can separate responsibilities (e.g. analysts, data engineers, reviewers) without interference.
* **Compliance readiness** — With clearly defined roles, you get better traceability for audits and compliance reviews.
* **Flexibility** — permissions are configurable per account, environment, and even per asset/test grouping, so you can support complex org structures.
## How it works in Elementary
| Level | What it controls | Typical use cases |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| **Account-level permissions** | **Global account configurations** (e.g. manage users, setup integrations, export audit logs) | Admins, Security / Compliance officers |
| **Environment-level permissions** | Access to **specific environments** (e.g. Prod, Staging) + Access to **environment pages and features** (catalog, lineage, PRs, etc.) | Data teams, Analytics engineers, Data governance |
| **Resources - Asset/Test-level permissions** | Access to **specific group of assets or tests** (e.g. only Marketing datasets, or only domain specific DQ rules) | Scoped analysts / data stewards or domain-specific business users |
A user's effective permissions are the sum (union) of all roles and permissions granted to them.
## Types of Roles
Roles in Elementary can be divided into two categories: built-in (Admin, editor, viewer) and custom.
### **Built-in Roles**
Elementary includes default roles most teams can use immediately - Admin, Editor, Viewer:
| Roles/ Permissions Level | **Account-level permissions** | **Environment-level permissions** | **Resources- Asset/Test-level permissions** |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| **Admin** | **Full (Can edit) access for account configurations** (e.g. manage users, setup integrations, export audit logs) | **Full (can edit) access for all environments, pages and features:** Can view and manage any configuration within the environments such as alert rules, manage incidents and PRs, etc. | **Full (can edit) access to all assets and tests:** can create/edit assets, run DQ (data-quality) rules, manage pipelines, etc |
| **Editor** | No permissions for account configurations | **Full (can edit) access for all environments, pages and features:** Can view and manage any configuration within the environments such as alert rules, manage incidents and PRs, etc. | **Full (can edit) access to all assets and tests:** can create/edit assets, run DQ (data-quality) rules, manage pipelines, etc |
| **Viewer** | No permissions for account configurations | **Read only access for all environments:** can view all environments and pages. can’t make any changes through the UI. | **Read-only access across data assets and tests:** can browse data assets, lineage, assets’ data health, etc. |
These roles cover most organizations, but you can extend them with custom roles.
**Custom Roles**
For each custom rule you can set limited permissions to a subset of environments, pages, or assets/tests (e.g. only Marketing data, or only Prod environments). It's great for domain-specific teams and business users, ensuring each team only has access to the environments, pages, assets and tests relevant to them.
For each role you need to define role Scope (account-level, environment-level, or asset-level) and permissions type (Read only, Can edit, No permissions).
Each role may include permissions at one, two, or all three levels:
1. **Account-Level**
Decide whether the role can **view** and whether it can **edit** account-wide configurations such as user management, integrations, and global configurations.
Without at least **view** permissions, account configurations won't appear for this role.
2. **Environment-Level**
By default, any role has access to **all environments**, but you can restrict it to specific ones.
Choose which environments the role applies to, and what the user can do there.
For each environment:
* **Feature access** — Select which pages (Catalog, Lineage, Alerts, etc.) and features (View/create PRs, AI Agents etc.) are visible.
* **Permission type** — Set it to **read-only,** **edit** or **No permissions**
3. **Resources filters - Assets & Tests (optional)**
Use metadata filters to restrict the role to specific datasets or tests.
* **Filters** — e.g., `Asset tag = #marketing`, `Test Owner = @data-team`.
* **Permission type** — Choose read-only or edit access (e.g., create tests, update metadata, modify configurations) for each filter group.
* **"Resources are by default"** — This controls access to resources that **don't match any filter**.
* Set to **Hidden** if you want users to only see the filtered resources (most common for scoped roles).
* Set to **Read-only** if you want unfiltered resources to still be visible (just not editable).
If you add filters but leave "Resources are by default" set to **Read-only**, users will still see all resources outside the filter. To truly restrict visibility, set this to **Hidden**.
**Custom role example: Finance Prod Editor**
**Description** - Can edit finance assets, cannot see PII assets
* Account level - No permissions to view/edit account configuration
* Environment level -
* For 'Prod' environment - Can edit
* Other environments - Hidden (No access)
* Assets/tests level -
* `Asset tag = #finance OR Asset tag = #finance-data-product` - Can edit
* `Asset tag = #PII` - Hidden (No access)
* All other assets → **Hidden** (set via "Resources are by default: Hidden")
## Getting started: How to set up roles and permissions
1. **Plan your organizational structure**
* List out personas in your org (e.g. "Data Engineer", "Analyst", "Marketing Analyst", "Business users", "DevOps")
* For each persona, decide:
* Whether they need access to **account-level configuration**
* **Which environments** they should be able to access (e.g., Dev, Staging, Prod)
* **Which pages, features, and capabilities** they should be able to use (Catalog, Lineage, Alerts, manage tests, update metadata, etc.)
* **Whether their access should be limited to specific assets or datasets**, using filters such as `Asset tag = #PII` or `Test Owner = @data-team`
2. **Create Custom Roles in Elementary (optional)**
1. Go to *Settings → Roles,* Click **"Create New Role"**
2. Set the Role Details - Role name - Give it a clear, descriptive name, such as *Marketing Analyst, Finance Viewer, Data Steward – Production*
3. Choose the Role Scope (account-level, environment-level, or asset-level) and permissions type (Read only, can edit)
**Each role may include permissions at one, two, or all three levels.**
3. **Assign Users or Groups to built in or custom Roles**
* Add individual users or entire groups/teams (via SSO/SCIM integration or manual).
* Users can have multiple roles, permissions are combined (union).
4. **Adjust and Evolve Roles over time**
* As your org grows or changes, update role definitions or add new roles (e.g. "IT team", "Business viewer").
* Reassign or remove roles when people change responsibilities or leave.
## Best Practices & Recommendations
* **Start with broad roles, then narrow down**
Begin with general roles (Admin, Data Engineer, Analyst), then create more scoped roles once patterns emerge (e.g. separate "Marketing Data Viewer," "Finance Data Viewer").
* **Use principle of least privilege**
Assign only the minimal permissions required. For example, an analyst who only needs to view data should not have edit or delete permissions.
* **Leverage asset/tag-based filters**
Use tags like `tag:#marketing`, `env:prod`, to automatically filter access. This lowers maintenance overhead as you add more assets.
* **Regularly review role assignments**
Periodically (e.g. quarterly) run the access snapshot and verify that users still need their assigned roles.
* **Avoid giving "Admin" widely**
Reserve account-level Admin or environment-wide write permissions for a small set of trusted users (e.g. data platform or security team).
## FAQ
**Q: What happens if a user has two roles with conflicting permissions (e.g. one role gives "read-only", another gives "edit")?**
A: Elementary uses a **union model** — the user gets the higher privilege. So in that case, they'd have "edit" access.
**Q: Can I let someone see only a subset of datasets (e.g. only marketing data)?**
A: Yes. When you create/edit a role, you can specify asset filters (e.g. by tag, owner, or dataset name). That way, scoped viewers or editors only see their allowed subset.
**Q: What if I need a temporary elevated permission for a user (e.g. for a one-time task)?**
A: You can assign a role temporarily (for example, a "Contractor – Write Access" role), and remove it once the task is done. We recommend logging the change and reviewing it later.
**Q: Does using Roles & Permissions affect performance or data loading?**
A: No — permissions are enforced efficiently at request time. There is no perceptible performance overhead under normal usage.
**Q: Can I export or audit roles and permissions?**
A: Yes — Elementary provides **Roles Audit Logs** that record all role changes and assignments. You can export to CSV or share with compliance teams.
**Q: I created a role with a resource filter, but the user can still see all assets — why?**
A: Check the **"Resources are by default"** setting in your role's Resource permissions. If it's set to **Read-only**, any resource that doesn't match your filter is still visible with read access. To hide everything outside your filter, change this to **Hidden**.
# Table usage
Source: https://docs.elementary-data.com/cloud/features/table-usage
Table usage shows how your data assets are **actually consumed** in the warehouse—how often they are queried and by who—based on **query activity** over a **rolling 30-day window**. That visibility helps you **prioritize incidents and data quality** on the assets that drive the most load and downstream impact, and get a **clearer end-to-end view** of what is happening in production.
## Usage in the Catalog
Open the **Catalog** and select the **Metadata** tab. Table assets include a **Usage** column with a **usage indicator** so you can compare relative activity at a glance.
### Tooltip details
Hover the usage indicator for a specific asset to open a tooltip with:
* **Percentile rank** — How this table’s query volume ranks against every other table. For example, **72nd percentile** means its query volume is **greater than or equal to** that of **72% of tables** in your environment.
* **Reporting period** — The exact **30-day** date range the metrics are based on (for example, March 15 – April 14, 2026).
* **Query count** — Total queries against the table in that period.
* **Unique users** — Distinct users who ran queries against the table in that period.
## AI agents
The same usage signals are available as **context** to Elementary **AI agents**, so you can ask natural-language questions that depend on real consumption—for example, **“Show me the top 5 most used tables.”**
## Related
* [Data Catalog](/cloud/features/collaboration-and-communication/catalog) — Overview of catalog navigation, metadata editing, and collaboration workflows.
* [AI agents](/cloud/ai-agents/overview) — How Elementary’s agents use catalog and observability context.
# AI Privacy Policy
Source: https://docs.elementary-data.com/cloud/general/ai-privacy-policy
**We are committed to maintaining the highest standards of data privacy and protection. This AI Privacy Policy outlines how we handle data in the context of our AI-powered features, including our [suite of AI agents](/cloud/ai-agents).**
Elementary’s AI features are designed to enhance the user experience across key workflows, enabling natural language responses, automated data exploration, intelligent issue triage and resolution, proactive test and governance recommendations, and query optimization. These features are strictly opt-in and must be explicitly enabled for each customer instance.
### Third-Party AI Providers
Elementary’s AI features are powered by leading large language models hosted through **Amazon Bedrock**, a secure AWS-managed environment. This architecture ensures that:
* **All AI processing stays entirely within the AWS infrastructure.**
* **No data is sent to external model providers directly by Elementary.**
* **No data used for AI inference is stored or used for training by any provider.**
Amazon Bedrock may access models from third-party providers such as **Anthropic** and **OpenAI**. However, these models are accessed **via Bedrock** - your data remains within AWS at all times.
### Data Handling & Anonymization
Elementary is committed to minimizing data exposure and ensuring user privacy in all AI-powered workflows. When using AI features, only the minimal context necessary to fulfill the requested functionality is processed, such as metadata, anonymized prompts, or structural test information.
No personally identifiable information (PII) or customer-specific identifiers are shared with AI models unless explicitly configured by the customer for a particular feature. Most features are designed to operate entirely on anonymized metadata and structured inputs.
All AI processing takes place within **Amazon Bedrock**, a secure environment within AWS. This ensures that your data remains fully contained within your cloud region and never leaves the AWS infrastructure.
### Customer Instance Isolation
AI interactions are scoped strictly to each customer’s instance. No data is shared between tenants. Responses are routed back only to the initiating instance. Our infrastructure uses per-instance storage and 2FA-protected admin access.
### Data Retention and Deletion
* No AI request data is stored beyond the scope of processing.
* General client data is retained for a maximum of one week, or deleted immediately upon account closure.
### Opt-Out Controls
AI functionality can be completely disabled at the instance level. Customers may also selectively enable only the features they wish to use.
### Compliance
Our AI processing aligns with:
* **GDPR**, **UK GDPR**, **CCPA**, and other global data privacy regulations.
* Security best practices, including those outlined in our [Data Protection Agreement](https://www.elementary-data.com/privacy).
## Have any more questions?
We would be happy to answer!
Reach out to us on [email](mailto:legal@elementary-data.com) or [Slack](https://elementary-data.com/community).
# Security and Privacy
Source: https://docs.elementary-data.com/cloud/general/security-and-privacy
For how data is handled by Elementary's AI features and AI agents, see the [AI Privacy Policy](/cloud/general/ai-privacy-policy).
## Security highlights
Our product is designed with security and privacy in mind.
* Elementary Cloud does not have read access to raw data in your data warehouse.
* Elementary Cloud only extracts and stores metadata, logs and aggregated metrics.
* All data is encrypted at rest and in transit using industry standard protocols.
* Elementary uses service accounts or authentication tokens with granular and minimal permissions.
* Elementary offers deployment options with no direct access from Elementary Cloud to your data warehouse and third-party tools.
Elementary Cloud is SOC2 type II certified and HIPAA compliant!
If you are interested in advanced authentication such as MFA, Okta SSO, Microsoft AD - please contact us at [cloud@elementary-data.com](mailto:cloud@elementary-data.com)
## Security principals
Elementary Cloud is designed with the core principle of least privilege.
Our cloud service does not require permissions to access the customer data.
Therefore, we instruct our customers to create a dedicated role for Elementary with `read only` access only to the Elementary schema in your data warehouse.
As long as you follow the onboarding process instructions, it will be impossible for Elementary Cloud to read data from your warehouse that does not reside in the Elementary schema.
This ensures that Elementary cloud will not mistakenly access your data, and minimizes the risk in case of a data breach.
Our product and architecture are always evolving, but our commitment to secure design always remains.
## How it works?
1. You install the Elementary dbt package in your dbt project and configure it to write to it's own schema, the Elementary schema.
2. The package writes test results, run results, logs and metadata to the Elementary schema.
3. The cloud service only requires `read access` to the Elementary schema, not to schemas where your sensitive data is stored.
4. The cloud service connects to sync the Elementary schema using an **encrypted connection** and a **static IP address** that you will need to add to your allowlist.
## What information does Elementary collect?
Elementary collects and stores metadata, aggregated metrics and logs.
The collected information is detailed in the table below.
You can see all the data that Elementary collects and stores in your local Elementary schema.
In general, Elementary does not collect any raw data. The only exception is the failed rows sample (stored in table `test_result_rows`) which can be disabled.
This is an opt-out feature that shows a sample of a few raw failed rows for failed tests, to help users triage and understand the problem.
To avoid this sampling, set the var `test_sample_row_count: 0` in your `dbt_project.yml` (default is 5 sample rows). You can also disable samples for specific tests, protect PII-tagged tables, and request environment-level controls. See [Test Result Samples](/data-tests/test-result-samples) for all available options.
| Information | Details | Usage |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Metadata | Table names, column names and types, names of BI dashboards, query code of tables, etc. | Present data issues with context, such as data lineage (dependencies). |
| Configuration | Technical configuration of how tables and tests should be executed, and configuration such as owners, descriptions, tags, etc. | Add context to data reliability issues and alerts for fast resolution. |
| Run results | Run results of tests and pipelines, including run time, errors, invocation origin, etc. | Alert on data reliability issues and identify performance bottle necks. |
| Query logs | Executed queries and their metadata such as user performing the query, resources, run time, etc. | Track usage to help troubleshooting and optimizing resources usage. |
| Table metrics | Row count, last modification time, size, and similar table-level metrics. | Monitor volume, freshness and similar data reliability factors. |
| Aggregated metrics | **Opt-in collection** of aggregated statistics on data in selected field and tables. For example: null rates, uniqueness rate, distribution of values, and other similar metrics. | Monitor data reliability using anomaly detection on the collected metrics. |
| Failed rows sample | **Opt-out collection** of small sample of raw rows or values that have a data reliability issue. These are attached to the issue for troubleshooting. | Enable users to quickly understand and investigate data issues. Users can opt-out completely or exclude specific tables, limit the sample size and limit the retention of samples. |
## Secrets and encryption
* **Tokens and credentials** - For customer secrets (tokens and credentials) we use [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html). Secrets Manager uses envelope encryption with AWS KMS keys and data keys to protect each secret value. Whenever the secret value in a secret changes, Secrets Manager generates a new data key to protect it. The data key is encrypted under a KMS key and stored in the metadata of the secret. [See this link for more details](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html).
* **Customer data (Elementary schema replica)** - The synced customer data is encrypted at rest using server-side encryption (AES-256).
* **Network connections** - All connections to Elementary Cloud are encrypted by default, in both directions using modern ciphers and cryptographic systems. We encrypt in transit utilizing TLS 1.2. Any attempt to connect over HTTP is redirected to HTTPS.
## Compliance
**SOC 2 certification:** Elementary Cloud is SOC2 type II certified!
Contact us at `legal@elementary-data.com` for auditing reports and penetration testing results.
## Privacy and data protection
We continuously implement evolving privacy and data protection processes, procedures, and best practices.
For more information, see our [Data Protection Agreement](https://www.elementary-data.com/privacy).
Contact us at `legal@elementary-data.com` if you need us to review and sign your company's DPA and MNDA.
## Samples storage in the US
As mentioned above, Elementary generally avoids storing raw data and relies on high level metrics and metadata.
That being said, one feature (that is opt-out) that may contain such data is storage of sample rows for failed tests in your pipeline.
For customers that require it, Elementary has the option to store such information in the US, instead of the EU which is the default.
In this case, the data will specifically be stored in S3 buckets in the us-east-1 (Virginia) AWS region, instead of eu-central-1 (Frankfurt).
The relevant datasets that will be stored in the US are:
1. The S3 bucket that contains the samples served by the Elementary UI.
2. The S3 bucket used for AI chat history - as these may potentially interact with sample data.
3. Intermediate S3 buckets used for Elementary's internal data pipeline, that may temporarily store samples.
This mechanism ensures, if enabled, that sample data and related datasets will never be persisted outside of the US, permanently or temporarily.
## Have more questions?
We would be happy to answer!
Reach out to us on [email](mailto:legal@elementary-data.com) or [Slack](https://elementary-data.com/community).
# Subprocessors
Source: https://docs.elementary-data.com/cloud/general/sub-processors
Elementary engages third-party service providers to deliver Elementary Cloud. A Subprocessor is a third-party that processes Customer Data or Personal Information on Elementary's behalf, in the framework of our engagement with the customer.
Prior to engaging such sub-processors, Elementary evaluates their data protection and security measures, and ensures a data protection agreement is in place per applicable data protection laws.
This page distinguishes between two types of data:
* **Customer Data** is the data Elementary processes on the customer's behalf under our Data Processing Agreement, such as metadata, test results, and query results from the customer's connected data environment.
* **Personal Information** is identifying information about the customer's personnel, such as names and email addresses, that Elementary processes to provide and support the service.
## Subprocessors that process Customer Data
| Subprocessor | Purpose | Location |
| ------------------------------------------ | --------------------------------------------------------------------------------------- | ---------------------------------------- |
| Amazon Web Services, Inc. | Cloud infrastructure hosting Elementary Cloud (compute, storage, databases, networking) | EU (default), US (upon customer request) |
| Amazon Bedrock (Amazon Web Services, Inc.) | AI model inference for Elementary's AI features and agents | Global (No data storage or retention) |
## Infrastructure and Development Subprocessors
These are subprocessors who have access to Elementary production systems and data by virtue of the services they provide. **Customer Data is not processed** by the sub-processors listed below as part of the on-going service.
| Subprocessor | Purpose | Location |
| ---------------------------------- | ----------------------------------------------------------------- | ------------- |
| Anthropic, PBC | AI assistants used in internal engineering and research workflows | United States |
| Cognition AI, Inc. | AI software engineering tooling | United States |
| Datadog, Inc. | Application and infrastructure monitoring | United States |
| Functional Software, Inc. (Sentry) | Error tracking | United States |
| LangChain, Inc. (LangSmith) | AI agent tracing and evaluation | United States |
| Frontegg, Inc. | User authentication, SSO, and access management | United States |
## Platform Sub-processors
These providers process Elementary's Platform data, which may include Personal Information such as customers email addresses and names. **Customer Data is not processed** by the sub-processors listed below.
| Subprocessor | Purpose | Location |
| --------------------------------- | -------------------------------------------------------------------- | -------------- |
| Twilio, Inc. (including SendGrid) | Transactional email delivery and SMS for multi-factor authentication | United States |
| Astrodon Corporation (Loops) | Product lifecycle and onboarding emails | United States |
| PostHog, Inc. | Product analytics | United States |
| Pylon Labs, Inc. | Customer support ticketing | United States |
| Slack (Salesforce, Inc.) | Customer support and shared communication channels | United States |
| tl;dv GmbH | Meeting recording and transcription | European Union |
| HubSpot, Inc. | CRM and customer lifecycle management | United States |
| Snowflake Inc. | Data warehouse for internal analytics and BI | European Union |
## Updates to this list
We update this page when we add or replace subprocessors. For questions about our subprocessors or our Data Processing Agreement, contact [legal@elementary-data.com](mailto:legal@elementary-data.com).
# Collect Jobs Info From Orchestrator
Source: https://docs.elementary-data.com/cloud/guides/collect-job-data
Elementary can collect metadata about your jobs from the orchestrator you are using, and enrich the Elementary UI with this information.
The goal is to provide context that is useful to triage and resolve data issues, such as:
* Is my freshness / volume issue related to a job that didn't complete? Which job?
* Which tables were built as part of the job that loaded data with issues?
* Which job should I rerun to resolve?
**Elementary collects the following job details:**
* Orchestrator name: `orchestrator`
* Job name: `job_name`
* Job ID: `job_id`
* Job results URL: `job_url`
* The ID of a specific run execution: `job_run_id`
* Job run results URL: `job_run_url`
## How Elementary collects jobs metadata
#### Environment variables
Elementary reads metadata at run time from environment variables. **Which fields fill in automatically depends on the orchestrator**—see the table in [Which orchestrators are supported?](#which-orchestrators-are-supported). For anything not supplied by the orchestrator, set the matching env vars in your orchestration tool, or pass [dbt vars](#dbt-vars).
These env vars are read when present:
`ORCHESTRATOR`, `JOB_NAME` or `DBT_JOB_NAME` (both map to `job_name`), `JOB_ID`, `JOB_URL`, `JOB_RUN_ID`, `JOB_RUN_URL`
To configure env vars for your orchestrator, use your orchestrator’s documentation. For **dbt Cloud** and the `job_name` column specifically, follow [Setting job name on dbt Cloud](#setting-job-name-on-dbt-cloud).
#### dbt vars
Elementary also supports passing job metadata as dbt vars. If both an env var and a `var` exist for the same field, **the var wins**.
To pass job data to Elementary using `var`, use the `--vars` flag in your invocations:
```shell theme={null}
dbt run --vars '{"orchestrator": "airflow", "job_name": "dbt_marketing_night_load"}'
```
If you already pass `--vars`, merge every key into **one** JSON object. Wrong: two `--vars` flags. Right: `dbt run --vars '{"job_name": "my_job", "other_key": "value"}'`.
#### Variables supported format
| var / env\_var | Format |
| -------------------------------- | ---------------------------------------------------------------------- |
| orchestrator | One of: `airflow`, `dbt_cloud`, `github_actions`, `prefect`, `dagster` |
| job\_name, job\_id, job\_run\_id | String |
| job\_url, job\_run\_url | Valid HTTP URL |
## Which orchestrators are supported?
You can pass job info to Elementary from any orchestration tool as long as you configure `env_vars` / `vars`.
The following default environment variables are supported out of the box:
| Orchestrator | Env vars |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| dbt cloud | orchestrator job\_id: `DBT_CLOUD_JOB_ID` job\_run\_id: `DBT_CLOUD_RUN_ID` job\_url: generated from `DBT_ACCOUNT_ID`, `DBT_CLOUD_PROJECT_ID`, `DBT_CLOUD_JOB_ID` job\_run\_url: generated from `ACCOUNT_ID`, `DBT_CLOUD_PROJECT_ID`, `DBT_CLOUD_RUN_ID` |
| Github actions | orchestrator job\_run\_id: `GITHUB_RUN_ID` job\_url: generated from `GITHUB_SERVER_URL`, `GITHUB_REPOSITORY`, `GITHUB_RUN_ID` |
| Airflow | orchestrator |
**dbt Cloud does not populate `job_name` for you.** The dbt Cloud row above lists what is wired automatically—it does **not** include job display name. If `JOB_NAME`, `DBT_JOB_NAME`, and the dbt `job_name` var are all unset, **`job_name` in `dbt_invocations` stays empty** (for example, filters like `WHERE job_name IS NOT NULL` return no rows). Use the next section to set it explicitly.
## Setting job name on dbt Cloud
Use one of the following. After you save, **only new runs** write `job_name`; past invocations are not updated.
In dbt Cloud, open the **deployment** or **job** that executes your dbt commands—the screen where you add **environment variables** for that run. Exact labels vary (for example **Environment variables**, **Deployment environment**, or **Job** → **Settings**).
Add **either**:
* **`JOB_NAME`** — set the value to the label you want in `dbt_invocations.job_name` (often the same as the job name in dbt Cloud), **or**
* **`DBT_JOB_NAME`** — same behavior as `JOB_NAME`; use whichever naming convention you prefer.
Save your changes.
Trigger a **new** job run. Confirm with something like:
```sql theme={null}
SELECT job_name, job_id, job_run_id
FROM .dbt_invocations
ORDER BY generated_at DESC
LIMIT 5;
```
You should see **`job_name`** populated on rows from runs **after** you added the variable.
### Alternative: `job_name` via dbt `--vars`
On the job’s dbt command, include `job_name` in `--vars`:
```shell theme={null}
dbt build --vars '{"job_name": "nightly_sales"}'
```
Combine with any other vars in the **same** JSON object. Remember: the dbt **`job_name`** var overrides **`JOB_NAME`** / **`DBT_JOB_NAME`** if both are set.
## What if I use dbt Cloud + orchestrator?
By default, Elementary will collect the dbt cloud jobs info.
If you wish to override that, change your dbt cloud invocations to pass the orchestrator job info using `--vars`:
```shell theme={null}
dbt run --vars '{"orchestrator": "airflow", "job_name": "dbt_marketing_night_load"}'
```
## Where can I see my job info?
* In your Elementary schema, the raw fields are stored in the table `dbt_invocations`. You could also use the view `job_run_results` which groups invocation by job.
* In the Elementary UI, if the info was collected successfully, you can filter the lineage by job and see the details in the node info.
## Can't find your orchestrator? Missing info?
We would love to support more orchestrators and collect more useful info!
Please [open an issue](https://github.com/elementary-data/elementary/issues/new/choose) and tell us what we should add.
# Collect dbt source freshness results
Source: https://docs.elementary-data.com/cloud/guides/collect-source-freshness
For users of dbt version 1.8 and above
Add the following flag to your `dbt_project.yml` file:
```yaml dbt_project.yml theme={null}
flags:
source_freshness_run_project_hooks: True
```
This flag enables Elementary to automatically collect `source-freshness` results, just like any other test results.
For dbt version under 1.8
In dbt versions lower than 1.8, the results of the command `dbt source-freshness` are not automatically collected.
You can collect the results using the Elementary CLI tool.
If dbt source freshness results are collected, they will be presented in the UI, and in alerts upon failure.
#### dbt core users
To alert on source freshness, you will need to run the following command:
```shell theme={null}
edr run-operation upload-source-freshness --project-dir
```
The command needs to be executed after you run `dbt source freshness` and before executing `edr monitor`.
This operation will upload the results to a table, and the execution of `edr monitor` will send the actual alert.
* Note that `dbt source freshness` and `upload-source-freshness` needs to run from the same machine.
* Note that `upload-source-freshness` requires passing `--project-dir` argument.
#### dbt Cloud users
The results can't be collected from dbt Cloud.
Here is a [suggestion from an Elementary user](https://elementary-community.slack.com/archives/C02CTC89LAX/p1688113609829869) for a solution you can implement.
## Need help?
Reach out to the Elementary team to consult on implementing dbt source freshness collection as an Elementary Cloud user.
# Elementary for Development and Production
Source: https://docs.elementary-data.com/cloud/guides/dev-prod-configuration
When working with Elementary Cloud, you'll want to configure Elementary differently based on your environment. This guide explains best practices for setting up Elementary across local development, dev/staging environments, and production.
## Overview
Elementary's `on-run-end` hooks collect test results and dbt artifacts, which are essential for monitoring but can slow down local development. Here's the recommended approach:
* **Local development**: Disable hooks to speed up your workflow
* **Dev/Staging environments**: Enable monitoring to catch issues early
* **Production**: Always enabled for full observability
## Disable Hooks in Local Development
When developing locally, the `on-run-end` hooks that upload run results and dbt artifacts can slow down your dbt runs. It's recommended to disable these hooks in your local environment.
You can disable specific hooks using variables in your `dbt_project.yml`:
```yaml dbt_project.yml theme={null}
vars:
disable_run_results: "{{ target.name not in ['prod','staging','dev'] }}"
disable_tests_results: "{{ target.name not in ['prod','staging','dev'] }}"
disable_dbt_artifacts_autoupload: "{{ target.name not in ['prod','staging','dev'] }}"
disable_dbt_invocation_autoupload: "{{ target.name not in ['prod','staging','dev'] }}"
```
This configuration ensures that:
* Hooks are **disabled** for local development (when `target.name` is not `prod`, `staging`, or `dev`)
* Hooks are **enabled** for your dev, staging, and production environments
## Monitor Dev and Staging Environments
While you should disable hooks in local development, you should **enable monitoring** for any dev or staging environments where you run tests. This allows you to:
* Catch data quality issues before they reach production
* Validate test coverage and test results in non-production environments
* Get early visibility into data problems
To ensure monitoring is enabled for your dev/staging environments:
1. **Configure your dbt targets** - Make sure your dev and staging targets are named consistently (e.g., `dev`, `staging`)
2. **Update the disablement vars** - Include your dev/staging target names in the condition, as shown in the example above
3. **Set up Elementary Cloud environments** - Create separate environments in Elementary Cloud for each of your dbt targets (dev, staging, prod)
## Production Configuration
In production, Elementary should always be enabled to provide full observability. The configuration above ensures hooks are enabled for production by including `prod` in the list of monitored environments.
## Complete Example Configuration
Here's a complete example that covers all scenarios:
```yaml dbt_project.yml theme={null}
vars:
# Disable hooks for local development, enable for dev/staging/prod
disable_run_results: "{{ target.name not in ['prod','staging','dev'] }}"
disable_tests_results: "{{ target.name not in ['prod','staging','dev'] }}"
disable_dbt_artifacts_autoupload: "{{ target.name not in ['prod','staging','dev'] }}"
disable_dbt_invocation_autoupload: "{{ target.name not in ['prod','staging','dev'] }}"
models:
elementary:
+schema: "elementary"
```
## Alternative: Disable Entire Package
If you prefer to disable the entire Elementary package in local development (note: this will also disable Elementary tests), you can use:
```yaml dbt_project.yml theme={null}
models:
elementary:
+schema: "elementary"
+enabled: "{{ target.name in ['prod','staging','dev'] }}"
```
Disabling the entire package will prevent Elementary tests from running in local development. We recommend using the hook disablement vars instead, which allows Elementary tests to run while skipping the artifact collection.
## Next Steps
After configuring your environments:
1. **Set up Elementary Cloud environments** - Create environments in Elementary Cloud for each of your monitored targets (dev, staging, prod)
2. **Configure sync scheduling** - Set up webhook-triggered syncs for real-time updates. See the [Sync Scheduling guide](/cloud/guides/sync-scheduling) for details
3. **Configure alerts** - Set up alerts for your dev and production environments to get notified of data quality issues
# Capture compiled code for microbatch models
Source: https://docs.elementary-data.com/cloud/guides/microbatch-compiled-code
Elementary can capture and store the compiled SQL of [dbt microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch) in `dbt_run_results.compiled_code`. By default dbt does not surface compiled code for the microbatch strategy, so this column is empty for microbatch models until you enable the setup below.
## How it works
Elementary provides the `capture_and_execute_microbatch_compiled_code_sql` macro that captures the compiled SQL of each batch as it runs. The captured code is cached during the invocation and later written to `dbt_run_results.compiled_code`, so microbatch models populate this column the same way other incremental strategies do.
## Enabling microbatch compiled code capture
Add a macro that delegates to Elementary's implementation. Place it under your project's `macros/` directory:
```sql filename="macros/get_incremental_microbatch_sql.sql" theme={null}
{% macro get_incremental_microbatch_sql(arg_dict) %}
{{ return(elementary.capture_and_execute_microbatch_compiled_code_sql(arg_dict)) }}
{% endmacro %}
```
Add the `require_batched_execution_for_custom_microbatch_strategy` flag to your `dbt_project.yml`:
```yaml filename="dbt_project.yml" theme={null}
flags:
require_batched_execution_for_custom_microbatch_strategy: True
```
This flag tells dbt to use your project-level override of the microbatch strategy with batched execution.
On the next `dbt run` or `dbt build`, Elementary captures the compiled SQL of each microbatch model and writes it to `dbt_run_results.compiled_code`.
## Unsupported configurations
The override flow is currently not supported on the following adapters:
* Spark
* BigQuery
* Athena
* ClickHouse
* Dremio
* Vertica
It is also not supported on dbt Fusion.
On unsupported adapters and on Fusion, microbatch models continue to run normally but `dbt_run_results.compiled_code` remains empty for them.
# Control On-Run-End Hooks
Source: https://docs.elementary-data.com/cloud/guides/reduce-on-run-end-time
Elementary's `on-run-end` hooks collect test results and dbt artifacts to provide comprehensive observability. However, for large projects or when running only a subset of models, these hooks can add significant time to your dbt runs. This guide explains how to control the timing of on-run-end hooks to make them more efficient and avoid unnecessary runs while maintaining the observability you need.
## Overview
The Elementary on-run-end hooks perform several operations that can impact run time:
* **Metadata artifacts upload**: Uploads all project metadata (models, tests, sources, etc.)
* **Run results upload**: Uploads test results and model execution results
* **dbt invocation upload**: Uploads dbt invocation metadata
For large projects, these operations can take several minutes. The following strategies help you control when these hooks run to optimize efficiency and reduce unnecessary uploads.
## Control Metadata Artifacts Upload Timing
The most effective way to reduce on-run-end time is to control when metadata artifacts are uploaded. Instead of uploading all project metadata (models, tests, etc.) on every run, you can upload artifacts on a schedule that matches when your project actually changes.
### Configuration
Add the following to your `dbt_project.yml`:
```yaml dbt_project.yml theme={null}
vars:
disable_dbt_artifacts_autoupload: true
```
### Upload Artifacts When Needed
When you need to update metadata (e.g., after schema changes or adding new models), upload artifacts either manually or via a recurring job (e.g., daily):
```bash theme={null}
dbt run --select elementary --vars '{"enable_elementary_models": true}'
```
For the `enable_elementary_models` pattern in `dbt_project.yml`, see [Can I disable / exclude the Elementary models?](/oss/general/faq#can-i-disable-exclude-the-elementary-models).
Metadata artifacts only need to be uploaded when your project structure changes (new models, tests, sources, etc.). For most runs, you only need run results, which are much faster to upload.
## Control Other Hooks Timing
**Important**: Disabling hooks (other than metadata artifacts) means you will lose this data for those runs. Only disable hooks for runs you don't want to monitor (e.g., local development).
You can further control when different parts of the on-run-end logic run by setting additional variables.
### Available Configuration Variables
```yaml dbt_project.yml theme={null}
vars:
disable_run_results: true
disable_tests_results: true
disable_dbt_invocation_autoupload: true
```
* **`disable_run_results`**: Controls when model execution results are uploaded
* **`disable_tests_results`**: Controls when test results are uploaded
* **`disable_dbt_invocation_autoupload`**: Controls when dbt invocation metadata is uploaded
Disabling these hooks means you will lose this data for those runs. Only disable hooks for runs you don't want to monitor (e.g., local development). This will reduce the amount of data processed and uploaded at the end of each run, but will impact the completeness of reports and monitoring for those runs.
## Disable test table cleanup during on-run-end
Elementary creates temporary tables during test execution and cleans them up at the end of each run. For large projects with many tests, this cleanup phase can become a significant portion of the `on_run_end` time.
If you observe in the logs that the slow phase is the test table cleanup, you can disable it during `on_run_end` and offload it to a dedicated scheduled job instead.
### Disable automatic cleanup
Add the following to your `dbt_project.yml`:
```yaml dbt_project.yml theme={null}
vars:
clean_elementary_temp_tables: false
```
### Run cleanup in a dedicated job
After disabling automatic cleanup, schedule a recurring dbt operation to clean up stale test tables:
```bash theme={null}
dbt run-operation elementary.cleanup_stale_test_tables
```
You can pass optional arguments to control the cleanup behavior:
```bash theme={null}
dbt run-operation elementary.cleanup_stale_test_tables --args '{"hours": 48, "limit": 1000}'
```
**Parameters:**
* **`hours`** (default: `24`): Only delete test tables that are older than this many hours. Increase this value if you want to retain temporary tables for longer before cleanup.
* **`limit`** (default: `2000`): Maximum number of tables to delete per run. Use this to cap the cleanup operation and avoid long-running jobs when many stale tables have accumulated.
We recommend scheduling this job to run daily. If a large number of stale tables have built up, the `limit` parameter prevents the first cleanup from running indefinitely — you can run the job multiple times until all stale tables are removed.
## Limit Hooks to Production or Specific Targets
If you only need full observability in production, configure Elementary to run hooks only for specific targets. This is especially useful when you want to:
* Speed up local development
* Reduce costs in non-production environments
* Focus monitoring on production workloads
### Configuration by Target
Configure hooks to run only for specific targets:
```yaml dbt_project.yml theme={null}
vars:
disable_dbt_artifacts_autoupload: "{{ target.name != 'prod' }}"
disable_run_results: "{{ target.name != 'prod' }}"
disable_tests_results: "{{ target.name != 'prod' }}"
disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"
```
### Disable Entire Package for Non-Production
Alternatively, you can disable the entire Elementary package for non-production targets:
```yaml dbt_project.yml theme={null}
models:
elementary:
+enabled: "{{ target.name == 'prod' }}"
```
Disabling the entire package will prevent Elementary tests from running. We recommend using the hook disablement vars instead, which allows Elementary tests to run while skipping the artifact collection.
For more details on configuring Elementary for different environments, see the [Dev/Prod Configuration guide](/cloud/guides/dev-prod-configuration).
## Next Steps
After configuring your on-run-end hooks timing:
1. **Set up metadata sync schedule**: If you disabled autoupload, create a process (manual or recurring job, e.g., daily) to sync metadata when your project changes
2. **Configure sync scheduling**: Set up webhook-triggered syncs for real-time updates. See the [Sync Scheduling guide](/cloud/guides/sync-scheduling) for details
3. **Monitor data completeness**: Verify that Elementary Cloud still receives the data needed for your monitoring use cases
For more information about configuring Elementary for different environments, see the [Dev/Prod Configuration guide](/cloud/guides/dev-prod-configuration).
# Set up Elementary
Source: https://docs.elementary-data.com/cloud/guides/set-up-elementary
Building reliable, trustworthy data pipelines shouldn't be painful. Elementary gives you a clear, fast path to understanding your data, monitoring its health, and responding before problems impact your business.
This guide walks you through everything you need to onboard successfully, from connecting your stack to configuration and setting up your first workflows. Once you've completed the technical setup, continue with [Start Using Elementary](/cloud/guides/start-using-elementary) to begin using Elementary in practice.
With Elementary, you'll quickly be able to:
* Catch issues proactively before they affect consumers
* Maintain reliable and compliant data across your ecosystem
* Establish clear ownership and accountability
* Help everyone discover and understand the data they use
Let's get you set up.
## Elementary's Core Principles
* **Code as the source of truth** - Elementary keeps your code as the system of record. Any change made in the app opens a pull request in your repository, fitting naturally into your CI/CD process and keeping engineers in control. You can work from code, from the UI, or through MCP and everything stays aligned.
* **Scaling reliability with AI** - Elementary uses AI to help you scale your reliability practices. Ella, our network of AI agents, assists with test creation, metadata enrichment, troubleshooting, and optimization so teams can stay proactive and focus on higher-value work.
* **Enabling all users** - Elementary is built for the entire organization. Business users get a clear catalog, ownership information, health scores, and AI assistance so everyone can find, understand, and trust the data without depending on engineering.
## Technical Setup
Here's a link to a checkbox version of this guide that you can download and use to track your progress: [Download as markdown](/cloud/guides/set-up-elementary-checklist.md).
### Setup and integrations
* [Create Elementary cloud account](/cloud/quickstart) (quick-start guide)
* [Install Elementary dbt package](/data-tests/dbt/quickstart-package)\
Collect dbt artifacts and enable Elementary's built-in tests (anomaly detection, schema change detection). If you already have the package deployed in the relevant environment you want to monitor, make sure it's up to date to the latest version, and upgrade it if not.
* [Connect your data warehouse](/cloud/integrations/elementary-integrations)\
Elementary reads metadata (dbt artifacts, information schema, query history) to power test results, run history, automated freshness/volume monitors, and column-level lineage.
* [Invite team members](/cloud/features/roles-and-permissions)
* [Code repository connection (optional)](/cloud/integrations/code-repo/connect-code-repo)\
Allow users and AI agents to open PRs and manage changes through your CI/CD process.
* [Integrate with your BI tool (optional)](/cloud/integrations/bi/connect-bi-tool)\
Enable column-level lineage and full context for BI assets, including the health of upstream sources and models.
* Connect to your external Catalog (Optional)\
Present Elementary data health context in another catalog, we support [Atlan](/cloud/integrations/governance/atlan).
* Non-dbt tables (optional)\
Elementary has a Python SDK for python-based transformations. If you need Elementary to monitor non-dbt tables, reach out to the Elementary team with a list of relevant schemas / datasets / databases. This is a beta feature, the team will guide you through the next steps.
### Alerts & Incidents
* Connect messaging app ([Slack](/cloud/integrations/alerts/slack) / [MS Teams](/cloud/integrations/alerts/ms-teams))\
Receive alerts where your team already works.
* Connect [incident management tool](/cloud/integrations/alerts) (optional)\
We support [Opsgenie](/cloud/integrations/alerts/opsgenie) and [PagerDuty](/cloud/integrations/alerts/pagerduty).
* Connect [ticketing system](/cloud/integrations/alerts) (optional)\
We support [Jira](/cloud/integrations/alerts/jira), [ServiceNow](/cloud/integrations/alerts/servicenow) and [Linear](/cloud/integrations/alerts/linear).
### AI Agents
* Sign [AI features consent form](https://2cwlc1.share-eu1.hsforms.com/2toUDbhpMRTK0WPmv2az_2g)\
Elementary AI uses only metadata, not raw or personal data. LLMs run via Amazon Bedrock with no data sharing to third parties. Learn more [here](/cloud/general/ai-privacy-policy).
* Elementary to open AI Features
## Advanced Setup
* [Schedule Elementary syncs](/cloud/guides/sync-scheduling) with your data warehouse.
* Contact the Elementary team to enable [Okta SSO](/cloud/integrations/security-and-connectivity/okta), [AWS PrivateLink](/cloud/integrations/security-and-connectivity/aws-privatelink-integration), or [Microsoft Entra ID](/cloud/integrations/security-and-connectivity/ms-entra).
* [Configuring Elementary for Development and Production](/cloud/guides/dev-prod-configuration)
* [Manage multiple environments](/cloud/features/multi-env)\
Use multiple dbt projects and targets (dev, staging, prod) through the multi-env feature.\
If you wish to monitor multiple environments in a single environment view (including unified lineage), reach out to the Elementary team.
* [Collect job information from orchestrator](/cloud/guides/collect-job-data)\
Elementary can collect metadata about your jobs from the orchestrator you are using, and enrich the Elementary UI with this information.
## Security and permissions
* Configure [roles and permissions](/cloud/features/roles-and-permissions) in Elementary (optional).
* Add SSO authentication with [Okta](/cloud/integrations/security-and-connectivity/okta) or [Azure AD](/cloud/integrations/security-and-connectivity/ms-entra) (optional).
* Connect using [AWS private link](/cloud/integrations/security-and-connectivity/aws-privatelink-integration) (optional)
* Export [user activity logs](/cloud/features/collaboration-and-communication/audit_logs/user-activity-logs) (optional).
# Start Using Elementary
Source: https://docs.elementary-data.com/cloud/guides/start-using-elementary
Here's a link to a checkbox version of this guide that you can download and use to track your progress: [Download as markdown](/cloud/guides/start-using-elementary-checklist.md).
With your environment connected (see [Set up Elementary](/cloud/guides/set-up-elementary) for technical onboarding), the next step is to see Elementary in action.
We'll take a few of your key assets and run them through the core reliability steps: setting clear ownership, adding the right tests, configuring alerts, and understanding how issues are detected and resolved. This gives you a practical, hands-on view of how Elementary keeps your most important data reliable.
## Identify Critical Assets
*Goal: Identify business-critical assets in your pipeline and make sure their documentation and ownership are clear.*
* In the [Catalog](/cloud/features/collaboration-and-communication/catalog), identify and mark three [critical assets](/cloud/features/data-governance/critical_assets)
* Add or update [descriptions, tags, and owners](/cloud/features/collaboration-and-communication/catalog) for these assets
* Leverage the [Governance AI agent](/cloud/ai-agents/governance-agent) - Complete missing metadata based on project context and according to your instructions.
* View [end-to-end, column-level lineage](/cloud/features/data-lineage/lineage) for each critical asset
## Setup monitoring on critical assets
*Goal: Ensure critical assets are covered with the right tests so issues are detected before they affect consumers.*
* Open the [Test Coverage](/cloud/features/data-tests/test-coverage-screen) view to understand existing coverage by dimension
* [Add missing tests](/cloud/features/data-tests/cloud-tests-overview) such as freshness, volume, uniqueness, [anomaly detection](/data-tests/how-anomaly-detection-works), [schema changes](/data-tests/schema-tests/schema-changes) or custom logic
* Leverage the [Test Recommendation AI agent](/cloud/ai-agents/test-recommendation-agent) for suggested tests based on patterns and lineage
* For source monitoring: Elementary automatically adds ML-based [freshness](/cloud/features/anomaly-detection/automated-freshness) and [volume](/cloud/features/anomaly-detection/automated-volume) tests to all sources to catch pipeline and ingestion issues early
## Setup Alerts
*Goal: Create a routing process that gets the right alerts to the right people while avoiding alert fatigue.*
* Use [tags, owners, and subscribers](/cloud/features/alerts-and-incidents/owners-and-subscribers) to define who should be notified and who is responsible for action
* Build an internal alert response plan: who fixes the issue, who needs awareness, and what the SLA should be (see the full playbook [here](https://www.elementary-data.com/post/breaking-alert-fatigue-the-enterprise-playbook-for-data-alerts))
* Create [alert rules](/cloud/features/alerts-and-incidents/alert-rules) that translate this plan into routing, using tags, owners, subscribers, and severity
## Triage & Resolution
*Goal: Investigate and resolve issues quickly and confidently.*
* Triage: Review lineage and upstream failures alongside recent commits and dbt run history to understand what changed and what the issue is impacting
* Use the [Triage & Resolution AI agent](/cloud/ai-agents/triage-resolution-agent) to run the investigation for you by analyzing lineage, failures, recent changes, and dependencies, and surfacing root causes
* Use the [Incidents](/cloud/features/alerts-and-incidents/incidents) screen to manage issues collaboratively and keep track of everything that's currently open
## Optimize Performance
*Goal: Meet your SLAs, improve run times, and ensure your pipelines operate efficiently.*
* Identify long-running tests and models using the [Performance pages](/cloud/features/performance-monitoring/performance-monitoring)
* Use the [Optimization AI agent](/cloud/ai-agents/performance-cost-agent) to optimize queries by surfacing inefficiencies, suggesting improved SQL patterns, and identifying opportunities to reduce data scans or simplify logic
## Measure Progress and Data Health
*Goal: Track improvements in reliability and identify where further attention is needed.*
* Use the [Data Health](/cloud/features/collaboration-and-communication/data-health) screen to monitor overall health scores for your assets
* Filter the screen by domain, owners, or critical assets to measure progress at a higher resolution and keep accountability clear
## Enable analysts and business users
*Goal: Make it easy for anyone to discover assets, understand what they represent, see how they're built, and know whether they can be trusted.*
* Use the [Catalog](/cloud/features/collaboration-and-communication/catalog) to explore assets, their definitions, ownership, and current health status
* Review [Lineage](/cloud/features/data-lineage/lineage) to see how the asset is built, what depends on it, and whether any upstream issues or test failures affect it
* Use the [Discovery AI agent](/cloud/ai-agents/catalog-agent) to get clear explanations of the asset, how it's calculated, and any current reliability concerns
## Advanced: Use MCP to Extend Elementary Everywhere
### For Engineers
*Goal: Make development safer and faster by bringing full pipeline context into your coding environment.*
* [Enable Elementary MCP](/cloud/mcp/setup-guide) inside your IDE or coding assistant (Cursor, Claude Code, etc.)
* Add metadata or validations at scale without touching the UI or code manually
* Check lineage, coverage, and asset health while you code so you can spot issues early, understand downstream impact, and prevent problems from reaching production
### For Analysts and Business Users
*Goal: Bring Elementary's context into the tools people already use.*
* [Connect Elementary MCP](/cloud/mcp/setup-guide) to any MCP-enabled client (Claude, ChatGPT, or internal AI agents)
* Allow users to ask about assets, see definitions and ownership, review lineage, and check health or incidents directly through their AI assistant
* (Optional) Connect additional MCPs such as dbt or Atlan so users can navigate across multiple systems from a single conversational interface
# Environment Syncs Schedule
Source: https://docs.elementary-data.com/cloud/guides/sync-scheduling
## Synchronizing the Elementary schema
The data on your Elementary Cloud environments is updated by syncing the local Elementary schema from the data warehouse.
There are 2 available scheduling options:
1. **Hourly syncs** - By default, Elementary Cloud syncs data from your Elementary schema on an hourly basis. This means that within the one-hour time window, your data may not be up-to-date.
2. **Triggered syncs** - Configure a webhook that will trigger a sync after your data pipeline has finished running. This configuration is recommended, as it's as close as possible to real time. As a backup, Elementary still keeps a sync that runs every 4 hours.
## Triggered syncs using webhook
When configuring a webhook, the environment will update as soon as new data is available, ensuring you see the most recent data while minimizing unnecessary updates and therefore cost.
To configure the webhook, navigate to the relevant environment on your [*Environments*](https://app.elementary-data.com/settings/environments) page in Elementary Cloud. Click the environment's *Schedule Settings* and change the environment update method to "Webhook triggered".
The next step depends on whether you're using dbt Cloud or dbt Core.
#### dbt Cloud
In the *Schedule Settings*, you're provided with a webhook URL. Next, you will need to provide a secret token that will be returned from dbt Cloud.
Heading to dbt Cloud, you can [create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks#create-a-webhook-subscription) that will trigger a sync after your jobs are done.
* Make sure the webhook is triggered on `Run completed` events.
* Select **only** the main jobs of the relevant environment.
Make sure to select only the main jobs of the relevant environment. Selecting all jobs will trigger a sync for each job, which may result in unnecessary updates and therefore increased cost on the data warehouse.
After completing the guide, you'll be provided with the secret token that you'll need to fill in the form on Elementary Cloud. Press *Save* and you're done!
Note that each environment has a different webhook
#### dbt Core
If you're using dbt Core and have a custom orchestration layer such as Airflow or Prefect, you can use the webhook URL and the secret generated by Elementary in the *Schedule Settings* to trigger a sync after your dbt job.
In the following code snippet,
replace `` and `` with the secret and webhook URL provided in the *Schedule Settings* and run it after your dbt job has finished.
```python theme={null}
import hmac
import requests
request_body = b""
secret = b""
webhook_url = ""
signature = hmac.new(secret, request_body, "sha256").hexdigest()
requests.post(webhook_url, headers={"Authorization": signature}, data=request_body)
```
Make sure to click *Save* after filling in the form.
# Troubleshooting
Source: https://docs.elementary-data.com/cloud/guides/troubleshoot
If you already connected your data warehouse to Elementary but are not seeing anything in the Elementary UI, there could be several reasons.
Try following these steps to troubleshoot:
**1. Validate elementary dbt package is deployed, working, and using the latest version**
* Refer to the [dbt package installation guide](/cloud/quickstart#install-the-dbt-package), and validate that your version in packages.yml is the one mentioned there. If not, upgrade and run `dbt deps`. Make sure to execute `dbt run --select elementary` for the package tables to be created.
**2. Check if the table `elementary_test_results` exists and has data**
* If the table does not exist - refer to the [dbt package installation guide](/cloud/quickstart#install-the-dbt-package). Make sure to execute `dbt run --select elementary` for the package tables to be created.
* If the table exists but has no data - Did you execute `dbt test` since deploying the package and creating the models?
* If you have, make sure the table was created as an incremental table (not a regular table or view).
* If not, there is a materialization configuration in your `dbt_project.yml` file that overrides the package config. Remove it, and run `dbt run --select elementary --full-refresh` to recreate the tables. After that run `dbt test` again and check if there is data.
**4. Still no data in the table? Reach out to the Elementary team by starting an intercom chat from the Elementary UI.**
When multiple dbt jobs run in parallel, they can contend on the same Elementary metadata artifact tables.
**Disable automatic metadata artifact uploads** in `dbt_project.yml`:
```yaml dbt_project.yml theme={null}
vars:
disable_dbt_artifacts_autoupload: true
```
**Add a separate scheduled job** (daily or hourly) to upload the artifacts:
```bash theme={null}
dbt run --select elementary --vars '{"enable_elementary_models": true}'
```
For the `enable_elementary_models` pattern in `dbt_project.yml`, see [Can I disable / exclude the Elementary models?](/oss/general/faq#can-i-disable-exclude-the-elementary-models). For more context, see [on-run-end hooks](/data-tests/dbt/on-run-end_hooks), [Reduce on-run-end time](/data-tests/dbt/reduce-on-run-end-time), and [Dev / prod configuration](/cloud/guides/dev-prod-configuration).
Let us know if you run into anything — start a chat from the Elementary UI or reach us on [Slack](https://elementary-data.com/community).
This error can happen because of a few reasons:
1. Check that your elementary dbt package version is 0.12.0 or higher.
2. Check that the user you are using to connect to your database has permission to access the information schema of all the schemas built or used by your dbt project.
For more information on the permissions required by each data warehouse:
[Snowflake](/cloud/integrations/dwh/snowflake#permissions-and-security)
[Bigquery](/cloud/integrations/dwh/bigquery#permissions-and-security)
[Redshift](/cloud/integrations/dwh/redshift#permissions-and-security)
[Databricks](/cloud/integrations/dwh/databricks#permissions-and-security)
[Postgres](/cloud/integrations/dwh/postgres#permissions-and-security)
Singular tests are sql queries that can reference more than one table, but are often intended to test a logic that is related to one table in particular.
In order to have that table name appear in the UI in the test results, test execution and more screens, you can set it up by adding the following to the config block of your singular test file:
```
{{ config(
override_primary_test_model_id="model_unique_id"
) }}
```
Note: Use the `model_unique_id`, not the model name.
The `model_unique_id` is the unique identifier of the model in dbt, and can be found by running the query:
```
SELECT unique_id
FROM .dbt_models
WHERE name=
```
# Email
Source: https://docs.elementary-data.com/cloud/integrations/alerts/email
Elementary supports sending alerts directly to email recipients, without requiring environment-level configuration. Email destinations can be added and managed directly within alert rules.
## Overview
* Email **does not** need to be configured as an integration on the Environment page.
* Email can be added **directly as a provider** when creating or editing alert rules.
* Users can enter individual emails or paste entire lists.
## Creating an Email Alert Destination
1. Go to **Alert Destinations**.
2. Select **Add new alert provider**.
3. Choose **Email**.
### Adding recipients
* Users may type or paste one or more email addresses.
* The system validates email format for each entry.
* When pasting combined fields (e.g., "Name [email@company.com](mailto:email@company.com)"), only the email address is stored.
* Previously used emails may be surfaced for autocomplete (optional enhancement).
## Visibility Across the UI
### Alert Rules
* Email recipients appear under the destination.
* If the list is long, the UI shows the first few emails and then a "+X more" indicator.
* Hovering displays the full recipient list.
### Test Overview
* Connected email destinations appear in the Destinations section for each test.
### Alert Destinations Tab
* All defined email alert destinations can be viewed and managed in this tab.
# incident.io
Source: https://docs.elementary-data.com/cloud/integrations/alerts/incident-io
Elementary's incident.io integration enables sending alerts to [incident.io](https://incident.io) when data issues are detected.
Alerts are sent as **alert events** to an incident.io [HTTP Alert Source](https://incident.io/docs/alerting/alert-sources). incident.io then creates or updates alerts based on your routing and escalation rules.
To send alerts to incident.io, you need to create [alert rules](/cloud/features/alerts-and-incidents/alert-rules) and select incident.io as the destination.
## Prerequisites
You need an incident.io account with access to create Alert Sources.
## Create an HTTP Alert Source in incident.io
Before connecting Elementary, you need to create an HTTP Alert Source in incident.io:
1. In incident.io, go to **Alerts** → **Configuration** → **Alert Sources**.
2. Click **Create alert source**, search for **HTTP**, and select it.
3. Give the alert source a name (e.g. "Elementary Cloud"), select **Default** as the type, and click **Continue**.
4. On the connection details page, you will see the **URL** and **authentication headers**. You will need the **alert source config ID** (from the URL) and the **API token** (from the Authorization header) when connecting in Elementary. Keep this page open — you will come back to it after sending a test alert.
The API token is a secret. Keep it secure and do not share it publicly.
## Connect incident.io in Elementary
1. Go to the **Environments** page on the sidebar.
2. Select an environment and click connect on the **Connect incident management tool** card, and select **incident.io**.
3. Fill in the connection details from your incident.io alert source:
* **Source name** — a display name for this alert source (e.g. "Elementary Cloud")
* **Alert source config ID** — the config ID from the alert source URL
* **API token** — the Bearer token from the authentication headers
4. Expand **Test connection** to send a test alert and verify the integration is working.
The test connection sends a real alert event to incident.io. You may need to manually resolve or close it after testing.
5. Click **Save**. **incident.io** will now be available as a destination on the [alert rules](/cloud/features/alerts-and-incidents/alert-rules) page.
You can add multiple alert sources by clicking **+ Add source** — for example, to route different alert rules to different incident.io alert sources.
## Configure the Alert Source in incident.io
After sending a test alert from Elementary, go back to the incident.io alert source setup page. You should see the test alert appear in the alerts list on the right side. Click **Continue**.
2. You will land on the alert source configuration page, where you can set up:
* **Title** — how the alert title is extracted from the payload
* **Connections** — routing rules for how alerts create incidents
* **Attributes** — map metadata fields from the alert payload to structured attributes (see below)
3. Click **Save** to finish the alert source setup.
## Configure Alert Attributes
Elementary sends metadata with each alert (such as severity, owners, and tags). To use these fields for routing, filtering, or displaying in incident.io, configure **Alert Attributes** in your alert source settings.
In the alert source configuration, click **Attributes** to map metadata fields. The alert payload is displayed on the right, making it easy to see which fields are available and map them.
### Available metadata fields
Elementary sends the following metadata fields depending on the alert type. All fields are optional — they are included only when the data is available.
| Field | Type | Description |
| --------------------------- | ---------------- | --------------------------------------------- |
| `metadata.severity` | String | Alert severity (e.g. `warn`, `fail`, `error`) |
| `metadata.owners` | Array of strings | Asset and test owners (email addresses) |
| `metadata.tags` | Array of strings | Asset and test tags |
| `metadata.environment` | String | Elementary environment name |
| `metadata.test_description` | String | Test description (test alerts only) |
| `metadata.source` | String | Pipeline tool name (pipeline alerts only) |
| `metadata.connector` | String | Connector name (pipeline alerts only) |
| `metadata.failure_reason` | String | Failure reason (pipeline alerts only) |
When configuring attributes, use the test alert payload as a reference — it includes sample values for the common metadata fields, making it easy to set up mappings.
### Recommended attribute mappings
We recommend mapping these fields in incident.io:
* **`metadata.severity`** → a **Severity** or text attribute — enables routing alerts by severity
* **`metadata.owners`** → a multi-value text attribute — enables filtering and routing by data owner
* **`metadata.tags`** → a multi-value text attribute — enables filtering by tag
For array fields like `owners` and `tags`, make sure to select a type that supports multiple values in incident.io's attribute configuration.
If you need additional metadata fields for your routing that are not currently included, [reach out to us](https://www.elementary-data.com/contact-us) and we'll work with you to add them.
## How alerts work
### Alert creation and deduplication
Each Elementary alert includes a **deduplication key** based on the alert's identity (test, model, or pipeline). incident.io uses this key to:
* **Create** a new alert when a new issue is detected
* **Deduplicate** — avoid creating duplicate alerts for the same ongoing issue
* **Auto-resolve** — when a test passes again, Elementary sends a resolved event with the same deduplication key, which closes the alert in incident.io
### Alert types
Elementary sends the following alert types to incident.io:
* **Test alerts** — dbt and Elementary test failures
* **Model alerts** — dbt model and snapshot failures
* **Source freshness alerts** — source freshness check failures
* **Pipeline alerts** — pipeline connector failures (Fivetran, etc.)
* **Resolved alerts** — automatically sent when a previously failing test/model passes
### Link to Elementary
Each alert includes a **source URL** linking back to the relevant page in Elementary Cloud, so you can quickly investigate the issue.
### Alerts and incidents in incident.io
Each alert sent by Elementary appears in incident.io with full metadata, attributes, and a link back to Elementary Cloud.
When an alert matches your incident.io routing rules, it automatically creates an incident with full context.
### Slack notifications
If you have Slack notifications configured in incident.io, alerts and incidents will appear in your Slack channels with the alert details, labels, and action buttons.
# Jira
Source: https://docs.elementary-data.com/cloud/integrations/alerts/jira
Elementary's Jira integration enables creating Jira issues from incidents.
## How to connect Jira
1. Go to the `Environments` page on the sidebar.
2. Select an environment and click connect on the `Connect ticketing system` card, and select `Jira`.
3. Authorize the Elementary app for your workspace. **This step may require a workspace admin approval.**
4. Select a default project for tickets
5. Click `Save` to finish the setup
Elementary Jira tickets include the basic ticket fields, test information, asset information and metadata. When connecting Jira, please make sure there's no required field that Elementary doesn't provide.
## Creating Jira issues from incidents
When an incident is created, you can generate a Jira issue directly from the incident page by clicking **Create Jira Ticket**. This opens a pre-filled form where you can review, update, or add additional fields before submitting. Once created, the issue is automatically added to the Jira team you selected when connecting Jira.
After the ticket is created, a link to the Jira issue appears on the incident page. The Jira ticket itself also includes a link back to the incident in Elementary for easy cross-referencing.
If you connected the app before December 2025, you’ll need to re-authenticate the Jira app to change the **Reporter** or **Assignee** fields. You can do this by clicking the re-authentication link in the ticket form, or by going to your environment page and re-authenticating the app for the relevant environment.Without a Jira webhook, Elementary does not automatically update the linked Jira issue when you resolve or change an incident. Use the webhook setup below for two-way status sync.
## Configure a Jira webhook for two-way sync
When a webhook is configured, Jira and Elementary stay aligned in real time: for example, closing the Jira issue can close the linked Elementary incident, and resolving the incident can update the Jira issue accordingly. Follow these steps in your Jira instance (Jira admin access required).
### Step 1: Open Jira webhook settings
1. Log in to Jira as an **Admin**.
2. Click the **Settings** (gear) icon in the top right.
3. Under Jira admin settings, select **System**.
4. In the left sidebar, scroll to the bottom and click **WebHooks**.
5. Click **Create a WebHook** (top right).
### Step 2: Fill in the webhook details
| Field | What to enter |
| ---------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Name** | A clear label (for example, `Elementary Sync`). |
| **Status** | **Enabled**. |
| **URL** | Paste the unique webhook URL shown in the Elementary product (environment / Jira integration UI). |
| **Secret** | Generate the secret in Elementary, copy it, and paste it into the **Secret** field in Jira so payloads can be verified. |
### Step 3: Select events
1. In the **Events** section, open the **Issue** category.
2. Enable **updated** only.
3. Leave all other event checkboxes unchecked.
Issue updates are enough for Elementary to receive the changes needed to keep tickets and incidents in sync.
### Step 4: Scope to your project (recommended)
Limit traffic to the projects you care about using a JQL filter:
1. Find the **JQL filter** field on the webhook form.
2. Enter:
```text theme={null}
project = "YOUR_PROJECT_KEY"
```
Replace `YOUR_PROJECT_KEY` with your Jira project key (for example, `PROJ`).
### Step 5: Save and verify
1. Click **Create** at the bottom of the page.
2. After Jira delivers its first event, the webhook status in the Elementary dashboard turns **green**, which indicates a successful connection.
# Linear
Source: https://docs.elementary-data.com/cloud/integrations/alerts/linear
Elementary's Linear integration enables creating Linear tickets from incidents.
## How to connect Linear
1. Go to the `Environments` page on the sidebar.
2. Select an environment and click connect on the `Connect ticketing system` card, and select `Linear`.
3. Authorize the Elementary app for your workspace. **This step may require a workspace admin approval.**
4. Select a default team for new tickets
5. Click `Save` to finish the setup
## Creating Linear ticket from incidents
When an incident is created, you can create a Linear ticket from the incident page by simply clicking on "Create Linear Ticket".
The ticket will automatically be created in Linear, in the team you chose upon connecting Linear.
After the ticket is created you can see the Linear ticket link in the incident page.
The ticket will also contain a link to the incident in Elementary.
Note: Elementary will not update the ticket in Linear when the incident is resolved or changed in any way
# Microsoft Teams (Legacy webhook)
Source: https://docs.elementary-data.com/cloud/integrations/alerts/ms-teams
The webhook integration is deprecated. For new setups, use the
[Microsoft Teams App](/cloud/integrations/alerts/ms-teams-app) instead —
it's easier to set up, requires no per-channel webhooks, and supports
richer alert cards.
Elementary's Microsoft Teams webhook integration enables sending alerts when data issues happen.
The alerts are sent using Adaptive Cards format, which provides rich formatting and interactive capabilities.
The alerts include rich context, and you can create [alert rules](/cloud/features/alerts-and-incidents/alert-rules) to distribute alerts to different channels and destinations.
## Enabling Microsoft Teams alerts
1. Go to the `Environments` page on the sidebar.
2. Select an environment and click connect on the `Connect messaging app` card (first card), and select `Microsoft Teams`.
3. For each MS Teams channel you want to connect, create a webhook using Power Automate. Power Automate is Microsoft's recommended way to receive webhook requests in Teams. You can set it up in two ways — both produce the same webhook URL:
**Option A — directly from Teams (faster):**
1. Go to your Teams channel
2. Click the three dots (...) next to the channel name
3. Select `Workflows`
4. Choose the template "Post to channel when a webhook request is received"
5. Copy the webhook URL
**Option B — from the Power Automate website:**
1. Go to [Power Automate](https://flow.microsoft.com)
2. Create a new instant cloud flow
3. Search for "When a HTTP request is received" as your trigger
4. Add a "Post adaptive card in a chat or channel" action
5. Configure the team and channel where you want to post
6. Save the flow and copy the HTTP POST URL
Elementary cannot directly verify if messages were successfully delivered via Power Automate. Monitor your workflow runs in Power Automate to check for any errors.
4. Configure your Microsoft Teams webhooks and give each one a name indicating its connected channel:
5. Select a default channel for alerts and set the suppression interval.
The default channel you select will automatically add a default [alert rule](/cloud/features/alerts-and-incidents/alert-rules) to send all failures to this channel. Alerts on warnings are not sent by default. To modify and add rules, navigate to the `Alert Rules` page.
## Microsoft 365 Connectors
Microsoft 365 Connectors (the legacy incoming webhook method) are no longer supported by Microsoft. If you previously connected Elementary using a Connector-based webhook, you'll need to recreate it using Power Automate Workflows above.
See [Microsoft's retirement announcement](https://devblogs.microsoft.com/microsoft365dev/retirement-of-office-365-connectors-within-microsoft-teams/) for more details.
# Microsoft Teams
Source: https://docs.elementary-data.com/cloud/integrations/alerts/ms-teams-app
Elementary's Microsoft Teams app sends alerts when data issues happen, as
[Adaptive Cards](https://adaptivecards.io/) with rich context. The app is
installed once per team and Elementary posts to any standard channel in that
team.
You can create [alert rules](/features/alerts-and-incidents/alert-rules) to
distribute alerts to different channels and destinations.
## Connecting the Microsoft Teams app
Install the [Elementary app from the Microsoft Teams App Store](https://marketplace.microsoft.com/en-us/product/teams-app/WA200011427)
and add it to the team where you want to receive alerts.
Once the bot is installed in a team, you should see this card.
Click **Connect to Elementary** to link this workspace to your
Elementary account.
This opens **Elementary** and connects the workspace automatically.
Connecting requires an Elementary user with the **Edit integrations**
permission — usually a user with the **Admin** role.
Once complete, you should see both:
**In Elementary** — a dialog confirming the workspace is connected.
Pick a **Default channel** to start receiving alerts right away.
Setting a default channel automatically adds a default
[alert rule](/features/alerts-and-incidents/alert-rules) that sends all
failures to that channel. Alerts on warnings are not sent by default — to
refine this, go to the `Alert Rules` page.
**In Microsoft Teams** — a confirmation card in the channel:
You can regenerate the connection link any time by typing
**`@Elementary connect`** in the channel.
Use [alert rules](/features/alerts-and-incidents/alert-rules) to control
which alerts go to which channels — for example, routing test failures to
one channel and freshness issues to another.
## Bot commands
**In Microsoft Teams**, mention the bot in any channel it's installed in:
* **`@Elementary help`** — what Elementary can do and how to get started.
* **`@Elementary connect`** — a fresh link to connect the team to your Elementary account.
## Supported channels
The app posts to **standard channels** in the team it's installed in. Private
and shared channels are not supported.
## Disconnecting
You can disconnect from either side:
* **From Elementary** — go to `Environments` → `Connect messaging app`, open the
**Microsoft Teams App** menu, and choose **Disconnect**. This unlinks the
workspace from your Elementary account and stops alerts. The app stays
installed in Teams — remove it there too if you want to fully uninstall.
* **In Microsoft Teams** — remove the Elementary app from the team
(`Manage team` → `Apps` → remove **Elementary**). This uninstalls the app, and
the integration then shows as disconnected **in Elementary**.
Either way, you can reconnect later from the welcome message or by typing
**`@Elementary connect`** in the channel.
Each Teams team can be connected to a single Elementary account. If a team is
already connected to a different account, connecting it again is rejected.
# Opsgenie
Source: https://docs.elementary-data.com/cloud/integrations/alerts/opsgenie
Elementary's Opsgenie integration enables sending alerts when data issues happen.
It is recommended to create [alert rules](/cloud/features/alerts-and-incidents/alert-rules) to filter and select the alerts that will create incidents in Opsgenei.
## Enabling Opsgenie alerts
### Create an Opsgenie API key
To create an `Opsgenie API key`, go to `Opsgenie` and follow the following steps:
* Create or select an `Opsgenie` team - this team will be responsible for alerts generated by Elementary.
* On the selected team go `Integrations` tab and press on `Add Integration`:
* Select `API` and press `Add`
* Select a name for the `API integration` - “Elementary” for example
* Make sure `Create` and `update access` are selected
* Press on `Save Integration`
* Copy the `API key` and provide it to Elementary UI.
### Add API key to an environment
1. Go to the `Environments` page on the sidebar.
2. Select an environment and click connect on the `Connect incident management tool` card (second card), and select `Opsgenie`.
3. Fill the `API key` and select the `API URL` and save the integration:
4. `Opsgenie` will now be available as a destination on the [`alert rules`](/cloud/features/alerts-and-incidents/alert-rules) page. You can add rules to create Opsgenie incidents out of alerts who match your rule.
# PagerDuty
Source: https://docs.elementary-data.com/cloud/integrations/alerts/pagerduty
Elementary's PagerDuty integration enables sending alerts when data issues happen.
It is recommended to create [alert rules](/cloud/features/alerts-and-incidents/alert-rules) to filter and select the alerts that will create incidents in PagerDuty.
## Enabling PagerDuty alerts
Important: The user connecting PagerDuty must have at least a "User" role (not "Limited User") in PagerDuty, as this role is required add "Events API v2" integrations to services.
1. Go to the `Environments` page on the sidebar.
2. Select an environment and click connect on the `Connect incident management tool` card (second card), and select `PagerDuty`.
3. Authorize the Elementary for your account. **This step may require admin approval.**
4. `PagerDuty` will now be available as a destination on the [`alert rules`](/cloud/features/alerts-and-incidents/alert-rules) page. You can add rules to create PagerDuty incidents out of alerts who match your rule.
# ServiceNow
Source: https://docs.elementary-data.com/cloud/integrations/alerts/servicenow
Elementary's ServiceNow integration enables creating ServiceNow incidents from incidents.
## Create a ServiceNow app (Admin required)
1. Connect to your ServiceNow instance
2. Navigate to the `System OAuth` -> `Application Registry` page
3. Create a new application by clicking `New` -> `Create an OAuth API endpoint for external clients`
4. Set the following fields:
* Name: `Elementary` (Recommended)
* Redirect URL: `https://prod.nango.elementary-data.com/oauth/callback`
* Logo URL: `https://cdn.prod.website-files.com/65ec9eebc532cfda4df5fcd7/6781525ba8657fba8fa35446_Vector.svg` (Optional)
5. Save the application
6. Go to the created application and copy the `Client ID` and `Client Secret`
## Add Elementary Link to the incidents table (Recommended)
1. Navigate to `System Definitions` -> `Table`
2. Select the `Incident` table
3. Under `Columns`, click `New`
4. Set the following fields:
* Type: `URL`
* Column Label: `Elementary Link`
* Column Name: `u_elementary_incident_url`
5. Click `Submit`
## How to connect your ServiceNow app
1. Go to the `Environments` page on the sidebar.
2. Select an environment and click connect on the `Connect ticketing system` card, and select `ServiceNow`.
3. Fill in the app details:
* Client ID: Client ID from the app you created in the previous step
* Client Secret: Client Secret from the app you created in the previous step
* Subdomain: The subdomain of your ServiceNow instance (e.g. for `https://my-company.service-now.com`, the subdomain is `my-company`)
4. Click `Connect`
5. Allow the app to access your ServiceNow instance. **This step may require admin approval.**
6. Choose a group to assign the incidents to
7. Click `Save`
## Creating ServiceNow incident from incidents
When an incident is created, you can create a ServiceNow incident from the incident page by simply clicking on "Create ticket".
The incident will automatically be created in ServiceNow, in the group you chose upon connecting ServiceNow.
After the ticket is created you can see the ServiceNow incident link in the incident page.
The incident will also contain a link to the incident in Elementary.
Note: Elementary will not update the incident in ServiceNow when the incident is resolved or changed in any way
# Slack
Source: https://docs.elementary-data.com/cloud/integrations/alerts/slack
Elementary's Slack integration enables sending Slack alerts when data issues happen.
The alerts include rich context, and you can change the incident status and asssigne from the alert itself.
You can also create [alert rules](/cloud/features/alerts-and-incidents/alert-rules) to distribute alerts to different channels and destinations.
## Enabling Slack alerts
1. Go to the `Environments` page on the sidebar.
2. Select an environment and click connect on the `Connect messaging app` card (first card), and select `Slack`.
4. Authorize the Elementary app for your workspace. **This step may require a workspace admin approval.**
5. Select a default channel for alerts, and set the suppression interval.
The default channel you select will automatically add a default [alert rule](/cloud/features/alerts-and-incidents/alert-rules)
to sends all failures to this channel. Alerts on warnings are not sent by default. To modify and add tules, navigate to `Alert Rules` page.
## Alerts to private channels
If the channel you want to send alerts to is private (🔒), it will not appear in the channels dropdown on the onboarding or the alert rules screen.
You will need to invite the Elementary bot to the private channel by typing `@Elementary` in the channel and clicking to invite the bot in, and then it will appear in the UI.
# Automated lineage to Data Visualization layer
Source: https://docs.elementary-data.com/cloud/integrations/bi/connect-bi-tool
Elementary will automatically and continuously extend the column-level-lineage to the dashboard level of your data visualization tool.
This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures.
### Why is end-to-end data lineage useful?
* **Incidents impact analysis**: You could explore which exposures are impacted by each data issue.
* **Exposure health**: By selecting an exposure and filtering on upstream nodes, you could see the status of all it’s upstream datasets.
* **Prioritize data issues**: Prioritize the triage and resolution of issues that are impacting your critical downstream assets.
* **Change impact**: Analyze which exposures will be impacted by a planned change.
* **Unused datasets**: Detect datasets that no exposure consumes, that could be removed to save costs.
### Supported BI tools
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
>
Click for details
}
>
Click for details
}
>
Click for details
### Automated dbt exposures.yml
Elementary can open PRs to your dbt project, to update your exposures yml files.
This means the lineage to the dashboards will also be visible in dbt docs and dbt cloud.
Why is it useful?
* **dbt docs enrichment** - As the exposures are created in code, they will be made available in your dbt docs lineage as well.
* **dbt cloud change impact** - If you develop in dbt cloud, you could visualize in your IDE the downstream exposures Elementary generates, and understand the impact of changes.
# Explo
Source: https://docs.elementary-data.com/cloud/integrations/bi/explo
}
>
Click for details
# Hex
Source: https://docs.elementary-data.com/cloud/integrations/bi/hex
After you connect Hex, Elementary will automatically and continuously extend the lineage to the project & cell level.
This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures.
### Prerequisites
The Hex integration relies on **query history** to discover which tables Hex projects query. It is supported on **BigQuery** and **Snowflake**.
Elementary's service account must have permissions to read query history. Refer to your warehouse's setup page for the required roles:
* [BigQuery permissions](/cloud/integrations/dwh/bigquery#grant-service-user-access)
* [Snowflake permissions](/cloud/integrations/dwh/snowflake)
If Elementary cannot read query history, the Hex integration will not discover any exposures.
### Create API Token
Elementary needs a workspace token in your account in order to access Hex's API on your behalf.
To create one, please follow the [official Hex documentation](https://learn.hex.tech/docs/api/api-overview#workspace-tokens).
Make sure you create a Hex Workspace token with read access for all categories and "read project queried tables".
### Connecting Hex to Elementary
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect Hex.
Choose the Hex connection and provide the following details to validate and complete the integration.
* **Base URL**: Your Hex workspace URL. For example: `https://app.hex.tech/my-workspace/`
* **Workspace Token**: The Hex workspace token you've created on the previous step.
### Hex queries on a different warehouse environment
By default, Elementary reads query history from the warehouse environments (e.g., BigQuery projects, Snowflake accounts) referenced in your dbt project. If Hex executes queries through a **different environment** (e.g., a dedicated Hex execution environment), you will need to:
1. **Grant query history permissions** — the same permissions required for your primary environment. See the [prerequisites](#prerequisites) above for links to your warehouse's permissions page.
2. **Let us know** — contact Elementary support with the environment details so we can configure Elementary to also read query history from it.
Without both steps, Elementary will not find any Hex queries in that environment and no Hex exposures will appear.
### Limitations
* **Hex assets will only be visible if their project has run within the last 14 days.** Elementary tracks Hex query history to determine asset visibility, and only projects with recent execution activity will appear in the lineage graph.
# Lightdash
Source: https://docs.elementary-data.com/cloud/integrations/bi/lightdash
After you connect Lightdash, Elementary will automatically and continuously extend the column-level-lineage to the dashboard and chart level.
This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures.
### Create a Personal Access Token
Elementary needs a Personal Access Token (PAT) to access the Lightdash API on your behalf.
1. In Lightdash, go to **Settings > Personal Access Tokens**.
2. Click **Generate Token**.
3. Give the token a descriptive name (e.g. "Elementary integration").
4. Copy and save the generated token securely — you will need it when connecting Lightdash to Elementary.
For more details, refer to the [official Lightdash documentation](https://docs.lightdash.com/references/workspace/personal-tokens).
The token inherits the permissions of the user who created it. Make sure the user has access to all the projects you want Elementary to sync.
### Connecting Lightdash to Elementary
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect Lightdash.
Choose the Lightdash BI connection and provide the following details to validate and complete the integration.
* **Connection Name:** A descriptive name for this Lightdash connection (e.g. "My Lightdash").
* **API Token:** The Personal Access Token you generated in the previous step.
* **Host URL** *(optional)*: Leave empty to use the default Lightdash Cloud URL (`https://app.lightdash.cloud`). Set this only if you are using a self-hosted Lightdash instance (e.g. `https://lightdash.mycompany.com`).
* **Project UUIDs** *(optional)*: Leave empty to sync all projects in your Lightdash organization. Add specific project UUIDs to sync only those projects. You can find a project's UUID in the URL when viewing it in Lightdash (e.g. `https://app.lightdash.cloud/projects//...`).
# Looker
Source: https://docs.elementary-data.com/cloud/integrations/bi/looker
After you connect Looker, Elementary will automatically and continuously extend the column-level-lineage to the dashboard level.
This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures.
### Create permission set and role
To follow the [least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) security principle, before creating user for Elementary, create a permission set that allows to do only what is needed for Elementary to work.
* **Create a Permission Set**
1. In Looker, go to Admin > Roles. Click **"New Permission Set"**.
2. Name the Permission Set "Elementary permissions" and **check the boxes for the following permissions:**
* `access_data`
* `see_lookml_dashboards`
* `see_looks`
* `see_user_dashboards`
* `see_sql`
* `see_lookml`
* `develop`
* **Create a role**
1. **Click "New Role".**
2. Name the role "Elementary Role" as well.
3. **Select the "**Elementary permissions**" permission set**
4. **Create the role.**
### Create a dedicated Elementary Looker user
In Looker, go to Admin > Users.
1. Click **Add Users**. Provide any email you want.
2. Under Roles, **check "Elementary Role".**
3. Click **Done.**
### Create an API key for the user
1. Find the user you just created and click **Edit.**
2. You can give this user a name like **Elementary**.
3. Under **API Keys**, click **Edit Keys**.
4. Generate a new API key by clicking **New API Key**.
5. Copy down the **client ID** and **client secret**.
### Connecting Looker to Elementary
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect Looker.
Choose the Looker BI connection and provide the following details to validate and complete the integration.
* **Host URL:** Copy and paste the URL you see when you visit the homepage of your Looker instance. e.g. [https://company.cloud.looker.com/](https://prod-useast-a.online.tableau.com/#/site/dev/projects)
* **Client ID**: The client ID you generated on Looker.
* **Client secret:** The client secret you generated on Looker.
* **LookML code repository**:
* Token
* [Github](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic)
* [Gitlab](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) - make sure the role is `developer` and the scopes include `read_api, read_repository`
* Repository - The repository name where your LookML code is.
# Metabase
Source: https://docs.elementary-data.com/cloud/integrations/bi/metabase
}
>
Click for details
# Mode
Source: https://docs.elementary-data.com/cloud/integrations/bi/mode
}
>
Click for details
# Power BI
Source: https://docs.elementary-data.com/cloud/integrations/bi/power-bi
After you connect Power BI, Elementary will automatically and continuously extend the lineage to the report/dashboard level.
This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures.
### Create & authorize a Service Principal
Elementary makes use of a service principal secret to authenticate with the Power BI API on your behalf.
Please follow the [official guide by Microsoft](https://learn.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal) for creating and authorizing a service principal for Power BI.
Make sure you're going through all steps up until but not including "Step 5 - Embed your content" (Which isn't currently our goal).
Also make sure that the security group you've created is allowed access to every relevant Workspace in your account.
Do NOTICE - "My Workspace" isn't accessible by service principals and thus can't be analyzed with our product.
### Allow Power BI's Admin API
Go to Power BI's Admin portal -> Tenant Settings -> Admin API settings
Enable the following features, and apply at least for the Service principal's security group (created on the last step):
* `Service principals can access read-only admin APIs`
* `Enhance admin APIs responses with detailed metadata`
* `Enhance admin APIs responses with DAX and mashup expressions`
Those features will allow Elementary to get all required info for computing the data lineage graph.
### Connecting Power BI to Elementary
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect Power BI.
Choose the Power BI connection and provide the following details to validate and complete the integration.
* **Tenant:** Your Microsoft tenant which is usaully your company's domain. e.g. `my-company.com`
* **Client ID**: The client ID of the new Microsoft Entra app you've created on the first step.
* **Client Secret:** The new client secret you've created on the first step.
### Limitations
* Due to service principal limitations - we cannot analyze the default "My Workspace" Workspace.
* The lineage currently doesn't apply to the following entities: (But would be in the future)
* Paginated Reports
* Dataflows
* Datamarts
# Sigma
Source: https://docs.elementary-data.com/cloud/integrations/bi/sigma
After you connect Sigma, Elementary will automatically and continuously extend the lineage to the workbook page & element level.
This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures.
### Create API Client Credentials
Elementary needs authorized client credentials in your account in order to access Sigma's API on your behalf.
To create those, Please follow the [official Sigma documentation](https://help.sigmacomputing.com/reference/generate-client-credentials#generate-api-client-credentials).
Make sure you enable 'REST API' privileges for that client.
### Connecting Sigma to Elementary
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect Sigma.
Choose the Sigma connection and provide the following details to validate and complete the integration.
* **Cloud Provider:** To determine your Sigma cloud provider, Navigate to **Account -> General Settings** under Sigma's **Administration** menu and look for **'Cloud: ...'**. Should be one of the following:
* `AWS US`
* `AWS Canada`
* `AWS Europe`
* `AWS UK`
* `Azure US`
* `GCP`
* **Client ID**: The Sigma client ID you've created on the previous step.
* **Client Secret:** The new Sigma client secret you've created on the previous step.
### Limitations
`Datasets` or `Data Models` are currently excluded from computed lineage graph - which will point from DWH directly to your Workbook Elements.
# Tableau
Source: https://docs.elementary-data.com/cloud/integrations/bi/tableau
After you connect Tableau, Elementary will automatically and continuously extend the lineage to the dashboard level.
This will provide end-to-end data lineage to help you understand your downstream dependencies, called exposures.
In order for Elementary to extract your metadata from Tableau you must meet all of the Tableau Metadata GraphQL requirements (most are set by default):
* Tableau Cloud / Server 2019.3 or later.
* Tableau REST API must not be disabled.
* The Metadata API must be enabled.
* The Metadata API must be accessed via an access token.
## Tableau Cloud
### Creating a Personal Access Token
Create a Personal Access Token in Tableau. For details on how to create a user token please refer to the **[Tableau guide](https://help.tableau.com/current/pro/desktop/en-us/useracct.htm#create-a-personal-access-token)**.
When you generate this token, **Tableau will display a token secret only once**.
Make sure you store this token securely for future use, you will need it when you connect Tableau to Elementary.
### Connecting Tableau to Elementary
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect Tableau.
Choose the Tableau BI connection and provide the following details to validate and complete the integration.
* **Token Name:** The name of the token you created in Tableau.
* **Token Value**: The token secret you saved when generating the token in Tableau
* **Site information:** The site URL and name can be found on the URL you see when you visit the homepage of your Tableau instance. e.g. `https://prod-useast-a.online.tableau.com/#/site/elementary/projects`
* **Site URL:** Copy and paste the base URL. e.g. `https://prod-useast-a.online.tableau.com/`
* **Site Name**: Copy and paste the site name. e.g. `elementary`
## Tableau Server
Elementary supports Tableau Server, reach out to our team for details.
# Thoughtspot
Source: https://docs.elementary-data.com/cloud/integrations/bi/thoughtspot
After you connect Thoughtspot, Elementary will automatically and continuously extend the lineage to the liveboard and answer level.
This will provide you end-to-end data lineage to understand your downstream dependencies, called exposures.
### Enable Trusted Authentication on a privileged user
For Elementary to access your Thoughtspot instance's API on your behalf of your user, your user should have Trusted Authentication enabled.
To enable Trusted Authentication on a user, please follow the [official Thoughtspot documentation](https://developers.thoughtspot.com/docs/trusted-auth-secret-key).
Make sure you copy the generated token (`Secret Key`) as you will need it to connect Thoughtspot to Elementary.
### User Privileges
For an easy integration, it's recommended for the connected user to be an administrator (`ADMINISTRATION` privilege), this will ensure Elementary can access all of your Liveboards and Answers.
It is also possible though to integrate with a regular user, just make sure it can download data (has `DATADOWNLOADING` privilege) for all the relevant ThoughtSpot entities you want Elementary to discover and show lineage for.
### Connecting Thoughtspot to Elementary
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect ThoughtSpot.
Choose the Thoughtspot connection and provide the following details to validate and complete the integration.
* **User Name:** The username of the user you want to use to connect to Thoughtspot.
* **Secret Key:** The token generated for the user you want to use to connect to Thoughtspot (from the previous step).
* **Base URL:** The URL of your Thoughtspot instance. This would be `'https://.thoughtspot.cloud'` by default, or your custom domain if [you've configured one](https://docs.thoughtspot.com/cloud/10.1.0.cl/custom-domains#_domain_url_customization). If you're unsure, just check the URL you use to access your Thoughtspot instance in the browser.
### Limitations
`Worksheets` or `Models` are currently excluded from computed lineage graph - which will point from DWH directly to your Liveboards or Answers.
# Azure DevOps Integration
Source: https://docs.elementary-data.com/cloud/integrations/code-repo/azure-devops
Elementary can integrate with Azure DevOps to connect to the code repository where your **dbt project code** is managed, and it opens pull requests with configuration changes.
## Connecting Through the Azure DevOps App
1. Navigate to **Environments** in Elementary Cloud, open your environment, then go to **Code repository**.
2. Click on **Connect** and select **Azure DevOps**.
3. Enter your Azure DevOps organization URL \
(e.g., `https://dev.azure.com/your-organization`).
4. Click **Save**.
5. Connect through OAuth to authenticate between Azure DevOps and Elementary Cloud. During this process, a temporary token is issued, which can be used to make API calls. Along with the temporary token, a refresh token is also provided. The refresh token is used when Azure DevOps indicates that the temporary token has expired. For Microsoft services, OAuth is managed by Microsoft Entra ID (formerly known as Active Directory).
## Updating tokens when they expire
When your Azure DevOps token expires, update the connection with new tokens:
1. Go to **Environments**, open your environment, and go to **Code repository**.
2. Click **Edit** on the code repository connection.
3. Paste the new generated tokens (from Azure DevOps) and save.
***
## Required Permissions
Elementary requires the following permissions in your Azure DevOps **dbt repository**:
* **Read and write** access to the repository
* Access to **file contents**
* Permission to **open and read pull requests**
## Project repository settings
Repository settings are configurable per project. You can set **labels**, **commit message prefix**, **branch prefix**, and **ticket ID requirements** for pull requests Elementary opens.
When **required ticket ID** is enabled, every pull request Elementary opens must include a ticket ID. The ticket ID is prepended to the branch name and the pull request title.
***
## Troubleshooting
If you encounter issues with the Azure DevOps integration, ensure the following:
1. Your **organization URL** is correct.
2. You have **sufficient permissions** in Azure DevOps.
3. Elementary is properly **authorized** in your Azure DevOps organization.
# Bitbucket
Source: https://docs.elementary-data.com/cloud/integrations/code-repo/bitbucket
Elementary connects to the code repository where your dbt project code is managed, and opens PRs with configuration changes.
## Recommended: Connect using Elementary Bitbucket App
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect the dbt project code repository.
Simply Click the blue button that says "Connect with Elementary Bitbucket App" and follow the instructions.
In the menu that opens up later on, select the repository where your dbt project is stored, and if needed the branch and path to the dbt project.
Requires a user with permissions to install new applications in the repository
### Alternative: Create a Bitbucket project token
If connecting the Elementary Bitbucket App isn't an option, you can connect using a token managed by your team instead.
## Repository connection settings
After the authentication, you need to fill in the following details:
* **Repository** - The full name of the code repo.
* *Optional* **Environment base branch** - If you want Elementary to open PRs in a target branch different than default, detail the branch name here.
* *Optional* **Project path** - If your dbt project isn't on the root directory of the repo, detail it's path here.
* *Optional* **Update token** - When the github token expires, regenerate a fine-grained token and paste it here.
## Project repository settings
Repository settings are configurable per project. You can set **labels**, **commit message prefix**, **branch prefix**, and **ticket ID requirements** for pull requests Elementary opens.
When **required ticket ID** is enabled, every pull request Elementary opens must include a ticket ID. The ticket ID is prepended to the branch name and the pull request title.
# Connect code repository
Source: https://docs.elementary-data.com/cloud/integrations/code-repo/connect-code-repo
We believe [configuration should be managed in code](/cloud/features/config-as-code).
With config-as-code you get: version control, CI, review process.
Adding and updating configuration becomes part of your development process.
Through integration and access to the dbt project code repository, Elementary opens pull requests to your dbt project.
## Project repository settings
Repository settings are configurable per project. You can set **labels**, **commit message prefix**, **branch prefix**, and **ticket ID requirements** for pull requests Elementary opens.
When **required ticket ID** is enabled, every pull request Elementary opens must include a ticket ID. The ticket ID is prepended to the branch name and the pull request title.
### Supported code repositories
}
/>
}
/>
}
/>
}
/>
# Github
Source: https://docs.elementary-data.com/cloud/integrations/code-repo/github
Elementary connects to the code repository where your dbt project code is managed, and opens PRs with configuration changes.
## Recommended: Connect using Elementary Github App
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect the dbt project code repository.
Simply Click the blue button that says "Connect with Elementary Github App" and follow the instructions.
In the menu that opens up later on, select the repository where your dbt project is stored, and if needed the branch and path to the dbt project.
Requires a user with permissions to install new applications in the repository.
## Alternative: Create a Github [fine-grained token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token)
If connecting the Elementary Github App isn't an option, you can connect to Github using a fine-grained token managed by your team instead.
1. In the upper-right corner of any page, click your profile photo, then click **Settings**.
2. On the bottom of the left sidebar, click **Developer settings**.
3. On the left sidebar, select **Personal access tokens > Fine-grained tokens**.
4. Click **Generate new token**.
5. Enter a token name, expiration (we recommend 90 days) and description.
6. Under **Resource owner** select the owner of the dbt project repo. If it is missing, you need to contact the resource owner administrator and ask him to approve fine-grained personal access tokens. Read more in For more information, see ["Setting a personal access token policy for your organization"](https://docs.github.com/en/organizations/managing-programmatic-access-to-your-organization/setting-a-personal-access-token-policy-for-your-organization).
7. Under **Repository access** select **Only selected repositories** and select the repo of your dbt project (see picture).
8. Under **Permissions**, grant the following scopes:
* *Pull Requests* (read and write)
* *Contents* (read and write)
9. Click **Generate token**.
## Repository connection settings
After the authentication, you need to fill in the following details:
* **Repository** - The full name of the code repo.
* *Optional* **Environment base branch** - If you want Elementary to open PRs in a target branch different than default, detail the branch name here.
* *Optional* **Project path** - If your dbt project isn't on the root directory of the repo, detail it's path here.
* *Optional* **Update token** - When the github token expires, regenerate a fine-grained token and paste it here.
## Project repository settings
Repository settings are configurable per project. You can set **labels**, **commit message prefix**, **branch prefix**, and **ticket ID requirements** for pull requests Elementary opens.
When **required ticket ID** is enabled, every pull request Elementary opens must include a ticket ID. The ticket ID is prepended to the branch name and the pull request title.
# Gitlab
Source: https://docs.elementary-data.com/cloud/integrations/code-repo/gitlab
Elementary connects to the code repository where your dbt project code is managed, and opens PRs with configuration changes.
## Recommended: Connect using Elementary Gitlab App
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect the dbt project code repository.
Simply Click the blue button that says "Connect with Elementary Gitlab App" and follow the instructions.
In the menu that opens up later on, select the repository where your dbt project is stored, and if needed the branch and path to the dbt project.
Requires a user with permissions to install new applications in the repository
### Alternative: Create a Gitlab project token
If connecting the Elementary Gitlab App isn't an option, you can connect using a token managed by your team instead.
You need to create a [project access token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) (token for a specific repository) with by following these steps:
1. On the left sidebar, at the top, select **Search GitLab** to find the dbt project repo.
2. Select **Settings > Access Tokens**.
3. Enter a name and expiry date for the token (default is 1 year).
4. Select the role **Maintainer** for the token.
5. Select the following scopes: `api`, `read_api`, `read_repository`, `write_repository`.
6. Select **Create project access token**.
## Repository connection settings
After the authentication, you need to fill in the following details:
* **Repository** - The full name of the code repo.
* *Optional* **Environment base branch** - If you want Elementary to open PRs in a target branch different than default, detail the branch name here.
* *Optional* **Project path** - If your dbt project isn't on the root directory of the repo, detail it's path here.
* *Optional* **Update token** - When the github token expires, regenerate a fine-grained token and paste it here.
## Project repository settings
Repository settings are configurable per project. You can set **labels**, **commit message prefix**, **branch prefix**, and **ticket ID requirements** for pull requests Elementary opens.
When **required ticket ID** is enabled, every pull request Elementary opens must include a ticket ID. The ticket ID is prepended to the branch name and the pull request title.
# Connect to Athena
Source: https://docs.elementary-data.com/cloud/integrations/dwh/athena
This guide contains the necessary steps to connect an Athena environment to your Elementary account.
### Create a user for Elementary cloud
On your dbt project, run:
```bash theme={null}
## Print the query you should run to generate a user
dbt run-operation create_elementary_user
```
This command will generate a query to create a user with the necessary permissions. Run this query on your data warehouse with **admin permissions** to create the user.
#### Permissions and security
Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following:
* Read-only access to the Elementary schema.
* Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project.
It is recommended to create a user using the instructions specified above to avoid granting excess privileges.
For more details, refer to [security and privacy](/cloud/general/security-and-privacy).
## AWS Setup
### 1. Create Required IAM Policy
First, you'll need to create an IAM policy with the following permissions:
* **AthenaPermissions**: Allows executing and retrieving query results from Athena
* **GluePermissions**: Enables reading metadata about databases and tables
* **S3AccessForStagingBuckets**: Provides full access to store Athena query results
* **S3AccessForElementarySchema**: Grants read-only access to your elementary schema
Here is an example of a JSON policy:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AthenaPermissions",
"Effect": "Allow",
"Action": [
"athena:StartQueryExecution",
"athena:GetQueryExecution",
"athena:GetQueryResults"
],
"Resource": "*"
},
{
"Sid": "GluePermissions",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetTable",
"glue:GetTables",
"glue:GetTableVersions",
"glue:GetPartition",
"glue:GetPartitions"
],
"Resource": "*"
},
{
"Sid": "S3AccessForStagingBuckets",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::your-query-results-bucket",
"arn:aws:s3:::your-query-results-bucket/*"
]
},
{
"Sid": "S3AccessForElementarySchema",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-elementary-schema-bucket",
"arn:aws:s3:::your-elementary-schema-bucket/*"
]
}
]
}
```
### 2. Choose Authentication Method
Elementary supports two authentication methods for connecting to Athena:
#### Option 1: AWS Role Authentication (Recommended)
This is the recommended approach as it provides better security and follows AWS best practices. [Learn more about AWS IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html).
1. **Create an IAM Role**:
* Go to AWS IAM Console
* Create a new role
* Select "Another AWS account" as the trusted entity
* Enter Elementary's AWS account ID: `743289191656`
* (Optional but recommended) Enable "Require external ID" and set a value
* Attach the policy created in step 1
2. **Note down the following information**:
* Role ARN
* External ID (if you enabled it) [Learn more about external IDs](https://aws.amazon.com/blogs/security/how-to-use-external-id-when-granting-access-to-your-aws-resources/).
#### Option 2: Access Key Authentication
This method is less secure as it requires permanent credentials. We recommend using AWS Role authentication instead.
1. **Create an IAM User**:
* Go to AWS IAM Console
* Create a new user, that will be used by elementary to query athena
* Enable programmatic access
* Attach the policy created in step 1
2. **Note down the following information**:
* AWS Access Key ID of the new elementary athena user
* AWS Secret Access Key of the new elementary athena user
## Elementary Configuration
### Connection Settings
Regardless of the authentication method you choose, you'll need to provide:
* **Region**: The AWS region where your Athena instance is located
* **Database**: The name of the database where your Elementary schema exist.
* **Schema**: The name of your Elementary schema. Usually \[schema name]\_elementary
* **S3 Staging Directory**: The S3 path where Athena query results will be stored
* **Workgroup**: (Optional) Your Athena workgroup name
### Authentication Details
Based on your chosen authentication method:
#### If using AWS Role Authentication:
* Select "AWS Role" as the authentication method
* Enter your role ARN
* Enter your external ID (if you enabled it)
#### If using Access Key Authentication:
* Select "Access Key" as the authentication method
* Enter your AWS Access Key ID
* Enter your AWS Secret Access Key
### Need help with onboarding?
We can provide [support on Slack](https://elementary-data.com/community) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
# Connect to Bigquery
Source: https://docs.elementary-data.com/cloud/integrations/dwh/bigquery
This guide contains the necessary steps to connect a BigQuery environment to your Elementary account.
## Choose an authentication method
Elementary supports two authentication methods for BigQuery. Pick the one that fits your security model:
* **Service account** — create a service account, download its JSON key, and upload the key to Elementary. Simplest to set up.
* **Workload Identity Federation (WIF)** — Elementary authenticates from its AWS role through a federated identity. No long-lived credentials are stored in Elementary.
Select a tab below and follow the steps for your chosen method.
### Create service account:
1. In the Cloud Console, go to: IAM & Admin > Service Accounts
2. Click on 'CREATE SERVICE ACCOUNT'
3. Fill in the service account name ('elementary') and account description ('Elementary Data') and click 'CREATE AND
CONTINUE':
4. Now we need to configure the relevant permissions for this new service account.
Select the following role: BigQuery Job User (you will need to grant read access to specific datasets later).
5. The last step is optional, skip it and press done.
6) Press on the dots icon to the right of your screen for your new service account and select 'Manage keys':
7. Press on 'ADD KEY' and select 'Create new key':
8. Use the 'JSON' option radio button and press 'CREATE':
9. This will automatically generate and download a JSON file with your private key information for this service account.
This JSON file provides the credentials to programmatically connect and work with your BigQuery environment.
### Grant service user access
In order for a service user to work with Elementary Cloud, it requires the following permissions:
* Role **"BigQuery Data Viewer"** on your Elementary dataset.
* Roles **"BigQuery Metadata Viewer"** and **"BigQuery Resource Viewer"** at the **project level** for the project containing your dbt models and any external projects it references.
"BigQuery Metadata Viewer" and "BigQuery Resource Viewer" **must be granted at the project level** (via IAM & Admin), not at the dataset level. These roles include permissions such as `bigquery.jobs.listAll` which are required for reading query history and metadata from `INFORMATION_SCHEMA` — they cannot be granted on individual datasets.
#### Step 1: Grant project-level roles
These roles are required for Elementary to read query history (`INFORMATION_SCHEMA.JOBS`) and table metadata. Without them, BI integrations (such as Hex) that rely on query history will not function.
1. Go to [IAM & Admin](https://console.cloud.google.com/iam-admin/iam) in the Google Cloud Console.
2. Select the project that contains your dbt models.
3. Click **"Grant Access"** at the top.
4. Fill out the form:
* In the "New principals" field, enter the **email address** of the Elementary service account.
* Add the role **"BigQuery Metadata Viewer"** (`roles/bigquery.metadataViewer`).
* Click **"Add another role"** and add **"BigQuery Resource Viewer"** (`roles/bigquery.resourceViewer`).
* Click **"Save"**.
5. **Repeat for any additional GCP projects** referenced in your dbt sources or used by BI tools (e.g., Hex, Looker) for query execution.
#### Step 2: Grant dataset-level role on the Elementary dataset
This role provides read access to your Elementary schema data.
1. Go to your project in [BigQuery console](https://console.cloud.google.com/bigquery).
2. In the "Explorer" tab, find your Elementary dataset.
3. Click on the three dots icon next to the dataset name, then Share.
4. Click the "ADD PRINCIPAL" button on the top right corner.
5. Fill out the form:
* In the "New principals" textbox, write the **email address** of your service account.
* In the "Select a role" dropdown menu, choose **"BigQuery Data Viewer"**.
* Click "Save".
### Create a Workload Identity pool and provider:
1. In the Cloud Console, go to: IAM & Admin > Workload Identity Federation. Click 'CREATE POOL' (or select an existing pool), give it a name (e.g. `elementary-identity-pool`), and continue.
2. On the pool, click 'ADD A PROVIDER' and choose 'AWS'. Set the provider name to `Elementary AWS` and the AWS account ID to `743289191656` (Elementary's AWS account), then continue.
3. In the provider's attribute conditions, add the following condition. This is the Elementary role for GCP — only Elementary can authenticate through this provider. Then save the provider:
```
attribute.aws_role == "arn:aws:sts::743289191656:assumed-role/elementary-gcp-wif-role-prod-elementary-cloud"
```
### Grant BigQuery permissions to the identity pool principal:
4. Go to IAM & Admin > IAM and click 'GRANT ACCESS'. In 'New principals', paste the following principalSet. Replace `PROJECT_NUMBER` with your GCP project number and `POOL_NAME` with the pool you created above:
```
principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/attribute.aws_role/arn:aws:sts::743289191656:assumed-role/elementary-gcp-wif-role-prod-elementary-cloud
```
5. Assign the following project-level roles to the principalSet and save. You will grant `BigQuery Data Viewer` at the dataset level in the next step:
* **BigQuery Job User**
* **BigQuery Metadata Viewer**
* **BigQuery Resource Viewer**
### Grant principal access to specific datasets:
6. In the [BigQuery console](https://console.cloud.google.com/bigquery), find your Elementary dataset in the 'Explorer' tab. Click the three dots next to it > Share > ADD PRINCIPAL. Paste the same principalSet from step 4, select `BigQuery Data Viewer`, and save. If your dbt dataset is in a different project, also grant it `BigQuery Metadata Viewer` and `BigQuery Resource Viewer`.
### Download the credential configuration file:
7. Back in IAM & Admin > Workload Identity Federation, select your pool and click 'GRANT ACCESS' at the top.
8. Choose 'Grant access using federated identities (Recommended)' and click 'DOWNLOAD CONFIG'.
9. In the 'Configure your application' panel, select the provider you created (e.g. `Elementary AWS`) from the Provider dropdown, then click 'DOWNLOAD CONFIG' to download the JSON configuration file.
10. Upload this JSON file to Elementary Cloud in the connection form below.
#### Permissions and security
Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following:
* Read-only access to the Elementary schema.
* Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project.
It is recommended to create a user using the instructions specified above to avoid granting excess privileges.
For more details, refer to [security and privacy](/cloud/general/security-and-privacy).
For BigQuery, this translates to:
* **BigQuery Data Viewer** on the Elementary dataset (for reading Elementary schema data).
* **BigQuery Metadata Viewer** and **BigQuery Resource Viewer** at the **project level** (for reading `INFORMATION_SCHEMA.JOBS` query history and table metadata). These project-level roles are required for BI integrations like Hex that rely on query history.
### Fill the connection form
Use the **Authentication method** toggle at the top of the form to select either **Service account** or **Workload Identity Federation**, matching the method you set up above. The credentials upload field changes based on your selection:
* **Service account file** (Service account method): the service account JSON key file you downloaded.
* **WIF credential file** (Workload Identity Federation method): the external account JSON configuration file you downloaded from the identity pool.
Then provide the remaining fields:
* **Project**: The name of your BigQuery project.
* **Elementary dataset**: The name of your Elementary dataset. Usually `[dataset name]_elementary`.
* **Location**: Use this field to configure the location of BigQuery datasets as per [the BigQuery documentation](https://cloud.google.com/bigquery/docs/locations).
### Add the Elementary IP to allowlist
Elementary IP for allowlist: `3.126.156.226`
### Need help with onboarding?
We can provide [support on Slack](https://elementary-data.com/community) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
# Connect to Clickhouse
Source: https://docs.elementary-data.com/cloud/integrations/dwh/clickhouse
This guide contains the necessary steps to connect an Clickhouse environment to your Elementary account.
You will connect Elementary Cloud to Clickhouse for syncing the Elementary schema (created by the [Elementary dbt package](/data-tests/dbt/quickstart-package)).
### Create a user for Elementary cloud
On your dbt project, run:
```bash theme={null}
## Print the query you should run to generate a user
dbt run-operation create_elementary_user
```
This command will generate a query to create a user with the necessary permissions. Run this query on your data warehouse with **admin permissions** to create the user.
#### Permissions and security
Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following:
* Read-only access to the Elementary schema.
* Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project.
It is recommended to create a user using the instructions specified above to avoid granting excess privileges.
For more details, refer to [security and privacy](/cloud/general/security-and-privacy).
### Fill the connection form
Provide the following fields:
* **Host**: The hostname of your Clickhouse account to connect to. This can either be an IP address or a hostname.
* **Port**: The port of your Clickhouse account to connect to. This is usually `8123`.
* **Elementary schema**: The name of your Elementary schema. Usually `[schema name]_elementary`.
* **User**: The name of the for Elementary user.
* **Password**: The password associated with the provided user.
### Add the Elementary IP to allowlist
Elementary IP for allowlist: `3.126.156.226`
### Connect via SSH tunnel
Elementary supports connecting via SSH or reverse SSH tunnel. Reach out to our team for details and support in this deployment.
### Need help with onboarding?
We can provide [support on Slack](https://elementary-data.com/community) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
# Connect to Databricks
Source: https://docs.elementary-data.com/cloud/integrations/dwh/databricks
This guide contains the necessary steps to connect a Databricks environment to your Elementary account.
### Create service principal
1. Open your Databricks console, and then open your relevant workspace.
2. Click on your Profile icon on the right and choose Settings.
3. On the sidebar, click on *Identity and access*, and then under the *Service Principals* row click on *Manage*.
4. Click on the *Add service principal* button, choose "Add new" and give a name to the service principal. This will be used by Elementary Cloud
to access your Databricks instance.
5. Click on your newly created service principal, add the "Databricks SQL access" entitlement, and click Update. Also, please copy the
"Application ID" field as it will be used later in the permissions section.
6. Next, generate credentials for your service principal. Choose one of the following methods:
**Option A: Generate an OAuth secret (Recommended)**
On the service principal page, go to the *Secrets* tab and click *Generate secret*. Copy the **Client ID** (this is the same as the "Application ID" from step 5) and the generated **Client secret** — you will need both when configuring the Elementary environment.
OAuth secrets are the recommended authentication method. They enable short-lived token generation
with automatic refresh, providing better security than long-lived personal access tokens.
**Option B: Create a personal access token (legacy)**
In order to generate a personal access token for your service principal, you may first need to allow Token Usage for it.
To do so, go to the settings menu and choose Advanced -> Personal Access Tokens -> Permission Settings, then make sure the service principal is in the list.
Then, create a personal access token for your service principal. For more details, please click [here](https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-service-principals).
7. Finally, in order to enable Elementary's automated monitors feature, please ensure [predictive optimization](https://docs.databricks.com/aws/en/optimizations/predictive-optimization#enable-or-disable-predictive-optimization-for-your-account) is enabled in your account.
This is required for table statistics to be updated (Elementary relies on this to obtain up-to-date row counts)
### Permissions and security
#### Required permissions
Elementary cloud requires the following permissions:
* **Elementary schema read-only access** - This is required by Elementary to read dbt metadata & test results collected by the Elementary dbt package as a part of your pipeline runs.
This permission does not give access to your data.
* **System metadata access** - Elementary needs access to the `system.information_schema.tables`, `system.information_schema.columns`, `system.query.history` and `system.access.table_lineage` system tables.
This access is used to get metadata about existing tables and columns, and to power features such as column-level lineage and automated volume & freshness monitors.
* **Billing metadata access** - Elementary needs access to the `system.billing.usage` and `system.billing.list_prices`. This allows Elementary to monitor the warehouse cost and alert on it.
* **Storage read-only access** - See details below.
#### Grants SQL template
Please use the following SQL statements to grant the permissions specified above (you should replace the placeholders with the correct values):
```sql theme={null}
-- Grant read access on the elementary schema (usually [your dbt target schema]_elementary)
GRANT USE CATALOG ON CATALOG TO ``;
GRANT USE SCHEMA, SELECT ON SCHEMA TO ``;
-- Grant access to system tables
GRANT USE CATALOG ON CATALOG system TO ``;
GRANT USE SCHEMA ON SCHEMA system.information_schema TO ``;
GRANT USE SCHEMA ON SCHEMA system.query TO ``;
GRANT USE SCHEMA ON SCHEMA system.access TO ``;
GRANT SELECT ON TABLE system.information_schema.tables TO ``;
GRANT SELECT ON TABLE system.information_schema.columns TO ``;
GRANT SELECT ON TABLE system.query.history TO ``;
GRANT SELECT ON TABLE system.access.table_lineage TO ``;
-- Grant access to billing metadata
GRANT USE SCHEMA ON SCHEMA system.billing TO ``;
GRANT SELECT ON TABLE system.billing.usage TO ``;
GRANT SELECT ON TABLE system.billing.list_prices TO ``;
```
### Storage Access
Elementary requires access to the table history in order to enable automated monitors such as volume and freshness monitors.
You can configure this in one of the following ways:
#### Option 1: Direct storage access
Elementary can access the storage directly using credentials that you configure.
In the Elementary UI, choose **Direct storage access** under **Storage access method**.
When using this option, Elementary does not read the table data itself. It only reads the Delta transaction log, which contains metadata about the transactions.
For S3-backed Databricks storage, you can configure access in one of the following ways:
**AWS Role authentication**
This is the recommended approach, as it provides better security and follows AWS best practices.
After choosing **Direct storage access**, select **AWS role ARN** under **Select S3 authentication method**.
1. Create an IAM role that Elementary can assume.
2. Select "Another AWS account" as the trusted entity.
3. Enter Elementary's AWS account ID: `743289191656`.
4. Optionally enable an external ID.
5. Attach a policy that grants read access to the Delta log files.
Use a policy similar to the following:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::databricks-metastore-bucket",
"arn:aws:s3:::databricks-metastore-bucket/*_delta_log*"
]
}
]
}
```
This policy is scoped to the bucket itself and objects matching `*_delta_log*`, so it does not grant access to other objects in the bucket.
Provide the role ARN in the Elementary UI, and the external ID as well if you configured one.
**AWS access keys**
If needed, you can instead provide direct AWS credentials.
After choosing **Direct storage access**, select **Secret access key** under **Select S3 authentication method**.
1. Create an IAM user that Elementary will use for storage access.
2. Enable programmatic access.
3. Attach the same read-only S3 policy shown above.
4. Provide the AWS access key ID and secret access key in the Elementary UI.
#### Option 2: Credentials vending
Elementary can access the storage using temporary credentials issued by Databricks through [credential vending](https://docs.databricks.com/aws/en/external-access/credential-vending).
In the Elementary UI, choose **Credentials vending** under **Storage access method**.
This requires granting `EXTERNAL USE SCHEMA` on the relevant schemas.
When using this option, Elementary does not read the table data itself. It only reads the Delta transaction log, which contains metadata about the transactions.
#### Option 3: Fetch history using `DESCRIBE HISTORY` - **DEPRECATED**
Elementary can fetch the table history by running `DESCRIBE HISTORY` queries on your Databricks warehouse.
In the Elementary UI, choose **None** under **Storage access method**.
This require granting SELECT access on your tables. This is a Databricks limitation - Elementary **never** reads any data from your tables, only metadata. However, there isn't
today any table-level metadata-only permission available in Databricks, so SELECT is required.
To grant the access, use the following SQL statements:
```sql theme={null}
GRANT USE CATALOG, USE SCHEMA, SELECT ON catalog to ``;
```
This option is deprecated, and will soon be removed.
### Add an environment in Elementary (requires an admin user)
In the Elementary platform, go to Environments in the left menu, and click on the "Create Environment" button.
Choose a name for your environment, and then choose Databricks as your data warehouse type.
Provide the following common fields in the form:
* **Server Host**: The hostname of your Databricks account to connect to.
* **Http path**: The path to the Databricks cluster or SQL warehouse.
* **Catalog (optional)**: The name of the Databricks Catalog.
* **Elementary schema**: The name of your Elementary schema. Usually `[your dbt target schema]_elementary`.
Then, select your authentication method:
#### OAuth (M2M) — Recommended
* **Client ID**: The Application (client) ID of the service principal (the "Application ID" you copied in [step 5](#create-service-principal)).
* **Client secret**: The OAuth secret you generated for the service principal (see [step 7](#create-service-principal)).
OAuth machine-to-machine (M2M) authentication is the recommended method for connecting to Databricks.
It uses short-lived tokens that are automatically refreshed, providing better security compared to
long-lived personal access tokens.
#### Access token (legacy)
* **Access token**: A personal access token generated for the Elementary service principal.
### Add the Elementary IP to allowlist
Elementary IP for allowlist: `3.126.156.226`
### Need help with onboarding?
We can provide [support on Slack](https://elementary-data.com/community) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
# Connect to Dremio
Source: https://docs.elementary-data.com/cloud/integrations/dwh/dremio
**Note:** We currently support **Dremio Cloud only**. If you are using Dremio Software, please contact us for assistance.
### Create a user for Elementary cloud
Create an email account for the Elementary user.
Example: **[elementary@your-organization.com](mailto:elementary@your-organization.com)**
On your dbt project, run:
```bash theme={null}
## Print the query you should run to generate a user.
dbt run-operation create_elementary_user --args '{user: the_mail_of_the_elementary_user}'
```
This command will generate a query to create a user with the necessary permissions. Run this query on your data warehouse with admin permissions to create the user. It will send an email invitation, you need to accept that invitation.
After the invitation was accepted, you need to sign in to Dremio as the Elementary user, and open "Account Settings" by clicking on the bottom-left corner of the Dremio UI.
Click **Generate Token**:
Set the **maximum lifetime** allowed by Dremio (currently **180 days**).
⚠️ Put a reminder to renew the token before it expires.
You can update the token in Elementary's UI at any time.
### Permissions and security
Elementary cloud doesn't require read permissions to your tables and schemas, but only the following:
* Read-only access to the elementary schema.
* Access to read metadata in information schema and query history, related to the tables in your dbt project.
It is recommended to create a user using the instructions specified above to avoid granting excess privileges. For more details, refer to
[**security and privacy**](https://docs.elementary-data.com/cloud/general/security-and-privacy)
### Fill the connection form
Provide the following fields:
* **Host:**
* US: `api.dremio.cloud`
* EU: `api.eu.dremio.cloud`
* **Object Storage:** Name of the object storage where the Elementary schema is stored.
* **Object Storage Path:** Path inside the object storage where the Elementary schema is stored.
* **Project ID:** Your Dremio Cloud project ID.
* **User:** The email address of the Elementary user.
* **Token:** The token you generated for the Elementary user.
### Connect your metadata store (optional)
If your Dremio sources contains Iceberg tables, you can connect your metadata store (Iceberg catalog) to your environment. This will allow automatically monitoring volume
and freshness of your Iceberg tables (even if they are ingested outside of Dremio).
Currently only [AWS Glue](/cloud/integrations/metadata-layer/glue) is supported, though more metadata integrations are planned.
### Need help with onboarding?
We can provide [support on Slack](https://elementary-data.com/community) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
# Connect to DuckDB
Source: https://docs.elementary-data.com/cloud/integrations/dwh/duckdb
}
>
Coming soon
# Connect to Fabric
Source: https://docs.elementary-data.com/cloud/integrations/dwh/fabric
}
>
Coming soon
# Connect to Postgres
Source: https://docs.elementary-data.com/cloud/integrations/dwh/postgres
You will connect Elementary Cloud to Postgres for syncing the Elementary schema (created by the [Elementary dbt package](/data-tests/dbt/quickstart-package)).
### Create a user for Elementary cloud
On your dbt project, run:
```bash theme={null}
## Print the query you should run to generate a user
dbt run-operation create_elementary_user
```
This command will generate a query to create a user with the necessary permissions. Run this query on your data warehouse with **admin permissions** to create the user.
#### Permissions and security
Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following:
* Read-only access to the Elementary schema.
* Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project.
It is recommended to create a user using the instructions specified above to avoid granting excess privileges.
For more details, refer to [security and privacy](/cloud/general/security-and-privacy).
This guide contains the necessary steps to connect a Postgres environment to your Elementary account.
### Create a user for Elementary cloud
On your dbt project, run:
```bash theme={null}
## Print the query you should run to generate a user
dbt run-operation create_elementary_user
```
This command will generate a query to create a user with the necessary permissions. Run this query on your data warehouse with **admin permissions** to create the user.
#### Permissions and security
Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following:
* Read-only access to the Elementary schema.
* Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project.
It is recommended to create a user using the instructions specified above to avoid granting excess privileges.
For more details, refer to [security and privacy](/cloud/general/security-and-privacy).
### Fill the connection form
Provide the following fields:
* **Host**: The hostname of your Postgres account to connect to. This can either be a hostname or an IP address.
* **Port**: The port of your Postgres account to connect to. This is usually `5432`.
* **Database name**: The name of the database where your Elementary schema exist. e.g. `analytics`.
* **Elementary schema**: The name of your Elementary schema. Usually `[schema name]_elementary`.
* **User**: The name of the for Elementary user.
* **Password**: The password associated with the provided user.
### Add the Elementary IP to allowlist
Elementary IP for allowlist: `3.126.156.226`
### Connect via SSH tunnel
Elementary supports connecting via SSH or reverse SSH tunnel. Reach out to our team for details and support in this deployment.
### Need help with onboarding?
We can provide [support on Slack](https://elementary-data.com/community) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
# Connect to Redshift
Source: https://docs.elementary-data.com/cloud/integrations/dwh/redshift
This guide contains the necessary steps to connect a Redshift environment to your Elementary account.
### Create a user for Elementary cloud
On your dbt project, run:
```bash theme={null}
## Print the query you should run to generate a user
dbt run-operation create_elementary_user
```
This command will generate a query to create a user with the necessary permissions. Run this query on your data warehouse with **admin permissions** to create the user.
#### Permissions and security
Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following:
* Read-only access to the Elementary schema.
* Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project.
It is recommended to create a user using the instructions specified above to avoid granting excess privileges.
For more details, refer to [security and privacy](/cloud/general/security-and-privacy).
### Fill the connection form
Provide the following fields:
* **Host**: The hostname of your Redshift account to connect to. This can either be a hostname or an IP address.
* **Port**: The port of your Redshift account to connect to. This is usually `5439`.
* **Database name**: The name of the database where your Elementary schema exist. e.g. `analytics`.
* **Elementary schema**: The name of your Elementary schema. Usually `[schema name]_elementary`.
* **User**: The name of the for Elementary user.
* **Password**: The password associated with the provided user.
### Add the Elementary IP to allowlist
Elementary IP for allowlist: `3.126.156.226`
### Connect via SSH tunnel
Elementary supports connecting via SSH or reverse SSH tunnel. Reach out to our team for details and support in this deployment.
### Need help with onboarding?
We can provide [support on Slack](https://elementary-data.com/community) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
# Connect to Snowflake
Source: https://docs.elementary-data.com/cloud/integrations/dwh/snowflake
This guide contains the necessary steps to connect a Snowflake environment to your Elementary account.
### Create a user for Elementary cloud
* Please create a Snowflake key-pair (private and public key) using [this](https://docs.snowflake.com/en/user-guide/key-pair-auth#configuring-key-pair-authentication) guide.
* Using the public key generated in the previous step, please run the following in your dbt project folder:
```bash theme={null}
## Store the public key in an environment variable
export SNOWFLAKE_PUBLIC_KEY=""
## Print the query you should run to generate a user
dbt run-operation create_elementary_user --args "{'public_key': '$SNOWFLAKE_PUBLIC_KEY'}"
```
This command will generate a query to create a user with the necessary permissions. Run this query on your data warehouse with **admin permissions** to create the user.
#### Permissions and security
Elementary Cloud doesn't require read permissions to your tables and schemas, but only the following:
* Read-only access to the Elementary schema.
* Access to read metadata in `INFORMATION_SCHEMA` (table metadata and query history), related to the tables in your dbt project.
It is recommended to create a user using the instructions specified above to avoid granting excess privileges.
For more details, refer to [security and privacy](/cloud/general/security-and-privacy).
### Grant permissions to monitor dynamic tables
To monitor [Snowflake dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables/overview), Elementary needs the `MONITOR` privilege on those dynamic tables, granted per database.
This privilege isn't included in the roles granted during user creation. Grant it separately for each database containing dynamic tables you want Elementary to monitor.
Run the following in a Snowflake worksheet with **admin permissions**. Set `databases` to the list of databases containing the dynamic tables you want Elementary to monitor, and `elementary_role` to your Elementary role name:
```sql theme={null}
DECLARE
databases ARRAY DEFAULT ARRAY_CONSTRUCT('DB1', 'DB2', 'DB3');
elementary_role STRING DEFAULT 'ELEMENTARY_ROLE';
BEGIN
FOR i IN 0 TO ARRAY_SIZE(databases) - 1 DO
LET db_name STRING := databases[i]::STRING;
GRANT MONITOR ON ALL DYNAMIC TABLES IN DATABASE IDENTIFIER(:db_name)
TO ROLE IDENTIFIER(:elementary_role);
GRANT MONITOR ON FUTURE DYNAMIC TABLES IN DATABASE IDENTIFIER(:db_name)
TO ROLE IDENTIFIER(:elementary_role);
END FOR;
RETURN 'Granted MONITOR on dynamic tables for ' || ARRAY_SIZE(databases) || ' database(s).';
END;
```
### Fill the connection form
Provide the following fields:
* **Account**: Find your account by using the Snowflake account URL and removing `snowflakecomputing.com`. e.g. `example.us-east-1`. For more information, see [Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) in the Snowflake docs.
* **Database name**: The name of the database where your Elementary schema exist. e.g. `analytics`.
* **Warehouse**: e.g. `ELEMENTARY_WAREHOUSE`.
* **Elementary schema**: The name of your Elementary schema. Usually `[schema name]_elementary`.
* **Role (optional)**: e.g. `ELEMENTARY_ROLE`.
Elementary Cloud supports the following authentication methods:
* **Key pair** (Recommended):
* User: The user created for Elementary.
* Private key: The private key you generated for Elementary. For more information, see [Generate Private Key](https://docs.snowflake.com/en/user-guide/key-pair-auth#configuring-key-pair-authentication) in the Snowflake docs.
* Private key passphrase (optional)
* **User password** (Deprecated, not recommended):
* User: The user created for Elementary.
* Password: The password you set when creating your Snowflake account.
Snowflake are in the process of deprecating single-factor username & password authentication for all human users. As a result, while still supported,
we recommend configuring the user in advance using key-pair authentication rather than username & password, and configuring the user as TYPE=SERVICE
(this is automatically handled in the user creation macro above).
See [here](https://docs.snowflake.com/en/user-guide/security-mfa-rollout) for more information regarding this change.
### Add the Elementary IP to allowlist
If you use network policies to restrict access, run this query on your data warehouse with **admin permissions**:
```sql theme={null}
CREATE NETWORK POLICY ELEMENTARY_CLOUD
ALLOWED_IP_LIST=('3.126.156.226')
BLOCKED_IP_LIST=()
COMMENT='IP used by Elementary Cloud to sync the Elementary schema';
```
After creating a network policy you would need to activate it. To activate a network policy for the elementary user simply run the following command -
```sql theme={null}
ALTER USER SET NETWORK_POLICY = ELEMENTARY_CLOUD;
```
### Need help with onboarding?
We can provide [support on Slack](https://elementary-data.com/community) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
# Connect to Spark
Source: https://docs.elementary-data.com/cloud/integrations/dwh/spark
}
>
Coming soon
# Connect to SQL Server
Source: https://docs.elementary-data.com/cloud/integrations/dwh/sqlserver
}
>
Coming soon
# Connect to Trino
Source: https://docs.elementary-data.com/cloud/integrations/dwh/trino
}
>
Click for details
# Connect to Vertica
Source: https://docs.elementary-data.com/cloud/integrations/dwh/vertica
}
>
Coming soon
# Elementary integrations
Source: https://docs.elementary-data.com/cloud/integrations/elementary-integrations
Elementary connects seamlessly to your data warehouse, BI tools, code repositories, alerting, and ticketing systems—giving you full context and true end-to-end, column-level lineage. This unified view makes it easy to trace issues back to their source, understand downstream impact, and resolve problems faster.
### Data warehouses
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
### Transformation and orchestration
}
/>
}
/>
}
/>
}
/>
}
/>
### Data visualization
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
>
Click for details
}
>
Click for details
}
>
Click for details
### Reverse ETL
}
>
Click for details
}
>
Click for details
### Code repositories
}
/>
}
/>
}
/>
}
/>
### Alerts & incidents
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
}
/>
### Log Streaming
}
/>
}
/>
}
/>
### Governance
}
>
Click for details
### Iceberg catalog
}
>
Click for details
# Atlan
Source: https://docs.elementary-data.com/cloud/integrations/governance/atlan
Elementary aims to meet business users where they live, so we believe it's important to allow anyone who uses data to
know whether the data is healthy quickly, easily and without any technical knowledge required. This is why we integrated with Atlan.
The integration works by pushing key insights to Atlan assets as custom metadata.
This metadata includes data health scores and open incidents, providing visibility into the quality and status of your
data.
### Atlan API Key
To generate an API key, follow these steps:
1. Create a new Persona in Atlan, with the following permissions:
* Assets: Read, Update
* Governance: Update custom metadata values
### Atlan Base Url
Your Atlan base URL, for example `https://my-company.atlan.com`
### Elementary Account Token (required for connecting through the Atlan app)
You can generate tokens directly from the Elementary UI: go to [Account → Account Tokens](https://app.elementary-data.com/settings/account-tokens).
Quick steps:
1. Open the **Account → Account Tokens** page and click **Generate token**.
2. (Optional) Add a name/description.
3. Copy the token and store it securely — it is shown once. Manage (revoke/rotate) anytime from the same page.
# Datadog
Source: https://docs.elementary-data.com/cloud/integrations/log-streaming/datadog
Elementary's Datadog integration enables streaming audit logs and system logs directly to your Datadog account for centralized log management and monitoring.
## Overview
When enabled, Elementary automatically streams your workspace's [audit logs](/cloud/features/collaboration-and-communication/audit_logs/overview) ([user activity logs](/cloud/features/collaboration-and-communication/audit_logs/user-activity-logs) and [system logs](/cloud/features/collaboration-and-communication/audit_logs/system-logs)) to Datadog using the [Datadog Logs API](https://docs.datadoghq.com/api/latest/logs/#send-logs). This allows you to:
* Centralize all logs in your Datadog dashboard
* Set up custom alerts and monitors on log events
* Correlate Elementary logs with other application logs
* Perform advanced log analysis and search
* Maintain long-term log retention in Datadog
## Prerequisites
Before configuring log streaming to Datadog, you'll need:
1. **Datadog API Key** - Your Datadog API key for authentication
* You can find or create an API key in your Datadog account under [Organization Settings > API Keys](https://app.datadoghq.com/organization-settings/api-keys)
2. **Datadog Site** (optional) - Your Datadog site region
* Default: `datadoghq.com` (US)
* EU: `datadoghq.eu`
* US3: `us3.datadoghq.com`
* US5: `us5.datadoghq.com`
* AP1: `ap1.datadoghq.com`
## Configuring Log Streaming to Datadog
1. Navigate to the [**Logs**](/cloud/features/collaboration-and-communication/audit_logs/overview) page:
* Click on your **account name** in the top-right corner of the UI
* Open the dropdown menu
* Select **Logs**
2. In the **External Integrations** section, click the **Connect** button
3. In the modal that opens, select **Datadog** as your log streaming destination
4. Enter your Datadog configuration:
* **API Key**: Your Datadog API key
* **Site** (optional): Your Datadog site region (defaults to `datadoghq.com` if not specified)
* **Service Name** (optional): Custom service name for logs in Datadog (defaults to `elementary`)
* **Source** (optional): Custom source tag for logs (defaults to `elementary-cloud`)
6. Click **Connect** to enable log streaming
7. After connecting, you will see the Datadog integration listed in the **External Integrations** section. You can edit or disable the integration at any time.
The log streaming configuration applies to your entire workspace. All logs matching your selected log types will be streamed to Datadog in real-time.
## Log Format in Datadog
Logs are sent to Datadog with the following structure:
* `timestamp`: ISO 8601 timestamp of the event
* `log_type`: The type of log (`audit` for user activity logs, `system` for system logs)
* `status`: Log level (`info` for successful actions, `error` for failed actions)
* `service`: Service name (configurable, defaults to `elementary`)
* `source`: Source tag (configurable, defaults to `elementary-cloud`)
* `event_name`: The specific action that was performed (e.g., `user_login`, `create_test`)
* `success`: Boolean indicating whether the action completed successfully
* `user_email`: User email address (only present in audit logs)
* `user_name`: User display name (only present in audit logs)
* `env_id`: Environment identifier (empty for account-level actions)
* `env_name`: Environment name (empty for account-level actions)
* `event_content`: Additional context-specific information as a JSON object
* `dd.tags`: Additional tags including:
* `log_type:` (e.g., `audit`, `system`)
* `event_name:` (e.g., `user_login`, `create_test`)
* `env_id:` (if applicable)
## Viewing Logs in Datadog
Once configured, logs will appear in your Datadog [Log Explorer](https://app.datadoghq.com/logs) within a few seconds of being generated.
You can filter logs using:
* `source:elementary-cloud` - All Elementary logs
* `log_type:audit` - User activity logs only
* `log_type:system` - System logs only
* `event_name:` - Specific action types
* `env_id:` - Logs from a specific environment
* `success:false` - Failed operations only
## Troubleshooting
### Logs not appearing in Datadog
1. **Verify API Key**: Ensure your Datadog API key is valid and has the necessary permissions
2. **Check Site Configuration**: Verify you've selected the correct Datadog site region
3. **Review Log Types**: Confirm the log types you want to stream are enabled
4. **Check Datadog Status**: Verify your Datadog account is active and not rate-limited
### Rate Limiting
Datadog has rate limits for log ingestion. If you're experiencing issues:
* Check your Datadog account's rate limits in the [Usage & Billing](https://app.datadoghq.com/billing/usage) page
* Consider filtering which log types you stream if you have high log volume
* Contact Datadog support if you need to increase your rate limits
## Disabling Log Streaming
To disable log streaming to Datadog:
1. Navigate to the [**Logs**](/cloud/features/collaboration-and-communication/audit_logs/overview) page
2. In the **External Integrations** section, find your Datadog integration
3. Click **Disable** or remove the Datadog configuration
4. Confirm the action
Disabling log streaming will stop sending new logs to Datadog immediately. Historical logs already sent to Datadog will remain in your Datadog account according to your retention settings.
# Google Cloud Storage (GCS)
Source: https://docs.elementary-data.com/cloud/integrations/log-streaming/gcs
Elementary's Google Cloud Storage (GCS) integration enables streaming audit logs and system logs directly to your GCS bucket for long-term storage, analysis, and integration with other Google Cloud services.
## Overview
When enabled, Elementary automatically streams your workspace's [audit logs](/cloud/features/collaboration-and-communication/audit_logs/overview) ([user activity logs](/cloud/features/collaboration-and-communication/audit_logs/user-activity-logs) and [system logs](/cloud/features/collaboration-and-communication/audit_logs/system-logs)) to your GCS bucket using the [Google Cloud Storage API](https://cloud.google.com/storage/docs/json_api). This allows you to:
* Store logs in your own GCS bucket for long-term retention
* Integrate logs with BigQuery, Dataflow, or other Google Cloud analytics services
* Maintain full control over log storage and access policies
* Process logs using Google Cloud data processing tools
* Archive logs for compliance and audit requirements
## Prerequisites
Before configuring log streaming to GCS, you'll need:
1. **GCS Bucket** - A Google Cloud Storage bucket where logs will be stored
* The bucket must exist and be accessible
* You'll need the bucket path (e.g., `gs://my-logs-bucket`)
2. **Authentication** - Either a Google Cloud service account or a Workload Identity Federation setup, with the **Storage Object User** (`roles/storage.objectUser`) role granted on the bucket. See [Authentication methods](#authentication-methods) below for both options.
## Authentication methods
Elementary supports two authentication methods for GCS. Pick the one that fits your security model:
* **Service account** — create a service account, download its JSON key, and upload the key to Elementary. Simplest to set up.
* **Workload Identity Federation (WIF)** — Elementary authenticates from its AWS role through a federated identity. No long-lived credentials are stored in Elementary.
Select a tab below and follow the steps for your chosen method.
1. Go to [Google Cloud Console > IAM & Admin > Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) and create a service account (or select an existing one).
2. Grant the service account the **Storage Object User** (`roles/storage.objectUser`) role on your GCS bucket.
3. Generate a JSON key for the service account:
1. Select your service account.
2. Click the three dots menu and select 'Manage keys'.
3. Click 'ADD KEY' and select 'Create new key'.
4. Choose 'JSON' format and click 'CREATE'. The JSON file downloads automatically.
4. You will upload this JSON key to Elementary in the connection form below, in the **Service account file** field.
The WIF setup is identical to the one documented for BigQuery — the only GCS-specific change is the permission you grant to the pool principal. Follow the Workload Identity Federation steps on the [BigQuery connection page](/cloud/integrations/dwh/bigquery) (select the **Workload Identity Federation** tab), with the following change:
* In the step where you grant IAM roles to the principalSet, **do not** grant the BigQuery roles. Instead, grant the **Storage Object User** (`roles/storage.objectUser`) role on your GCS bucket. You can do this from the bucket's 'Permissions' tab: 'GRANT ACCESS' → paste the same principalSet → assign `Storage Object User`.
* You can skip the dataset-level grant step entirely — it is BigQuery-specific.
Once you have the downloaded WIF credential configuration file, upload it to Elementary in the connection form below, in the **WIF credential file** field.
## Configuring Log Streaming to GCS
1. Navigate to the [**Logs**](/cloud/features/collaboration-and-communication/audit_logs/overview) page:
* Click on your **account name** in the top-right corner of the UI
* Open the dropdown menu
* Select **Logs**
2. In the **External Integrations** section, click the **Connect** button
3. In the modal that opens, select **Google Cloud Storage (GCS)** as your log streaming destination
4. Enter your GCS configuration:
* **Bucket Path**: The full GCS bucket path (e.g., `gs://my-logs-bucket`)
* **Authentication method**: Use the toggle to select **Service account** or **Workload Identity Federation**, matching the method you set up in [Authentication methods](#authentication-methods) above.
* **Service account file** (Service account method) or **WIF credential file** (Workload Identity Federation method): Upload the JSON file you prepared.
5. Click **Save** to enable log streaming
The log streaming configuration applies to your entire workspace. Both user activity logs and system logs will be streamed to your GCS bucket in batches.
## Log Batching
Logs are automatically batched and written to GCS files based on the following criteria:
* **Time-based batching**: A new file is created every **15 minutes**
* **Size-based batching**: A new file is created when the batch reaches **100MB**
**Whichever condition is met first** triggers a new file to be created. This ensures efficient storage while maintaining reasonable file sizes for processing.
## File Path Format
Logs are stored at the root of your bucket using a Hive-based partitioning structure for efficient querying and organization:
```
log_type={log_type}/date={YYYY-MM-DD}/hour={HH}/file_{timestamp}_{batch_id}.ndjson
```
Where:
* `{log_type}`: Either `audit` (for user activity logs) or `system` (for system logs)
* `{YYYY-MM-DD}`: Date in ISO format (e.g., `2024-01-15`)
* `{HH}`: Hour in 24-hour format (e.g., `14`)
* `{timestamp}`: Unix timestamp when the file was created
* `{batch_id}`: Unique identifier for the batch
### Example File Paths
```
log_type=audit/date=2024-01-15/hour=14/file_1705320000_batch_abc123.ndjson
log_type=system/date=2024-01-15/hour=14/file_1705320900_batch_def456.ndjson
```
This Hive-based structure allows you to:
* Efficiently query logs by date and hour using BigQuery or other tools
* Filter logs by type (`audit` or `system`)
* Process logs in parallel by partition
## Log Format
Logs are stored as **line-delimited JSON** (NDJSON), where each line represents a single log entry as a JSON object.
### User Activity Logs
Each user activity log entry includes:
```json theme={null}
{
"timestamp": "2024-01-15T14:30:45.123456Z",
"log_type": "audit",
"event_name": "user_login",
"success": true,
"user_email": "john.doe@example.com",
"user_name": "John Doe",
"env_id": "env_7890123456abcdef",
"env_name": "Production",
"event_content": {
"additional": "context"
}
}
```
### System Logs
Each system log entry includes:
```json theme={null}
{
"timestamp": "2024-01-15T14:30:45.123456Z",
"log_type": "system",
"event_name": "dbt_data_sync_completed",
"success": true,
"env_id": "env_7890123456abcdef",
"env_name": "Production",
"event_content": {
"environment_id": "env_789",
"environment_name": "Production"
}
}
```
### Field Descriptions
* `timestamp`: ISO 8601 timestamp of the event (UTC)
* `log_type`: Either `"audit"` for user activity logs or `"system"` for system logs
* `event_name`: The specific action that was performed (e.g., `user_login`, `create_test`, `dbt_data_sync_completed`)
* `success`: Boolean indicating whether the action completed successfully
* `user_email`: User email address (only present in audit logs)
* `user_name`: User display name (only present in audit logs)
* `env_id`: Environment identifier (empty string for account-level actions)
* `env_name`: Environment name (empty string for account-level actions)
* `event_content`: Additional context-specific information as a JSON object
## Disabling Log Streaming
To disable log streaming to GCS:
1. Navigate to the [**Logs**](/cloud/features/collaboration-and-communication/audit_logs/overview) page
2. In the **External Integrations** section, find your GCS integration
3. Click **Disable** or remove the GCS configuration
4. Confirm the action
Disabling log streaming will stop sending new logs to GCS immediately. Historical logs already written to GCS will remain in your bucket.
# Splunk
Source: https://docs.elementary-data.com/cloud/integrations/log-streaming/splunk
Elementary's Splunk integration enables streaming audit logs and system logs directly to your Splunk instance via HTTP Event Collector (HEC) for centralized log management, monitoring, and analysis.
## Overview
When enabled, Elementary automatically streams your workspace's [audit logs](/cloud/features/collaboration-and-communication/audit_logs/overview) ([user activity logs](/cloud/features/collaboration-and-communication/audit_logs/user-activity-logs) and [system logs](/cloud/features/collaboration-and-communication/audit_logs/system-logs)) to Splunk using the [Splunk HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/HECExamples). This allows you to:
* Centralize all logs in your Splunk instance
* Set up custom alerts and dashboards on log events
* Correlate Elementary logs with other application logs
* Perform advanced log analysis and search using Splunk's powerful query language
* Maintain long-term log retention in Splunk
## Prerequisites
Before configuring log streaming to Splunk, you'll need:
1. **Splunk Instance** - A Splunk Enterprise or Splunk Cloud instance with HTTP Event Collector (HEC) enabled
* HEC must be configured and accessible from Elementary's servers
* You'll need the HEC URL (e.g., `https://splunk.example.com:8088`)
2. **HEC Token** - An HTTP Event Collector token for authentication
* You can create a token in Splunk under **Settings > Data Inputs > HTTP Event Collector**
* The token must have write permissions
3. **Splunk Index** (optional) - A specific index where logs should be stored
* If not specified, logs will be sent to the default index configured for the HEC token
## Configuring Log Streaming to Splunk
1. Navigate to the [**Logs**](/cloud/features/collaboration-and-communication/audit_logs/overview) page:
* Click on your **account name** in the top-right corner of the UI
* Open the dropdown menu
* Select **Logs**
2. In the **External Integrations** section, click the **Connect** button
3. In the modal that opens, select **Splunk** as your log streaming destination
4. Enter your Splunk configuration:
* **HEC URL**: Your Splunk HTTP Event Collector URL (e.g., `https://splunk.example.com:8088` or `https://example.splunkcloud.com:8088`)
* **HEC Token**: Your Splunk HEC authentication token
* **Index** (optional): The Splunk index where logs should be stored (defaults to the token's configured index if not specified)
5. Click **Save** to enable log streaming
The log streaming configuration applies to your entire workspace. Both user activity logs and system logs will be streamed to Splunk in batches.
## Log Format in Splunk
Logs are sent to Splunk with the following structure:
### Event Structure
Each log entry is sent as a JSON event with the following fields:
* `event`: The log data as a JSON object
* `sourcetype`: `_json` (indicating JSON format)
* `source`: `elementary-cloud` (identifying the source)
* `time`: Unix timestamp of the event
* `index`: The Splunk index (if specified in configuration)
### User Activity Logs
Each user activity log entry includes:
```json theme={null}
{
"timestamp": "2024-01-15T14:30:45.123456Z",
"log_type": "audit",
"event_name": "user_login",
"success": true,
"user_email": "john.doe@example.com",
"user_name": "John Doe",
"env_id": "env_7890123456abcdef",
"env_name": "Production",
"event_content": {
"additional": "context"
}
}
```
### System Logs
Each system log entry includes:
```json theme={null}
{
"timestamp": "2024-01-15T14:30:45.123456Z",
"log_type": "system",
"event_name": "dbt_data_sync_completed",
"success": true,
"env_id": "env_7890123456abcdef",
"env_name": "Production",
"event_content": {
"environment_id": "env_789",
"environment_name": "Production"
}
}
```
### Field Descriptions
* `timestamp`: ISO 8601 timestamp of the event (UTC)
* `log_type`: Either `"audit"` for user activity logs or `"system"` for system logs
* `event_name`: The specific action that was performed (e.g., `user_login`, `create_test`, `dbt_data_sync_completed`)
* `success`: Boolean indicating whether the action completed successfully
* `user_email`: User email address
* `user_name`: User display name
* `env_id`: Environment identifier (empty for account-level actions)
* `env_name`: Environment name (empty for account-level actions)
* `event_content`: Additional context-specific information as a JSON object
## Viewing Logs in Splunk
Once configured, logs will appear in your Splunk instance within a few seconds of being generated.
You can search logs using Splunk Search Processing Language (SPL):
```
# Search for all Elementary logs
source="elementary-cloud"
# Filter by log type
source="elementary-cloud" log_type="audit"
source="elementary-cloud" log_type="system"
# Search for specific actions
source="elementary-cloud" event_name="user_login"
source="elementary-cloud" event_name="dbt_data_sync_completed"
# Filter by environment
source="elementary-cloud" env_name="Production"
# Search for failed operations
source="elementary-cloud" success=false
# Search by user email
source="elementary-cloud" user_email="john.doe@example.com"
```
## Disabling Log Streaming
To disable log streaming to Splunk:
1. Navigate to the [**Logs**](/cloud/features/collaboration-and-communication/audit_logs/overview) page
2. In the **External Integrations** section, find your Splunk integration
3. Click **Disable** or remove the Splunk configuration
4. Confirm the action
Disabling log streaming will stop sending new logs to Splunk immediately. Historical logs already sent to Splunk will remain in your Splunk instance according to your retention settings.
# AWS Glue
Source: https://docs.elementary-data.com/cloud/integrations/metadata-layer/glue
The AWS Glue integration in Elementary will allow you to automatically monitor volume & freshness anomalies in your Iceberg tables in Glue, by continouously syncing metadata about Iceberg snapshots.
This integration is currently only supported with the Dremio engine. We plan to add other engines in the near future.
## AWS Setup
### 1. Create Required IAM Policy
First, you'll need to create an IAM policy with the following permissions:
* **GluePermissions**: Enables reading metadata about tables in your Glue catalog.
* **S3IcebergMetadataReadAccess**: Grants access metadata-only access to files of your Iceberg tables. These metadata files contain statistics about Iceberg snapshots,
such as update cadence and row count changes.
Here is an example of a JSON policy:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GluePermissions",
"Effect": "Allow",
"Action": [
"glue:GetTable",
"glue:GetTables"
],
"Resource": "*"
},
{
"Sid": "S3IcebergMetadataReadAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::your-iceberg-tables-bucket/*metadata.json"
]
}
]
}
```
### 2. Choose Authentication Method
Elementary supports two authentication methods for connecting to Glue:
#### Option 1: AWS Role Authentication (Recommended)
This is the recommended approach as it provides better security and follows AWS best practices. [Learn more about AWS IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html).
1. **Create an IAM Role**:
* Go to AWS IAM Console
* Create a new role
* Select "Another AWS account" as the trusted entity
* Enter Elementary's AWS account ID: `743289191656`
* (Optional but recommended) Enable "Require external ID" and set a value
* Attach the policy created in step 1
2. **Note down the following information**:
* Role ARN
* External ID (if you enabled it) [Learn more about external IDs](https://aws.amazon.com/blogs/security/how-to-use-external-id-when-granting-access-to-your-aws-resources/).
#### Option 2: Access Key Authentication
This method is less secure as it requires permanent credentials. We recommend using AWS Role authentication instead.
1. **Create an IAM User**:
* Go to AWS IAM Console
* Create a new user, that will be used by elementary to connect to Glue
* Enable programmatic access
* Attach the policy created in step 1
2. **Note down the following information**:
* AWS Access Key ID of the new elementary glue user
* AWS Secret Access Key of the new elementary glue user
## Elementary Configuration
Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect AWS Glue.
Under the "Metadata Layer" section, please choose Glue.
### Connection Settings
Regardless of the authentication method you choose, you'll need to provide:
* **Connection Name**: A descriptive name for your connection (e.g. "Datalake"). Needs to be unique if you're adding more than one metadata integration.
* **Region**: The AWS region where your Glue catalog is located
### Authentication Details
Based on your chosen authentication method:
#### If using AWS Role Authentication:
* Select "AWS Role" as the authentication method
* Enter your role ARN
* Enter your external ID (if you enabled it)
#### If using Access Key Authentication:
* Select "Access Key" as the authentication method
* Enter your AWS Access Key ID
* Enter your AWS Secret Access Key
# Fivetran
Source: https://docs.elementary-data.com/cloud/integrations/pipeline/fivetran
Elementary connects to your Fivetran account to automatically create incidents when connector syncs fail.
When a sync ends, Elementary evaluates the outcome and opens or updates an incident accordingly.
**Connector filtering is not yet supported.** Incidents will be created for all connectors in the monitored destination groups. Filtering by specific connectors will be added in a future release.
## How it works
1. You connect your Fivetran account to Elementary with a Fivetran API key and select which destination groups to monitor.
2. Elementary subscribes to Fivetran's sync\_end events for the selected groups automatically - no manual configuration in Fivetran is required.
3. Each time a connector sync completes, Elementary evaluates the outcome and creates or updates an incident when the sync fails.
4. The incident appears in Elementary's Incidents view, where you can investigate, comment, and resolve it.
## Enabling Fivetran
Connecting Fivetran requires permission to edit integrations in Elementary.
In Fivetran, go to your account settings and generate an API key. Copy the **API key** and **API secret** .
**Which permissions should I provide to the API key?**
* Read permission to all resources
* Manage permission to the webhook resource
See Fivetran's [API key documentation](https://fivetran.com/docs/rest-api/getting-started) for details.
In Elementary, go to **Settings → Environments** and select the environment you want to connect. Under **Integrations**, click **Connect Fivetran**.
Paste the **API key** and **API secret** and continue. Elementary validates the credentials and loads the destination groups available in your Fivetran account.
Choose the destination groups you want to monitor, then save. Elementary connects to Fivetran and automatically starts syncing and monitoring the selected groups.
## Where to see Fivetran incidents
Fivetran sync failures appear in the **Incidents** view in Elementary. Each incident includes:
* Connector name and ID
* Sync status and failure reason (when available)
* Event timestamp
* Destination group ID
# Census
Source: https://docs.elementary-data.com/cloud/integrations/reverse-etl/census
Elementary will automatically and continuously extend the column-level-lineage to your reverse ETL connectors.
This will enable to understand your downstream dependencies.
}
>
Click for details
# Hightouch
Source: https://docs.elementary-data.com/cloud/integrations/reverse-etl/hightouch
Elementary will automatically and continuously extend the column-level-lineage to your reverse ETL connectors.
This will enable to understand your downstream dependencies.
}
>
Click for details
# AWS PrivateLink
Source: https://docs.elementary-data.com/cloud/integrations/security-and-connectivity/aws-privatelink-integration
## What is AWS PrivateLink?
**AWS PrivateLink** is a secure and scalable networking technology that enables private connectivity between Virtual Private Clouds (VPCs), AWS services, and on-premises applications—without exposing traffic to the public internet. By leveraging PrivateLink, organizations can simplify their network architecture, reduce data exposure risks, and ensure secure communication between services.
With PrivateLink, services are exposed as **private endpoints** within a VPC, allowing consumers to connect to them using private IP addresses. This minimizes the need for complex networking configurations like VPC peering or VPNs, and reduces the risk of data leakage by keeping traffic within the AWS network.
In the context of our integration, AWS PrivateLink enables Elementary Cloud to securely and privately communicate with supported services, ensuring data privacy, compliance, and a streamlined user experience. We support cross-region PrivateLink and can connect to any region where your cloud is hosted, using VPC peering to link different regions to our production environment securely. Elementary Data maintains a global network of regional VPCs designed for PrivateLink, with strict security controls.
## Architecture overview
Elementary’s PrivateLink setup consists generally from two parts:
1. **AWS PrivateLink connection** -
1. Provider side (Customer / 3rd party) - **A VPC endpoint service** is set up at the customer’s AWS account (or a 3rd party AWS account in the case of Snowflake). This provides access to a particular service in that account.
2. Consumer side (Elementary) - Elementary sets up a dedicated VPC interface that will connect to the integrated service, in the same AWS region as the service.
This is done through a dedicated regional VPC created for this purpose.
2. **AWS VPC Peering:**
1. Elementary’s production servers are located in the **eu-central-1** (Frankfurt) region. For us to be able to access the service exposed through PrivateLink, we connect our main production VPC with the regional VPC mentioned above.
## Supported integrations
### Snowflake
Snowflake has support for connecting to AWS-hosted Snowflake accounts via PrivateLink. This setup is entirely managed by Snowflake, so Elementary connects with an endpoint service hosted on Snowflake’s AWS account for this purpose.
In order to set up a PrivateLink connection with Snowflake, please follow the steps below:
1. **Open a support case to Snowflake Support**
1. Ask to authorize Elementary’s AWS account for PrivateLink access.
2. Provide Elementary’s account ID in the request - `743289191656`
2. **Obtain the PrivateLink configuration**
1. Once Snowflake’s support team approves the request, obtain the PrivateLink configuration by invoking the following commands (admin access is required):
```sql theme={null}
USE ROLE ACCOUNTADMIN;
SELECT SYSTEM$GET_PRIVATELINK_CONFIG();
```
3. **Provide Elementary with the configuration obtained in the previous step.**
1. Elementary will then setup the required infrastructure to connect to Snowflake via PrivateLink.
4. **Add a Snowflake environment in Elementary**
1. Follow the instructions [here](/cloud/integrations/dwh/snowflake) to set up a Snowflake environment in Elementary.
1. When supplying the account, use `.privatelink` , where the account identifier is the result of the following query:
```sql theme={null}
SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME();
```
2. In the Snowflake instructions, under the *Add the Elementary IP to allowlist* section, please add the following private subnets *instead* of the IP mentioned there:
* 10.0.1.x
* 10.0.2.x
* 10.0.3.x
### Databricks
Databricks has support for connecting to AWS-hosted Databricks workspaces via PrivateLink. This setup is entirely managed by Databricks, so Elementary connects with an endpoint service hosted on Databrick's AWS account for this purpose.
**Note**:
1. You must be a Databricks account admin to perform this setup.
2. Your databricks workspace must be deployed on a customer-managed VPC. PrivateLink is not supported with Databricks-managed VPCs.
In order to set up a PrivateLink connection with Databricks, please follow the steps below:
1. **Please provide Elementary with the following details:**
* Your databricks workspace URL.
* Your AWS account ID.
* Your AWS region.
Elementary will then provide you with a VPC Endpoint ID that will be used in the next step.
2. **Register your VPC endpoint**
In the account management portal (not your workspace), go to Security -> Networking -> VPC Endpoints,
then click on the "Register VPC Endpoint" button.
You should fill in:
1. A name for the VPC endpoint - e.g. "Elementary".
2. Your AWS region.
3. The VPC Endpoint ID provided to you by Elementary.
3. **Configure a private access setting**
Go to Security -> Private Access Settings.
* If you've set up private link with your Databricks instance before, you should already have a private access setting configured.
In that case, please ensure that the endpoint allows access to the VPC endpoint created in step (2).
* If this is the first time you are setting PrivateLink for your databricks workspace:
* Click on "Add private access config".
* Please fill in the following details:
* A name for your setting: e.g. "Privatelink settings"
* Your AWS region
* Whether or not to allow public access - only set this as False if all your systems and users access your Databricks workspace through privatelink.
* Private access level - either leave as "Account", or allow-list specific VPCs including the Elementary VPC created in the previous step.
4. **Add the private access setting to your Databricks workspace**
**Note**: If you have already set up Privatelink with Databricks in the past, you can skip this step.
Under the Databricks account management portal, go to Workspaces, click on your workspace and then on "Update Workspace".
Then go to "Advanced Configurations", and under "Private Link", please attach the setting created in the previous step.
5. **Add a Databricks environment in Elementary**
After all the previous steps are completed, please reach out to the Elementary team to verify that your Databricks cluster is accessible via PrivateLink.
Once verified, please add a Databricks environment to Elementary by following [this guide](/cloud/integrations/dwh/databricks).
Under the *Add the Elementary IP to allowlist* section, please add the following private subnets *instead* of the IP mentioned there:
* 10.0.1.x
* 10.0.2.x
* 10.0.3.x
### Github Enterprise Server
Github Enterprise Server can be connected to Elementary Cloud via AWS PrivateLink. This setup requires creating a VPC endpoint service in your AWS account that exposes your GitHub Enterprise Server instance.
**Prerequisites:**
* Your Github Enterprise Server instance must be accessible from within your AWS VPC
* You must have administrative access to your AWS account
* Ensure you are working in the correct AWS region where your Github Enterprise Server is deployed
In order to set up a PrivateLink connection with Github Enterprise Server, please follow the steps below:
1. **Create a VPC Endpoint Service**
* Follow the detailed instructions in the [Creating a VPC Endpoint Service](#creating-a-vpc-endpoint-service) section below to set up the endpoint service for your GitHub Enterprise Server instance.
2. **Add a Github integration in Elementary** - Once the VPC endpoint service setup is completed, please proceed to adding a [GitHub integration](https://docs.elementary-data.com/cloud/integrations/code-repo/github). Note:
* OAuth is not currently supported for Github Enterprise Server, so you should generate a fine-grained token.
* You should use the same Github hostname as you would internally (Elementary will resolve that host to the privatelink endpoint)
## Creating a VPC Endpoint Service
The setup below is only required for services that are hosted in your own VPC (e.g. Github Enterprise Server). It is not required for Snowflake or Databricks which manage themselves the server-side PrivateLink setup.
Each integration above explicitly states if it requires setting up a VPC endpoint service.
In order to expose services that are hosted within your own VPC, it is required to create a VPC endpoint service. This is essentially the server-side component of PrivateLink, and the destination Elementary's VPC endpoint will connect to.
> **Note:** You should create the VPC endpoint service in the same region your service is located in.
You can follow the steps below to create an endpoint service via the AWS console.
### Create a network load balancer to your service
Before setting up the VPC endpoint service, it is required to set up an internal network load balancer pointing to the service you are exposing.
Setting up a load balancer consists of the following sub-steps:
**1. Create a target group**
Under the EC2 page in AWS, navigate to the Target Groups menu and click "Create target group". Please follow the wizard and fill the following details:
1. **Target type** - You should select this based on the method you used to deploy your service - e.g. based on Instance or IP address.
2. **Target group name** - Choose a name for the target group
3. **Protocol : Port** - Choose this based on the service you are exposing (normally should be HTTP / HTTPS)
4. **IP address type** - IPv4.
5. **VPC** - The VPC your service is deployed in.
**2. Create a Network Load Balancer (NLB)**
Under the EC2 page in AWS, navigate to "Load Balancers" and click on "Create load balancer". Choose **Network Load Balancer** and proceed with the creation. Please follow the wizard and fill the following details:
1. **Load balancer name** - Choose a name for the load balanver (e.g. "github-lb")
2. **Scheme** - Internal.
3. **Load balancer IP address type** - IPv4.
4. **VPC** - Choose same VPC as the one you used for the target group above.
5. **Mappings** - Select one or more private subnets.
6. **Security groups** - Select a security group with access to your service. Please grant access to the relevant ports for your service, to the following IP ranges (these are the internal IPs Elementary may connect to your service from):
* 10.0.1.x
* 10.0.2.x
* 10.0.3.x
7. **Listeners** - Select the target group, protocol, and port from Step 1.
**3. Verify the Target Group is Healthy**
Once the load balancer from the previous step is ready, please navigate to the target group you created above. It should be listed as "Healthy".
If it appears as "Unhealthy" for some reason, please ensure the security group provides access to the service, that the health check is configured correctly, and of course that the service itself is available.
**4. Enable Cross-Zone Load Balancing**
If you selected more than one subnet for the load balancer above:
* Navigate back to the "Load Balancer" screen
* Choose the load balancer you created.
* Click on Actions -> Edit load balancer attributes.
* Enable the setting "Enable cross-zone load balancing".
If you selected more than one subnet (availability zone) when creating the NLB in Step 2, navigate back to the Load Balancer, then select Actions → Edit load balancer attributes. From this page, select "Enable cross-zone load balancing" and save your changes.
### Create a VPC Endpoint Service and approve access for Elementary
**1. Create a VPC Endpoint Service**
Navigate to the VPC page in AWS, go to "Endpoint services," and select "Create endpoint service." Please follow the wizard and fill in the following details:
1. **Name** - Choose a name for your VPC endpoint service.
2. **Load balancer type** - Network.
3. **Available load balancers** - Select the network load balancer (NLB) created above.
4. **Require acceptance for endpoint** - Yes (so new connections will require approval, see below)
5. **Enable private DNS name** - No.
6. **Supported IP address types** - IPv4.
Once the service is created, please go to the "Details" tab and save the "Service name" attribute, you will need later to provide it to the Elementary team.
**2. Allow the Elementary Principal**
Once the VPC endpoint service is successfully created, navigate to the "Allow principals tab" and click on "Allow Principals". Then add the following principal:
```
arn:aws:iam::743289191656:root
```
After the endpoint service finishes creating, navigate to the "Allow principals" section and select "Allow principals." Add Elementary's AWS account ID: `743289191656`.
**3. Contact the Elementary team to configure the PrivateLink connection**
Please provide the Elementary team with the following details:
1. Your AWS account ID.
2. Your AWS region.
3. The VPC endpoint service name (from step 1).
4. The relevant service / integration (e.g. Github).
5. The hostname you use internally to connect to your service.
**4. Accept the endpoint connection request**
Once you got confirmation from the Elementary team that the private link connection is set up, you need to approve
the VPC endpoint connection from Elementary. You can do so with the following steps:
* In the VPC page in your AWS console, go to "Endpoint Services", and then choose the endpoint service that you created in step 1.
* Under the "Endpoint Connections" tab, you should see a pending connection, select it.
* Click on Actions->Accept endpoint connection request to accept the connection.
After a couple of minutes the connection should change from "Pending" to "Available".
**5. Notify the elementary team**
Once the endpoint connection is approved and shows as "Available", please reach out to the Elementary team, so we will ensure the connection is ready and working.
# Microsoft Entra ID
Source: https://docs.elementary-data.com/cloud/integrations/security-and-connectivity/ms-entra
## Enabling SAML
In order to enable SAML using Microsoft Entra ID (Previously Azure AD SSO), we need the following steps to be taken:
* Go to the [Microsoft Entra portal](https://entra.microsoft.com/)
* On the left, choose Applications → Enterprise Applications
* Click on “New Application”
* Click on “Create your own application”
* Choose the last option in the side-window that opens and click “Create”
* In the App window that opens, click on “Single Sign-On”
* Choose SAML
* Click on Edit on the “Basic SAML Configuration” section
* Fill the following entries:
* Identifier (Entity ID) - `elementary`
* Reply URL - [`https://elementary-data.frontegg.com/auth/saml/callback`](https://elementary-data.frontegg.com/auth/saml/callback)
* Download the Federation Metadata XML.
* Go to your account settings page in Elementary (Your avatar in the top right corner -> Account -> Settings)
* In the SSO section, click on "Configure connection"
* Fill in the form with the following details:
* SAML Metadata: Choose "Upload file", upload the Federation Metadata XML you downloaded.
* Domains: Add the domains you want to allow access to Elementary.
* Click on "Save" to save the configuration.
Make sure to verify that login works in an incognito window or with another user before logging out.
If it does not, disable the SSO configuration immediately and contact the Elementary team.
## Provisioning
Elementary supports user provisioning via SCIM to automate user management. If you want to enable automatic provisioning, follow these steps:
* Go to your account settings page in Elementary (Your avatar in the top right corner -> Account -> Settings)
* In the Provisioning section, click on "Configure"
* Choose "Azure AD", and click "Create" to create a new URL and token for provisioning
* DO NOT close this dialog until you have configured SCIM in Azure AD
* In the **Microsoft Entra portal**, go to **Enterprise Applications** and select the newly created SAML application.
* Navigate to **Provisioning** and click **Get Started**.
* Set the **Provisioning Mode** to **Automatic**.
* Configure the **Tenant URL** and **Secret Token** - *value from Elementary Provisioning section*
* Click **Test Connection** to validate the setup.
* Enable provisioning and save changes.
This setup ensures that users are automatically created, updated, and deactivated in Elementary based on their status in Microsoft Entra ID. You can always reach out if you need any help.
# Okta
Source: https://docs.elementary-data.com/cloud/integrations/security-and-connectivity/okta
## Authentication & SSO Integration
### Supported Authentication Protocols
Elementary Cloud supports **Okta Single Sign-On (SSO)** via multiple authentication protocols:
* **SAML 2.0** (Security Assertion Markup Language)
* **OIDC (OpenID Connect)**
These protocols enable seamless authentication, reducing the need for manual credential management.
### SCIM for Automated Provisioning
Elementary Cloud supports **SCIM (System for Cross-domain Identity Management)** for automated user provisioning and deprovisioning:
* **Automated User Creation**: Users added in Okta can be provisioned automatically in Elementary Cloud.
* **Deprovisioning Support**: When a user is removed from Okta, their access to Elementary Cloud is revoked automatically.
* **Group-Based Provisioning**: Okta groups can be mapped to roles in Elementary Cloud by the Elementary team.
For more details on SCIM setup, refer to Okta’s SCIM integration guide: [Okta SCIM Guide](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_scim.htm).
## Security & Access Control
### Multi-Factor Authentication (MFA)
Elementary Cloud does not enforce MFA directly, but any MFA policies configured through Okta will automatically apply once Okta SSO is enabled.
### Role-Based Access Control (RBAC) and Group Sync
* Supports **RBAC with predefined roles** (**Admin, Can Write, Can Read**).
* **Role mappings for group names** can be pre-defined if sent in advance.
* **Role Assignment**:
* The account creator will have a default **Admin** role.
* For provisioned users, If no configuration is made, the **default role will be Can Read**.
* Manually invited users will have the role defined during the invite process.
* **Custom roles** are currently not supported.
## How to Set Up Okta SSO for Elementary Cloud
Please follow the steps below to configure an SSO connection in Elementary.
* Go to Applications → Applications, and click on **Create App Integration**
Choose SAML 2.0 as the sign-in method:
* Name the app “Elementary Data” and click “Next”
* Under **SAML Settings**, please enter the following details:
* Single sign-on URL: **[https://elementary-data.frontegg.com/auth/saml/callback](https://elementary-data.frontegg.com/auth/saml/callback)**
* Audience URI (SP Entity ID): elementary
* Name ID Format: EmailAddress
* Application Username: Email
* Update application username on: Create and update
* Click “Finish” on the next screen, and the app will be created!
* Now, let’s configure users / groups that have access to the app. To do so, please go to the “Assignments” tab and add relevant groups / users.
* Note -
* It is recommended to set up an “Elementary Users” group dedicated for this purpose, though you can also add access for individual users.
* The setting below is for **assignment** of users to your app, e.g. users that are permitted to login to Elementary via Okta. This does not cover actually provisioning the user in Elementary (this is covered in the next section).
* Go to the “Sign On” tab, and copy the link under Metadata URL:
* Go to your account settings page in Elementary (Your avatar in the top right corner -> Account -> Settings)
* In the SSO section, click on "Configure connection"
* Fill in the form with the following details:
* SAML Metadata: Choose "URL", paste the link you copied from Okta into the URL field, and click "Fetch".
* Domains: Add the domains you want to allow access to Elementary.
* Click on "Save" to save the configuration.
Make sure to verify that login works in an incognito window or with another user before logging out.
If it does not, disable the SSO configuration immediately and contact the Elementary team.
## How to Set Up SCIM for Automated Provisioning
This section covers how to automatically provision users and groups from Okta in Elementary. If you prefer, it is also possible to set up the SSO part without provisioning. In that case, users can be invited to the platform via the Team page in Elementary.
Please follow the steps below to configure SCIM provisioning within Elementary:
* Go to your account settings page in Elementary (Your avatar in the top right corner -> Account -> Settings)
* In the Provisioning section, click on "Configure"
* Choose "Okta", and click "Create" to create a new URL and token for provisioning
* DO NOT close this dialog until you have configured SCIM in Okta
* Under the **Elementary Data** app, go to the **General** tab, and click **Edit**. Then modify the **Provisioning** setting to **SCIM** and click **Save**.
A new Provisioning tab should appear, click it and then click Edit.
* Please fill the following details:
* **SCIM connector base URL** - *value from Elementary Provisioning section*
* **Unique identifier field for users** - email
* **Supported provisioning actions** - mark all the “Push” settings (New users, Profile updates and Groups).
* **Authentication Mode -** HTTP Header
* **Authorization** - *access token from Elementary Provisioning section*
When you are done, click on **Test Connector Configuration**
Ensure that all the marked provisioning actions were successful:
* Click **Save** to update the provisioning configuration.
* Click the **To App** section on the left and click **Edit**:
* Please enable the settings:
* Create Users
* Update User Attributes
* Deactivate Users
And click **Save.**
If you already created an “Elementary Users” group under the Assignments tab in the previous section, you may want to remove and re-add it to ensure all the users there are created successfully in Elementary.
## **Pushing groups to Elementary**
As a part of the provisioning setup for Elementary, you can also choose to provision **user groups** to control permissions within Okta.
These can be mapped to roles within Elementary (such as **Can Edit** or **Admin**).
To do so, under the Elementary Data app, do the following
* Click on **Push Groups**
* Add the groups you would like to push:
* If the **Push Status** appears as **Active** - it means the groups were successfully pushed to Elementary.
* Please ask the Elementary team to map the groups you pushed to roles within Elementary. In this case the mapping is clear:
* **Elementary Admins** - Admin.
* **Elementary Editors** - Can Edit.
* Once this is done, you should be able to see in the **Team** page in Elementary all the users and their correct roles.
# Snowflake Semantic Views
Source: https://docs.elementary-data.com/cloud/integrations/semantic-layer/snowflake
Elementary discovers and catalogs your Snowflake Semantic Views as first-class assets, connecting them to your lineage graph and exposing their metrics, dimensions, and facts alongside your dbt models and sources.
## What Elementary surfaces
For each Snowflake Semantic View, Elementary exposes:
* **Metrics, dimensions, and facts** — listed in a dedicated semantic elements table on the asset detail page.
* **Asset-level lineage** — the semantic view is connected to the physical tables it depends on, so you can trace data issues upstream and assess downstream impact.
* **Column-level lineage** — individual metrics, dimensions, and facts are traced back to the specific columns in the underlying physical tables.
Semantic views appear in the **catalog tree** under their Snowflake database and schema, with a distinct teal color and Snowflake icon in the lineage graph so they are easy to identify at a glance.
## How it works
Once Elementary is connected to your Snowflake environment, semantic views are discovered automatically on each sync. No manual configuration is required.
Elementary reads semantic view definitions directly from Snowflake's account usage, extracting the structure of each view and resolving lineage to the physical tables it depends on.
Semantic layer support is currently available for **Snowflake only**. Support for additional platforms (dbt Semantic Layer, Cube, etc.) is planned.
## Prerequisites
* Snowflake connection configured in your Elementary environment. See [Snowflake setup](/cloud/integrations/dwh/snowflake).
* `ACCOUNT_USAGE` privilege granted to the Elementary Snowflake user.
# Airflow
Source: https://docs.elementary-data.com/cloud/integrations/transformation-and-orchestration/airflow
## Jobs information
Elementary can collect jobs data from Airflow, including `job run url`, `job name`, `job id`.
These are presented in the lineage node info tab.
To collect jobs info for Airflow, check out [this guide](/cloud/guides/collect-job-data).
## Syncs schedule
For near-real-time results, it is recommended to change the sync configuration from the default hourly syncs to [webhook triggered syncs](/cloud/guides/sync-scheduling) and invoke using Airflow.
# dbt cloud
Source: https://docs.elementary-data.com/cloud/integrations/transformation-and-orchestration/dbt-cloud
Elementary integrates with dbt Cloud for the supported data warehouses.
[Install the Elementary dbt package](/data-tests/dbt/quickstart-package) in your monitored project. It collects results and artifacts at runtime using on-run-end hooks, saving them to a dedicated schema. Elementary syncs through the DWH connection.
## Jobs information
Elementary automatically collects jobs data from dbt Cloud, including `job run url` and `job id`. These are presented in the lineage node info tab.
To collect additional job info such as `job name`, see the [collect job data guide](/cloud/guides/collect-job-data).
## Connect dbt Cloud to the agent
Connect dbt Cloud to Elementary's agent so it has complete context when triaging issues. The agent can then pull dbt Cloud run and job context directly through its MCP tools, so root cause analysis reaches across your transformation layer.
Go to **Account → Agent Settings → MCP Connections**, add a dbt Cloud connection, and paste your dbt Cloud MCP server URL and a service token. Tokens are stored encrypted.
## Syncs schedule
For near-real-time results, it is recommended to change the sync configuration from the default hourly syncs to [webhook triggered syncs](/cloud/guides/sync-scheduling).
## Setup guides
Configure webhook-triggered syncs for near-real-time results.
Collect additional job metadata like job name from dbt Cloud.
Set up separate environments for development and production.
Connect and manage multiple dbt environments in Elementary.
# dbt core
Source: https://docs.elementary-data.com/cloud/integrations/transformation-and-orchestration/dbt-core
Elementary integrates with dbt core (1.3.0 and above) for the supported data warehouses.
[Install the Elementary dbt package](/data-tests/dbt/quickstart-package) in your monitored project. It collects results and artifacts at runtime using on-run-end hooks, saving them to a dedicated schema. Elementary syncs through the DWH connection.
## Syncs schedule
For near-real-time results, it is recommended to change the sync configuration from the default hourly syncs to [webhook triggered syncs](/cloud/guides/sync-scheduling).
## Setup guides
Configure webhook-triggered syncs for near-real-time results.
Set up separate environments for development and production.
Connect and manage multiple dbt environments in Elementary.
Sync dbt source freshness results into Elementary.
# dbt Fusion (Beta)
Source: https://docs.elementary-data.com/cloud/integrations/transformation-and-orchestration/dbt-fusion
Note: dbt-fusion support in Elementary is still in beta, as is dbt-fusion itself. Please see below a [list of features that are not yet implemented](#missing-features-in-dbt-fusion).
Elementary's dbt package integrates with dbt-fusion, starting with version 0.20.
Fusion is a complete rewrite of the dbt engine, and provides many benefits, including enhanced performance and static analysis.
For more details about dbt-fusion capabilities please consult the [dbt-fusion docs](https://docs.getdbt.com/docs/fusion/about-fusion).
## Upgrading to dbt-fusion
As a part of the migration to dbt-fusion, it is required to remove deprecated syntax from various areas of the dbt project, and YAMLs in particular.
Specifically for tests the following are important:
1. Test arguments must be encapsulated under an `arguments` field.
2. Configuration fields such as `meta`, `tags` or `severity` must be encapsulated under a `config` field.
3. Recommended (but not a must) - change the `tests` field to `data_tests` to conform with the current dbt guidelines.
Here's an example of an Elementary anomaly test with the old and new syntax:
```yml Old syntax theme={null}
models:
- name: login_events
config:
elementary:
timestamp_column: "loaded_at"
tests:
- elementary.volume_anomalies:
where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
time_bucket:
period: day
count: 1
tags: ["elementary"]
severity: warn
meta:
owner: "@jessica.jones"
- name: users
tests:
- elementary.volume_anomalies:
tags: ["elementary"]
```
```yml New syntax theme={null}
models:
- name: login_events
config:
elementary:
timestamp_column: "loaded_at"
data_tests:
- elementary.volume_anomalies:
arguments:
where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
time_bucket:
period: day
count: 1
config:
tags: ["elementary"]
severity: warn
meta:
owner: "@jessica.jones"
- name: users
data_tests:
- elementary.volume_anomalies:
config:
tags: ["elementary"]
```
To facilitate the process of making these changes within your project, dbt introduced a tool called [dbt-autofix](https://github.com/dbt-labs/dbt-autofix) that
can be used to automatically migrate your project to the new syntax.
Before running this tool, **please upgrade** to the most recent dbt-core version - as some of the syntax changes are not supported on older versions.
For full information on upgrading from dbt-core to dbt-fusion, please check [dbt-fusion's official upgrade guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-fusion).
## Supported Elementary capabilities
Most of the main capabilities of the Elementary dbt package are supported in Fusion, including:
1. Anomaly detection tests.
2. Schema tests.
3. Artifacts collection.
However, the following capabilities are not supported right now for dbt Fusion:
1. Python tests.
2. JSON schema tests.
3. The missing dbt-fusion features listed below.
## Missing features in dbt-fusion
In addition to the above, there are some features that are currently missing from dbt-fusion, and therefore are not yet available in the Elementary package.
For each one of the issues below, we've included a link to the dbt-fusion github repository - **please upvote these features if they are important to you!**
1. [Tests with error status are not being reported](https://github.com/dbt-labs/dbt-fusion/issues/686) - If a test fails "normally" (e.g. not\_null fails on rows with null values), Elementary will report it as expected with a "fail" status.
However, if there is a compilation error / any error that comes before the actual test query ("error" status in dbt-core), it will currently be missing.
2. [Source freshness results are not reported](https://github.com/dbt-labs/dbt-fusion/issues/720)
3. [Exposure artifacts are not reported](https://github.com/dbt-labs/dbt-fusion/issues/859)
4. [Group artifacts are not reported](https://github.com/dbt-labs/dbt-fusion/issues/25)
5. [Compiled code is missing from dbt artifact tables](https://github.com/dbt-labs/dbt-fusion/issues/723)
6. [Failed row count is missing from dbt artifact tables](https://github.com/dbt-labs/dbt-fusion/issues/724)
## Setup guides
Configure webhook-triggered syncs for near-real-time results.
Set up separate environments for development and production.
Connect and manage multiple dbt environments in Elementary.
# Orchestration tools
Source: https://docs.elementary-data.com/cloud/integrations/transformation-and-orchestration/orchestration-tools
## Jobs information
Elementary can collect jobs data from your orchestration tool, including `job run url`, `job name`, `job id`.
These are presented in the lineage node info tab.
To collect jobs info for your orchestrator, check out [this guide](/cloud/guides/collect-job-data).
## Syncs schedule
For near-real-time results, it is recommended to change the sync configuration from the default hourly syncs to [webhook triggered syncs](/cloud/guides/sync-scheduling) and invoke using your orchestration tool.
# Elementary Cloud Platform
Source: https://docs.elementary-data.com/cloud/introduction
Elementary Cloud is a data observability platform for data teams that need to ship reliable data at scale. It monitors your data quality, manages incidents, enforces governance, and runs AI agents that automate coverage, triage, and documentation — all connected through a unified lineage graph across your warehouse, dbt project, and BI tools.
Automates triage, test coverage, governance, and performance workflows.
The knowledge layer that connects your stack and powers the agent.
Expose your data context to Claude, Cursor, and any MCP-compatible tool.
## Get started
# MCP Tools
Source: https://docs.elementary-data.com/cloud/mcp/mcp-tools
This is full list of tools currently exposed by the Elementary MCP Server.
These tools allow agents and interfaces to retrieve information from your Elementary environment, such as models, tests, lineage, incidents, and coverage, in a structured and consistent way.
You won’t need to use these tools directly. They’re used behind the scenes by any MCP-compatible client.
| **Category** | **Tool Name** | **Purpose** | **Output** |
| -------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- |
| **Discovery & Lineage** | `get_asset` | When you need info about any asset type, from source to BI (table assets, BI assets) | Complete asset details with general information |
| | `get_upstream_assets` | When investigating data lineage or debugging issues upstream | Asset dependencies with metadata (id, name, type, criticality, tags, owners, description, path) |
| | `get_downstream_assets` | When assessing impact of changes or understanding data consumers | Asset dependencies with metadata (id, name, type, criticality, tags, owners, description, path) |
| | `get_column_upstream_columns` | When tracing column-level lineage for specific data fields | Column dependencies with their metadata (id, name, type, table info) |
| | `get_column_downstream_columns` | When understanding column usage and impact analysis | Column dependencies with their metadata (id, name, type, table info) |
| | `get_table_asset` | When you need table-specific details like schema, columns, and tests | Table asset with columns, materialization, execution status, and associated tests |
| | `get_table_assets` | When searching for tables by name, tags, owners, or other criteria | List of table assets matching filter criteria |
| | `get_table_asset_compiled_query` | When you need to see the actual SQL code being executed | Compiled SQL query from the latest execution |
| | `get_table_asset_execution_history` | When investigating table execution patterns or failures over time | Historical execution records with timestamps and status |
| | `get_table_asset_execution` | When debugging a specific table execution failure | Detailed execution logs, timing, and error information |
| **Test & Coverage** | `get_assets_sorted_by_coverage` | When prioritizing which assets need more test coverage | Assets ranked by test coverage percentage |
| | `get_test` | When investigating test configuration or recent test results | Test details including config, execution status, and quality dimensions |
| | `get_tests` | When searching for tests by name, type, or asset association | List of tests matching filter criteria |
| | `get_test_execution_metrics` | When analyzing specific test failure metrics that caused anomaly | Test execution results, numerical metrics, and anomaly details |
| | `get_test_execution_history` | When reviewing test performance patterns over time | Historical test execution records and results |
| | `get_test_execution` | When debugging a specific test execution failure | Detailed test execution logs, results, and error information |
| | `get_tests_catalog` | When helping users add tests to their dbt project | Available test types that can be defined in the dbt project |
| **Incidents** | `get_incidents` | When searching for incidents by time range, status, severity, or assignee | List of incidents with timing, assignee, status, severity, tags, and ticket information |
| | `get_incident` | When investigating details of a specific incident | Complete incident details including source asset/test information and execution context |
| | `get_asset_incidents_history` | When reviewing past incident patterns for a specific asset | Historical incident records for an asset with status and timing |
| **Environment Management** | `get_environments` | When starting a session to discover available data environments | Dictionary mapping environment IDs to environment names |
# Elementary MCP Server
Source: https://docs.elementary-data.com/cloud/mcp/overview
The Elementary MCP Server allows you to connect your Elementary environment to any client that supports MCP (Model Context Protocol). This lets AI agents, copilots, or natural language interfaces query and act on your data stack using live context.
## What is MCP?
MCP (Model Context Protocol) is an open protocol introduced by Anthropic. It defines a way for AI systems to retrieve context and trigger actions from external tools in a structured way.
The Elementary MCP Server exposes an interface to query key parts of your data environment — including:
* Models
* Tests
* Incidents
* Test coverage
* Lineage (dbt + BI), including column-level
* Incidents
This enables workflows like:
* Asking “What’s the status of the model feeding the revenue dashboard?”
* Automatically creating a freshness test for a column
* Browsing lineage to find upstream causes of issues
* Triggering updates or syncs without opening a UI
## How it works
The MCP Server runs as a remote service and is exposed via a single authenticated endpoint. It is compatible with any MCP-enabled client, such as Claude, Cursor IDE and Custom agents and LLM copilots.
## Supported operations
You can use the Elementary MCP Server for:
* **Full asset details** - Metadata, column definitions, and test coverage
* **Lineage** - Explore table + column-level lineage across dbt and BI tools
* **Incidents -** View open incidents and their context
* **Tests -** Browse and add tests using the test catalog
* **Models -** Inspect model metadata and status
* **Execution History -** View historical runs and performance details
## Coming soon
We're expanding support to include:
* Data health summaries
* Volume and freshness metrics
* Sync triggers
* Cloud test config updates
# Recommended Rules
Source: https://docs.elementary-data.com/cloud/mcp/recommended-rules
asset-discovery-and-metadata.mdc
```mdc theme={null}
---
description: "Use Elementary MCP for authoritative asset and metadata discovery."
alwaysApply: true
---
## When to prefer MCP
- You need to find or summarize assets (models, tables, columns) with owners, tags, subscribers, freshness, and incident history.
- You're answering "What is this asset?", "Who owns it?", or "What depends on it?"
- You want up-to-date, production metadata rather than local project assumptions.
## Guidance
- Query Elementary MCP for full metadata and lineage pointers instead of scanning project files.
- Return a concise summary: asset name, type, owners, freshness, criticality, recent incidents, and related tests.
- If nothing matches, say so clearly and suggest refining filters (name, owner, tag, time window).
### Examples
**Good** → "Fetching authoritative metadata for `customer_activity_daily` via observability service; summarizing owners, freshness, and downstream consumers."
**Avoid** → "Parsing local SQL files or repo comments to guess owners or status."
```
elementary-best-practices.mdc
```mdc theme={null}
---
description: "General best practices when interacting with Elementary MCP."
alwaysApply: true
---
## Guidance
- Work in a context of a specific environment. If not provided in advance, ask the user for the environment they want to work in.
- Apply filters (name, tag, owner, time range) to narrow scope.
- Before applying any detailed filters, begin by retrieving a count of assets. This provides a sense of the dataset size, allowing you to iteratively apply filters (such as by name, tag, owner, or time window) to efficiently narrow your results.
- Use lightweight mode or limit fields for large result sets.
- Handle missing assets gracefully - return a clear message.
### Examples
**Good** → "Query MCP with owner='Data Platform', time window=7d, lightweight=true."
**Avoid** → "Fetching entire environment data without filters."
```
impact-analysis-and-lineage.mdc
```mdc theme={null}
---
description: "Leverage Elementary MCP to assess upstream and downstream impact — including cross-project dependencies, and to analyze column-level lineage for safe, comprehensive change management."
alwaysApply: true
---
## When to prefer MCP
- Before modifying, renaming, or removing a model/table/column.
- When asked "What breaks if we change X?" or "Where does this column come from?"
- When downstream consumers may exist **outside this repo** (e.g., dashboards, jobs in other projects).
## Guidance
- Use MCP to map:
- **Downstream consumers** (including dashboards/assets external to this project).
- **Upstream sources** and transformations.
- **Column-level lineage** to explain how columns are populated and which downstream columns depend on them.
- Provide a **safe change plan**: sequence updates, notify owners, define rollback steps.
### Examples
**Good** → "End-to-end lineage: `finance_orders.net_revenue` feeds an external revenue dashboard. coordinate before renaming."
**Avoid** → "Grepping this repo and assuming there are no external consumers."
```
incidents-and-data-health.mdc
```mdc theme={null}
---
description: "Use Elementary MCP to investigate incidents, data-quality issues and assess asset health."
alwaysApply: true
---
## When to prefer MCP
- An incident or test failure is mentioned
- You need to check the current health score for an asset.
- You need **history** (when it started, status changes, recurrence) and **related context** (impacted assets/tests).
- You're doing daily or post-deploy health checks.
## Guidance
- Use MCP to:
- List active or recent incidents (filter by severity, status, or time window).
- Retrieve details: affected downstream assets, related tests/monitors, incident timeline and status.
- Prefer MCP summaries over reading raw logs/stdout directly.
- If none found, state that and suggest adjusting filters/time window.
### Examples
**Good** → "Using MCP: show all incidents in the last 24h with affected assets, related tests, and current resolution status."
**Avoid** → "Inspecting raw process logs to infer which asset failed."
```
pre-deploy-impact-guardrails.mdc
```mdc theme={null}
---
description: "Before merge/deploy, use Elementary MCP to detect downstream breakage—including external dashboards."
alwaysApply: true
---
## When to prefer MCP
- During review or deployment to ensure changes won't break consumers.
- When verifying dashboards, jobs, or models that rely on changed assets.
- When downstream assets lie **outside this repo**.
## Guidance
- Run a **downstream impact check** via MCP:
- Identify all dependent assets (dashboards, other models).
- Highlight high-risk paths where changes alter downstream **columns** or semantics.
- Use **column-level lineage** to describe how each change propagates.
- If risk exists, recommend: coordination with owners, temporary compatibility (dual-writing/shim columns), or deferring merge until dependents update.
### Examples
**Good** → "Pre-deploy: MCP shows `orders_enriched.ltv` feeds 'Revenue Overview'—notify owners and add a temporary alias before merge."
**Avoid** → "Assuming safety because tests here pass; skipping checks for external dashboards."
```
## Paste into cursor to apply rules!
```placeholder theme={null}
Add the rules from here to the project rules
https://docs.elementary-data.com/cloud/mcp/recommended-rules
```
# MCP Setup Guide
Source: https://docs.elementary-data.com/cloud/mcp/setup-guide
This guide walks you through connecting any MCP-compatible client (Cursor, Claude Desktop, and others) to Elementary's remote MCP server.
The server is available at:
```
https://prod.api.elementary-data.com/mcp/
```
## Authentication
Elementary's MCP server supports the **Streamable HTTP** transport with **OAuth 2.1** and dynamic client registration.
When you start the MCP server for the first time, your client opens a browser window so you can log in with your Elementary account. The MCP server uses your authenticated user's permissions directly (user-scoped).
Use the following config in the client setup steps below:
```json filename="mcp.json" theme={null}
{
"mcpServers": {
"Elementary": {
"url": "https://prod.api.elementary-data.com/mcp/"
}
}
}
```
Re-authentication is only required when your session expires or you explicitly sign out.
Use this method for headless setups, CI environments, or any client that doesn't support browser-based OAuth.
#### Prerequisites
| Requirement | Why it's needed |
| --------------------------- | -------------------------------------------------------- |
| **Node.js 18 LTS or newer** | `mcp-remote` runs on Node. Verify with `node --version`. |
| **npm / npx** | `npx` fetches `mcp-remote` on demand. Bundled with Node. |
If you don't have Node.js installed, use [nvm](https://github.com/nvm-sh/nvm) for easy installation:
```bash theme={null}
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
# Restart your terminal, then:
nvm install --lts && nvm use --lts
```
#### Generate an access token
Go to [User → Personal Tokens](https://app.elementary-data.com/settings/user-tokens) or [Account → Account Tokens](https://app.elementary-data.com/settings/account-tokens) and click **Generate token**. Copy the token and store it securely — it is shown only once.
* **Personal tokens** are user-scoped and inherit your workspace permissions.
* **Account tokens** are account-scoped with "Can View" permissions.
Treat tokens like passwords: do not share or commit them, rotate regularly, and revoke immediately if compromised.
#### Install `mcp-remote` (optional)
Our configs use `npx` to fetch `mcp-remote` automatically — no global install required. If you prefer a global install:
```bash theme={null}
npm install -g mcp-remote@latest
```
Use the following config in the client setup steps below:
```jsonc filename="mcp.json" theme={null}
{
"mcpServers": {
"elementary-remote": {
"command": "npx",
"args": [
"-y",
"mcp-remote@latest",
"https://prod.api.elementary-data.com/mcp/",
"--header",
"Authorization:${AUTH_HEADER}"
],
"env": {
"AUTH_HEADER": "Bearer "
}
}
}
}
```
Replace `` with the token you generated above.
***
## Configure your client
Adding a custom connector is gated by your Claude organization's admin policy, not by Elementary. If you see a **"Missing permissions"** banner when opening the add-connector dialog, your Claude workspace role isn't allowed to add custom connectors. Ask a Claude **organization admin** to either enable custom connectors for members or add the Elementary connector org-wide.
Click the link below to open Claude with the Elementary connector pre-filled:
[Add Elementary connector →](https://claude.ai/customize/connectors?modal=add-custom-connector\&connectorName=Elementary\&connectorUrl=https%3A%2F%2Fprod.api.elementary-data.com%2Fmcp)
Click **Add**. The Elementary connector will appear in your connectors list.
Click **Connect**. A browser window opens for you to log in with your Elementary account.
In the menu bar, choose **Claude → Settings → Developer** and click **Edit** under **Local MCP Servers**.
Paste the JSON block from the authentication tab above and save.
Quit and reopen Claude Desktop. If using OAuth, a browser window opens on first launch for authentication.
Go to **Settings → Model Context Protocol → Add Custom Server**, or edit `~/.cursor/mcp.json` (global) / `/.cursor/mcp.json` (workspace).
Paste the JSON block from the authentication tab above and save. Cursor restarts its MCP agent automatically.
If using OAuth, a browser window opens on first launch. Log in with your Elementary account to complete authentication.
# Python SDK Overview
Source: https://docs.elementary-data.com/cloud/python-sdk/overview
More teams are shifting their data quality checks out of dashboards and into the transformation layer itself. It's obvious why: the transformation code is the first place that touches real data. If a check fails here, the pipeline stops before corrupted rows ever land downstream. No backfills, no detective work, no "how long has this been wrong?" scramble.
And the value cuts both ways:
* **You catch issues before they ever hit the data warehouse or lake**, right at the ingestion and preprocessing layers.
* **You catch issues after the data warehouse too**, in the pipelines that stream data to downstream destinations, models, APIs, and operational systems.
## Python: The Backbone of Modern Data Engineering
Python has become the backbone of modern data engineering - especially in pipelines that go beyond SQL. It now drives:
* Ingestion and storage of **unstructured data**
* **Vectorization and embedding pipelines** for AI systems
* **ML model training** and feature generation
* Monitoring of **model inputs and outputs**
* Hybrid pipelines that mix structured, semi-structured, and free-form data
As these pipelines multiply, Python becomes the glue. It runs wherever data flows — before the DWH, inside the DWH, and after the DWH — making it the natural place for data quality and observability to live.
## Wrapping Existing Tools Instead of Inventing New Ones
Engineers already have strong opinions about how they want to write tests. Some rely on Great Expectations, others on DQX, pytest-based workflows, or homegrown frameworks. Reinventing a new test engine or DSL would just fragment the landscape - so we didn't.
We focused on the simplest possible layer: **a lightweight Python SDK that captures any Python test result, from any framework**, and reports it to Elementary. You keep your code - we handle the metadata, structure, and visibility.
This means full observability without dictating how you build.
## Built for Teams That Treat Their Data Pipelines Like Software
Elementary has always leaned into engineering-first workflows. Our deep integration with dbt set that foundation. Extending this into Python is the natural continuation of that approach.
As more transformations shift into Python (Pyspark, SQL generation, AI/ML pipelines, unstructured data processing), teams want the same capabilities they rely on when using Elementary with dbt:
* Understand what ran
* Track when it ran
* Measure how long it took
* Identify which upstream assets fed it
* Trace which downstream assets it produced
* Run data quality checks on the product and see the results
* Get alerts on data issues as soon as they happen
The SDK provides exactly that by wrapping the transformation code itself. You get execution metadata, lineage, run context, and full test surface — directly from inside your existing codebase.
This unifies **Analytics Engineering, Data Science, and AI/ML Operations** into a single observability platform. Python + dbt + cloud tests now all land in one place.
## What You'll See in Elementary Once You Report Through the SDK
When a Python pipeline reports assets, test results, and execution metadata, everything shows up in Elementary unified with your dbt and cloud tests:
* **All test results appear together** — Python validations, dbt tests, cloud tests — in a single, consistent interface.
* **Alerts fire through your existing channels** (Slack, PagerDuty, email), ensuring that pipeline-level issues trigger the same operational flow as warehouse-level ones.
* **Incidents are created automatically** for detected issues, including opening Jira tickets. Elementary's agentic tools then investigate root cause, assess downstream impact, and guide resolution.
* **Lineage becomes fully connected**, tying together Python assets, dbt models, warehouse tables, unstructured data, vectors, and ML outputs.
* **Every table, view, file, or vector store entity produced by Python becomes discoverable** through the Elementary catalog, data discovery agent, and MCP server — giving analysts, DS, and AI teams a shared understanding of the entire data ecosystem.
This closes the gap between ingestion pipelines, warehouse transformations, ML prep code, and AI workloads — all observed in one place.
## Next Steps
Learn how to install and configure the Python SDK
See how to report assets and test results from your Python pipelines
## Get Started
The SDK is now in beta and already surfacing surprisingly rich insights from Python-based workflows. If you want early access or want to see how it fits your implementation, [reach out to the team](https://meetings-eu1.hubspot.com/joost-boonzajer-flaes/intro-call-docs?uuid=17a4a61f-d0d3-4cbc-9362-56e37483f6f5) — we're shaping this with real teams using it at scale.
# Python SDK Setup Guide
Source: https://docs.elementary-data.com/cloud/python-sdk/setup-guide
This guide walks you through installing and configuring the Elementary Python SDK to report assets, test results, and execution metadata from your Python pipelines.
## Installation
The Python SDK is installed via pip. Installation instructions and package requirements will be provided when you get access to the beta.
## Configuration
The Python SDK requires configuration to connect to your Elementary Cloud environment. You'll need:
1. **Elementary Cloud API Key** - Get this from your Elementary Cloud account settings
2. **Environment ID** - Your Elementary environment identifier
Configuration can be done via environment variables or a configuration file, similar to how the Elementary CLI is configured.
## Basic Concepts
The SDK is designed to work with any Python testing framework. The core functionality includes:
* **Reporting assets** - Tables, files, vector stores, or any data entity produced by your Python pipeline
* **Reporting test results** - Results from any testing framework (Great Expectations, pytest, custom frameworks, etc.)
* **Tracking execution metadata** - Pipeline runs, timing, status, and errors
* **Reporting lineage** - Connecting Python assets to upstream and downstream dependencies
## Integration with Existing Frameworks
The SDK works with any Python testing framework. You can wrap your existing tests to report results to Elementary without changing your test logic:
* **Great Expectations** - Report GE validation results
* **Pytest** - Report pytest test outcomes
* **Custom frameworks** - Report results from any homegrown testing solution
* **DQX and other tools** - The SDK is framework-agnostic
## Reporting Execution Metadata
Track pipeline execution details including:
* Pipeline name and environment
* Start and end times
* Duration
* Success/failure status
* Error messages and stack traces
## Reporting Lineage
Connect your Python assets to upstream and downstream dependencies, creating a complete lineage graph that includes:
* Python pipeline outputs
* dbt models
* Warehouse tables
* Unstructured data sources
* Vector stores
* ML model outputs
## Next Steps
* See [Usage Examples](/cloud/python-sdk/usage-examples) for conceptual examples
* Learn about the [SDK Overview](/cloud/python-sdk/overview)
## Need Help?
If you need assistance with setup or have questions about the SDK, [reach out to the team](https://meetings-eu1.hubspot.com/joost-boonzajer-flaes/intro-call-docs?uuid=17a4a61f-d0d3-4cbc-9362-56e37483f6f5).
# Python SDK Usage Examples
Source: https://docs.elementary-data.com/cloud/python-sdk/usage-examples
This page provides conceptual examples of how the Elementary Python SDK can be used in different scenarios.
## Reporting Assets
### Tables and Views
Report tables or views created by your Python pipeline. Include metadata like schema, database, and description to make them discoverable in the Elementary catalog.
### Files and Unstructured Data
Report files, blobs, or unstructured data stored in object storage (S3, GCS, Azure Blob, etc.). Include location, format, and other relevant metadata.
### Vector Stores
Report vector stores used in AI/ML pipelines. Include information about the store type (Pinecone, Weaviate, etc.), index names, and dimensions.
## Reporting Test Results
### Basic Test Results
Report simple test outcomes - whether a test passed or failed, along with the test name and type.
### Detailed Test Results
Report comprehensive test information including:
* Test name and type
* Pass/fail status
* Actual vs expected values
* Column-level details
* Failed row counts
* Sample data from failed rows
### Framework Integration
Report test results from any framework:
* Wrap Great Expectations validations
* Report pytest outcomes
* Capture results from custom test frameworks
* Integrate with DQX or other data quality tools
## Complete Pipeline Example
A typical Python pipeline using the SDK would:
1. **Start tracking** - Begin a pipeline run with metadata (name, environment)
2. **Report input assets** - Document what data sources the pipeline consumes
3. **Execute transformations** - Run your existing Python code
4. **Report output assets** - Document what the pipeline produces
5. **Run and report tests** - Execute data quality checks and report results
6. **End tracking** - Complete the run with success/failure status and timing
This creates a complete observability record in Elementary, unified with your dbt and cloud tests.
## Integration with Orchestrators
The SDK can be integrated with any orchestrator:
* **Airflow** - Wrap your Python tasks to report execution and test results
* **Prefect** - Use the SDK in Prefect flows and tasks
* **Dagster** - Report assets and tests from Dagster ops
* **Custom orchestrators** - Works with any Python-based orchestration system
## ML Pipeline Example
For ML pipelines, you can:
* Report training data assets
* Report model artifacts
* Report test/validation datasets
* Report model performance metrics as test results
* Track model training runs
* Connect models to their training data and downstream consumers
This provides full observability for ML workflows alongside your data engineering pipelines.
## Next Steps
* Review the [Setup Guide](/cloud/python-sdk/setup-guide) for installation and configuration
* Learn about the [SDK Overview](/cloud/python-sdk/overview)
# Quickstart
Source: https://docs.elementary-data.com/cloud/quickstart
Welcome to Elementary!
Elementary is the platform for data observability, quality, and governance. Connect your stack and you get a team of AI agents that handle test coverage, incident triage, governance, and performance monitoring — continuously, without manual configuration.
There are two ways to get started:
* **Start a 30-day free trial** and set everything up yourself — perfect for smaller teams or hands-on users.
* **For larger teams or more tailored needs, [reach out to the team](https://meetings-eu1.hubspot.com/joost-boonzajer-flaes/intro-call-docs?uuid=17a4a61f-d0d3-4cbc-9362-56e37483f6f5)** and we'll help you get set up.
### Self-serve setup
[Sign up to Elementary](https://elementary-data.frontegg.com/oauth/account/sign-up) using Google SSO or email.
[Add the Elementary dbt package](/data-tests/dbt/quickstart-package) to your dbt project. It collects test results, run metadata, and dbt artifacts automatically at runtime using on-run-end hooks.
In Elementary, create your first environment and [connect it to your data warehouse](/cloud/integrations/elementary-integrations). Elementary only needs access to the Elementary schema, not your underlying data.
Connect your [code repository](/cloud/integrations/code-repo/connect-code-repo), [Slack or other alert channels](/cloud/integrations/alerts/slack), and [BI tools](/cloud/integrations/bi/connect-bi-tool) to get full visibility across your data workflows.
[Invite team members](/cloud/features/roles-and-permissions) and assign roles so everyone has access to what they need.
### Get the most out of Elementary
* [Take our product tour](https://www.elementary-data.com/product-tour-signup) for a guided walkthrough of key capabilities
* Follow the full [technical setup guide](/cloud/guides/set-up-elementary) for a detailed checklist
### Need help?
We can provide [support on Slack](https://elementary-data.com/community) or hop on a [guided onboarding call](https://meetings-eu1.hubspot.com/joost-boonzajer-flaes/intro-call-docs?uuid=17a4a61f-d0d3-4cbc-9362-56e37483f6f5).
# Community
Source: https://docs.elementary-data.com/cloud/resources/community
# Pricing
Source: https://docs.elementary-data.com/cloud/resources/pricing
# AI Data Validations
Source: https://docs.elementary-data.com/data-tests/ai-data-tests/ai_data_validations
**Beta Feature**: AI data validation tests is currently in beta. The functionality and interface may change in future releases.
**Version Requirement**: This feature requires Elementary dbt package version 0.18.0 or above.
# AI Data Validation with Elementary
## What is AI Data Validation?
Elementary's `elementary.ai_data_validation` test allows you to validate any data column using AI and LLM language models. This test is more flexible than traditional tests as it can be applied to any column type and uses natural language to define validation rules.
With `ai_data_validation`, you can simply describe what you expect from your data in plain English, and Elementary will check if your data meets those expectations. This is particularly useful for complex validation rules that would be difficult to express with traditional SQL or dbt tests.
## How It Works
Elementary leverages the AI and LLM capabilities built directly into your data warehouse. When you run a validation test:
1. Your data stays within your data warehouse
2. The warehouse's built-in AI and LLM functions analyze the data
3. Elementary reports whether each value meets your expectations based on the prompt
## Required Setup for Each Data Warehouse
Before you can use Elementary's AI data validations, you need to set up AI and LLM capabilities in your data warehouse:
### Snowflake
* **Prerequisite**: Enable Snowflake Cortex AI LLM functions
* **Recommended Model**: `claude-3-5-sonnet`
* [View Snowflake's Guide](/data-tests/ai-data-tests/supported-platforms/snowflake)
### Databricks
* **Prerequisite**: Ensure Databricks AI Functions are available
* **Recommended Model**: `databricks-meta-llama-3-3-70b-instruct`
* [View Databrick's Setup Guide](/data-tests/ai-data-tests/supported-platforms/databricks)
### BigQuery
* **Prerequisite**: Configure BigQuery to use Vertex AI models
* **Recommended Model**: `gemini-1.5-pro`
* [View BigQuery's Setup Guide](/data-tests/ai-data-tests/supported-platforms/bigquery)
### Redshift
* Support coming soon
### Data Lakes
* Currently supported through Snowflake, Databricks, or BigQuery external object tables
* [View Data Lakes Information](/data-tests/ai-data-tests/supported-platforms/data-lakes)
## Using the AI Data Validation Test
The test requires one main parameter:
* `expectation_prompt`: Describe what you expect from the data in plain English
Optionally, you can also specify:
* `llm_model_name`: Specify which AI model to use (see recommendations above for each warehouse)
This test works with any column type, as the data will be converted to a string format for validation. This enables natural language data validations for dates, numbers, and other structured data types.
```yml Models theme={null}
version: 2
models:
- name: < model name >
columns:
- name: < column name >
data_tests:
- elementary.ai_data_validation:
arguments:
expectation_prompt: "Description of what the data should satisfy"
llm_model_name: "model_name" # Optional
```
```yml Example - Date Validation theme={null}
version: 2
models:
- name: crm
description: "A table containing contract details."
columns:
- name: contract_date
description: "The date when the contract was signed."
data_tests:
- elementary.ai_data_validation:
arguments:
expectation_prompt: "There should be no contract date in the future"
```
```yml Example - Numeric Validation theme={null}
version: 2
models:
- name: sales
description: "A table containing sales data."
columns:
- name: discount_percentage
description: "The discount percentage applied to the sale."
data_tests:
- elementary.ai_data_validation:
arguments:
expectation_prompt: "The discount percentage should be between 0 and 50, and should only be a whole number."
llm_model_name: "claude-3-5-sonnet"
config:
severity: warn
```
```yml Example - Complex Validation theme={null}
version: 2
models:
- name: customer_accounts
description: "A table containing customer account information."
columns:
- name: account_status
description: "The current status of the customer account."
data_tests:
- elementary.ai_data_validation:
arguments:
expectation_prompt: "The account status should be one of: 'active', 'inactive', 'suspended', or 'pending'. If the account is 'suspended', there should be a reason code in the suspension_reason column."
llm_model_name: "gemini-1.5-pro"
```
# BigQuery Vertex AI
Source: https://docs.elementary-data.com/data-tests/ai-data-tests/supported-platforms/bigquery
Learn how to configure BigQuery to use Vertex AI models for unstructured data validation tests
# BigQuery Setup for Unstructured Data Tests
Elementary's unstructured data validation tests leverage BigQuery ML and Vertex AI models to perform advanced AI-powered validations. This guide will walk you through the setup process.
## Prerequisites
Before you begin, ensure you have:
* A Google Cloud account with appropriate permissions
* Access to BigQuery and Vertex AI services
* A BigQuery dataset where you'll create your model, that will be used by Elementary's data validation tests. This is the dataset where you have unstructured data stored and that you want to apply validations on.
## Step 1: Enable the Vertex AI API
1. Navigate to the Google Cloud Console
2. Go to **APIs & Services** > **API Library**
3. Search for "Vertex AI API"
4. Click on the API and select **Enable**
## Step 2: Create a Remote Connection to Vertex AI
Elementary's unstructured data validation tests use BigQuery ML to access pre-trained Vertex AI models. To establish this connection:
1. Navigate to the Google Cloud Console > **BigQuery**
2. In the Explorer panel, click the **+** button
3. Select **Connections to external data sources**
4. Change the connection type to **Vertex AI remote models, remote functions and BigLake (Cloud Resource)**
5. Select the appropriate region:
* If your model and dataset are in the same region, select that specific region
* Otherwise, select multi-region
After creating the connection:
1. In the BigQuery Explorer, navigate to **External Connections**
2. Find and click on your newly created connection
3. Copy the **Service Account ID** for the next step
## Step 3: Grant Vertex AI Access Permissions
Now you need to give the connection's service account permission to access Vertex AI:
1. In the Google Cloud Console, go to **IAM & Admin**
2. Click **+ Grant Access**
3. Under "New principals", paste the service account ID you copied
4. Assign the **Vertex AI User** role
5. Click **Save**
## Step 4: Create an LLM Model Interface in BigQuery
1. In the BigQuery Explorer, navigate to **External Connections**
2. Find again your newly created connection from previous step and clikc on it
3. Copy the **Connection ID** (format: `projects//locations//connections/`)
4. [Select a model endpoint](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-remote-model#gemini-api-multimodal-models). You can use `gemini-1.5-pro-002` as a default endpoint.
5. Run the following SQL query to create a model in your dataset:
```sql theme={null}
CREATE OR REPLACE MODEL
`..`
REMOTE WITH CONNECTION
``
OPTIONS (
endpoint = ''
);
```
### Example
```sql theme={null}
CREATE OR REPLACE MODEL
`my-project.my-dataset.gemini-1.5-pro`
REMOTE WITH CONNECTION
`projects/my-project/locations/us/connections/my-remote-connection-model-name`
OPTIONS (
endpoint = 'gemini-1.5-pro-002'
);
```
> **Note:** During development, we used `gemini-1.5-pro` and recommend it as the default model for unstructured data tests in BigQuery.
### Additional Resources
* [Available models and endpoints](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-remote-model#gemini-api-multimodal-models)
* [Documentation on creating remote models](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-remote-model)
## Step 5: Running an Unstructured Data Test
Once your model is set up, you can reference it in your Elementary tests:
```yaml theme={null}
models:
- name: table_with_unstructured_data
description: "A table containing unstructured text data."
columns:
- name: text_data
description: "Unstructured text data stored as a string."
data_tests:
- elementary.validate_unstructured_data:
arguments:
expectation_prompt: "The text data should represent an example of unstructured data."
llm_model_name: "gemini-1.5-pro"
```
# Data lakes
Source: https://docs.elementary-data.com/data-tests/ai-data-tests/supported-platforms/data-lakes
Currently, you can apply Elementary's unstructured data validation tests on data lakes using Snowflake, Databricks, or BigQuery external object tables.
Native and direct support for data lakes is coming soon. Please reach out if you would like to discuss this integration and use case.
# Databricks AI Functions
Source: https://docs.elementary-data.com/data-tests/ai-data-tests/supported-platforms/databricks
# Setting Up Databricks AI Functions
Elementary unstructured data validation tests run on top of Databricks AI Functions for Databricks users.
This guide provides details on the prerequisites to use Databricks AI Functions.
## What are Databricks AI Functions?
Databricks AI Functions are built-in SQL functions that allow you to apply AI capabilities directly to your data using SQL. These functions enable you to leverage large language models and other AI capabilities without complex setup or external dependencies, making them ideal for data validation tests.
## Availability and Prerequisites
To use Databricks AI Functions, your environment must meet the following requirements:
### Runtime Requirements
* **Recommended**: Databricks Runtime 15.3 or above for optimal performance
### Environment Requirements
* Your workspace must be in a supported Model Serving region.
* For Pro SQL warehouses, AWS PrivateLink must be enabled.
* Databricks SQL does support AI functions but Databricks SQL Classic does not support it.
### Models
Databricks AI functions can run on foundation models hosted in Databricks, external foundation models (like OpenAI's models) and custom models.
Currently Elementary's unstructured data validations support only foundation models hosted in Databricks. Adding support for external and custom models is coming soon.
> **Note**: While developing the tests we worked with `databricks-meta-llama-3-3-70b-instruct` so we recommend using this model as a default when running unstructured data validation tests in Databricks.
## Region Considerations
When using AI functions, be aware that some models are limited to specific regions (US and EU). Make sure your Databricks workspace is in a supported region for the Databricks AI functions.
# Redshift
Source: https://docs.elementary-data.com/data-tests/ai-data-tests/supported-platforms/redshift
Elementary's unstructured data validation tests do not currently support Redshift.
On Redshift setting up LLM functions is more complex and requires deploying a lambda function to call external LLM models. Documentation and support for this integration is coming soon. Please reach out if you'd like to discuss this use case and integration options.
# Snowflake Cortex AI
Source: https://docs.elementary-data.com/data-tests/ai-data-tests/supported-platforms/snowflake
# Snowflake Cortex AI LLM Functions
This guide provides instructions on how to enable Snowflake Cortex AI LLM functions, which is a prerequisite for running Elementary unstructured data validation tests on Snowflake.
## What is Snowflake Cortex?
Snowflake Cortex is a fully managed service that brings cutting-edge AI and ML solutions directly into your Snowflake environment. It allows you to leverage the power of large language models (LLMs) without any complex setup or external dependencies.
Snowflake provides LLMs that are fully hosted and managed by Snowflake, using them requires no setup and your data stays within Snowflake.
## Cross-Region Model Usage
> **Important**: It is always better to use models in the same region as your dataset to avoid errors and optimize performance.
To learn where each model is located we recommend checking this [models list](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions#availability).
If you encounter a "model not found" error, it may be because the model you're trying to use is not available in your current region. In such cases, you can enable cross-region model access with the following command (requires ACCOUNTADMIN privileges):
```sql theme={null}
-- Enable access to models in any region
ALTER ACCOUNT SET CORTEX_ENABLED_CROSS_REGION = 'ANY_REGION';
```
This setting allows your account to use models from any region, which can be helpful when the model you need is not available in your current region. However, be aware that cross-region access may impact performance and could have additional cost implications.
## Supported LLM Models
Snowflake Cortex provides access to various industry-leading LLM models with different capabilities and context lengths. Here are the key models available:
### Native Snowflake Models
* **Snowflake Arctic**: An open enterprise-grade model developed by Snowflake, optimized for business use cases.
### External Models (Hosted within Snowflake)
* **Claude Models (Anthropic)**: High-capability models for complex reasoning tasks.
* **Mistral Models**: Including mistral-large, mixtral-8x7b, and mistral-7b for various use cases.
* **Llama Models (Meta)**: Including llama3.2-1b, llama3.2-3b, llama3.1-8b, and llama2-70b-chat.
* **Gemma Models (Google)**: Including gemma-7b for code and text completion tasks.
> **Note**: While developing the tests we worked with `claude-3-5-sonnet` so we recommend using this model as a default when running unstructured data tests in Snowflake.
## Permissions
> **Note**: By default, all users in your Snowflake account already have access to Cortex AI LLM functions through the PUBLIC role. In most cases, you don't need to do anything to enable access.
The `CORTEX_USER` database role in the SNOWFLAKE database includes all the privileges needed to call Snowflake Cortex LLM functions. This role is automatically granted to the PUBLIC role, which all users have by default.
The following commands are **only needed if** your administrator has revoked the default access from the PUBLIC role or if you need to set up specific access controls. If you can already use Cortex functions, you can skip this section.
```sql theme={null}
-- Run as ACCOUNTADMIN
USE ROLE ACCOUNTADMIN;
-- Create a dedicated role for Cortex users
CREATE ROLE cortex_user_role;
-- Grant the database role to the custom role
GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO ROLE cortex_user_role;
-- Grant the role to specific users
GRANT ROLE cortex_user_role TO USER ;
-- Optionally, grant warehouse access to the role
GRANT USAGE ON WAREHOUSE TO ROLE cortex_user_role;
```
# Unstructured Data Validations
Source: https://docs.elementary-data.com/data-tests/ai-data-tests/unstructured_data_validations
**Beta Feature**: Unstructured data validation tests is currently in beta. The functionality and interface may change in future releases.
**Version Requirement**: This feature requires Elementary dbt package version 0.18.0 or above.
# Validating Unstructured Data with Elementary
## What is Unstructured Data Validation?
Elementary's `elementary.unstructured_data_validation` test allows you to validate unstructured data using AI and LLM language models. Instead of writing complex code, you can simply describe what you expect from your data in plain English, and Elementary will check if your data meets those expectations.
For example, you can verify that customer feedback comments are in English, product descriptions contain required information, or support tickets follow a specific format or a sentiment.
## How It Works
Elementary leverages the AI and LLM capabilities built directly into your data warehouse. When you run a validation test:
1. Your unstructured data stays within your data warehouse
2. The warehouse's built-in AI and LLM functions analyze the data
3. Elementary reports whether each text value meets your expectations
## Required Setup for Each Data Warehouse
Before you can use Elementary's unstructured data validations, you need to set up AI and LLM capabilities in your data warehouse:
### Snowflake
* **Prerequisite**: Enable Snowflake Cortex AI LLM functions
* **Recommended Model**: `claude-3-5-sonnet`
* [View Snowflake's Guide](/data-tests/ai-data-tests/supported-platforms/snowflake)
### Databricks
* **Prerequisite**: Ensure Databricks AI Functions are available
* **Recommended Model**: `databricks-meta-llama-3-3-70b-instruct`
* [View Databrick's Setup Guide](/data-tests/ai-data-tests/supported-platforms/databricks)
### BigQuery
* **Prerequisite**: Configure BigQuery to use Vertex AI models
* **Recommended Model**: `gemini-1.5-pro`
* [View BigQuery's Setup Guide](/data-tests/ai-data-tests/supported-platforms/bigquery)
### Redshift
* Support coming soon
### Data Lakes
* Currently supported through Snowflake, Databricks, or BigQuery external object tables
* [View Data Lakes Information](/data-tests/ai-data-tests/supported-platforms/data-lakes)
## Using the Validation Test
The test requires two main parameters:
* `expectation_prompt`: Describe what you expect from the text in plain English
* `llm_model_name`: Specify which AI model to use (see recommendations above for each warehouse)
This test works with any column containing unstructured text data such as descriptions, comments, or other free-form text fields. It can also be applied to structured columns that can be converted to strings, enabling natural language data validations.
```yml Models theme={null}
version: 2
models:
- name: < model name >
columns:
- name: < column name >
data_tests:
- elementary.unstructured_data_validation:
arguments:
expectation_prompt: "Description of what the text should contain or represent"
llm_model_name: "model_name"
```
```yml Example theme={null}
version: 2
models:
- name: table_with_unstructured_data
description: "A table containing unstructured text data."
columns:
- name: text_data
description: "Unstructured text data stored as a string."
data_tests:
- elementary.unstructured_data_validation:
arguments:
expectation_prompt: "The text data should represent an example of unstructured data."
llm_model_name: "test_model"
```
```yml Example - Validating Customer Feedback theme={null}
version: 2
models:
- name: customer_feedback
description: "A table containing customer feedback comments."
columns:
- name: feedback_text
description: "Customer feedback in free text format."
data_tests:
- elementary.unstructured_data_validation:
arguments:
expectation_prompt: "The text should be a customer feedback comment in English, it should describe only a bug or a feature request."
llm_model_name: "claude-3-5-sonnet"
config:
severity: warn
```
## Usage Examples
Here are some powerful ways you can apply unstructured data validations:
### Validating Structure
```yml theme={null}
models:
- name: medicine_prescriptions
description: "A table containing medicine prescriptions."
columns:
- name: doctor_notes
description: "A column containing the doctor notes on the prescription"
data_tests:
- elementary.unstructured_data_validation:
arguments:
expectation_prompt: "The prescription has to include a limited time period and recommendations to the patient"
llm_model_name: "claude-3-5-sonnet"
```
Test fails if: A doctor's note does not specify a time period or lacks recommendations for the patient.
### Validating Sentiment
```yml theme={null}
models:
- name: customer_feedback
description: "A table containing customer feedback."
columns:
- name: negative_feedbacks
description: "A column containing negative feedbacks about our product."
data_tests:
- elementary.unstructured_data_validation:
arguments:
expectation_prompt: "The customer feedback's sentiment has to be negative"
llm_model_name: "claude-3-5-sonnet"
```
Test fails if: Any feedback in `negative_feedbacks` is not actually negative.
### Validating Similarities Coming Soon
```yml theme={null}
models:
- name: summarized_pdfs
description: "A table containing a summary of our ingested PDFs."
columns:
- name: pdf_summary
description: "A column containing the main PDF's content summary."
data_tests:
- elementary.validate_similarity:
arguments:
to: ref('pdf_source_table')
column: pdf_content
match_by: pdf_name
```
Test fails if: A PDF summary does not accurately represent the original PDF's content. The validation will use the pdf name as the key to match a summary from the pdf\_summary table to the pdf\_content in the pdf\_source\_table.
```yml theme={null}
models:
- name: jobs
columns:
- name: job_title
data_tests:
- elementary.validate_similarity:
arguments:
column: job_description
```
Test fails if: The job title does not align with the job description.
### Accepted Categories Coming Soon
```yml theme={null}
models:
- name: support_tickets
description: "A table containing customer support tickets."
columns:
- name: issue_description
description: "A column containing customer-reported issues."
data_tests:
- elementary.accepted_categories:
arguments:
categories: ['billing', 'technical_support', 'account_access', 'other']
```
Test fails if: A support ticket does not fall within the predefined categories.
### Accepted Entities Coming Soon
```yml theme={null}
models:
- name: news_articles
description: "A table containing news articles."
columns:
- name: article_text
description: "A column containing full article text."
data_tests:
- elementary.extract_and_validate_entities:
arguments:
entities:
organization:
required: true
accepted_values: ['Google', 'Amazon', 'Microsoft', 'Apple']
location:
required: false
accepted_values: {{ run_query('select zip_code from locations') }}
```
Test fails if:
* The required entity (e.g., `organization`) is missing.
* Extracted entities do not match the expected values.
### Compare Numeric Values Coming Soon
```yml theme={null}
models:
- name: board_meeting_summaries
description: "A table containing board meeting summary texts."
columns:
- name: meeting_notes
description: "A column containing the full summary of the board meeting."
data_tests:
- elementary.extract_and_validate_numbers:
arguments:
entities:
revenue:
compare_with: ref('crm_financials')
column: sum(revenue)
required: true
net_profit:
compare_with: ref('crm_financials')
column: sum(net_profit)
customer_count:
compare_with: ref('crm_customers')
column: count(customers)
required: true
```
Test fails if:
* Required entities are missing
* The numerical entities do not match the structured CRM data
# anomaly_direction
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-direction
Let our Slack chatbot create the anomaly test you need.
`anomaly_direction: both | spike | drop`
By default, data points are compared to the expected range and check if these are below or above it.
For some data monitors, you might only want to flag anomalies if they are above the range and not under it, and vice versa.
For example - when monitoring for freshness, we only want to detect data delays and not data that is “early”.
The anomaly\_direction configuration is used to configure the direction of the expected range, and can be set to both, spike or drop.
* *Default: `both`*
* *Supported values: `both`, `spike`, `drop`*
* *Relevant tests: All anomaly detection tests*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
anomaly_direction: drop
- elementary.all_columns_anomalies:
arguments:
column_anomalies:
- null_count
- missing_count
- zero_count
anomaly_direction: spike
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
anomaly_direction: drop
```
```yml dbt_project theme={null}
vars:
anomaly_direction: both
```
# anomaly_exclude_metrics
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-exclude-metrics
Let our Slack chatbot create the anomaly test you need.
`anomaly_exclude_metrics: [SQL where expression on fields metric_date / metric_time_bucket / metric_value]`
By default, data points are compared to the all data points in the training set.
Using this param, you can exclude metrics from the training set, to improve the test accuracy.
The filter can be configured using an SQL where expression syntax, and the following fields:
1. `metric_date` - The date of the relevant bucket (even if the bucket is not daily).
2. `metric_time_bucket` - The exact time bucket.
3. `metric_value` - The value of the metric.
* *Supported values: valid SQL where expression on the columns metric\_date / metric\_time\_bucket / metric\_value*
* *Relevant tests: All anomaly detection tests*
```yml test theme={null}
models:
- name: this_is_a_model
tests:
- elementary.volume_anomalies:
anomaly_exclude_metrics: metric_value < 10
- elementary.all_columns_anomalies:
column_anomalies:
- null_count
- missing_count
- zero_count
anomaly_exclude_metrics: metric_time_bucket >= '2023-10-01 06:00:00' and metric_time_bucket <= '2023-10-01 07:00:00'
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
anomaly_exclude_metrics: metric_date = '2023-10-01'
```
```yml dbt_project theme={null}
vars:
anomaly_exclude_metrics: metric_date = '2023-10-01'
```
# Anomaly Tests configuration params
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-params
Let our Slack chatbot create the anomaly test you need.
If your data set has a timestamp column that represents the creation time of a
field, it is highly recommended configuring it as a `timestamp_column`.
## Example configurations
```yml Models theme={null}
version: 2
models:
- name:
config:
elementary:
timestamp_column: < model timestamp column >
data_tests: < here you will add elementary monitors as tests >
- name:
## if no timestamp is configured, elementary will monitor without time filtering
data_tests:
```
```yml Models example theme={null}
version: 2
models:
- name: login_events
config:
elementary:
timestamp_column: updated_at
data_tests:
- elementary.freshness_anomalies:
config:
tags: ["elementary"]
- elementary.all_columns_anomalies:
config:
tags: ["elementary"]
- name: users
## if no timestamp is configured, elementary will monitor without time filtering
data_tests:
- elementary.volume_anomalies:
config:
tags: ["elementary"]
```
```yml Sources theme={null}
version: 2
sources:
- name: < some name >
database: < database >
schema: < schema >
tables:
- name: < table_name >
config:
elementary:
timestamp_column: < source timestamp column >
data_tests:
```
```yml Sources example theme={null}
version: 2
sources:
- name: "my_non_dbt_table"
database: "raw_events"
schema: "product"
tables:
- name: "raw_product_login_events"
config:
elementary:
timestamp_column: "loaded_at"
data_tests:
- elementary.volume_anomalies
- elementary.all_columns_anomalies:
arguments:
column_anomalies:
- null_count
- missing_count
- zero_count
columns:
- name: user_id
data_tests:
- elementary.column_anomalies
```
# anomaly_sensitivity
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-sensitivity
Let our Slack chatbot create the anomaly test you need.
`anomaly_sensitivity: [float]`
Configuration to define how the expected range is calculated.
A sensitivity of 3 means that the expected range is within 3 standard deviations from the average of the training set.
Smaller sensitivity means this range will be reduced and more values will be potentially flagged as anomalies.
Larger values will have the opposite effect and will reduce the number of anomalies as the expected range will be larger.
* *Default: 3*
* *Relevant tests: All anomaly detection tests*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
anomaly_sensitivity: 2.5
- elementary.all_columns_anomalies:
arguments:
column_anomalies:
- null_count
- missing_count
- zero_count
anomaly_sensitivity: 4
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
anomaly_sensitivity: 3.5
```
```yml dbt_project theme={null}
vars:
anomaly_sensitivity: 3
```
# column_anomalies
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/column-anomalies
Let our Slack chatbot create the anomaly test you need.
`column_anomalies: [column monitors list]`
Select which monitors to activate as part of the test.
* *Default: default monitors*
* *Relevant tests: `all_columns_anomalies`, `column_anomalies`*
* *Configuration level: test*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.column_anomalies:
arguments:
column_anomalies:
- null_count
- missing_count
- average
```
#### Supported column monitors
**Default monitors by type:**
| Data quality metric | Column Type |
| -------------------- | ----------- |
| `null_count` | any |
| `null_percent` | any |
| `min_length` | string |
| `max_length` | string |
| `average_length` | string |
| `missing_count` | string |
| `missing_percent` | string |
| `min` | numeric |
| `max` | numeric |
| `average` | numeric |
| `zero_count` | numeric |
| `zero_percent` | numeric |
| `standard_deviation` | numeric |
| `variance` | numeric |
**Opt-in monitors by type:**
| Data quality metric | Column Type |
| ------------------- | ----------- |
| `sum` | numeric |
# detection_delay
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-delay
Let our Slack chatbot create the anomaly test you need.
```
detection_delay:
period: < time period > # supported periods: hour, day, week, month
count: < number of periods >
```
The duration for retracting the detection period.
That's useful in cases which the latest data should be excluded from the test. For example, this can happen because of scheduling issues- if the test is running before the table is populated for some reason. The detection delay is the period of time to ignore, after the detection period.
* *Default: 0*
* *Relevant tests: Anomaly detection tests with `timestamp_column`*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
detection_delay:
period: day
count: 1
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
detection_delay:
period: day
count: 1
```
```yml dbt_project.yml theme={null}
vars:
detection_delay:
period: day
count: 1
```
#### How it works?
The `detection_delay` param only works for tests that have `timestamp_column` configuration.
It does not affect the other duration parameters, like `detection_period` or `training_period`.
# detection_period
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-period
Let our Slack chatbot create the anomaly test you need.
```
detection_period:
period: < time period > # supported periods: day, week, month
count: < number of periods >
```
Configuration to define the detection period.
If the detection\_period are set to 2 days, only data points in the last 2 days will be included in the detection period and could be flagged anomalous.
If detection\_period is set to 7 days, the detection period will be 7 days long.
For incremental models, this is also the period for re-calculating metrics.
If metrics for buckets in the detection period were already calculated, Elementary will overwrite them. The reason behind it is to monitor recent backfills of data, if there were any.
This configuration should be changed according to your data delays.
* *Default: 2 days*
* *Relevant tests: Anomaly detection tests with `timestamp_column`*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
detection_period:
period: day
count: 30
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
detection_period:
period: month
count: 1
```
```yml dbt_project.yml theme={null}
vars:
detection_period:
period: week
count: 2
```
#### How it works?
The `detection_period` param only works for tests that have `timestamp_column` configuration.
It works differently according to the table materialization:
* **Regular tables and views** - `detection_period` defines the detection period.
* **Incremental models and sources** - `detection_period` defines the detection period, and the period for which metrics will be re-calculated.
**Overlap with training period:**
When `detection_period` spans multiple time buckets (e.g., 7 days), it can overlap with the training period. By default, values in the detection period are included in the training calculation, which can lead to false negatives. To prevent this overlap, use [`exclude_detection_period_from_training`](/data-tests/anomaly-detection-configuration/exclude_detection_period_from_training) set to `true`.
# dimensions
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions
Let our Slack chatbot create the anomaly test you need.
`dimensions: [list of SQL expressions]`
The test will group the results by a given column / columns / valid select sql expression.
Under `dimensions` you can configure the group by expression.
Using this param segments the tested data per dimension, and each dimension is monitored separately.
For example -
A `column_anomalies` test monitoring for `null_rate` with `dimensions` configured will monitor the
`null_rate` of values in the column, grouped by dimension, and will fail if in a specific dimension there is an anomaly in `null_rate`.
It is best to configure low-cardinality fields as `dimensions`.
* *Default: None*
* *Relevant tests: `dimension_anomalies`, `column_anomalies`, `all_columns_anomalies`*
* *Configuration level: test*
```yml test theme={null}
models:
- name: model_name
config:
elementary:
timestamp_column: updated_at
data_tests:
- elementary.dimension_anomalies:
arguments:
dimensions:
- device_os
- device_browser
```
# event_timestamp_column
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/event_timestamp_column
Let our Slack chatbot create the anomaly test you need.
`event_timestamp_column: [column name]`
Configuration for the test `event_freshness_anomalies`.
This test complements the freshness\_anomalies test and is primarily intended for data that is updated in a continuous / streaming fashion.
The test can work in a couple of modes:
* If only an event\_timestamp\_column is supplied, the test measures over time the difference between the current timestamp (“now”) and the most recent event timestamp.
* If both an event\_timestamp\_column and an update\_timestamp\_column are provided, the test will measure over time the difference between these two columns.
* *Default: None*
* *Relevant tests: `event_freshness_anomalies`*
* *Configuration level: test*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.event_timestamp_column:
arguments:
event_timestamp_column: "event_timestamp"
update_timestamp_column: "created_at"
```
# exclude_final_results
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/exclude-final-results
Let our Slack chatbot create the anomaly test you need.
`exclude_final_results: [SQL where expression on fields value / average]`
Failures in dimension anomaly tests consist of outliers in row count of each dimension.
Some dimensions may be considered insignificant compared to others, and you may prefer not to receive alerts for them.
With this parameter, you can exclude these dimensions from the results set and avoid such failures.
1. `value` - Max row count of a dimension during the detection period.
2. `average` - The average rows count of a dimension during the training period.
* *Supported values: valid SQL where expression on the columns value / average*
* *Relevant tests: Dimension anomalies*
```yml test theme={null}
models:
- name: model_name
config:
elementary:
timestamp_column: updated_at
data_tests:
- elementary.dimension_anomalies:
arguments:
dimensions:
- device_os
- device_browser
exclude_final_results: 'value > 1000 or average > 10'
```
# exclude_detection_period_from_training
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/exclude_detection_period_from_training
Let our Slack chatbot create the anomaly test you need.
`exclude_detection_period_from_training: true | false`
When the detection period spans multiple values, there can be overlap between the training period and the detection period. By default, values in the detection period are included in the training calculation, which can lead to false negatives because the detection period values influence the expected range used to evaluate those same values.
Setting `exclude_detection_period_from_training: true` ensures that no values from the detection period are used in the training calculation, preventing this overlap and improving anomaly detection accuracy.
**Example use case:**
When `detection_period` is set to more than 1 time bucket (e.g., `detection_period: 7 days`), the detection period overlaps with the training period. Without excluding the detection period from training, values being evaluated for anomalies are also contributing to the expected range calculation, which can mask actual anomalies and result in false negatives.
* *Default: false*
* *Supported values: `true`, `false`*
* *Relevant tests: Anomaly detection tests with `timestamp_column` and `detection_period` greater than 1 time bucket*
#### How it works?
* When `exclude_detection_period_from_training: false` (default), all values within both the training period and detection period are used to calculate the expected range.
* When `exclude_detection_period_from_training: true`, values within the detection period are excluded from the training calculation, ensuring the expected range is based solely on historical data that is not being evaluated.
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
exclude_detection_period_from_training: true
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
exclude_detection_period_from_training: true
```
```yml dbt_project.yml theme={null}
vars:
exclude_detection_period_from_training: true
```
# exclude_prefix
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/exclude_prefix
Let our Slack chatbot create the anomaly test you need.
`exclude_prefix: [string]`
Param for the `all_columns_anomalies` test only, which enables to exclude a column from the tests based on prefix match.
* *Default: None*
* *Relevant tests: `all_columns_anomalies`*
* *Configuration level: test*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.column_anomalies:
arguments:
exclude_prefix: "id_"
```
# exclude_regexp
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/exclude_regexp
Let our Slack chatbot create the anomaly test you need.
`exclude_regexp: [regex]`
Param for the `all_columns_anomalies` test only, which enables to exclude a column from the tests based on regular expression match.
* *Default: None*
* *Relevant tests: `all_columns_anomalies`*
* *Configuration level: test*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.column_anomalies:
arguments:
exclude_regexp: ".*SDC$"
```
# fail_on_zero
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/fail_on_zero
Let our Slack chatbot create the anomaly test you need.
`fail_on_zero: true/false`
Elementary anomaly detection tests will fail if there is a zero metric value within the detection period.
If undefined, default is false.
* *Default: false*
* *Relevant tests: All anomaly detection tests*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
fail_on_zero: true
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
fail_on_zero: true
```
```yml dbt_project.yml theme={null}
vars:
fail_on_zero: true
```
# ignore_small_changes
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes
Let our Slack chatbot create the anomaly test you need.
```
ignore_small_changes:
spike_failure_percent_threshold: [int]
drop_failure_percent_threshold: [int]
```
If defined, an anomaly test will fail only if all the following conditions hold:
* The z-score of the metric within the detection period is anomoulous
* One of the following holds:
* The metric within the detection period is higher than `spike_failure_percent_threshold` percentages of the mean value in the training period, if defined.
* The metric within the detection period is lower than `drop_failure_percent_threshold` percentages of the mean value in the training period, if defined
Those settings can help to deal with situations where your metrics are stable and small changes causes to high z-scores, and therefore to anomaly.
If undefined, default is null for both spike and drop.
* *Default: none*
* *Relevant tests: All anomaly detection tests*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
ignore_small_changes:
spike_failure_percent_threshold: 2
drop_failure_percent_threshold: 50
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
ignore_small_changes:
spike_failure_percent_threshold: 2
```
```yml source theme={null}
sources:
- name: my_non_dbt_tables
schema: raw
tables:
- name: source_table
config:
elementary:
ignore_small_changes:
drop_failure_percent_threshold: 50
```
```yml dbt_project.yml theme={null}
vars:
ignore_small_changes:
spike_failure_percent_threshold: 10
```
# seasonality
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/seasonality
Let our Slack chatbot create the anomaly test you need.
`seasonality: day_of_week | hour_of_day | hour_of_week`
Some data sets have patterns that repeat over a time period, and are expected.
This is the normal behavior of these data sets.
This means that when we try to detect outliers from the normal and expected range, ignoring this patterns might cause false positives or make us miss anomalies.
The seasonality configuration is used to overcome this challenge and account for expected patterns.
**Supported seasonality configurations:**
* `day_of_week` - Uses the same day of week as a training set for each daily bucket (Compares Sunday to Sundays, Monday to Mondays, etc.).
* `hour_of_day` - Uses the same hour as a training set for each hourly bucket (For example will compare 10:00-11:00AM to 10:00-11:00AM on previous days, instead of any previous hour).
* `hour_of_week` - Uses the same hour and day of week as a training set for each hourly bucket (For example will compare 10:00-11:00AM on Sunday to 10:00-11:00AM on previous sundays).
**Example use case:**
Many data sets have lower volume over the weekend, and higher volume over the week days.
This means that the expected range for different days of the week is different.
The `day_of_week` seasonality uses the same day of week as a training set for each daily time bucket data point.
The expected range for Monday will be based on a training set of previous Mondays, and so on.
* *Default: none*
* *Supported values: `day_of_week`, `hour_of_day`, `hour_of_week`*
* *Relevant tests: Anomaly detection tests with `timestamp_column` and 1 day `time_bucket`*
#### How it works?
* The test will compare the value of a bucket to previous bucket with the same seasonality attribute, and not to the adjacent previous data points.
* **The `training_period` of the test will be changed by default to assure a minimal training set. When `seasonality: day_of_week` is configured, `training_period` is by default multiplied by 7.**
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
seasonality: day_of_week
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
seasonality: day_of_week
```
```yml dbt_project.yml theme={null}
vars:
seasonality: day_of_week
```
# time_bucket
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/time-bucket
Let our Slack chatbot create the anomaly test you need.
```
time_bucket:
period: < time period > # supported periods: hour, day, week, month
count: < number of periods >
```
This configuration controls the duration of the time buckets.
To calculate how data changes over time and detect issues, we split the data into consistent time buckets.
For example, if we use daily (period=`day`, count=`1`) time bucket and monitor for row count anomalies, we will count new rows per day.
Depending on the nature of your data, it may make sense to modify this parameter.
For example, if you want to detect volume anomalies in an hourly resolution, you should set the time bucket to period=`hour` and count=`1`.
* *Default: daily buckets. `time_bucket: {period: day, count: 1}`*
* *Relevant tests: Anomaly detection tests with `timestamp_column`*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
time_bucket:
period: day
count: 2
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
time_bucket:
period: hour
count: 4
```
```yml dbt_project.yml theme={null}
vars:
time_bucket:
period: hour
count: 12
```
#### How it works?
* The `training_period` and `detection_period` of the test might be extended to ensure full time buckets (for example, full week Sunday-Saturday).
* Weekly buckets start at the day that is configured as week start on the data warehouse.
# timestamp_column
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column
Let our Slack chatbot create the anomaly test you need.
`timestamp_column: [column name]`
If your data set has a timestamp column that represents the creation time of a
field, it is highly recommended configuring it as a `timestamp_column`.
Elementary anomaly detection tests will use this column to create time buckets and filter the table.
It is highly recommended to configure a timestamp column (if there is one).
The best column for this would be an `updated_at`/`created_at`/`loaded_at` timestamp for each row (date type also works).
* When you specify a `timestamp_column`, when the test runs it splits the data to buckets according to the timestamp in this column, calculates the metric for each bucket and checks for anomalies between these buckets.
This also means that if the table has enough historical data, the test can start working right away.
* When you do not specify a `timestamp_column`, each time the test runs it calculates the metric for all of the data in the table, and checks for anomalies between the metric from previous runs.
This also means that it will take the test `training_period` days to start working, as it needs to the time to collect the necessary metrics.
If undefined, default is null (no time buckets).
* *Default: none*
* *Relevant tests: All anomaly detection tests*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
timestamp_column: created_at
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
timestamp_column: updated_at
```
```yml source theme={null}
sources:
- name: my_non_dbt_tables
schema: raw
tables:
- name: source_table
config:
elementary:
timestamp_column: loaded_at
```
```yml dbt_project.yml theme={null}
vars:
timestamp_column: loaded_at
```
# training_period
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period
Let our Slack chatbot create the anomaly test you need.
```
training_period:
period: < time period > # supported periods: day, week, month
count: < number of periods >
```
The maximal timeframe for which the test will collect data.
This timeframe includes the training period and detection period. If a detection delay is defined, the whole training period is being delayed.
* *Default: 14 days*
* *Relevant tests: Anomaly detection tests with `timestamp_column`*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
training_period:
period: day
count: 30
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
detection_delay:
period: week
count: 1
```
```yml dbt_project.yml theme={null}
vars:
detection_delay:
period: month
count: 1
```
#### How it works?
The `training_period` param only works for tests that have `timestamp_column` configuration.
It works differently according to the table materialization:
* **Regular tables and views** - The values of the full `training_period` period is calculated on each run.
* **Incremental models and sources** - The values of the full `training_period` period is calculated on the first test run, and on full refresh. The following test runs will only calculate the values of the `detection_period` period.
**Changes from default:**
* **Full time buckets** - Elementary will increase the `training_period` automatically to insure full time buckets. For example if the `time_bucket` of the test is `period: week`, and 14 days `training_period` result in Tuesday, the test will collect 2 more days back to complete a week (starting on Sunday).
* **Seasonality training set** - If seasonality is configured, Elementary will increase the `training_period` automatically to ensure there are enough training set values to calculate an anomaly. For example if the `seasonality` of the test is `day_of_week`, `training_period` will be increased to ensure enough Sundays, Mondays, Tuesdays, etc. to calculate an anomaly for each.
**Overlap with detection period:**
When the `detection_period` spans multiple time buckets, it can overlap with the training period. By default, values in the detection period are included in the training calculation, which can lead to false negatives because detection period values influence the expected range used to evaluate those same values. To prevent this overlap and improve anomaly detection accuracy, use [`exclude_detection_period_from_training`](/data-tests/anomaly-detection-configuration/exclude_detection_period_from_training) set to `true`.
#### The impact of changing `training_period`
If you **increase `training_period`** your test training set will be larger. This means a larger sample size for calculating the expected range, which should make the test less sensitive to outliers. This means less chance of false positive anomalies, but also less sensitivity so anomalies have a higher threshold.
If you **decrease `training_period`** your test training set will be smaller. This means a smaller sample size for calculating the expected range, which might make the test more sensitive to outliers. This means more chance of false positive anomalies, but also more sensitivity as anomalies have a lower threshold.
# update_timestamp_column
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/update_timestamp_column
Let our Slack chatbot create the anomaly test you need.
`update_timestamp_column: [column name]`
Configuration for the test `event_freshness_anomalies`.
This test complements the freshness\_anomalies test and is primarily intended for data that is updated in a continuous / streaming fashion.
The test can work in a couple of modes:
* If only an event\_timestamp\_column is supplied, the test measures over time the difference between the current timestamp (“now”) and the most recent event timestamp.
* If both an event\_timestamp\_column and an update\_timestamp\_column are provided, the test will measure over time the difference between these two columns.
* *Default: None*
* *Relevant tests: `event_freshness_anomalies`*
* *Configuration level: test*
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.event_timestamp_column:
arguments:
event_timestamp_column: "event_timestamp"
update_timestamp_column: "created_at"
```
# where_expression
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/where-expression
Let our Slack chatbot create the anomaly test you need.
`where_expression: [sql expression]`
Filter the tested data using a valid sql expression.
* *Default: None*
* *Relevant tests: All anomaly detection tests*
The `where_expression` is used to filter out the data that you want to test. For example, to only test weekdays for anomalies, you can filter on the day of the week:
```yml theme={null}
where_expression: EXTRACT(DOW FROM timestamp_column) BETWEEN 2 AND 6
#Test Monday through Friday
```
```yml test theme={null}
models:
- name: this_is_a_model
data_tests:
- elementary.volume_anomalies:
arguments:
where_expression: "user_name != 'test'"
```
```yml model theme={null}
models:
- name: this_is_a_model
config:
elementary:
where_expression: "loaded_at is not null"
```
```yml dbt_project.yml theme={null}
vars:
where_expression: "loaded_at > '2022-01-01'"
```
# OSS vs Cloud Anomaly Detection
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-tests-oss-vs-cloud
Elementary OSS and Elementary Cloud Platform both offer data anomaly detection. However, there are significant differences in implementation.
There are two types of anomaly detection tests:
* **Pipeline health monitors** - Monitor the pipeline runs, ensuring timely and complete data ingestion and transformation. These monitors monitor metadata to detect volume and freshness issues.
* **Data quality metrics tests** - Run as part of the pipeline, collect metrics by querying the data itself. These include various data quality metrics such as nullness, cardinality, average, length, etc.
Here is a comparison between the implementation of these tests in Elementary Cloud and OSS:
## Pipeline Health Monitors - Freshness and Volume
| | OSS | Cloud |
| ----------------------- | --------------------------------------------- | ------------------------------------------------- |
| **Implementation** | dbt tests | Elementary Cloud monitors |
| **Tests execution** | Run in dbt | Run in Cloud |
| **Coverage** | Manually added in code | Automated, out-of-the-box full coverage |
| **Configuration** | Manual, many parameters required for accuracy | No configuration, automated ML models |
| **Detection mechanism** | Z-score, statistical | ML anomaly detection, various models |
| **What is monitored?** | Data | Metadata (query history, information schema) |
| **Time to detection** | Only when dbt runs | As soon as the problem happens, including sources |
| **Cost** | DWH compute | No cost, only metadata is leveraged |
## Data Quality Metrics
| | OSS | Cloud |
| ----------------------- | --------------------------------------------- | ---------------------------------------------------- |
| **Implementation** | dbt tests | Metrics collection in dbt, Elementary Cloud monitors |
| **Tests execution** | Run in dbt | Metrics collection in dbt, detection in Cloud |
| **Coverage** | Manually added in code | Opt-in, can be added in bulk in Cloud |
| **Configuration** | Manual, many parameters required for accuracy | Automated ML models |
| **Detection mechanism** | Z-score, statistical | ML anomaly detection, various models |
| **What is monitored?** | Data | Data |
# Anomaly Tests Troubleshooting
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide
Let our Slack chatbot create the anomaly test you need.
First, check if your test uses a timestamp column:
```yaml theme={null}
# In your YAML configuration
data_tests:
- elementary.volume_anomalies:
arguments:
timestamp_column: created_at# If this is configured, you have a timestamp-based test
```
* Metrics are calculated by grouping data into time buckets (default: 'day')
* Detection period (default: 2 days) determines how many buckets are being tested
* Training period data (default: 14 days) comes from historical buckets, allowing immediate anomaly detection with sufficient history
Verify data collection:
```sql theme={null}
-- Check if metrics are being collected in time buckets
SELECT
bucket_end,
metric_value,
COUNT(*) as metrics_per_bucket
FROM your_schema.data_monitoring_metrics
WHERE full_table_name = 'your_table'
GROUP BY bucket_end, metric_value
ORDER BY bucket_end DESC;
```
* Each bucket should represent one time bucket (e.g., daily metrics)
* Gaps in `metric_timestamp` might indicate data collection issues
* Training uses historical buckets for anomaly detection
* The format for full\_table\_name is DATABASE.SCHEMA.TABLE\_NAME
**Common collection issues:**
* Missing or null values in timestamp column
* Timestamp column not in expected format
* No data in specified training period
* Training period data builds up over multiple test runs, using the test run time as its timestamp column. This requires time to collect enough points; for a 14 day training period, the test would need 14 different runs on different days to have a full training set.
* Metrics are calculated for the entire table in each test run
* Detection period (default: 2 days) determines how many buckets are being tested
Check metric collection across test runs:
```sql theme={null}
-- Check metrics from different test runs
SELECT
updated_at,
metric_value
FROM your_schema.data_monitoring_metrics
WHERE full_table_name = 'your_table'
ORDER BY updated_at DESC;
```
* Should see one metric per test run and per dimension
* Training requires multiple test runs over time
* Each new test run creates the training point for a time bucket. A second test run within the same bucket will override the first one.
* The format for full\_table\_name is DATABASE.SCHEMA.TABLE\_NAME
**Common collection issues:**
* Test hasn't run enough times
* Previous test runs failed
* Metrics not being saved between runs
Anomaly detection is influenced by:
* Detection period (default: 2 days) - the time window being tested
* Sensitivity (default: 3.0) - how many standard deviations from normal before flagging
* Training data from previous periods/runs
* `metrics_anomaly_score` calculates the anomaly based on the data in `data_monitoring metrics`.
Check calculations in `metrics_anomaly_score`:
```sql theme={null}
-- Check how anomalies are being calculated
SELECT
metric_name,
latest_metric_value,
training_avg,
training_stddev,
anomaly_score,
is_anomaly
FROM your_schema.metrics_anomaly_score
WHERE full_table_name = 'your_table'
ORDER BY bucket_end DESC;
```
* `anomaly_score`: The standardized score that measures how many standard deviations a data point is from the mean
* `is_anomaly`: A boolean field that indicates whether the anomaly score exceeds the configured threshold
This occurs when there are fewer than 7 training data points. To resolve:
### For timestamp-based tests:
* Check if your timestamp column has enough historical data
* Verify time buckets are being created correctly in `data_monitoring_metrics`
* Look for gaps in your data that might affect bucket creation
### For non-timestamp tests:
* Run your tests multiple times to build up training data.
* Check `data_monitoring_metrics` to verify the data collection. The test will need data for at least 7 time buckets (e.g 7 days) to calculate the anomaly.
If your test isn't appearing in `data_monitoring_metrics`:
Verify test configuration:
```yaml theme={null}
data_tests:
- elementary.volume_anomalies:
arguments:
timestamp_column: created_at# Check if specified correctly
```
### Common causes:
* Incorrect timestamp column name
* Timestamp column contains null values or is not of type timestamp or date
* For non-timestamp tests: Test hasn't run successfully
* Incorrect test syntax
If you change it after executing elementary tests, you will need to run a full refresh to the metrics collected. This will make the next tests collect data for the new **`training_period`** timeframe. The steps are:
1. Change var **`training_period`** in your **`dbt_project.yml`**.
2. Full refresh of the model ‘data\_monitoring\_metrics’ by running **`dbt run --select data_monitoring_metrics --full-refresh`**.
3. Running the elementary tests again.
If you want the Elementary UI to show data for a longer period of time, use the days-back option of the CLI: **`edr report --days-back 45`**
# all_columns_anomalies
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-tests/all-columns-anomalies
Let our Slack chatbot create the anomaly test you need.
`elementary.all_columns_anomalies`
Executes column level monitors and anomaly detection on all the columns of the table.
Specific monitors are detailed in the table below and can be configured using the `columns_anomalies` configuration.
The test checks the data type of each column and only executes monitors that are relevant to it.
You can use `column_anomalies` param to override the default monitors, and `exclude_prefix` / `exclude_regexp` to exclude columns from the test.
**Default monitors by type:**
| Data quality metric | Column Type |
| -------------------- | ----------- |
| `null_count` | any |
| `null_percent` | any |
| `min_length` | string |
| `max_length` | string |
| `average_length` | string |
| `missing_count` | string |
| `missing_percent` | string |
| `min` | numeric |
| `max` | numeric |
| `average` | numeric |
| `zero_count` | numeric |
| `zero_percent` | numeric |
| `standard_deviation` | numeric |
| `variance` | numeric |
**Opt-in monitors by type:**
| Data quality metric | Column Type |
| ------------------- | ----------- |
| `sum` | numeric |
### Test configuration
No mandatory configuration, however it is highly recommended to configure a `timestamp_column`.
```yml Models theme={null}
models:
- name: < model name >
config:
elementary:
timestamp_column: < timestamp column >
data_tests:
- elementary.all_columns_anomalies:
arguments:
column_anomalies: < specific monitors, all if null >
where_expression: < sql expression >
time_bucket: # Daily by default
period: < time period >
count: < number of periods >
```
```yml Models example theme={null}
models:
- name: login_events
config:
elementary:
timestamp_column: "loaded_at"
data_tests:
- elementary.all_columns_anomalies:
config:
tags: ["elementary"]
arguments:
where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
time_bucket:
period: day
count: 1
# optional - change global sensitivity
anomaly_sensitivity: 3.5
```
# column_anomalies
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies
Let our Slack chatbot create the anomaly test you need.
`elementary.column_anomalies`
Executes column level monitors and anomaly detection on the column.
Specific monitors are detailed in the table below and can be configured using the `columns_anomalies` configuration.
The test checks the data type of the column and only executes monitors that are relevant to it.
**Default monitors by type:**
| Data quality metric | Column Type |
| -------------------- | ----------- |
| `null_count` | any |
| `null_percent` | any |
| `min_length` | string |
| `max_length` | string |
| `average_length` | string |
| `missing_count` | string |
| `missing_percent` | string |
| `min` | numeric |
| `max` | numeric |
| `average` | numeric |
| `zero_count` | numeric |
| `zero_percent` | numeric |
| `standard_deviation` | numeric |
| `variance` | numeric |
**Opt-in monitors by type:**
| Data quality metric | Column Type |
| ------------------- | ----------- |
| `sum` | numeric |
### Test configuration
No mandatory configuration, however it is highly recommended to configure a `timestamp_column`.
```yml Models theme={null}
models:
- name: < model name >
config:
elementary:
timestamp_column: < timestamp column >
columns:
- name: < column name >
data_tests:
- elementary.column_anomalies:
arguments:
column_anomalies: < specific monitors, all if null >
where_expression: < sql expression >
time_bucket: # Daily by default
period: < time period >
count: < number of periods >
- name: < model name >
## if no timestamp is configured, elementary will monitor without time filtering
columns:
- name: < column name >
data_tests:
- elementary.column_anomalies:
arguments:
column_anomalies: < specific monitors, all if null >
where_expression: < sql expression >
```
```yml Models example theme={null}
models:
- name: login_events
config:
elementary:
timestamp_column: 'loaded_at'
columns:
- name: user_name
data_tests:
- elementary.column_anomalies:
arguments:
column_anomalies:
- missing_count
- min_length
where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
time_bucket:
period: day
count: 1
config:
tags: ['elementary']
- name: users
## if no timestamp is configured, elementary will monitor without time filtering
data_tests:
elementary.volume_anomalies
config:
tags: ['elementary']
columns:
- name: user_id
data_tests:
- elementary.column_anomalies:
config:
tags: ['elementary']
arguments:
timestamp_column: 'updated_at'
where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
time_bucket:
period: < time period >
count: < number of periods >
- name: user_name
data_tests:
- elementary.column_anomalies:
arguments:
column_anomalies:
- missing_count
- min_length
config:
tags: ['elementary']
```
# dimension_anomalies
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies
Let our Slack chatbot create the anomaly test you need.
`elementary.dimension_anomalies`
The test counts rows grouped by given `dimensions` (columns/expressions).
This test practically monitors the frequency of values in the configured dimension over time, and alerts on unexpected changes in the distribution.
It is best to configure it on low-cardinality fields.
If `timestamp_column` is configured, the distribution is collected per `time_bucket`. If not, it counts the total rows per dimension.
### Test configuration
*Required configuration: `dimensions`*
```yml Models theme={null}
models:
- name: < model name >
config:
elementary:
timestamp_column: < timestamp column >
data_tests:
- elementary.dimension_anomalies:
arguments:
dimensions: < columns or sql expressions of columns >
# optional - configure a where a expression to accurate the dimension monitoring
where_expression: < sql expression >
time_bucket: # Daily by default
period: < time period >
count: < number of periods >
```
```yml Models example theme={null}
models:
- name: login_events
config:
elementary:
timestamp_column: "loaded_at"
data_tests:
- elementary.dimension_anomalies:
arguments:
dimensions:
- event_type
- country_name
where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
time_bucket:
period: hour
count: 4
config:
# optional - use tags to run elementary tests on a dedicated run
tags: ["elementary"]
# optional - change severity
severity: warn
- name: users
# if no timestamp is configured, elementary will monitor without time filtering
data_tests:
- elementary.dimension_anomalies:
arguments:
dimensions:
- event_type
config:
tags: ["elementary"]
```
# event_freshness_anomalies
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-tests/event-freshness-anomalies
Let our Slack chatbot create the anomaly test you need.
`elementary.event_freshness_anomalies`
Monitors the freshness of event data over time, as the expected time it takes each event to load -
that is, the time between when the event actually occurs (the `event timestamp`), and when it is loaded to the
database (the `update timestamp`).
This test complements the `freshness_anomalies` test and is primarily intended for data that is updated in a continuous / streaming fashion.
The test can work in a couple of modes:
* If only an `event_timestamp_column` is supplied, the test measures over time the difference between the current
timestamp ("now") and the most recent event timestamp.
* If both an `event_timestamp_column` and an `update_timestamp_column` are provided, the test will measure over time
the difference between these two columns.
### Test configuration
*Required configuration: `event_timestamp_column`*
*Default configuration: `anomaly_direction: spike` to alert only on delays.*
```yml Models theme={null}
models:
- name: < model name >
data_tests:
- elementary.event_freshness_anomalies:
arguments:
event_timestamp_column: < timestamp column > # Mandatory
update_timestamp_column: < timestamp column > # Optional
where_expression: < sql expression >
time_bucket: # Daily by default
period: < time period >
count: < number of periods >
```
```yml Models example theme={null}
models:
- name: login_events
data_tests:
- elementary.event_freshness_anomalies:
arguments:
event_timestamp_column: "occurred_at"
update_timestamp_column: "updated_at"
config:
# optional - use tags to run elementary tests on a dedicated run
tags: ["elementary"]
# optional - change severity
severity: warn
```
# freshness_anomalies
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-tests/freshness-anomalies
Let our Slack chatbot create the anomaly test you need.
`elementary.freshness_anomalies`
Monitors the freshness of your table over time, as the expected time between data updates.
Upon running the test, your data is split into time buckets (daily by default, configurable with the `time bucket` field),
and then we compute the maximum time without updates (in seconds) as the freshness value per bucket for the last `training_period` (by default 14 days).
The test then compares the freshness of each bucket within the detection period (last 2 days by default, controlled by the
`detection_period` var), and compares it to the freshness of the previous time buckets.
If there were any anomalies during the detection period, the test will fail.
### Test configuration
*Required configuration: `timestamp_column`*
*Default configuration: `anomaly_direction: spike` to alert only on delays.*
```yml Models theme={null}
models:
- name: < model name >
data_tests:
- elementary.freshness_anomalies:
arguments:
timestamp_column: < timestamp column > # Mandatory
where_expression: < sql expression >
time_bucket: # Daily by default
period: < time period >
count: < number of periods >
```
```yml Models example theme={null}
models:
- name: login_events
data_tests:
- elementary.freshness_anomalies:
arguments:
timestamp_column: "updated_at"
config:
# optional - use tags to run elementary tests on a dedicated run
tags: ["elementary"]
# optional - change severity
severity: warn
```
# volume_anomalies
Source: https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies
Let our Slack chatbot create the anomaly test you need.
`elementary.volume_anomalies`
Monitors the row count of your table over time per time bucket (if configured without `timestamp_column`, will count table total rows).
Upon running the test, your data is split into time buckets (daily by default, configurable with the `time bucket` field),
and then we compute the row count per bucket for the last [`training_period`](/data-tests/anomaly-detection-configuration/training-period) days (by default 14).
The test then compares the row count of each bucket within the detection period (last 2 days by default, configured as [`detection_period`](/data-tests/anomaly-detection-configuration/detection-period)),
and compares it to the row count of the previous time buckets.
**The test will only run on completed time buckets**, so if you run it with daily buckets in the middle of today, the test would only count yesterday as a complete bucket.
If there were any anomalies during the detection period, the test will fail.
### Test configuration
No mandatory configuration, however it is highly recommended to configure a `timestamp_column`.
```yml Models theme={null}
models:
- name: < model name >
data_tests:
- elementary.volume_anomalies:
arguments:
timestamp_column: < timestamp column >
where_expression: < sql expression >
time_bucket: # Daily by default
period: < time period >
count: < number of periods >
```
```yml Models example theme={null}
models:
- name: login_events
config:
elementary:
timestamp_column: "loaded_at"
data_tests:
- elementary.volume_anomalies:
arguments:
where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
time_bucket:
period: day
count: 1
config:
# optional - use tags to run elementary tests on a dedicated run
tags: ["elementary"]
# optional - change severity
severity: warn
- name: users
# if no timestamp is configured, elementary will monitor without time filtering
data_tests:
- elementary.volume_anomalies:
config:
tags: ["elementary"]
```
# Data anomaly detection method
Source: https://docs.elementary-data.com/data-tests/data-anomaly-detection
Elementary uses "[standard score](https://en.wikipedia.org/wiki/Standard_score)", also known as "Z-score" for anomaly detection. This score represents the number of standard deviations of a value from the average of a set of values.
According to the [empirical rule](https://www.statisticshowto.com/probability-and-statistics/statistics-definitions/empirical-rule-2/), in a standard normal distribution:
* **\~68%** of values have an absolute **z-score of 1 or less.**
* **\~95%** of values have an absolute **z-score of 2 or less.**
* **\~99.7%** of values have an absolute **z-score of 3 or less.**
Values with a **standard score of 3 and above are [considered outliers](https://www.ctspedia.org/do/view/CTSpedia/OutLier)**, and this is a recommended threshold for anomaly detection.
This is the default Elementary uses as well, and it can be changed using the var `anomaly_score_threshold` in the [global configuration](/data-tests/elementary-tests-configuration).
You can use the model `anomaly_sensitivity` to see if values of metrics from your last run would have been considered anomalies in different scores. **This can help you decide if there is a need to adjust the sensitivity:**
# data_freshness_sla
Source: https://docs.elementary-data.com/data-tests/data-freshness-sla
Let our Slack chatbot create the anomaly test you need.
`elementary.data_freshness_sla`
Verifies that data in a model was updated before a specified SLA deadline time.
This test checks the maximum timestamp value of a specified column in your data to determine whether the data was actually refreshed before your deadline. Unlike `freshness_anomalies` (which uses z-score based anomaly detection as a dbt test, or ML-based detection in Elementary Cloud), this test validates against a fixed, explicit SLA time, making it ideal when you have a concrete contractual or operational deadline.
Unlike `execution_sla` (which only checks if the dbt model *ran* on time), `data_freshness_sla` checks whether the actual *data* is fresh. A pipeline can run successfully but still serve stale data if, for example, an upstream source didn't update. This test catches that.
### Use Case
"Was the data in my model updated before 7 AM Pacific today?"
### Test Logic
1. If today is not a scheduled check day → **PASS** (skip)
2. Query the model for the maximum value of `timestamp_column`
3. If the max timestamp is from today → **PASS** (data is fresh)
4. If the SLA deadline hasn't passed yet → **PASS** (still time)
5. If the max timestamp is from a previous day → **FAIL** (DATA\_STALE)
6. If no data exists in the table → **FAIL** (NO\_DATA)
### Test configuration
*Required configuration: `timestamp_column`, `sla_time`, `timezone`*
data\_tests:
-- elementary.data\_freshness\_sla:
arguments:
timestamp\_column: column name # Required - timestamp column to check for freshness
sla\_time: string # Required - e.g., "07:00", "7am", "2:30pm", "14:30"
timezone: string # Required - IANA timezone name, e.g., "America/Los\_Angeles"
day\_of\_week: string | array # Optional - Day(s) to check: "Monday" or \["Monday", "Wednesday"]
day\_of\_month: int | array # Optional - Day(s) of month to check: 1 or \[1, 15]
where\_expression: sql expression # Optional - filter the data before checking
```yml Models theme={null}
models:
- name: < model name >
data_tests:
- elementary.data_freshness_sla:
arguments:
timestamp_column: < column name > # Required
sla_time: < deadline time > # Required - e.g., "07:00", "7am", "2:30pm"
timezone: < IANA timezone > # Required - e.g., "America/Los_Angeles"
day_of_week: < day or array > # Optional
day_of_month: < day or array > # Optional
where_expression: < sql expression > # Optional
```
```yml Daily check theme={null}
models:
- name: daily_revenue
data_tests:
- elementary.data_freshness_sla:
arguments:
timestamp_column: updated_at
sla_time: "07:00"
timezone: "America/Los_Angeles"
config:
tags: ["elementary"]
severity: error
```
```yml With filter expression theme={null}
models:
- name: daily_events
data_tests:
- elementary.data_freshness_sla:
arguments:
timestamp_column: event_timestamp
sla_time: "6am"
timezone: "Europe/Amsterdam"
where_expression: "event_type = 'completed'"
config:
tags: ["elementary"]
```
```yml Weekly - only Mondays theme={null}
models:
- name: weekly_report_data
data_tests:
- elementary.data_freshness_sla:
arguments:
timestamp_column: report_date
sla_time: "09:00"
timezone: "Asia/Tokyo"
day_of_week: ["Monday"]
config:
tags: ["elementary"]
```
### Features
* **Data-level freshness**: Checks actual data timestamps, not just pipeline execution time
* **Flexible time formats**: Supports `"07:00"`, `"7am"`, `"2:30pm"`, `"14:30"`, and other common formats
* **IANA timezone support**: Uses standard timezone names like `"America/Los_Angeles"`, `"Europe/Amsterdam"`, etc.
* **Automatic DST handling**: Uses `pytz` for timezone conversions with automatic daylight saving time handling
* **Database-agnostic**: All timezone logic happens at compile time
* **Schedule filters**: Optional `day_of_week` and `day_of_month` parameters to check only specific days
* **Filter support**: Use `where_expression` to check freshness of a specific subset of data
### Parameters
| Parameter | Required | Description |
| ------------------ | -------- | -------------------------------------------------------- |
| `timestamp_column` | Yes | Column name containing timestamps to check for freshness |
| `sla_time` | Yes | Deadline time (e.g., `"07:00"`, `"7am"`, `"2:30pm"`) |
| `timezone` | Yes | IANA timezone name (e.g., `"America/Los_Angeles"`) |
| `day_of_week` | No | Day(s) to check: `"Monday"` or `["Monday", "Wednesday"]` |
| `day_of_month` | No | Day(s) of month to check: `1` or `[1, 15]` |
| `where_expression` | No | SQL expression to filter the data before checking |
### Comparison with other freshness tests
| Feature | `data_freshness_sla` | `freshness_anomalies` | `execution_sla` |
| ------------------ | ---------------------------------------------- | ---------------------------------------------- | ---------------------------------------- |
| What it checks | Actual data freshness (timestamps in the data) | Actual data freshness (timestamps in the data) | Pipeline execution (did the model run?) |
| Detection method | Fixed SLA deadline | Z-score (dbt test) / ML (Cloud) | Fixed SLA deadline |
| Best for | Contractual/operational deadlines on data | Detecting unexpected delays in data updates | Ensuring the pipeline itself ran on time |
| Works with sources | Yes | Yes | No (models only) |
### Notes
* The `timestamp_column` values are assumed to be in **UTC** (or timezone-naive timestamps that represent UTC). If your data stores local timestamps, the comparison may be incorrect.
* If both `day_of_week` and `day_of_month` are set, the test uses OR logic (checks if either matches)
* The test passes if the SLA deadline hasn't been reached yet, giving your data time to be updated
# dbt artifacts
Source: https://docs.elementary-data.com/data-tests/dbt/dbt-artifacts
The [Elementary dbt package](/cloud/quickstart#install-the-dbt-package) includes uploading and modeling of dbt artifacts.
Each dbt invocation generates artifacts, with details about the project resources, configuration and execution. Most are in json format. The artifacts are practically the metadata and logs of the dbt project.
## How Elementary uploads dbt artifacts
The Elementary dbt package includes macros that extract specific fields from the artifacts during the execution, and insert to tables. These tables are defined as models of the package.
Some artifacts are updated by an [on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end) hook when you run `dbt run / test / build`, and some only when you run the models:
* **On run end:**
* `elementary_test_results` - Results of all dbt tests (including elementary and other packages, such as dbt\_expectations and dbt\_utils).
* `dbt_run_results` - Results of all dbt executions.
* **`dbt run` that includes elementary models:**
* `dbt_models`, `dbt_tests`, `dbt_sources`, `dbt_exposures`, `dbt_metrics` - Metadata and configuration. It is recommended to run these models when you make changes in your projects.
Read more about [Elementary on-run-end hooks](/data-tests/dbt/on-run-end_hooks)
## Which artifacts are uploaded and modeled?
Elementary uploads fields from the run-results, and the graph object (includes data of manifest, catalog and sources).
You can disable or enable the relevant models in the Elementary package if you want to limit the artifacts that are loaded to your data warehouse (under `elementary/models/edr/dbt_artifacts`).
## dbt artifacts models
Elementary loads data from dbt artifacts to models that can be found under `dbt_artifacts` folder of the package.
For `dbt_run_results` data is inserted at the on-run-end, and for the rest of the models it's a post-hook on the model itself.
The dbt artifacts models include:
* **Metadata tables** - `dbt_models`, `dbt_tests`, `dbt_sources`, `dbt_exposures`, `dbt_metrics`, `dbt_snapshots`, `dbt_seeds`, `dbt_columns`, `dbt_groups` - These provide a comprehensive view of your dbt project structure and configurations. Each time these models are executed, the data is replaced with data from the project's current graph.
* **Run results tables** - `dbt_run_results`, `dbt_invocations`, `elementary_test_results`, `model_run_results`, `snapshot_run_results`, `seed_run_results`, `job_run_results`, `dbt_source_freshness_results` - These track execution details, test outcomes, and performance metrics from your dbt runs.
For detailed schema documentation including all columns and their descriptions, see the [package models documentation](/data-tests/dbt/package-models).
# Elementary dbt package
Source: https://docs.elementary-data.com/data-tests/dbt/dbt-package
The Elementary dbt package serves as a collector of logs and metadata from your dbt project and offers a set of data anomaly detection and schema tests.
To gain the most value from the dbt package, we recommend using it with the [Elementary Cloud Platform](/cloud/introduction) or with [Elementary open-source CLI tool](/oss/oss-introduction).
Quickstart the Elementary dbt package in minutes [here](/data-tests/dbt/quickstart-package). The repository and source code of the can be [found here](https://github.com/elementary-data/dbt-data-reliability).
## Package Features
The Elementary dbt package is designed to power data observability use cases for dbt pipelines.
This package will upload logs and metadata generated from your runs as dbt artifacts into tables in your data warehouse.
Additionally, it offers a wide range of tests, including anomalies in volume, freshness, columns and different dimensions of your data.
The impact of the package on **`dbt run`** is minimal, and most of the processing happens as part of the data tests that are executed on **`dbt test`**.
A dbt package is additional Jinja and SQL code that is added to your project, for additional functionality. In fact, each package is a dbt project. By adding a package to your project, you are adding the package code to be part of your project, you can reference its macros, execute its models, and so on.
Add packages to your project by creating a `packages.yml` file under the main project directory (where your `
dbt_project.yml` is), and adding the relevant package. After you add a new package, run `dbt deps` to actually pull its
code to your project. This is also how you update packages.
Some packages we recommend you check out: [dbt\_utils](https://github.com/dbt-labs/dbt-utils/tree/0.8.2/)
, [dbt\_date](https://github.com/calogica/dbt-date/tree/0.5.4/)
, [codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.5.0/).
After you deploy the dbt package, you can use Elementary tests, and your dbt artifacts will be uploaded automatically with on-run-end hooks:
## Elementary schema
The Elementary package creates various models to store information about collected dbt artifacts and test results.
To avoid mixing with your existing models, we recommend configuring a dedicated schema for the Elementary models using
the [dbt custom schema](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-schemas)
option.
Here is an example configuration, that creates a schema with the suffix '\_elementary' for elementary models:
```yml dbt_project.yml theme={null}
models:
elementary:
+schema: elementary
```
## Package full refresh
Elementary incremental models are not full-refreshed by default. This is because these models contain
information such as historical runs and test results, which typically you'd want to maintain even when full refreshing
the models of your main dbt project.
However, if you wish to change this behavior and include Elementary models as a part of the full refresh, you can set
the following var:
```yaml theme={null}
vars:
elementary_full_refresh: true
```
# Elementary dbt package on-run-end hooks
Source: https://docs.elementary-data.com/data-tests/dbt/on-run-end_hooks
Elementary dbt package uses `on-run-end` [hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end) to log results and metadata to tables in the Elementary schema.
## Why Elementary uses `on-run-end` hooks?
Elementary report and alerts are generated from the data in the Elementary schema.
The solution relies on the Elementary schema being up-to-date and complete to be able to provide reliable and accurate observability.
By leveraging `on-run-end` hooks, we add a built-in collection of the latest results and metadata as part of your runs.
This means the results you see in Elementary report and the alerts you receive are full, up-to-date and accurate.
We strongly recommend not to disable the hooks for environments you want to monitor using Elementary.
## What happens on the `on-run-end` hooks?
On the `on-run-end` hooks Elementary extracts data from the dbt `results` and `graph` objects, and runs SQL queries to load this data to the Elementary models.
There are 2 types of models that Elementary updates :
1. **Metadata models** - Such as `dbt_models`, `dbt_tests`, `dbt_sources`.
2. **Result models** - Such as `dbt_run_results`, `elementary_test_results`, `dbt_invocations`.
#### Updates of metadata models
These models store the current resources and configuration in your dbt projects (models, snapshots, sources, tests, etc.).
The metadata in the models only represents the project state on the latest run, so upon changes the metadata is replaced.
The `on-run-end` hook runs SQL queries with the new metadata and updates the relevant tables.
#### Updates of result models
These models store a log of results of dbt invocations, and of the specific executed resources.
The `on-run-end` hook runs SQL queries with the run results and invocation details.
## Performance impact of `on-run-end` hooks
We give a lot of thought and effort to making Elementary efficient in both cost and performance.
We only run the hooks that are relevant to each run, and each hook creates a minimal amount of queries possible.
#### Metadata models
**For `dbt 1.4.0` and above**, we maintain a metadata cache. This means each of these models are only updated with changes in your project (new model, change in config, etc.).
The first time you execute Elementary the initial update might take a while, but the following updates should be quick.
**For `dbt 1.3.0` and lower**, these models would be fully updated on each run.
The performance impact depends on the size of your dbt project.
**We strongly recommend Elementary users to use a dbt version of 1.4.0 or above.**
Elementary implemented an "artifacts cache" to improve performance drastically.
A change to dbt-core was required to achieve that, which was only included in 1.4.0 release.
This means that if you upgrade to dbt 1.4.0 or above you will get a great improvement in Elementary hooks runtime.
**If you can't upgrade, the alternative is** -
1. Disable the auto uploading of artifacts.
2. Make sure to upload artifacts yourself anytime you make a change to the project (merge a PR).
**How?**
1. Add this to dbt\_project.yml:
```yaml dbt_project.yml theme={null}
vars:
disable_dbt_artifacts_autoupload: true
```
2. Make sure to run `dbt run --select elementary --vars '{"enable_elementary_models": true}'` upon merging PRs.
#### Result models
The size of the queries depends on the amount of models/tests executed in the run.
The time the run results adds to the invocation shouldn't be significant.
## Can I disable the `on-run-end` hooks?
Yes, but note that this may cause missing results and/or outdated metadata in Elementary report and alerts.
#### Disable metadata models updates
Configure the following var:
```yaml dbt_project.yml theme={null}
vars:
disable_dbt_artifacts_autoupload: true
```
If you only want to disable dbt\_columns but leave all other artifacts, configure the following:
```yaml dbt_project.yml theme={null}
vars:
disable_dbt_columns_autoupload: true
```
If you disable the artifacts autoupload, we recommend you run `dbt run --select elementary --vars '{"enable_elementary_models": true}'` every time you deploy changes to your project.
#### Disable result models updates
Configure the following vars (you can also disable with conditions):
```yaml dbt_project.yml theme={null}
vars:
disable_run_results: true
disable_tests_results: true
disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"
```
Please note you can also use the vars in your command line rather than in the dbt\_project.yml file, by adding `--vars '{"disable_dbt_artifacts_autoupload": true}'` to your command line.
# Elementary dbt package models
Source: https://docs.elementary-data.com/data-tests/dbt/package-models
## Run Results Tables
These tables track execution details, test outcomes, and performance metrics from your dbt runs.
### dbt\_run\_results
*Incremental model*
Run results of dbt invocations, inserted at the end of each invocation.
Each row is the invocation result of a single resource (model, test, snapshot, etc).
New data is loaded to this model on an on-run-end hook named `elementary.upload_run_results` from each invocation that produces a result object.
* `model_execution_id` (string) - Execution id generated by joining the unique\_id of the resource and the invocation\_id. This is the unique key of each row.
* `unique_id` (string) - The unique id of the resource (would be similar for all executions of the same resource).
* `invocation_id` (string) - The unique id of the invocation (would be similar for all resources executed on the same invocation). Foreign key to `dbt_invocations`.
* `name` (string) - Resource name.
* `status` (string) - Execution result status (success, error, pass, fail).
* `resource_type` (string) - Resource type (model, test, snapshot, seed, etc).
* `execution_time` (float) - Resource execution duration in seconds.
* `execute_started_at` (string) - Start time of the execution.
* `execute_completed_at` (string) - End time of the execution.
* `compile_started_at` (string) - Start time of resource compile action.
* `compile_completed_at` (string) - End time of resource compile action.
* `rows_affected` (int) - Number of rows affected by the execution.
* `full_refresh` (boolean) - Whether this was a full refresh execution.
* `compiled_code` (string) - The compiled code (SQL / Python) executed against the database. For microbatch incremental models, this column requires [extra setup](/cloud/guides/microbatch-compiled-code).
* `failures` (int) - Number of failures in this run.
* `query_id` (string) - Query ID in the data warehouse, if returned by the adapter (currently only supported in Snowflake, is null for any other adapter).
* `thread_id` (string) - Id of the thread of this resource run.
* `adapter_response` (string) - Response returned by the adapter (Fields will be different for each adapter).
* `message` (string) - Execution results message returned by dbt.
* `generated_at` (string) - Timestamp when the result was generated.
### dbt\_invocations
*Incremental model*
Attributes associated with each dbt invocation, inserted at the end of each invocation.
Each row is the result of a single invocation (dbt run, dbt test, dbt build, etc).
New data is loaded to this model on an on-run-end hook named `elementary.upload_dbt_invocation`.
It also contains information about your job or what triggered the invocation such as `pull_request_id`, `git_sha`, or `cause`.
If you're using an orchestrator that Elementary natively supports such as dbt Cloud or GitHub Actions,
this data is automatically populated, otherwise, you can populate it by using environment variables in the form of `DBT_`.
For instance, adding `DBT_JOB_NAME` will populate `dbt_invocations.job_name` with the value of the environment variable.
* `invocation_id` (string) - Primary key of this table.
* `run_started_at` (string) - Timestamp the invocation was started.
* `run_completed_at` (string) - Timestamp the invocation was completed.
* `generated_at` (string) - The time this invocation was uploaded to the database.
* `command` (string) - dbt command that was used (e.g., run, test, build).
* `dbt_version` (string) - Version of dbt that was used in this invocation.
* `elementary_version` (string) - Version of the elementary package that was used in this invocation.
* `full_refresh` (boolean) - Whether or not this invocation was executed as a full-refresh.
* `target_name` (string) - Name of the target used in this invocation.
* `target_database` (string) - Name of the target database that was used in this invocation.
* `target_schema` (string) - Name of the target schema that was used in this invocation.
* `target_profile_name` (string) - Name of the dbt profile that was used in this invocation.
* `threads` (integer) - Number of threads that were used to run this dbt invocation.
* `selected` (string) - The selected resources in the dbt command. While this is a string in the database, this can easily be converted to an array.
* `yaml_selector` (string) - The yaml selector that was passed in this invocation.
* `job_id` (string) - The ID of a job, defined in the `job_id` var or in the `JOB_ID` env var or by the orchestrator.
* `job_name` (string) - The name of a job, defined in the `job_name` var or in the `JOB_NAME` env var.
* `job_run_id` (string) - The run ID of a job, defined in the `job_run_id` var or in the `DBT_JOB_RUN_ID` env var or by the orchestrator.
* `env` (string) - The environment's name, defined in the `DBT_ENV` env var.
* `env_id` (string) - The ID of an environment, defined in the `DBT_ENV_ID` env var.
* `project_id` (string) - The ID of a project, defined in the `DBT_PROJECT_ID` env var or by the orchestrator.
* `cause_category` (string) - The category of the cause of the invocation (e.g., schedule, manual).
* `cause` (string) - The cause of the invocation (e.g., "Kicked off by Joe").
* `pull_request_id` (string) - The ID of a pull request, defined in the `DBT_PULL_REQUEST_ID` env var or by the orchestrator.
* `git_sha` (string) - The git SHA of the commit that was used in this invocation.
* `orchestrator` (string) - The orchestrator that was used to run this invocation (e.g., dbt Cloud, GitHub Actions).
* `job_url` (string) - The URL of the job, defined in the `job_url` var or in the `JOB_URL` env var or by the orchestrator.
* `account_id` (string) - The ID of the account, defined in the `account_id` var or in the `ACCOUNT_ID` env var or by the orchestrator.
* `invocation_vars` (string) - Dictionary of the variables (and values) that were declared in the invocation.
* `vars` (string) - Dictionary of all variables (and values) in the dbt project.
### model\_run\_results
*View*
Run results of dbt models, enriched with models metadata.
Each row is the result of a single model.
This is a view that joins data from `dbt_run_results` and `dbt_models`.
**Columns:** Combines all columns from `dbt_run_results` with metadata from `dbt_models` such as `database_name`, `schema_name`, `tags`, `owner`, `materialization`, `package_name`, `path`, `original_path`, and `alias`.
### snapshot\_run\_results
*View*
Run results of dbt snapshots, enriched with snapshots metadata.
Each row is the result of a single snapshot.
This is a view that joins data from `dbt_run_results` and `dbt_snapshots`.
**Columns:** Combines all columns from `dbt_run_results` with metadata from `dbt_snapshots` such as `database_name`, `schema_name`, `tags`, `owner`, `materialization`, `package_name`, `path`, `original_path`, and `alias`.
### seed\_run\_results
*View*
Run results of dbt seeds, enriched with seeds metadata.
Each row is the result of a single seed.
This is a view that joins data from `dbt_run_results` and `dbt_seeds`.
**Columns:** Combines all columns from `dbt_run_results` with metadata from `dbt_seeds` such as `database_name`, `schema_name`, `tags`, `owner`, `package_name`, `path`, `original_path`, and `alias`.
### job\_run\_results
*View*
Run results of dbt invocations, enriched with jobs metadata.
Each row is the result of a single job.
This is a view on `dbt_invocations`.
**Columns:** All columns from `dbt_invocations` table.
### elementary\_test\_results
*Incremental model*
Run results of all dbt tests, with fields and metadata needed to produce
the [Elementary report](/cloud/features/collaboration-and-communication/data-observability-dashboard).
Each row is the result of a single test, including native dbt tests, packages tests and elementary tests.
New data is loaded to this model on an on-run-end hook named `elementary.handle_tests_results`.
* `id` (string) - Unique identifier for the test result.
* `test_unique_id` (string) - The unique id of the test.
* `invocation_id` (string) - The unique id of the invocation. Foreign key to `dbt_invocations`.
* `detected_at` (timestamp) - When the test result was detected.
* `created_at` (timestamp) - When the test result record was created.
* `status` (string) - Test result status (pass, fail, error, warn).
* `result_rows` (int) - Number of rows that failed the test (for failing tests).
* `failures` (int) - Number of failures.
* Additional columns include test metadata, execution details, and test-specific information.
### test\_result\_rows
*Incremental model*
Failed test row samples. Each row contains a sample of data that caused a test to fail, including the test result ID that links to the parent test result, the actual sample data stored as JSON, and timestamps for detection and creation. By default, up to 5 sample rows are stored per failed test (configurable via [test\_sample\_row\_count](https://docs.elementary-data.com/oss/general/faq#can-i-see-more-result-samples-in-the-report)).
* `id` (string) - Unique identifier for the test result row.
* `test_result_id` (string) - Links to the parent test result in `elementary_test_results`.
* `result_row` (string/JSON) - Sample data that caused the test to fail, stored as JSON.
* `detected_at` (timestamp) - When the failing row was detected.
* `created_at` (timestamp) - When the record was created.
### dbt\_source\_freshness\_results
*Incremental model*
Results from dbt source freshness checks. Tracks when source data was last updated and whether it meets freshness thresholds.
* `source_freshness_execution_id` (string) - Unique identifier for the freshness check execution.
* `unique_id` (string) - The unique id of the source.
* `max_loaded_at` (string) - The maximum loaded\_at timestamp found in the source data.
* `snapshotted_at` (string) - When the freshness check was performed.
* `generated_at` (string) - When the result was generated.
* `created_at` (timestamp) - When the record was created.
* `max_loaded_at_time_ago_in_s` (float) - How many seconds ago the max\_loaded\_at timestamp was.
* `status` (string) - Freshness check status (pass, warn, error).
* `error` (string) - Error message if the check failed.
* `compile_started_at` (string) - Start time of compilation.
* `compile_completed_at` (string) - End time of compilation.
* `execute_started_at` (string) - Start time of execution.
* `execute_completed_at` (string) - End time of execution.
* `invocation_id` (string) - The unique id of the invocation. Foreign key to `dbt_invocations`.
* `warn_after` (string) - Freshness warning threshold.
* `error_after` (string) - Freshness error threshold.
* `filter` (string) - Filter expression used in the freshness check.
## Metadata Tables - dbt Artifacts
These tables provide a comprehensive view of your dbt project structure and configurations.
The dbt artifacts models are created as empty tables, and a post-hook macro inserts data from the dbt graph object to
the table.
**Each time the model is executed, the data is replaced with the project's current graph.**
It is recommended to execute these models every time a change is merged to the project.
### dbt\_models
*Table*
Metadata about all the models in the project and project packages.
Each row contains information about the properties of a single model, including columns like tags, owner,
materialization, depends\_on, and description.
* `unique_id` (string) - The unique id of the model.
* `name` (string) - Model name.
* `alias` (string) - Model alias.
* `database_name` (string) - The model database name.
* `schema_name` (string) - The model schema name.
* `materialization` (string) - The model materialization config (e.g., table, view, incremental).
* `tags` (string) - Model tags property (stored as JSON array string).
* `meta` (string) - The content of 'meta' property key (stored as JSON string).
* `owner` (string) - Model owner property (configured under 'meta' key).
* `description` (string) - Model description.
* `package_name` (string) - Package name of the model.
* `path` (string) - Short path of the model file.
* `original_path` (string) - Full path of the model file.
* `checksum` (string) - Model file checksum.
* `depends_on_macros` (string) - The macros the model directly depends on (stored as JSON array string).
* `depends_on_nodes` (string) - The nodes the model directly depends on (stored as JSON array string).
* `generated_at` (string) - Update time of the table.
### dbt\_tests
*Table*
Metadata about all the tests in the project and project packages.
Each row contains information about the properties of a single test, including columns like severity, parent model
unique id, tags and owner of the parent model, test params, and the test compiled query.
* `unique_id` (string) - The unique id of the test.
* `name` (string) - The test name.
* `short_name` (string) - Short name of the test.
* `alias` (string) - Test alias.
* `type` (string) - Test type (e.g., singular, generic).
* `test_namespace` (string) - Namespace of the test (e.g., elementary, dbt\_utils).
* `database_name` (string) - The tested model database name.
* `schema_name` (string) - The tested model schema name.
* `test_column_name` (string) - The name of the tested column (null for table-level tests).
* `parent_model_unique_id` (string) - The unique id of the model this test is attached to.
* `severity` (string) - Test severity (error, warn).
* `warn_if` (string) - Warning condition.
* `error_if` (string) - Error condition.
* `test_params` (string) - Test parameters (stored as JSON string).
* `tags` (string) - Test tags (stored as JSON array string).
* `model_tags` (string) - Tags from the parent model (stored as JSON array string).
* `model_owners` (string) - Owners from the parent model (stored as JSON array string).
* `meta` (string) - Test metadata (stored as JSON string).
* `description` (string) - Test description.
* `package_name` (string) - Package name of the test.
* `path` (string) - Short path of the test file.
* `original_path` (string) - Full path of the test file.
* `depends_on_macros` (string) - The macros the test directly depends on (stored as JSON array string).
* `depends_on_nodes` (string) - The nodes the test directly depends on (stored as JSON array string).
* `generated_at` (string) - Update time of the table.
### dbt\_sources
*Table*
Metadata about the sources configured in the project and project packages.
Each row contains information about the properties of a single source, including columns like tags, owner, freshness
configuration, database and schema.
* `unique_id` (string) - The unique id of the source.
* `source_name` (string) - The name of the source.
* `name` (string) - The name of the source table.
* `identifier` (string) - The identifier of the source table.
* `database_name` (string) - The source database name.
* `schema_name` (string) - The source schema name.
* `relation_name` (string) - Full relation name (database.schema.table).
* `loaded_at_field` (string) - Field used for freshness checks.
* `freshness_warn_after` (string) - Freshness warning threshold.
* `freshness_error_after` (string) - Freshness error threshold.
* `freshness_filter` (string) - Filter expression for freshness checks.
* `tags` (string) - Source tags (stored as JSON array string).
* `meta` (string) - Source metadata (stored as JSON string).
* `owner` (string) - Source owner (configured under 'meta' key).
* `description` (string) - Source description.
* `source_description` (string) - Description of the source itself.
* `package_name` (string) - Package name of the source.
* `path` (string) - Short path of the source file.
* `original_path` (string) - Full path of the source file.
* `generated_at` (string) - Update time of the table.
### dbt\_exposures
*Table*
Metadata about the exposures configured in the project and project packages.
Each row contains information about the properties of a single exposure, including columns like tags, owner, url and
depends on.
* `unique_id` (string) - The unique id of the exposure.
* `name` (string) - Exposure name.
* `type` (string) - Exposure type (e.g., dashboard, notebook, application).
* `maturity` (string) - Exposure maturity level.
* `url` (string) - URL of the exposure.
* `owner_email` (string) - Email of the exposure owner.
* `owner_name` (string) - Name of the exposure owner.
* `description` (string) - Exposure description.
* `tags` (string) - Exposure tags (stored as JSON array string).
* `meta` (string) - Exposure metadata (stored as JSON string).
* `package_name` (string) - Package name of the exposure.
* `path` (string) - Short path of the exposure file.
* `original_path` (string) - Full path of the exposure file.
* `depends_on_macros` (string) - The macros the exposure directly depends on (stored as JSON array string).
* `depends_on_nodes` (string) - The nodes the exposure directly depends on (stored as JSON array string).
* `generated_at` (string) - Update time of the table.
### dbt\_metrics
*Table*
Metadata about the metrics configured in the project and project packages.
Each row contains information about the properties of a single metric, including columns like tags, owner, sql, and
depends on.
* `unique_id` (string) - The unique id of the metric.
* `name` (string) - Metric name.
* `label` (string) - Metric label.
* `model` (string) - The model this metric is based on.
* `type` (string) - Metric type (e.g., simple, derived).
* `sql` (string) - SQL expression for the metric.
* `timestamp` (string) - Timestamp column for the metric.
* `filters` (string) - Metric filters (stored as JSON string).
* `time_grains` (string) - Time grains for the metric (stored as JSON array string).
* `dimensions` (string) - Metric dimensions (stored as JSON array string).
* `description` (string) - Metric description.
* `tags` (string) - Metric tags (stored as JSON array string).
* `meta` (string) - Metric metadata (stored as JSON string).
* `package_name` (string) - Package name of the metric.
* `path` (string) - Short path of the metric file.
* `original_path` (string) - Full path of the metric file.
* `depends_on_macros` (string) - The macros the metric directly depends on (stored as JSON array string).
* `depends_on_nodes` (string) - The nodes the metric directly depends on (stored as JSON array string).
* `generated_at` (string) - Update time of the table.
### dbt\_snapshots
*Table*
Metadata about all the snapshots in the project and project packages.
Each row contains information about the properties of a single snapshot, including columns like tags, owner, depends\_on,
and description.
* `unique_id` (string) - The unique id of the snapshot.
* `name` (string) - Snapshot name.
* `alias` (string) - Snapshot alias.
* `database_name` (string) - The snapshot database name.
* `schema_name` (string) - The snapshot schema name.
* `materialization` (string) - The snapshot materialization config.
* `tags` (string) - Snapshot tags (stored as JSON array string).
* `meta` (string) - Snapshot metadata (stored as JSON string).
* `owner` (string) - Snapshot owner (configured under 'meta' key).
* `description` (string) - Snapshot description.
* `package_name` (string) - Package name of the snapshot.
* `path` (string) - Short path of the snapshot file.
* `original_path` (string) - Full path of the snapshot file.
* `checksum` (string) - Snapshot file checksum.
* `depends_on_macros` (string) - The macros the snapshot directly depends on (stored as JSON array string).
* `depends_on_nodes` (string) - The nodes the snapshot directly depends on (stored as JSON array string).
* `generated_at` (string) - Update time of the table.
### dbt\_seeds
*Table*
Metadata about seed files in the dbt project and project packages.
Each row contains information about the properties of a single seed, including columns like tags, owner, database, schema, and description.
* `unique_id` (string) - The unique id of the seed.
* `name` (string) - Seed name.
* `alias` (string) - Seed alias.
* `database_name` (string) - The seed database name.
* `schema_name` (string) - The seed schema name.
* `tags` (string) - Seed tags (stored as JSON array string).
* `meta` (string) - Seed metadata (stored as JSON string).
* `owner` (string) - Seed owner (configured under 'meta' key).
* `description` (string) - Seed description.
* `package_name` (string) - Package name of the seed.
* `path` (string) - Short path of the seed file.
* `original_path` (string) - Full path of the seed file.
* `checksum` (string) - Seed file checksum.
* `group_name` (string) - Group name if the seed belongs to a group.
* `metadata_hash` (string) - Hash of the metadata for change detection.
* `generated_at` (string) - Update time of the table.
### dbt\_columns
*Table*
Stores detailed information about columns across the dbt project.
Each row contains information about a single column from a model, source, or snapshot.
* `unique_id` (string) - The unique id of the column (format: `column.{parent_unique_id}.{column_name}`).
* `parent_unique_id` (string) - The unique id of the parent table (model, source, or snapshot).
* `name` (string) - Column name.
* `data_type` (string) - Column data type.
* `database_name` (string) - The database name of the parent table.
* `schema_name` (string) - The schema name of the parent table.
* `table_name` (string) - The table name (alias) of the parent table.
* `resource_type` (string) - Type of the parent resource (model, source, snapshot).
* `description` (string) - Column description.
* `tags` (string) - Column tags (stored as JSON array string).
* `meta` (string) - Column metadata (stored as JSON string).
* `metadata_hash` (string) - Hash of the metadata for change detection.
* `generated_at` (string) - Update time of the table.
### dbt\_groups
*Table*
Metadata about the groups configured in the project and project packages.
Each row contains information about the properties of a single group, including columns like group name and owner.
* `unique_id` (string) - The unique id of the group.
* `name` (string) - Group name.
* `owner` (string) - Group owner.
* `package_name` (string) - Package name of the group.
* `path` (string) - Short path of the group file.
* `original_path` (string) - Full path of the group file.
* `generated_at` (string) - Update time of the table.
## Alerts views
### alerts\_dbt\_models
*View*
A view that is used by the Elementary CLI to generate models alerts, including all the fields the alert will include
such as owner, tags, error message, etc.
It joins data about models and snapshots run results, and filters alerts according to configuration.
### alerts\_dbt\_tests
*View*
A view that is used by the Elementary CLI to generate dbt tests alerts, including all the fields the alert will include
such as owner, tags, error message, etc.
This view includes data about all dbt tests except elementary tests.
It filters alerts according to configuration.
### alerts\_anomaly\_detection
*View*
A view that is used by the Elementary CLI to generate alerts on data anomalies detected using the elementary anomaly
detection tests.
The view filters alerts according to configuration.
### alerts\_schema\_changes
*View*
A view that is used by the Elementary CLI to generate alerts on schema changes detected using elementary tests.
The view filters alerts according to configuration.
## Anomaly detection
### data\_monitoring\_metrics
*Incremental model*
Elementary anomaly detection tests monitor metrics such as volume, freshness and data quality metrics.
This incremental table is used to store the metrics over time.
On each anomaly detection test, the test queries this table for historical metrics, and compares to the latest values.
The table is updated with new metrics on the on-run-end named `handle_test_results` that is executed at the end of dbt
test invocations.
### metrics\_anomaly\_score
*View*
This is a view on `data_monitoring_metrics` that runs the same query the anomaly detection tests run to calculate
anomaly scores.
The purpose of this view is to provide visibility to the results of anomaly detection tests.
### anomaly\_threshold\_sensitivity
*View*
This is a view on `metrics_anomaly_score` that calculates if values of metrics from the latest runs would have been
considered anomalies in different anomaly scores.
This can help you decide if there is a need to adjust the `anomaly_score_threshold`.
### monitors\_runs
*View*
This is a view on `data_monitoring_metrics` that is used to determine when a specific anomaly detection test was last
executed.
Each anomaly detection test queries this view to decide on a start time for collecting metrics.
## Schema changes
### schema\_columns\_snapshot
*Incremental model*
Stores the schema details for tables that are monitored with elementary schema changes test.
In order to compare current schema to previous state, we must store the previous state.
The data is from a view that queries the data warehouse information schema.
# Quickstart: Install Elementary dbt package
Source: https://docs.elementary-data.com/data-tests/dbt/quickstart-package
To start using Elementary, you need to add our dbt package to your dbt project.
**Note:** elementary dbt package has to be installed in all of the connected projects in your environment.
A dbt package is additional Jinja and SQL code that is added to your project, for additional functionality. In fact, each package is a dbt project. By adding a package to your project, you are adding the package code to be part of your project, you can reference its macros, execute its models, and so on.
Add packages to your project by creating a `packages.yml` file under the main project directory (where your `dbt_project.yml` is), and adding the relevant package. After you add a new package, run `dbt deps` to actually pull its code to your project. This is also how you update packages.
Some packages we recommend you check out: [dbt\_utils](https://github.com/dbt-labs/dbt-utils/tree/0.8.2/), [dbt\_date](https://github.com/calogica/dbt-date/tree/0.5.4/), [codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.5.0/).
## Step-by-step: How to install the Elementary dbt package?
## Install Elementary dbt package
Add the following to your `packages.yml` file (if missing, create it where `dbt_project.yml` is):
```yml packages.yml theme={null}
packages:
- package: elementary-data/elementary
version: 0.25.1
## Docs: https://docs.elementary-data.com
```
This means Elementary models will have their own schema.
Depending on your project [custom schema macro](https://docs.getdbt.com/docs/build/custom-schemas), the schema will be named `elementary` or `_elementary`.
Make sure your user has permissions to create schemas.
For elementary to work, it needs to create some of the models as incremental tables.
Make sure that there are no global materialization configurations that affect elementary, such as:
```yml dbt_project.yml theme={null}
materialized: "{{ 'table' if target.name == 'prod-cloud' else 'view' }}"
```
Make sure to place the 'elementary' configuration under the models key, and other configs under your project name.
Example:
```yml dbt_project.yml theme={null}
models:
my_project:
materialized: "{{ 'table' if target.name == 'prod-cloud' else 'view' }}"
elementary:
+schema: "elementary"
```
**If you change materialization settings, make sure to run `dbt run -s elementary --full-refresh`.**
In order for these features to work, add the following file: elementary\_materialization.sql to your macros folder so it will look like this:
`macros/elementary_materialization.sql`
In the file copy and paste the following code:
If you use Snowflake:
```
{% materialization test, adapter='snowflake' %}
{{ return(elementary.materialization_test_snowflake()) }}
{% endmaterialization %}
```
If you use any other DWH:
```
{% materialization test, default %}
{{ return(elementary.materialization_test_default()) }}
{% endmaterialization %}
```
```yml dbt_project.yml theme={null}
models:
## see docs: https://docs.elementary-data.com/
elementary:
## elementary models will be created in the schema '_elementary'
+schema: "elementary"
## To disable elementary for dev, uncomment this:
# enabled: "{{ target.name in ['prod','analytics'] }}"
```
```shell Terminal theme={null}
dbt deps
dbt run --select elementary
```
This will mostly create empty tables, that will be updated with artifacts, metrics and test results in your future dbt executions.
Validate the installation by running some tests:
```shell Terminal theme={null}
dbt test
```
After you ran your tests, we recommend that you ensure that the results were loaded to `elementary_test_results` table.
## What happens now?
Once the elementary dbt package has been installed and configured, your test results, run results and [dbt artifacts](/data-tests/dbt/dbt-artifacts) will be loaded to elementary schema tables.
If you see data in these models you completed the package deployment (Congrats! 🎉).
Updating environment settings can take up to a few minutes.
## What's next?
Take a moment to ⭐️ [star our Github repo!](https://github.com/elementary-data/elementary) ⭐️ (It helps us a lot!)
# Control On-Run-End Hooks
Source: https://docs.elementary-data.com/data-tests/dbt/reduce-on-run-end-time
Elementary's `on-run-end` hooks collect test results and dbt artifacts to provide comprehensive observability. However, for large projects or when running only a subset of models, these hooks can add significant time to your dbt runs. This guide explains how to control the timing of on-run-end hooks to make them more efficient and avoid unnecessary runs while maintaining the observability you need.
## Overview
The Elementary on-run-end hooks perform several operations that can impact run time:
* **Metadata artifacts upload**: Uploads all project metadata (models, tests, sources, etc.)
* **Run results upload**: Uploads test results and model execution results
* **dbt invocation upload**: Uploads dbt invocation metadata
For large projects, these operations can take several minutes. The following strategies help you control when these hooks run to optimize efficiency and reduce unnecessary uploads.
## Control Metadata Artifacts Upload Timing
The most effective way to reduce on-run-end time is to control when metadata artifacts are uploaded. Instead of uploading all project metadata (models, tests, etc.) on every run, you can upload artifacts on a schedule that matches when your project actually changes.
### Configuration
Add the following to your `dbt_project.yml`:
```yaml dbt_project.yml theme={null}
vars:
disable_dbt_artifacts_autoupload: true
```
### Upload Artifacts When Needed
When you need to update metadata (e.g., after schema changes or adding new models), upload artifacts either manually or via a recurring job (e.g., daily):
```bash theme={null}
dbt run --select elementary --vars '{"enable_elementary_models": true}'
```
For the `enable_elementary_models` pattern in `dbt_project.yml`, see [Can I disable / exclude the Elementary models?](/oss/general/faq#can-i-disable-exclude-the-elementary-models).
Metadata artifacts only need to be uploaded when your project structure changes (new models, tests, sources, etc.). For most runs, you only need run results, which are much faster to upload.
## Control Other Hooks Timing
**Important**: Disabling hooks (other than metadata artifacts) means you will lose this data for those runs. Only disable hooks for runs you don't want to monitor (e.g., local development).
You can further control when different parts of the on-run-end logic run by setting additional variables.
### Available Configuration Variables
```yaml dbt_project.yml theme={null}
vars:
disable_run_results: true
disable_tests_results: true
disable_dbt_invocation_autoupload: true
```
* **`disable_run_results`**: Controls when model execution results are uploaded
* **`disable_tests_results`**: Controls when test results are uploaded
* **`disable_dbt_invocation_autoupload`**: Controls when dbt invocation metadata is uploaded
Disabling these hooks means you will lose this data for those runs. Only disable hooks for runs you don't want to monitor (e.g., local development). This will reduce the amount of data processed and uploaded at the end of each run, but will impact the completeness of reports and monitoring for those runs.
## Disable test table cleanup during on-run-end
Elementary creates temporary tables during test execution and cleans them up at the end of each run. For large projects with many tests, this cleanup phase can become a significant portion of the `on_run_end` time.
If you observe in the logs that the slow phase is the test table cleanup, you can disable it during `on_run_end` and offload it to a dedicated scheduled job instead.
### Disable automatic cleanup
Add the following to your `dbt_project.yml`:
```yaml dbt_project.yml theme={null}
vars:
clean_elementary_temp_tables: false
```
### Run cleanup in a dedicated job
After disabling automatic cleanup, schedule a recurring dbt operation to clean up stale test tables:
```bash theme={null}
dbt run-operation elementary.cleanup_stale_test_tables
```
You can pass optional arguments to control the cleanup behavior:
```bash theme={null}
dbt run-operation elementary.cleanup_stale_test_tables --args '{"hours": 48, "limit": 1000}'
```
**Parameters:**
* **`hours`** (default: `24`): Only delete test tables that are older than this many hours. Increase this value if you want to retain temporary tables for longer before cleanup.
* **`limit`** (default: `2000`): Maximum number of tables to delete per run. Use this to cap the cleanup operation and avoid long-running jobs when many stale tables have accumulated.
We recommend scheduling this job to run daily. If a large number of stale tables have built up, the `limit` parameter prevents the first cleanup from running indefinitely — you can run the job multiple times until all stale tables are removed.
## Limit Hooks to Production or Specific Targets
If you only need full observability in production, configure Elementary to run hooks only for specific targets. This is especially useful when you want to:
* Speed up local development
* Reduce costs in non-production environments
* Focus monitoring on production workloads
### Configuration by Target
Configure hooks to run only for specific targets:
```yaml dbt_project.yml theme={null}
vars:
disable_dbt_artifacts_autoupload: "{{ target.name != 'prod' }}"
disable_run_results: "{{ target.name != 'prod' }}"
disable_tests_results: "{{ target.name != 'prod' }}"
disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"
```
### Disable Entire Package for Non-Production
Alternatively, you can disable the entire Elementary package for non-production targets:
```yaml dbt_project.yml theme={null}
models:
elementary:
+enabled: "{{ target.name == 'prod' }}"
```
Disabling the entire package will prevent Elementary tests from running. We recommend using the hook disablement vars instead, which allows Elementary tests to run while skipping the artifact collection.
For more information about configuring Elementary for different environments:
* **Elementary Cloud users**: See the [Dev/Prod Configuration guide](/cloud/guides/dev-prod-configuration)
* **Elementary OSS users**: See the [OSS configuration documentation](/oss/general/faq)
# dbt Package 0.18.1
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.18.1
This update includes:
* 📝 Updated README
* 🐛 Report the same test status as dbt
* 🔧 Added resource type option to generate\_schema\_baseline\_test
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.18.1](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.18.1)
# dbt Package 0.18.2
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.18.2
This update includes:
* 🔧 Allow contributor PRs to run tests
* 🗄️ Temp tables macro improvements
* 🗄️ Support for Athena in integration tests
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.18.2](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.18.2)
# dbt Package 0.18.3
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.18.3
This update includes:
* 🗄️ ClickHouse integration - artifacts only
* 🐛 Fixed Databricks temp relations
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.18.3](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.18.3)
# dbt Package 0.19.0
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.19.0
This update includes:
* 📊 Added dbt group artifact and column support
* ⏰ Added seasonality parameter to test\_event\_freshness\_anomalies
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.0](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.0)
# dbt Package 0.19.1
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.19.1
This update includes:
* ☁️ BigQuery - save execution\_project in adapter-specific fields
* 🐛 Fixed dbt metric collection
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.1](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.1)
# dbt Package 0.19.2
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.19.2
This update includes:
* 📊 Updated tests quality dimension
* 🔧 Added metadata comment to all queries run with elementary.run.query
* 🔐 Snowflake user creation - use public key
* 📝 Added runtime log to dbt package
* 🐛 Fixed schema name generation for branch names with forward slashes
* 🔒 Added configuration to disable sample collection for PII tables
* 🛡️ Implemented column-level PII protection for sample collection
* 🗄️ Dremio improvements
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.2](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.2)
# dbt Package 0.19.3
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.19.3
This update includes:
* 🗄️ Dremio types mapping improvements
* 🔧 dbt-fusion fix: meta under config in dbt\_run\_results
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.3](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.3)
# dbt Package 0.19.4
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.19.4
This update includes:
* 📊 Dimension anomalies visualization
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.4](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.19.4)
# dbt Package 0.20.0
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.20.0
This update includes:
* 🗄️ dbt Fusion support
* ⚡ Performance improvements with indices for faster loading times
* 🐛 Fixed datetime casting macro for ClickHouse
* 🐛 Fixed Athena DROP TABLE parsing error with backtick quoting
* 🗄️ Dremio improvements: timestamp formatting, metadata permissions, and special character escaping
* 🐛 Fixed Databricks temp relations and Fusion bugfixes
* 📊 Index on test\_result\_rows.detected\_at column for better performance
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.20.0](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.20.0)
# dbt Package 0.20.1
Source: https://docs.elementary-data.com/data-tests/dbt/release-notes/releases/0.20.1
This update includes:
* 🔧 Fixed dbt Fusion version check to require major version 2
Check out the full details here: [https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.20.1](https://github.com/elementary-data/dbt-data-reliability/releases/tag/0.20.1)
# Upgrade Elementary dbt package
Source: https://docs.elementary-data.com/data-tests/dbt/upgrade-package
On new releases, you will need to upgrade the Elementary dbt package.
## Upgrade Elementary dbt package
1. On your `packages.yml` file, change the version to the latest:
```yml packages.yml theme={null}
packages:
- package: elementary-data/elementary
version: 0.25.1
```
2. Run the command:
```shell theme={null}
dbt deps
```
3. When there's a change in the structure of the Elementary tables, the minor version is raised. If you're updating a minor version (for example 0.22.X -> 0.23.X), run this command to rebuild the Elementary tables:
```shell theme={null}
dbt run --select elementary
```
**Note:** For CLI upgrades, refer to the [Elementary OSS upgrade guide](/oss/release-notes/upgrading-elementary#upgrade-elementary-cli).
# Tests configuration
Source: https://docs.elementary-data.com/data-tests/elementary-tests-configuration
In Elementary Cloud you can configure your tests in the UI.
This requires you to [connect a code repository](/cloud/integrations/code-repo/connect-code-repo) to your project.
Configuring a test will open a pull request / merge request in your code repository.
**Step 1 - Choose a target**
**Step 2 - Choose a test**
**Step 3 - Configure the test**
**Step 4 - Review and submit**
The anomaly detection tests configuration is defined in `.yml` files in your dbt project, just like in native dbt tests.
The configuration of Elementary is dbt native and follows the same priorities and inheritance.
The more granular and specific configuration overrides the less granular one.
Elementary searches and prioritizes configuration in the following order:
**For models tests:**
1. Test arguments.
2. Tests path configuration under `tests` key in `dbt_project.yml`.
3. Model configuration.
4. Path configuration under `models` key in `dbt_project.yml`.
5. Global vars in `dbt_project.yml`.
**For sources tests:**
1. Test arguments.
2. Tests path configuration under `tests` key in `dbt_project.yml`.
3. Table configuration.
4. Source configuration.
5. Global vars in `dbt_project.yml`.
# execution_sla
Source: https://docs.elementary-data.com/data-tests/execution-sla
Let our Slack chatbot create the anomaly test you need.
`elementary.execution_sla`
Verifies that dbt models are executed successfully before a specified SLA deadline time.
This test checks whether your pipeline *ran* before a specified deadline on the days you care about. It queries `dbt_run_results` for successful runs of the model and validates that at least one run completed before the SLA deadline.
Note that this test only verifies that the model executed, not that the data is actually fresh. If you need to verify that the underlying data was updated (e.g., an upstream source refreshed), use [`data_freshness_sla`](/data-tests/data-freshness-sla) instead.
### Use Case
"Did my pipeline complete before 7 AM Pacific on the days I care about?"
### Test Logic
1. If today is not a scheduled check day → **PASS** (skip)
2. Query `dbt_run_results` for successful runs of the model today
3. If any run completed before the SLA deadline → **PASS**
4. If SLA deadline hasn't passed yet → **PASS** (still time)
5. If model ran but after deadline → **FAIL** (MISSED\_SLA)
6. If all runs failed → **FAIL** (ALL\_FAILED)
7. If model didn't run today → **FAIL** (NOT\_RUN)
### Test configuration
*Required configuration: `sla_time`, `timezone`*
data\_tests:
-- elementary.execution\_sla:
arguments:
sla\_time: string # Required - e.g., "07:00", "7am", "2:30pm", "14:30"
timezone: string # Required - IANA timezone name, e.g., "America/Los\_Angeles", "Europe/Amsterdam"
day\_of\_week: string | array # Optional - Day(s) to check: "Monday" or \["Monday", "Wednesday"]
day\_of\_month: int | array # Optional - Day(s) of month to check: 1 or \[1, 15]
```yml Models theme={null}
models:
- name: < model name >
data_tests:
- elementary.execution_sla:
arguments:
sla_time: < deadline time > # Required - e.g., "07:00", "7am", "2:30pm"
timezone: < IANA timezone > # Required - e.g., "America/Los_Angeles"
day_of_week: < day or array > # Optional
day_of_month: < day or array > # Optional
```
```yml Daily check (default) theme={null}
models:
- name: daily_revenue
data_tests:
- elementary.execution_sla:
arguments:
sla_time: "07:00"
timezone: "America/Los_Angeles"
config:
tags: ["elementary"]
severity: error
```
```yml Weekly - only Mondays and Wednesdays theme={null}
models:
- name: weekly_report
data_tests:
- elementary.execution_sla:
arguments:
sla_time: "06:00"
timezone: "Europe/London"
day_of_week: ["Monday", "Wednesday"]
config:
tags: ["elementary"]
```
```yml Monthly - only 1st and 15th theme={null}
models:
- name: monthly_close
data_tests:
- elementary.execution_sla:
arguments:
sla_time: "09:00"
timezone: "Asia/Tokyo"
day_of_month: [1, 15]
config:
tags: ["elementary"]
```
### Features
* **Flexible time formats**: Supports `"07:00"`, `"7am"`, `"2:30pm"`, `"14:30"`, and other common formats
* **IANA timezone support**: Uses standard timezone names like `"America/Los_Angeles"`, `"Europe/Amsterdam"`, etc.
* **Automatic DST handling**: Uses `pytz` for timezone conversions with automatic daylight saving time handling
* **Database-agnostic**: All timezone logic happens at compile time
* **Schedule filters**: Optional `day_of_week` and `day_of_month` parameters to check only specific days
### Parameters
| Parameter | Required | Description |
| -------------- | -------- | -------------------------------------------------------- |
| `sla_time` | Yes | Deadline time (e.g., `"07:00"`, `"7am"`, `"2:30pm"`) |
| `timezone` | Yes | IANA timezone name (e.g., `"America/Los_Angeles"`) |
| `day_of_week` | No | Day(s) to check: `"Monday"` or `["Monday", "Wednesday"]` |
| `day_of_month` | No | Day(s) of month to check: `1` or `[1, 15]` |
### Notes
* This test only works with **models**, not sources
* The test automatically skips on non-scheduled days (when `day_of_week` or `day_of_month` filters are set)
* If both `day_of_week` and `day_of_month` are set, the test uses OR logic (checks if either matches)
* The test passes if the SLA deadline hasn't been reached yet, giving your pipeline time to complete
# Elementary anomaly detection tests
Source: https://docs.elementary-data.com/data-tests/how-anomaly-detection-works
Elementary data anomaly detection tests monitor a specific metric (like row count, null rate, average value, etc.) and compare recent values to historical values.
This is done to detect [significant changes and deviations](/data-tests/data-anomaly-detection), that are probably data reliability issues.
## What happens on each test?
Upon running a test, your data is split into time buckets based on the [`time_bucket`](/data-tests/anomaly-detection-configuration/time-bucket) field and is limited by
the [`training_period`](/data-tests/anomaly-detection-configuration/training-period) var. The test then compares a certain metric (e.g. row count) of the buckets that are within the detection
period ([`detection_period`](/data-tests/anomaly-detection-configuration/detection-period)) to the row count of all the previous time buckets within the [`training_period`](/data-tests/anomaly-detection-configuration/training-period) period.
If there were any anomalies in the detection period, the test will fail.
On each test elementary package executes the relevant monitors, and searches for anomalies by comparing to historical metrics.
To learn more, refer to [core concepts](/data-tests/how-anomaly-detection-works).
## What does it mean when a test fails?
When a test fail, it means that an anomaly was detected on this metric and dataset. To learn more, refer to [anomaly detection method](/data-tests/data-anomaly-detection).
## Core concepts
### Anomaly
A value in the detection set that is an outlier comparing to the expected range calculated based on the training set.
### Monitored data set
The data set we run the data monitor against, and includes the training set values and detection set values.
### Data monitors
When we use anomaly detection tests we can monitor different metrics to detect problems - freshness, volume, nullness, uniqueness, distribution, etc.
Each different metric we collect is a ‘data monitor’.
### Training set
The set of values used as a reference point to calculate the expected range.
### Detection set
The set of values that are compared to the expected range.
If a value in the detection set is an outlier to the expected range, it will be flagged as an anomaly.
### Expected range
Based of the values in the training test, we calculate an expected range for the monitor.
Each data point in the detection period will be compared to the expected range calculated based on its training set.
### Training period
The period of time for which the training set is collected.
As data changed over time, we don’t consider the entire history of the metric, just a recent period.
### Detection period
The values in the detection period will be compared to the expected range calculated using the training set.
If a data point is outside the expected range and is part of the detection period, it is flagged as an anomaly.
### Time bucket
To calculate how data changes over time and detect issues, we split the data into consistent time buckets.
For example, if we use daily time bucket and monitor for row count anomalies, we will count new rows per day.
### Detection algorithm
Read about it in [data anomaly detection method](/data-tests/data-anomaly-detection).
## Tests configuration and core concepts
Not all data sets are the same, so Elementary offers configuration options for the anomaly detection tests.\
To detect data issues with high accuracy, it is important to leverage the configuration options.
Configuration params related directly to the test's core concepts:
**Data monitors**
* [column\_anomalies](/data-tests/anomaly-detection-configuration/column-anomalies)
**Expected range**
* [anomaly\_sensitivity](/data-tests/anomaly-detection-configuration/anomaly-sensitivity)
* [anomaly\_direction](/data-tests/anomaly-detection-configuration/anomaly-direction)
**Detection period and detection set**
* [detection\_period](/data-tests/anomaly-detection-configuration/detection-period)
**Training period and training set**
* [training\_period](/data-tests/anomaly-detection-configuration/training-period)
* [seasonality](/data-tests/anomaly-detection-configuration/seasonality)
**Time buckets**
* [timestamp\_column](/data-tests/anomaly-detection-configuration/timestamp-column)
* [time\_bucket](/data-tests/anomaly-detection-configuration/time-bucket)
**Monitored data set**
* [where\_expression](/data-tests/anomaly-detection-configuration/where-expression)
* [dimensions](/data-tests/anomaly-detection-configuration/dimensions)
# Elementary Data Tests
Source: https://docs.elementary-data.com/data-tests/introduction
Elementary provides anomaly tests for detection of data quality issues.
Elementary data tests are configured and executed like native tests in your dbt project.
Elementary tests can be used in addition to dbt tests, packages tests (such as dbt-expectations), and custom tests.
All of these test results will be presented in the Elementary UI and alerts.
The Elementary dbt package offers two test types:
* **Pipeline tests:** Monitor the health of data pipelines, ensuring timely and smooth data ingestion, transformation, and loading.
* **Data quality tests:** Validate data accuracy, completeness, and correctness, detect anomalies and schema changes, and ensure the data meets predefined business rules.
Together, these tests ensure reliable pipelines and trusted data.
In addition to the mentioned dbt package tests, the [Elementary Cloud Platform](https://docs.elementary-data.com/cloud/introduction) offers **automated pipeline tests.** While traditional tests query the dbt tables directly, automated pipeline tests analyze **query history metadata**. This method is both **faster and more cost-efficient**, as it eliminates the need to query large datasets, focusing solely on the metadata layer. Learn more about [automated tests](https://docs.elementary-data.com/features/anomaly-detection/automated-monitors).
* Volume
* Freshness
* Event freshness
* Column anomalies
* Dimensions
* Schema changes
* Baseline schema
* JSON schema
* Exposure schema
* Python tests
## Anomaly detection tests
Tests to detect anomalies in data quality metrics such as volume, freshness, null rates, and anomalies in specific dimensions.
[How Elementary anomaly detection tests
work?](/data-tests/how-anomaly-detection-works)
Monitors table row count over time to detect drops or spikes in volume.
Monitors the latest timestamp of a table to detect data delays.
Monitors the gap between the latest event timestamp and its loading time, to
detect event freshness issues.
Monitors the row count per dimension over time, and alerts on unexpected
changes in the distribution. It is best to configure it on low-cardinality
fields.
Monitors a column for anomalies in metrics such as null rate, length, max and
min, and more. Read more about [specific column
metrics](/data-tests/anomaly-detection-configuration/column-anomalies).
Activates the column anomalies test on all the columns of the table. It's
possible to exclude specific columns.
## Schema tests
Fails on changes in schema: deleted or added columns, or change of data type
of a column.
Fails if the table schema is different in columns names or column types than a
configured baseline (can be generated with a macro).
Monitors a JSON type column and fails if there are JSON events that don't
match a configured JSON schema (can be generated with a macro).
Monitors changes in your models' columns that break schema for downstream
exposures, such as BI dashboards.
## Other tests
Write your own custom tests using Python scripts.
## dbt tests outcomes
In dbt, there are three possible outcomes when running models or tests: errors, failures, and warnings.
* An **error** means dbt could not run the SQL at all (e.g., syntax mistake, missing table, broken macro). This stops execution (on `dbt build`).
* A **failure** happens when a test runs successfully but its condition isn’t met, and if the test’s severity is set to error, it will fail the pipeline.
* A **warning** is the same as a failure in terms of data quality, but with severity: warn, dbt exits successfully and does not break pipelines.
# Python tests
Source: https://docs.elementary-data.com/data-tests/python-tests
## Python in dbt
In dbt v1.3.0, Python models were introduced in order to solve use cases that you can't solve with SQL.
Using Python allows you to perform analyses using tools available in the open source Python ecosystem, including
state-of-the-art packages for data science and statistics.
Before, you would have needed separate infrastructure and orchestration to run Python.
For more information, review
the [Python models documentation](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/python-models).
## What about tests?
Just like with models, we'd also like to be able to test our data using those powerful tools and the convenient Python
runtime.
Certain problems such as validating JSONs or comparing tables become tremendously easier to solve.
Currently, dbt doesn't provide an out-of-the-box mechanism to run tests using Python. Elementary now provides the
ability to do so using dbt's building blocks.
## Elementary Python tests
*This is a beta feature, introduced in version 0.5.3 of Elementary's dbt package.*
Elementary introduces a generic dbt test called `elementary.python` that allows users to run Python code and validate
their data.
### How does it work?
Let's start with a basic example and proceed gradually to show the full potential of Python tests.
A Python test is defined like any other dbt test.
```yaml models/schema.yml theme={null}
- name: orders
data_tests:
- elementary.python:
arguments:
code_macro: check_undelivered_orders
```
Then, we need to define a macro under `macros/.sql` that contains the Python code the test will execute.
```python macros/py_check_undelivered_orders.sql theme={null}
{% macro check_undelivered_orders(args) %}
def test(model_df, ref, session):
return model_df.filter(model_df['ORDER_STATUS'] != 'delivered')
{% endmacro %}
```
As you can see, the macro's name is equivalent to the `code_macro` argument that is passed to the test.
The macro needs to contain a `def test(model_df, ref, session)` function with the test's logic.
A test function receives the following parameters:
1. `model_df`: The model's `DataFrame` object.
2. `ref`: A function that allows you to ref to other resources - `ref('my_other_model')`.
3. `session`: A `Session` object,
either [Snowpark](https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/_autosummary/snowflake.snowpark.html#snowflake.snowpark.Session)
or [PySpark](https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.SparkSession.html).
In practice, you'll most likely only need the `model_df` when testing a model.
We'll show examples below for the usage of the other parameters.
You can return either of the following data types:
* `DataFrame`: Either a Spark or a [Pandas](https://pandas.pydata.org/) `DataFrame`.
The test's failed results count will be the amount of rows in the `DataFrame`.
* `int` (number): The amount of failed results.
* `bool`: Whether the test passed (`True`) or failed (`False`).
In the example above, the test filters out rows that their `ORDER_STATUS` column is not `delivered` and returns the
new `DataFrame`. Let's move on to demonstrate Python's true power when it comes to testing.
## Example use cases
### Comparing tables
Let's compare two different tables, or views, within our warehouse.
```yaml models/schema.yml theme={null}
- name: orders
data_tests:
- elementary.python:
arguments:
code_macro: compare_tables
macro_args:
other_table: raw_orders
```
We're passing an additional argument to the test called `macro_args`.
This argument allows us to make our tests generic,
meaning that we can simply change the `other_table` field and the compared table will be different without ever touching
the Python code.
Here's the actual implementation.
```python macros/py_compare_tables.sql theme={null}
{% macro compare_tables(args) %}
import pandas as pd
def test(model_df, ref, session):
original_table = model_df.toPandas().drop_duplicates()
other_table = ref('{{ args.other_table }}').toPandas().drop_duplicates()
return pd.concat([original_table, other_table]).drop_duplicates(keep=False)
{% endmacro %}
```
Note how the `macro` *itself* receives an `args` which is the `macro_args` that we passed in the test's definition.
We're using `ref('{{ args.other_table }}')` in order to get the other table, just like we would with SQL.
Afterwards, we return a new Pandas `DataFrame` that holds the difference between the two tables.
If the `DataFrame` is not empty, the test will fail with the amount of failed results equal to the amount of rows.
### Validating JSONs
In this example we'll validate a JSON column according to a pre-defined schema.
```yaml models/schema.yml theme={null}
- name: login_events
data_tests:
- elementary.python:
arguments:
code_macro: validate_json
macro_args:
schema: "{'type': 'object', 'properties': {'country': {'type': 'string'}}}"
column: geolocation_json
packages: ["jsonschema"]
```
Here we'll be testing that the `country` that is provided in the `geolocation_json` column in the `login_events`
model is a `string`.
```python macros/py_validate_json.sql theme={null}
{% macro validate_json(args) %}
import json
import jsonschema
def test(model_df, ref, session):
model_df = model_df.toPandas()
invalid_row_count = 0
for json_row in model_df['{{ args.column }}']:
try:
jsonschema.validate(json.loads(json_row), {{ args.schema }})
except (json.JSONDecodeError, jsonschema.ValidationError):
invalid_row_count += 1
return invalid_row_count
{% endmacro %}
```
In the test's definition, we're passing a `packages` argument which lists additional PyPI Python packages that are
required for the code to work.
Here, we've added `jsonschema` to our requirements. Using the `macro_args`, we're specifying the JSON schema that we
want to test against, and the column name.
In our `validate_json` macro, we'll do the formatting accordingly to our Python code.
The code itself iterates over the column's rows, loads the rows into a JSON and validates them.
If it failed to either load the row into a JSON or that it was out of schema, it'll mark the row as invalid.
At the end, we return the amount of invalid rows. IN this example we're returning an `int` compared to the other in
which we returned a `DataFrame`.
## API Reference
* `elementary.python` dbt test:
* `code_macro: str`: Name of the macro that returns the Python code.
* `macro_args: dict`: Arguments that are supplied to the macro - `{{ code_macro(macro_args) }}`.
* `where_expression: str`: SQL WHERE clause to limit the test scope.
* All available configuration in Python models such as:
* `packages: list[str]`: PyPI package requirements.
* `submission_method: str`: `cluster` or `serverless` for BigQuery.
* ...
* `def test(model_df, ref, session)` Python function:
* Arguments
* `model_df: DataFrame`: The model's `DataFrame` object.
* `ref: Function`: A function that allows you to ref to other resources - `ref('my_other_model')`.
* `session: Session`: A `Session` object,
either [Snowpark](https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/_autosummary/snowflake.snowpark.html#snowflake.snowpark.Session)
or [PySpark](https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.SparkSession.html).
* Returns
* `DataFrame`: Either a Spark or a [Pandas](https://pandas.pydata.org/) `DataFrame`.
The test's failed results count will be the amount of rows in the `DataFrame`.
* `int`: The amount of failed results.
* `bool`: Whether the test passed (`True`) or failed (`False`).
## Setup
Additional setup might be required depending on your warehouse.
Given that Elementary's Python tests base on Python models' building blocks, the requirements are the same and can be
found [here](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/python-models#specific-data-warehouses)
.
## Beta feature
Running Python within the warehouse is a new feature in dbt and even newer in Elementary.
Therefore, there might be some bumps along the road.
Please contact us in any of the mediums you find fit, and we'll be glad to help.
# Exposure validation test
Source: https://docs.elementary-data.com/data-tests/schema-tests/exposure-tests
Elementary dbt package includes **exposure validation tests, implemented as [dbt tests](https://docs.getdbt.com/docs/building-a-dbt-project/tests)**.
These tests can detect changes in your models' columns that break downstream exposures, such as BI dashboards.
## Configure your elementary exposure validation tests
Within your `models` directory, add a file called `exposures.yml`
```yml theme={null}
version: 2
exposures:
- name: customers
label: Customer Dashboard
type: dashboard
maturity: high
url: https://your.bi.tool/dashboards/1
description: >
Shows customer growth
depends_on:
- ref('customers')
owner:
name: Callum McData
email: data@jaffleshop.com
```
You can read up about exposures and how to add them at [dbt
documentation](https://docs.getdbt.com/docs/build/exposures)
### Adding the Elementary exposure information
For each exposure you wish to verify, add a new property called `meta`:
```yml theme={null}
version: 2
exposures:
- name: customers
label: Customer Dashboard
type: dashboard
maturity: high
url: https://your.bi.tool/dashboards/1
description: >
Shows customer growth
depends_on:
- ref('customers')
owner:
name: Callum McData
email: data@jaffleshop.com
meta:
referenced_columns:
- column_name: "customer_id"
```
In Elementary Cloud, the exposures and exposure validation tests are added
automatically, leveraging an integration with your BI tool.
You can optionally also specify the column data type (`data_type`)
```yml theme={null}
version: 2
- name: returned_orders
label: Returned Orders
type: dashboard
maturity: high
url: https://your.bi.tool/dashboards/2
description: >
Returned orders over time
depends_on:
- ref('returned_orders')
owner:
name: Callum McData
email: data@jaffleshop.com
meta:
referenced_columns:
- column_name: "order_id"
data_type: "numeric"
```
In addition, if your exposure depends on several nodes, you can depend explicitly per column which source should be tested:
```yml theme={null}
version: 2
- name: returned_orders
label: Returned Orders
type: dashboard
maturity: high
url: https://your.bi.tool/dashboards/2
description: >
Returned orders over time
depends_on:
- ref('returned_orders')
- ref('orders')
owner:
name: Callum McData
email: data@jaffleshop.com
meta:
referenced_columns:
- column_name: "order_id"
data_type: "numeric"
node: ref('returned_orders')
- column_name: "customer_id"
data_type: "numeric"
node: ref('orders')
```
### Adding the Elementary exposure tests for your module
For each module schema you wish to verify the exposure dependencies, add the elementary exposure tests:
```yml theme={null}
...
- name: returned_orders
description: This table contains all of the returned orders
config:
tags: ["finance"]
data_tests:
- elementary.volume_anomalies:
config:
tags: ["table_anomalies"]
arguments:
timestamp_column: "order_date"
- elementary.exposure_schema_validity:
config:
tags: [elementary]
```
We recommend adding a tag to the tests so you can execute these in a dedicated run using the selection parameter `--select tag:elementary`.
Upon running the tests, if breaking changes are detected in the model, the test will fail and in the test result query you'll be able to see the reasons:
```sql theme={null}
SELECT 'customers' as exposure, 'https://your.bi.tool/dashboards/1' as url, 'different data type for the column customer_id numeric vs numeric' as error
UNION ALL SELECT 'customers' as exposure, 'https://your.bi.tool/dashboards/1' as url, 'order_id column missing in the model' as error
```
## What does it mean when a test fails?
When a test fails, it means that an exposure is potentially broken due to a missing or wrongly-typed column. Open your BI tool to validate, and if you make any changes to your dashboards be sure to update the `exposures.yml` and your model schema accordingly.
# JSON schema
Source: https://docs.elementary-data.com/data-tests/schema-tests/json-schema
`elementary.json_schema`
Allows validating that a string column matches a given JSON schema. The test expects a JSON schema as input,
defined according to the [JSON schema standard](https://json-schema.org/), defined in YAML format (see an example below).
This test along with the relevant JSON schema can be auto-generated (see details below).
This test relies on our [Python tests capability](/data-tests/python-tests), and is currently only supported for
Snowflake and BigQuery data warehouses.
### Auto-generate JSON schema tests
Elementary provides the `generate_json_schema_test` macro in order to auto-generate the JSON schema for a given
column using existing data.
Example usage:
```shell theme={null}
dbt run-operation elementary.generate_json_schema_test --args '{"node_name": "login_events", "column_name": "raw_event_data"}'
```
Will print:
```shell theme={null}
Please add the following test to your model configuration:
----------------------------------------------------------
columns:
- name: raw_event_data
data_tests:
- elementary.json_schema:
arguments:
type: object
properties:
event_id:
type: integer
event_name:
type: string
event_args:
type: array
items:
type: string
required:
- event_id
- event_name
```
*Note: The `generate_json_schema_test` macro relies on a 3rd-party python library called `genson`. If you are using
BigQuery, you will need to pre-install this library in your Dataproc cluster (See [dbt's documentation on Python models](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/python-models#specific-data-platforms)
for more details)*
```yml Models theme={null}
version: 2
models:
- name: < model name >
columns:
- name: < column name >
data_tests:
- elementary.json_schema:
arguments:
```
```yml Models example theme={null}
version: 2
models:
- name: login_events
columns:
- name: raw_event_data
data_tests:
- elementary.json_schema:
arguments:
type: object
properties:
event_id:
type: integer
event_name:
type: string
event_args:
type: array
items:
type: string
required:
- event_id
- event_name
```
# Schema changes
Source: https://docs.elementary-data.com/data-tests/schema-tests/schema-changes
`elementary.schema_changes`
Executes schema changes monitor that alerts on deleted table, deleted or added columns, or change of data type of a column.
Data Types note: Snowflake, Redshift and Postgres all has "synonym" data types, like `VARCHAR` and `TEXT` in Snowflake. These will be considered as equals by the tests. Types are converted internally to deal with this.
Supported in Databricks with Unity Catalog only.
Not supported in Athena or Trino.
```yml Models theme={null}
version: 2
models:
- name: < model name >
data_tests:
- elementary.schema_changes
```
```yml Models example theme={null}
version: 2
models:
- name: login_events
data_tests:
- elementary.schema_changes:
config:
tags: ["elementary"]
severity: warn
```
# Schema changes from baseline
Source: https://docs.elementary-data.com/data-tests/schema-tests/schema-changes-from-baseline
`elementary.schema_changes_from_baseline`
Checks for schema changes against baseline columns defined in a source's or model's configuration.
For this test to work, the configuration should contain columns and data types.
The initial configuration needed for this test can be auto-generated (see details below).
Supported parameters for the test:
* `fail_on_added` - If set, the test will fail if there are columns in the table that do not exist in the baseline (default:
False - meaning added columns won't cause the test to fail).
* `enforce_types` - If set, the test will raise an error if there are columns that are defined without a data type (default:
False - in this case the test will not fail, but instead will only verify that the column exists and not its type)
Supported in Databricks with Unity Catalog only.
Not supported in Athena or Trino.
### Auto-generate baseline schema
To make it easier to configure schema tests, Elementary provides dbt operations to auto generate tests configuration based on the existing schemas.
In order for the schema changes from baseline test to work, a baseline needs to be generated from an initial state of
the table and it should be added to the configuration of the source / model under "columns". The baseline consists
of the name and data type for each column.
In order to generate the baseline, Elementary provides the `generate_schema_baseline_test` macro. By default,
running it will generate a schema\_changes\_from\_baseline test for all sources, but it can be customized with
the following arguments:
* `name` - run on a specific source / model
* `include_models` - whether or not to generate tests for models (default - false)
* `include_sources` - whether or not to generate tests for sources (default - true)
* `fail_on_added` - if set, the "fail\_on\_added" parameter will be added to the configuration of the tests with the supplied setting
* `enforce_types` - if set, the "enforce\_types" parameter will be added to the configuration of the tests with the supplied setting
Examples:
```shell theme={null}
# Generate a schema changes from baseline test for all sources
dbt run-operation elementary.generate_schema_baseline_test
# Generate a schema changes from baseline test for a specific model / source named "orders"
dbt run-operation elementary.generate_schema_baseline_test --args '{"name": "orders"}'
# Generate a schema changes from baseline test for all sources and all models
dbt run-operation elementary.generate_schema_baseline_test --args '{"include_models": true}'
# Generate a schema changes from baseline test with the "fail_on_added" and "enforce_types" parameters set to true
dbt run-operation elementary.generate_schema_baseline_test --args '{"fail_on_added": true, "enforce_types": true}'
# Output:
#> Generating schema changes from baseline test for ...:
#>
#> sources:
#> - name:
#> columns:
#> ...
#> data_tests:
#> - elementary.schema_changes_from_baseline:
#> ...
```
This command will output the generated configuration for the schema changes from baseline test, which can be copied and pasted into the relevant `yml` file.
```yml Sources theme={null}
version: 2
sources:
- name: < source name >
database: < database name >
schema: < schema name >
tables:
- name: < table name >
columns:
- name: < column 1 >
data_type: < data type 1 >
- name: < column 2 >
data_type: < data type 2 >
data_tests:
- elementary.schema_changes_from_baseline
```
```yml Sources example theme={null}
version: 2
sources:
- name: 'my_non_dbt_data'
database: 'raw_events'
schema: 'product'
tables:
- name: 'raw_product_login_events'
columns:
- name: event_name
data_type: text
- name: event_id
data_type: integer
data_tests:
- elementary.schema_changes_from_baseline
config:
tags: ["elementary"]
```
```yml Models theme={null}
version: 2
models:
- name: < model name >
columns:
- name: < column 1 >
data_type: < data type 1 >
- name: < column 2 >
data_type: < data type 1 >
data_tests:
- elementary.schema_changes_from_baseline
```
```yml Models example theme={null}
version: 2
models:
- name: login_events
columns:
- name: event_name
data_type: text
- name: event_id
data_type: integer
data_tests:
- elementary.schema_changes_from_baseline:
config:
tags: ["elementary"]
```
# Test Result Samples
Source: https://docs.elementary-data.com/data-tests/test-result-samples
When a test fails, Elementary captures a sample of the failing rows and stores them in the `test_result_rows` table. These samples help you quickly understand and investigate data issues without manually running queries.
By default, Elementary saves **5 sample rows per failed test**.
This page describes all the available controls for managing test result samples -- both self-service configuration in your dbt project and options available through the Elementary team for Cloud users.
## Configuring sample size
### Global setting
Set the number of sample rows saved per failed test across your entire project by adding the `test_sample_row_count` variable to your `dbt_project.yml`:
```yaml theme={null}
vars:
test_sample_row_count: 10
```
Or pass it as a flag when running dbt:
```shell theme={null}
dbt test --vars '{"test_sample_row_count": 10}'
```
Set to `0` to disable sample collection entirely:
```yaml theme={null}
vars:
test_sample_row_count: 0
```
The larger the number of rows you save, the more data you will store in your data warehouse. This can affect the performance and cost of your Elementary schema, depending on your database.
### Per-test override
You can override the global sample size for individual tests using the `test_sample_row_count` meta configuration:
```yaml theme={null}
models:
- name: orders
data_tests:
- unique:
config:
meta:
test_sample_row_count: 20 # Save more samples for this specific test
- not_null:
column_name: order_id
config:
meta:
test_sample_row_count: 0 # Disable samples for this test
```
The per-test setting takes precedence over the global variable.
## Disabling samples for specific tests
Use the `disable_test_samples` meta configuration to completely disable sample collection for a specific test:
```yaml theme={null}
models:
- name: user_profiles
data_tests:
- elementary.volume_anomalies:
config:
meta:
disable_test_samples: true
```
## PII protection
Elementary provides built-in protection for sensitive data by automatically disabling test sample collection when a PII tag is detected.
### Enable PII protection
Add these variables to your `dbt_project.yml`:
```yaml theme={null}
vars:
disable_samples_on_pii_tags: true # Enable PII protection (default: false)
pii_tags: ['pii', 'sensitive'] # Tags that identify PII data (default: ['pii'])
```
PII tag matching is **case-insensitive** -- `PII`, `pii`, and `Pii` are all equivalent.
### Tag levels
PII tags are evaluated at three levels:
**Model level** -- disables samples for all tests on that model:
```yaml theme={null}
models:
- name: customer_data
config:
tags: ['pii']
```
**Column level** -- disables samples for tests that reference that column:
```yaml theme={null}
models:
- name: customer_data
columns:
- name: email
tags: ['pii']
```
**Test level** -- disables samples for that specific test:
```yaml theme={null}
models:
- name: customer_data
data_tests:
- elementary.volume_anomalies:
config:
tags: ['pii']
```
You can also tag entire directories in `dbt_project.yml`:
```yaml theme={null}
models:
my_project:
sensitive_data:
+tags: ['pii']
```
## Showing samples selectively with `show_sample_rows`
`show_sample_rows` is the inverse of PII protection. When enabled, **all samples are hidden by default** and only shown for models, columns, or tests that are explicitly tagged.
### Enable selective sample display
```yaml theme={null}
vars:
enable_samples_on_show_sample_rows_tags: true # Hide all samples by default (default: false)
```
By default, the tag `show_sample_rows` is used to opt in. You can customize this with the `show_sample_rows_tags` variable -- just like `pii_tags` lets you customize which tags trigger PII protection:
```yaml theme={null}
vars:
enable_samples_on_show_sample_rows_tags: true
show_sample_rows_tags: ['show_sample_rows', 'debug'] # default: ['show_sample_rows']
```
Tag matching is **case-insensitive**.
### Tag levels
Works at the same three levels as PII:
**Model level** -- shows samples for all tests on that model:
```yaml theme={null}
models:
- name: orders
config:
tags: ['show_sample_rows']
```
**Column level** -- shows samples for tests that target that column:
```yaml theme={null}
models:
- name: orders
columns:
- name: status
tags: ['show_sample_rows']
```
**Test level** -- shows samples for that specific test:
```yaml theme={null}
models:
- name: orders
data_tests:
- elementary.volume_anomalies:
config:
tags: ['show_sample_rows']
```
## Behavior matrix
### `enable_samples_on_show_sample_rows_tags: true`
| Level | Tags | Samples shown? |
| ------ | ------------------ | ---------------------- |
| model | none | No (hidden by default) |
| model | `show_sample_rows` | Yes |
| model | `pii` | No (hidden by default) |
| test | `show_sample_rows` | Yes |
| column | `show_sample_rows` | Yes |
### `disable_samples_on_pii_tags: true`
| Level | Tags | Samples shown? |
| ------ | --------------------------- | ------------------------------- |
| model | none | Yes |
| model | `pii` | No |
| test | `pii` | No |
| column | `pii` | No |
| column | `show_sample_rows` (no pii) | Yes (excluded from PII columns) |
## Configuration precedence
When multiple settings apply, Elementary follows this order (highest priority first):
1. **`disable_test_samples` in test meta** -- per-test on/off switch
2. **`show_sample_rows` tag** -- when `enable_samples_on_show_sample_rows_tags: true` and the model, column, or test has a matching tag
3. **`enable_samples_on_show_sample_rows_tags`** -- hides all samples by default when enabled
4. **PII tag detection** -- when `disable_samples_on_pii_tags: true` and the model, column, or test has a matching tag
5. **`test_sample_row_count` global var** -- project-wide sample size
6. **Default** -- 5 rows
## Elementary Cloud: additional controls
For Elementary Cloud users, there are additional environment-level controls that can be enabled by the Elementary team.
The controls below are managed by Elementary and apply to how test samples are handled after they are synced from your data warehouse. To request changes, contact the Elementary team via Slack or email.
### Disable test samples for an environment
The Elementary team can disable test samples entirely for a specific environment. When enabled:
* Test samples will **not be synced** from your Elementary schema.
* Test samples will **not appear** in the UI or in alerts, even if they exist in your warehouse.
This is useful for environments that contain highly sensitive data where no sample rows should ever leave the warehouse.
### Skip database storage of sample rows
The Elementary team can configure an environment so that the `test_result_rows` data is stored only in the data lake (S3) and **not loaded into the application database**. This reduces database size while keeping the raw data available for debugging if needed.
## Summary of all controls
| Control | Scope | Where to configure | Default |
| ----------------------------------------- | --------------- | ----------------------- | ---------------------- |
| `test_sample_row_count` | Global | `dbt_project.yml` vars | `5` |
| `test_sample_row_count` | Per-test | Test meta | Inherits global |
| `disable_test_samples` | Per-test | Test meta | `false` |
| `disable_samples_on_pii_tags` | Global | `dbt_project.yml` vars | `false` |
| `pii_tags` | Global | `dbt_project.yml` vars | `['pii']` |
| `enable_samples_on_show_sample_rows_tags` | Global | `dbt_project.yml` vars | `false` |
| `show_sample_rows_tags` | Global | `dbt_project.yml` vars | `['show_sample_rows']` |
| Disable samples for environment | Per-environment | Contact Elementary team | Disabled |
| Skip DB storage of sample rows | Per-environment | Contact Elementary team | Disabled |
# volume_threshold
Source: https://docs.elementary-data.com/data-tests/volume-threshold
Let our Slack chatbot create the anomaly test you need.
`elementary.volume_threshold`
Monitors row count changes between time buckets using configurable percentage thresholds with multiple severity levels.
Unlike `volume_anomalies` (which uses z-score based anomaly detection as a dbt test, or ML-based detection in Elementary Cloud), this test lets you define explicit percentage thresholds for warnings and errors, giving you precise control over when to be alerted. It uses Elementary's metric caching infrastructure to avoid recalculating row counts for buckets that have already been computed.
### Use Case
"Alert me if my table's row count drops or spikes by more than 10% compared to the previous period."
### Test Logic
1. Collect row count metrics per time bucket (using Elementary's incremental metric caching)
2. Compare the most recent completed bucket against the previous bucket
3. Calculate the percentage change between the two
4. If the previous bucket has fewer rows than `min_row_count` → **PASS** (insufficient baseline)
5. If the absolute change exceeds `error_threshold_percent` → **ERROR**
6. If the absolute change exceeds `warn_threshold_percent` → **WARN**
7. Otherwise → **PASS**
### Test configuration
*Required configuration: `timestamp_column`*
data\_tests:
-- elementary.volume\_threshold:
arguments:
timestamp\_column: column name # Required
warn\_threshold\_percent: int # Optional - default: 5
error\_threshold\_percent: int # Optional - default: 10
direction: \[both | spike | drop] # Optional - default: both
time\_bucket: # Optional
period: \[hour | day | week | month]count: intwhere\_expression: sql expression # Optional
days\_back: int # Optional - default: 14
backfill\_days: int # Optional - default: 2
min\_row\_count: int # Optional - default: 100
```yml Models theme={null}
models:
- name: < model name >
data_tests:
- elementary.volume_threshold:
arguments:
timestamp_column: < column name > # Required
warn_threshold_percent: < int > # Optional - default: 5
error_threshold_percent: < int > # Optional - default: 10
direction: < both | spike | drop > # Optional - default: both
```
```yml Default thresholds (5% warn, 10% error) theme={null}
models:
- name: daily_orders
data_tests:
- elementary.volume_threshold:
arguments:
timestamp_column: created_at
config:
tags: ["elementary"]
```
```yml Custom thresholds - drop only theme={null}
models:
- name: critical_transactions
data_tests:
- elementary.volume_threshold:
arguments:
timestamp_column: transaction_time
warn_threshold_percent: 3
error_threshold_percent: 8
direction: drop
config:
tags: ["elementary"]
```
```yml With time bucket and filter theme={null}
models:
- name: hourly_events
data_tests:
- elementary.volume_threshold:
arguments:
timestamp_column: event_timestamp
warn_threshold_percent: 10
error_threshold_percent: 25
direction: both
time_bucket:
period: hour
count: 1
where_expression: "event_type = 'purchase'"
config:
tags: ["elementary"]
```
### Features
* **Dual severity levels**: Separate thresholds for warnings and errors, giving you graduated alerting
* **Directional monitoring**: Choose to monitor `both` directions, only `spike` (increases), or only `drop` (decreases)
* **Incremental metric caching**: Uses Elementary's `data_monitoring_metrics` table to avoid recalculating row counts for previously computed time buckets
* **Minimum baseline protection**: The `min_row_count` parameter prevents false alerts when the baseline is too small
* **Configurable time buckets**: Works with hourly, daily, weekly, or monthly buckets
### Parameters
| Parameter | Required | Default | Description |
| ------------------------- | -------- | ------------------------- | ----------------------------------------------------------------- |
| `timestamp_column` | Yes | - | Column to determine time periods |
| `warn_threshold_percent` | No | 5 | Percentage change that triggers a warning |
| `error_threshold_percent` | No | 10 | Percentage change that triggers an error |
| `direction` | No | `both` | Direction to monitor: `both`, `spike`, or `drop` |
| `time_bucket` | No | `{period: day, count: 1}` | Time bucket configuration |
| `where_expression` | No | - | SQL expression to filter the data |
| `days_back` | No | 14 | Days of metric history to retain |
| `backfill_days` | No | 2 | Days to recalculate on each run |
| `min_row_count` | No | 100 | Minimum rows in the previous bucket required to trigger the check |
### Comparison with volume\_anomalies
| Feature | `volume_threshold` | `volume_anomalies` |
| ---------------- | --------------------------- | ------------------------------- |
| Detection method | Fixed percentage thresholds | Z-score (dbt test) / ML (Cloud) |
| Severity levels | Dual (warn + error) | Single (pass/fail) |
| Best for | Known acceptable ranges | Unknown/variable patterns |
| Configuration | Explicit thresholds | Sensitivity tuning |
| Baseline | Previous bucket | Training period average |
### How severity levels work
This test has built-in dual severity using dbt's `warn_if` / `error_if` config. You do **not** need to set `config.severity` yourself. The behavior is:
* Change exceeds `warn_threshold_percent` but not `error_threshold_percent` → **dbt warning**
* Change exceeds `error_threshold_percent` → **dbt error** (test fails)
* Change is below `warn_threshold_percent` → **pass**
For example, with `warn_threshold_percent: 3` and `error_threshold_percent: 8`:
* A 2% drop → pass
* A 5% drop → warning
* A 10% drop → error
Do not set `config.severity: error` on this test. That would override the built-in dual severity and turn all warnings into errors, defeating the purpose of having separate thresholds.
### Notes
* The `warn_threshold_percent` must be less than or equal to `error_threshold_percent`
* The test uses Elementary's metric caching infrastructure. Row counts for previously computed time buckets are reused across runs
* If the previous bucket has fewer rows than `min_row_count`, the test passes (insufficient data for a meaningful comparison)
* The test only evaluates completed time buckets
# Tests with context
Source: https://docs.elementary-data.com/data-tests/with-context-tests
A set of generic tests that extend common dbt, dbt-utils, and dbt-expectations tests with a `context_columns` parameter.
When a test fails, the failing rows are returned together with the columns you care about — making it much easier to investigate the root cause directly from the test results.
If `context_columns` is omitted, **all columns** are returned alongside failing rows.
If a column listed in `context_columns` does not exist on the model, a warning
is logged and that column is skipped. The test continues and will not error.
***
## not\_null\_with\_context
`elementary.not_null_with_context`
Validates that there are no null values in a column. Extends dbt's built-in `not_null` test.
### Parameters
| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ----------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test for null values. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |
```yml With context columns theme={null}
models:
- name: orders
columns:
- name: order_id
data_tests:
- elementary.not_null_with_context:
context_columns: [customer_id, order_date, amount]
```
```yml Return all columns theme={null}
models:
- name: orders
columns:
- name: order_id
data_tests:
- elementary.not_null_with_context
```
***
## accepted\_range\_with\_context
`elementary.accepted_range_with_context`
Validates that column values fall within an accepted range. Extends `dbt_utils.accepted_range`.
### Parameters
| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ----------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test. |
| `min_value` | No\* | `none` | Minimum accepted value (inclusive by default). At least one bound must be provided. |
| `max_value` | No\* | `none` | Maximum accepted value (inclusive by default). At least one bound must be provided. |
| `inclusive` | No | `true` | Whether the bounds are inclusive. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |
\* At least one of `min_value` or `max_value` must be provided.
```yml With context columns theme={null}
models:
- name: orders
columns:
- name: amount
data_tests:
- elementary.accepted_range_with_context:
min_value: 0
max_value: 10000
context_columns: [order_id, customer_id, order_date]
```
```yml Min bound only theme={null}
models:
- name: orders
columns:
- name: amount
data_tests:
- elementary.accepted_range_with_context:
min_value: 0
inclusive: false
```
***
## expect\_column\_values\_to\_not\_be\_null\_with\_context
`elementary.expect_column_values_to_not_be_null_with_context`
Expects column values to not be null. Extends `dbt_expectations.expect_column_values_to_not_be_null`.
### Parameters
| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ----------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test for null values. |
| `row_condition` | No | `none` | Optional SQL filter applied before testing (e.g. `"status = 'active'"`). |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |
```yml With row condition and context columns theme={null}
models:
- name: subscriptions
columns:
- name: end_date
data_tests:
- elementary.expect_column_values_to_not_be_null_with_context:
row_condition: "status = 'active'"
context_columns: [subscription_id, customer_id, start_date]
```
***
## expect\_column\_values\_to\_be\_unique\_with\_context
`elementary.expect_column_values_to_be_unique_with_context`
Expects column values to be unique. Returns all duplicate rows (not just a count), so you can see the full context of each duplicate. Extends `dbt_expectations.expect_column_values_to_be_unique`.
### Parameters
| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ----------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test for uniqueness. |
| `row_condition` | No | `none` | Optional SQL filter applied before testing. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |
```yml With context columns theme={null}
models:
- name: customers
columns:
- name: email
data_tests:
- elementary.expect_column_values_to_be_unique_with_context:
context_columns: [customer_id, created_at, name]
```
***
## expect\_column\_values\_to\_match\_regex\_with\_context
`elementary.expect_column_values_to_match_regex_with_context`
Expects column values to match a given regular expression. Extends `dbt_expectations.expect_column_values_to_match_regex`.
Requires `dbt_expectations` to be installed in your project.
### Parameters
| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ----------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test. |
| `regex` | Yes | — | The regular expression pattern to match. |
| `row_condition` | No | `none` | Optional SQL filter applied before testing. |
| `is_raw` | No | `false` | Whether the regex is a raw string. |
| `flags` | No | `""` | Optional regex flags (adapter-dependent). |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |
```yml Email format validation with context theme={null}
models:
- name: customers
columns:
- name: email
data_tests:
- elementary.expect_column_values_to_match_regex_with_context:
regex: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
context_columns: [customer_id, name, created_at]
```
***
## relationships\_with\_context
`elementary.relationships_with_context`
Validates referential integrity between a child and parent table. Extends dbt's built-in `relationships` test.
### Parameters
| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The foreign key column in the child model. |
| `to` | Yes | — | The parent model (use `ref()` or `source()`). |
| `field` | Yes | — | The column in the parent model to join to. |
| `context_columns` | No | `none` | List of additional columns from the child model to return with failing rows. Omit to return all columns. |
```yml With context columns theme={null}
models:
- name: orders
columns:
- name: customer_id
data_tests:
- elementary.relationships_with_context:
to: ref('customers')
field: id
context_columns: [order_id, order_date, amount]
```
```yml Return all columns theme={null}
models:
- name: orders
columns:
- name: customer_id
data_tests:
- elementary.relationships_with_context:
to: ref('customers')
field: id
```
# Home
Source: https://docs.elementary-data.com/home
The Data & AI
Control Plane
Elementary is built for and trusted by
1000+ engineering and analytics teams.
Whether you need a lightweight start or an enterprise-ready platform, Elementary has a solution for you.
Featured guides
Learn from the Elementary team and thousands of users from our community how to set up
effective, scalable data observability.
Join the community
Join the Elementary Community for AI and team support, explore best practices, and stay up to
date on OSS and Cloud. Connect with other data professionals and help shape what's next.
# CLI commands and usage
Source: https://docs.elementary-data.com/oss/cli-commands
With any of the CLI commands, you can use the `--help` flag and get instructions and available options:
```shell theme={null}
edr --help
```
## CLI commands
Read from the test results table and [send new alerts](/oss/guides/alerts/alerts-configuration):
```shell theme={null}
edr monitor
```
Read from the test results table and generate the Elementary report:
```shell theme={null}
edr report
```
Read from the test results table and generate the Elementary report and send it to external platforms such as Slack, S3, GCS:
```shell theme={null}
edr send-report
```
## CLI Options
You can define options either via the CLI or using environment variables in the following
format: `EDR__