Data Tests Overview

Data tests are useful for validating and enforcing explicit expectations on your data.

Elementary enables data validation and result tracking by leveraging dbt tests and dbt packages such as dbt-utils, dbt-expectations, and Elementary. This rich ecosystem of tests covers various use cases, and is widely adopted as a standard for data validations. Any custom dbt generic or singular test you develop will also be included.
Additionally, users can create custom SQL tests in Elementary, and add tests in bulk from the UI.

The combination of dbt tests, Elementary monitors, custom SQL tests and the rich dbt testing ecosystem provides the ability to achieve wide and comprehensive coverage.

​

Supported data tests

​

Test configuration

One of the design principals in Elementary is that users should manage configuration in code. This enables to maintain the same workflow for building the pipeline and configuring coverage, it makes observability and governance part of the development cycle, and provides control in review process and version management.

However, adding many tests in code is tedious, and configuration in code isn’t usable for everyone.

In Elementary, we designed a flow to incorporate the good of both worlds:

• Configuration in code or in UI - The UI test configuration flow opens pull requests to the code base through the code repository integration.
• The code is the single source of truth - As configuration from UI goes to code, the code remains the place where configuration is managed and maintained.

​

Create new tests from the UI

​

Edit existing tests from the UI

Tests can be edited from the UI, and the changes will create a pull request in the code repository. The pull request will be reviewed and merged by the team, and the changes will be applied to the tests after dbt pipeline is run and Elementary is synced. Simply open the test side panel from the test results / incidents page, and navigate to the Configuration tab. Then Click “Edit test” and make the necessary changes.

A pull request will be opened in the code repository, and a link to the PR will be provided in the UI.

​

Benefits of leveraging dbt tests

dbt tests are very powerful. The ease of use, simplicity, and usefulness in the dev process is unmatched. When you adopt any observability tool, you will still use dbt tests. This is why in Elementary, dbt tests are first class citizens.

There are several benefits to this approach:

• Single interface for all observability - Prevent the distribution of monitoring between different tools. All configuration is in code, all the results are in one interface.
• Avoid duplicate work and vendor lock in - The tests you implemented already are effective in Elementary, as well as additional configuration. The future tests you add will remain in your code if you decide to offboard.
• Control of schedule and cost - You have control of configuration and scheduling, tests can be executed when data is actually loaded and validation is needed.
• Prevent bad data from propagating - As tests are in pipeline, you can leverage dbt build and fail the pipeline on critical test failures.
• Rich ecosystem - The community of dbt users developes and supports various testing use cases.

​

dbt Test Hub

To help you find the test that is right for your use case, we created the dbt Test Hub. It's a searchable catalog of all the tests supported in Elementary, with their descriptions and example use cases.

The tests are also segmented to use cases, so you can easily find the different options for addressing your detection use case.