To start using Elementary to monitor you dbt tests, executions and data, you need to add our dbt package to your dbt project.
A dbt package is additional Jinja and SQL code that is added to your project, for additional functionality. In fact, each package is a dbt project. By adding a package to your project, you are adding the package code to be part of your project, you can reference it's macros, execute it's models, and so on.
Add packages to your project by creating a ‘packages.yml’ file under the main project directory (where your ‘dbt_project.yml is), and adding the relevant package. After you add a new package, run dbt deps to actually pull it’s code to your project. This is also how you update packages.
Some packages we recommend you check out: dbt_utils, dbt_date, codegen.
Add the following to your packages.yml file (if missing, create it where dbt_project.yml is):
packages.yml
Copied
packages:-package: elementary-data/elementary
version: 0.4.8
## compatible with Elementary CLI version 0.4.8## see docs: https://docs.elementary-data.com/
This means Elementary models will have their own schema.
Make sure your user has permissions to create schemas.
dbt_project.yml
Copied
models:## elementary models will be created in the schema '<your_schema>_elementary'## see docs: https://docs.elementary-data.com/elementary:+schema:"elementary"
This will mostly create empty tables, that will be updated with artifacts, metrics and test results in your future dbt executions.
The short answer is yes.
We recommend that Elementary models will have their own schema, but it is not mandatory.
You can change the schema name by using dbt custom schema configuration on your dbt_project.yml.
In short, the default dbt generate_schema_name macro concatenate the value provided in schema configuration key to the target schema, as in: target_schema_custom_schema.
If you want a different behaviour, like configuring a full name for the Elementary schema, you can override the default generate_schema_name macro with your logic.
Before you do that, make sure that there isn't already a macro named generate_schema_name.sql in your project.
Here is a macro you can use that would search for a config under meta named schema_name.
If it exists, that would be the schema name. If not - the original dbt logic would be followed:
generate_schema_name.sql
Copied
{% macro generate_schema_name(custom_schema_name, node)-%}
{%-set default_schema = target.schema-%}
{%set config_meta = node.config.get('meta')%}
{%if config_meta and config_meta is mapping %}
{%set schema_name = config_meta.get('schema_name')%}
{%if schema_name and schema_name is string %}
{{ return(schema_name) }}
{% endif %}
{%- elif custom_schema_name is none -%}
{{ default_schema }}
{%-else-%}
{{ default_schema }}_{{ custom_schema_name | trim }}
{%- endif -%}
{%- endmacro %}
If you implement this macro and want to name the Elementary schema elementary_data_observability:
For elementary to work, it needs to create some of the models as incremental tables.
Make sure that there are no global materialization configurations that affect elementary, such as:
dbt_project.yml
Copied
materialized:"{{ 'table' if target.name == 'prod-cloud' else 'view' }}"
Make sure to place the ‘elementary’ configuration under the models key, and other configs under your project name.
Example:
dbt_project.yml
Copied
models:my_project:materialized:"{{ 'table' if target.name == 'prod-cloud' else 'view' }}"elementary:+schema:"elementary"
If you change materialization settings, make sure to run dbt run -s elementary --full-refresh.
Running your tests will populate Elementary’s models with the relevant information that is needed to monitor your data.
You can also choose to run only some of your tests, but keep in mind that only tests that ran after Elementary was installed will be monitored.
After you ran your tests, we recommend that you ensure that the results were uploaded to the elementary_test_results table.
Once the elementary dbt package has been installed and configured, your test results, run results and dbt artifacts will be loaded to elementary schema tables.
If you see data in these models you completed the package deployment (Congrats! 🎉).
