Elementary dbt package uses on-run-end hooks to log results and metadata to tables in the Elementary schema.

​
Why Elementary uses on-run-end hooks?

Elementary report and alerts are generated from the data in the Elementary schema. The solution relies on the Elementary schema being up-to-date and complete to be able to provide reliable and accurate observability.

By leveraging on-run-end hooks, we add a built-in collection of the latest results and metadata as part of your runs. This means the results you see in Elementary report and the alerts you receive are full, up-to-date and accurate.

We strongly recommend not to disable the hooks for environments you want to monitor using Elementary.

​
What happens on the on-run-end hooks?

On the on-run-end hooks Elementary extracts data from the dbt results and graph objects, and runs SQL queries to load this data to the Elementary models.

There are 2 types of models that Elementary updates :

  1. Metadata models - Such as dbt_models, dbt_tests, dbt_sources.
  2. Result models - Such as dbt_run_results, elementary_test_results, dbt_invocations.

​
Updates of metadata models

These models store the current resources and configuration in your dbt projects (models, snapshots, sources, tests, etc.). The metadata in the models only represents the project state on the latest run, so upon changes the metadata is replaced. The on-run-end hook runs SQL queries with the new metadata and updates the relevant tables.

​
Updates of result models

These models store a log of results of dbt invocations, and of the specific executed resources. The on-run-end hook runs SQL queries with the run results and invocation details.

​
Performance impact of on-run-end hooks

We give a lot of thought and effort to making Elementary efficient in both cost and performance. We only run the hooks that are relevant to each run, and each hook creates a minimal amount of queries possible.

​
Metadata models

For dbt 1.4.0 and above, we maintain a metadata cache. This means each of these models are only updated with changes in your project (new model, change in config, etc.). The first time you execute Elementary the initial update might take a while, but the following updates should be quick.

For dbt 1.3.0 and lower, these models would be fully updated on each run. The performance impact depends on the size of your dbt project.

We strongly recommend Elementary users to use a dbt version of 1.4.0 or above.

​
Result models

The size of the queries depends on the amount of models/tests executed in the run. The time the run results adds to the invocation shouldn’t be significant.

​
Can I disable the on-run-end hooks?

Yes, but note that this may cause missing results and/or outdated metadata in Elementary report and alerts.

​
Disable metadata models updates

Configure the following var:

dbt_project.yml
vars:
  disable_dbt_artifacts_autoupload: true

If you only want to disable dbt_columns but leave all other artifacts, configure the following:

dbt_project.yml
vars:
  disable_dbt_columns_autoupload: true

If you disable the artifacts autoupload, we recommend your run dbt run --select elementary.edr.dbt_artifacts every time you deploy changes to your project.

​
Disable result models updates

Configure the following vars (you can also disable with conditions):

dbt_project.yml
vars:
  disable_run_results: true
  disable_tests_results: true
  disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"
Please note you can also use the vars in your command line rather than in the dbt_project.yml file, by adding --vars '{"disable_dbt_artifacts_autoupload": true}' to your command line.
dbt artifactsPackage Models