Elementary Databricks integration is supports:
- Installing the dbt package to upload dbt artifacts, test results and run results.
- Data observability report generation.
- Slack alerts on failures in models and tests.
- Anomaly detection tests.
Elementary schema changes tests are not supported yet.
Databricks connection profile
After installing Elementary’s dbt package upon running
you can generate Elementary’s profile for usage with
edr by running the following command within your project:
dbt run-operation elementary.generate_elementary_cli_profile
The command will print to the terminal a partially filled template of the profile that’s needed for
edr to work.
Fill in the missing authentication fields
in the output and copy it to
profiles.yml file to connect to Databricks should contain a profile in the following format:
## DATABRICKS ## ## By default, edr expects the profile name 'elementary'. ## elementary: outputs: default: type: databricks host: [hostname, like <ID>.cloud.databricks.com] http_path: [like /sql/1.0/endpoints/<ID>] schema: [schema name] # elementary schema, usually [schema name]_elementary token : [token] threads: [number of threads like 8]