Elementary can collect metadata about your jobs from the orchestrator you are using, and enrich the Elementary UI with this information.
The goal is to provide context that is useful to triage and resolve data issues, such as:
Elementary collects the following job details:
orchestrator
job_name
job_id
job_url
job_run_id
job_run_url
Elementary collects jobs metadata in run time from env_vars
.
Orchestration tools usually have default environment variables, so this might happen automatically. The list of supported orchestrators and default env vars is in the following section.
These are the env vars that are collected:
ORCHESTRATOR
, JOB_NAME
, JOB_ID
, JOB_URL
, JOB_RUN_ID
, JOB_RUN_URL
To configure env_var
for your orchestrator, refer to your orchestrator's docs.
Elementary also supports passing job metadata as dbt vars. If env_var
and var
exist, the var
will be prioritized.
To pass job data to Elementary using var
, use the --vars
flag in your invocations:
var / env_var | Format |
---|---|
orchestrator | One of: airflow , dbt_cloud , github_actions , prefect , dagster |
job_name, job_id, job_run_id | String |
job_url, job_run_url | Valid HTTP URL |
You can pass job info to Elementary from any orchestration tool as long as you configure env_vars
/ vars
.
The following default environment variables are supported out of the box:
Orchestrator | Env vars |
---|---|
dbt cloud | orchestrator job_id: DBT_CLOUD_JOB_ID job_run_id: DBT_CLOUD_RUN_ID job_url: generated from DBT_ACCOUNT_ID , DBT_CLOUD_PROJECT_ID , DBT_CLOUD_JOB_ID job_run_url: generated from ACCOUNT_ID , DBT_CLOUD_PROJECT_ID , DBT_CLOUD_RUN_ID |
Github actions | orchestrator job_run_id: GITHUB_RUN_ID job_url: generated from GITHUB_SERVER_URL , GITHUB_REPOSITORY , GITHUB_RUN_ID |
Airflow | orchestrator |
By default, Elementary will collect the dbt cloud jobs info.
If you wish to override that, change your dbt cloud invocations to pass the orchestrator job info using --vars
:
dbt_invocations
. You could also use the view job_run_results
which groups invocation by job.We would love to support more orchestrators and collect more useful info! Please open an issue and tell us what we should add.
Elementary can collect metadata about your jobs from the orchestrator you are using, and enrich the Elementary UI with this information.
The goal is to provide context that is useful to triage and resolve data issues, such as:
Elementary collects the following job details:
orchestrator
job_name
job_id
job_url
job_run_id
job_run_url
Elementary collects jobs metadata in run time from env_vars
.
Orchestration tools usually have default environment variables, so this might happen automatically. The list of supported orchestrators and default env vars is in the following section.
These are the env vars that are collected:
ORCHESTRATOR
, JOB_NAME
, JOB_ID
, JOB_URL
, JOB_RUN_ID
, JOB_RUN_URL
To configure env_var
for your orchestrator, refer to your orchestrator's docs.
Elementary also supports passing job metadata as dbt vars. If env_var
and var
exist, the var
will be prioritized.
To pass job data to Elementary using var
, use the --vars
flag in your invocations:
var / env_var | Format |
---|---|
orchestrator | One of: airflow , dbt_cloud , github_actions , prefect , dagster |
job_name, job_id, job_run_id | String |
job_url, job_run_url | Valid HTTP URL |
You can pass job info to Elementary from any orchestration tool as long as you configure env_vars
/ vars
.
The following default environment variables are supported out of the box:
Orchestrator | Env vars |
---|---|
dbt cloud | orchestrator job_id: DBT_CLOUD_JOB_ID job_run_id: DBT_CLOUD_RUN_ID job_url: generated from DBT_ACCOUNT_ID , DBT_CLOUD_PROJECT_ID , DBT_CLOUD_JOB_ID job_run_url: generated from ACCOUNT_ID , DBT_CLOUD_PROJECT_ID , DBT_CLOUD_RUN_ID |
Github actions | orchestrator job_run_id: GITHUB_RUN_ID job_url: generated from GITHUB_SERVER_URL , GITHUB_REPOSITORY , GITHUB_RUN_ID |
Airflow | orchestrator |
By default, Elementary will collect the dbt cloud jobs info.
If you wish to override that, change your dbt cloud invocations to pass the orchestrator job info using --vars
:
dbt_invocations
. You could also use the view job_run_results
which groups invocation by job.We would love to support more orchestrators and collect more useful info! Please open an issue and tell us what we should add.