Connect to Databricks

​

Create service principal

1. Open your Databricks console, and then open your relevant workspace.
2. Click on your Profile icon on the right and choose Settings.

3. On the sidebar, click on Identity and access, and then under the Service Principals row click on Manage.

4. Click on the Add service principal button, choose “Add new” and give a name to the service principal. This will be used by Elementary Cloud to access your Databricks instance.

5. Click on your newly created service principal, add the “Databricks SQL access” entitlement, and click Update. Also, please copy the “Application ID” field as it will be used later in the permissions section.

6. Next, generate credentials for your service principal. Choose one of the following methods: Option A: Generate an OAuth secret (Recommended) On the service principal page, go to the Secrets tab and click Generate secret. Copy the Client ID (this is the same as the “Application ID” from step 5) and the generated Client secret — you will need both when configuring the Elementary environment.
   OAuth secrets are the recommended authentication method. They enable short-lived token generation with automatic refresh, providing better security than long-lived personal access tokens.
   Option B: Create a personal access token (legacy) In order to generate a personal access token for your service principal, you may first need to allow Token Usage for it. To do so, go to the settings menu and choose Advanced -> Personal Access Tokens -> Permission Settings, then make sure the service principal is in the list. Then, create a personal access token for your service principal. For more details, please click here.
7. Finally, in order to enable Elementary’s automated monitors feature, please ensure predictive optimization is enabled in your account. This is required for table statistics to be updated (Elementary relies on this to obtain up-to-date row counts)

​

Permissions and security

​

Required permissions

• Elementary schema read-only access - This is required by Elementary to read dbt metadata & test results collected by the Elementary dbt package as a part of your pipeline runs. This permission does not give access to your data.
• Information schema metadata access - Elementary needs access to the system.information_schema.tables and system.information_schema.columns system tables, to get metadata about existing tables and columns in your data warehouse. This is used to power features such as column-level lineage and automated volume & freshness monitors.
• Read access needed for some metadata operations (optional) - In order to enable Elementary’s automated volume & freshness monitors, Elementary needs access to query history, as well as Databricks APIs to obtain table statistics. These operations require granting SELECT access on your tables. This is a Databricks limitation - Elementary never reads any data from your tables, only metadata. However, there isn’t today any table-level metadata-only permission available in Databricks, so SELECT is required.

​

Grants SQL template

-- Grant read access on the elementary schema (usually [your dbt target schema]_elementary)
GRANT USE CATALOG ON CATALOG <catalog> TO `<service_principal_app_id>`;
GRANT USE SCHEMA, SELECT ON SCHEMA <elementary_schema> TO `<service_principal_app_id>`;

-- Grant access to information schema tables
GRANT USE CATALOG ON CATALOG system TO `<service_principal_app_id>`;
GRANT USE SCHEMA ON SCHEMA system.information_schema TO `<service_principal_app_id>`;
GRANT SELECT ON TABLE system.information_schema.tables TO `<service_principal_app_id>`;
GRANT SELECT ON TABLE system.information_schema.columns TO `<service_principal_app_id>`;

-- Grant select on tables for history & statistics access
-- (Optional, required for automated volume & freshness tests - see explanation above. You can also limit to specific schemas used by dbt instead of granting on the full catalog)
GRANT USE CATALOG, USE SCHEMA, SELECT ON catalog <catalog> to `<service_principal_app_id>`;

​

Add an environment in Elementary (requires an admin user)

• Server Host: The hostname of your Databricks account to connect to.
• Http path: The path to the Databricks cluster or SQL warehouse.
• Catalog (optional): The name of the Databricks Catalog.
• Elementary schema: The name of your Elementary schema. Usually [your dbt target schema]_elementary.

​

OAuth (M2M) — Recommended

• Client ID: The Application (client) ID of the service principal (the “Application ID” you copied in step 5).
• Client secret: The OAuth secret you generated for the service principal (see step 7).

​

Access token (legacy)

• Access token: A personal access token generated for the Elementary service principal.

​

Add the Elementary IP to allowlist

​

Need help with onboarding?