​
Synchronizing the Elementary schema

The data on your Elementary Cloud environments is updated by syncing the local Elementary schema from the data warehouse.

There are 2 available scheduling options:

  1. Hourly syncs - By default, Elementary Cloud syncs data from your Elementary schema on an hourly basis. This means that within the one-hour time window, your data may not be up-to-date.
  2. Triggered syncs - Configure a webhook that will trigger a sync after your data pipeline has finished running. This configuration is recommended, as it’s as close as possible to real time.

​
Triggered syncs using webhook

When configuring a webhook, the environment will update as soon as new data is available, ensuring you see the most recent data while minimizing unnecessary updates and therefore cost.

To configure the webhook, navigate to the relevant environment on your Environments page in Elementary Cloud. Click the environment’s Schedule Settings and change the environment update method to “Webhook triggered”. The next step depends on whether you’re using dbt Cloud or dbt Core.

​
dbt Cloud

In the Schedule Settings, you’re provided with a webhook URL. Next, you will need to provide a secret token that will be returned from dbt Cloud.

Heading to dbt Cloud, you can create a webhook subscription that will trigger a sync after your jobs are done.

  • Make sure the webhook is triggered on Run completed events.
  • Select only the main jobs of the relevant environment.
Make sure to select only the main jobs of the relevant environment. Selecting all jobs will trigger a sync for each job, which may result in unnecessary updates and therefore increased cost on the data warehouse.

After completing the guide, you’ll be provided with the secret token that you’ll need to fill in the form on Elementary Cloud. Press Save and you’re done!

Note that each environment has a different webhook

​
dbt Core

If you’re using dbt Core and have a custom orchestration layer such as Airflow or Prefect, you can use the webhook URL and the secret generated by Elementary in the Schedule Settings to trigger a sync after your dbt job.

In the following code snippet, replace <SECRET> and <WEBHOOK_URL> with the secret and webhook URL provided in the Schedule Settings and run it after your dbt job has finished.

import hmac
import requests

request_body = b""
secret = b"<SECRET>"
webhook_url = "<WEBHOOK_URL>"
signature = hmac.new(secret, request_body, "sha256").hexdigest()
requests.post(webhook_url, headers={"Authorization": signature}, data=request_body)

Make sure to click Save after filling in the form.