Configuration: Connection profile
In order to connect to your data warehouse, Elementary Data Lineage CLI requires connection details and credentials. These are provided to the tool using a file named profiles.yml. You can store as many profiles as you need in this file. Typically, you would have one profile for each warehouse you use. If you use dbt, you don't even need to create the file, you already have it. In general, we support the same profiles format as dbt for our supported integrations, with the exception that in BigQuery we require 'location', and in dbt it is optional.

How to create 'profiles.yml'?

    1.
    Choose a directory that does not sync with any version control system
    2.
    Create a new file and name it profiles.yml
    3.
    Use a template of the relevant integration to create a profile in the file, and save it.
1
# example profiles.yml file
2
3
## SNOWFLAKE ##
4
## profile name, replace 'my_snowflake_profile' with a name of your choice ##
5
my_snowflake_profile:
6
outputs:
7
default:
8
type: snowflake
9
account: [account id]
10
11
## User/password auth, other options (Keypair/SSO) require other configs ##
12
user: [username]
13
password: [password]
14
15
role: [user role]
16
database: [database name]
17
warehouse: [warehouse name]
18
## Schema is used as filter by default, you can use the 'ignore schema'
19
## option to see cross schema lineage
20
schema: [schema name]
21
22
## OPTIONAL - if you want to create the lineage based on queries from more
23
## than the last 7 days or you have 10k or more queries in the history pulled during
24
## the requested dates range, add this parameter (NOTE: account_usage requires more permissions, see note below).
25
query_history_source: account_usage
26
27
## BIGQUERY ##
28
## profile name, replace 'my_bigquery_profile' with a name of your choice ##
29
my_bigquery_profile:
30
outputs:
31
default:
32
type: bigquery
33
34
## Service account auth, other options require other configs ##
35
method: service-account
36
keyfile: [full path to your keyfile]
37
project: [project id]
38
## Dataset is used as filter by default, you can use the 'ignore schema'
39
## option to see cross dataset lineage (only in the same location)
40
dataset: [dataset name]
41
threads: 1
42
## Location is mandatory, can be one of US or EU, or a regional location
43
location: [dataset location]
44
priority: interactive
Copied!

Connection details and permissions

Each warehouse schema you connect to requires a profile with connection details, and the provided credentials need to have permissions to read the query history for this schema. If the credentials provided will not have access to the entire query history, a partial lineage graph will be generated based on the available queries (can be exported using the export query history option).
The required connection details and permissions for each data warehouse is detailed in the relevant integration documentation under integrations.

dbt users

If you are a dbt user, you already have a profiles.yml file that you can use. The default path of the dbt profiles file is ~/.dbt/ directory.
    For further details, please refer to dbt's documentation about the file.
    Note: in BigQuery we require 'location' (details), and in dbt it is optional.

Anonymous usage tracking

We want to keep building and improving, and for that, we need to understand how users work with Elementary (and data is fun!). For that we added an anonymous tracking of events using Posthog (open-source product analytics, highly recommended).
We only track start, end, platform, number of queries and the size of the graph. No credentials, queries content, table names or anything private (not now and not ever).
By default this completely anonymous tracking is turned on. You can opt-out at any time by adding the following to your profiles.yml file:
1
anonymous_usage_tracking: False
Copied!
Last modified 3d ago