Quickstart

Install

Start by downloading and installing elementary-lineage:
1
pip install elementary-lineage
2
3
# The tool is named edl (Elementary Data Lineage),
4
# run it to validate the installation:
5
edl --help
Copied!
We recommend using pip, for other installation options refer to the download and install guide.

Requirements

In order to connect to start, we need a simple profiles.yml file to configure the connection. In this file, make sure to provide credentials that have the permissions level detailed on each integration page.
If you use dbt, you should already have it under ~/.dbt/profiles.yml, just note that in BigQuery 'location' is required and not optional.
If not, here is an example for Snowflake and BigQuery profiles:
1
# example profiles.yml file
2
3
## SNOWFLAKE ##
4
## profile name, replace 'my_snowflake_profile' with a name of your choice ##
5
my_snowflake_profile:
6
outputs:
7
default:
8
type: snowflake
9
account: [account id]
10
11
## User/password auth, other options (Keypair/SSO) require other configs ##
12
user: [username]
13
password: [password]
14
15
role: [user role]
16
database: [database name]
17
warehouse: [warehouse name]
18
## Schema is used as filter by default, you can use the 'ignore schema'
19
## option to see cross schema lineage
20
schema: [schema name]
21
## OPTIONAL - if you want to create the lineage based on queries from more
22
## than the last 7 days or you have 10k or more queries in the history pulled during
23
## the requested dates range, add this parameter (NOTE: account_usage requires more permissions, see note below).
24
query_history_source: account_usage
25
26
## BIGQUERY ##
27
## profile name, replace 'my_bigquery_profile' with a name of your choice ##
28
my_bigquery_profile:
29
outputs:
30
default:
31
type: bigquery
32
33
## Service account auth, other options require other configs ##
34
method: service-account
35
keyfile: [full path to your keyfile]
36
37
project: [project id]
38
## Dataset is used as filter by default, you can use the 'ignore schema'
39
## option to see cross dataset lineage (only in the same location)
40
dataset: [dataset name]
41
threads: 1
42
## Location is mandatory, can be one of US or EU, or a regional location
43
location: [dataset location]
44
priority: interactive
Copied!
For further information and other authentication options, refer to the connection profile guide.
Note that the credentials provided in the profiles file need to have the relevant permissions in order for Elementary to work.
For details, please refer to:

Generate lineage graph

For details about the CLI operation go to the usage guide, here is a usage example:
1
# Creates a lineage graph from queries executed between 7 days ago and current time,
2
# using the my_snowflake profile from ~/profiles_dir/profiles.yml
3
4
edl -d ~/profiles_dir -p my_snowflake
Copied!
All the options for the CLI operation are also available by simply running:
1
edl --help
Copied!

Last modified 7d ago