Project overview
General
Quickstart
Install
Start by downloading and installing elementary-lineage:
1
pip install elementary-lineage
2
3
# The tool is named edl (Elementary Data Lineage),
4
# run it to validate the installation:
5
edl --help
Copied!
Requirements
In order to connect to start, we need a simple
profiles.yml file to configure the connection. In this file, make sure to provide credentials that have the permissions level detailed on each integration page. If you use dbt, you should already have it under
~/.dbt/profiles.yml, just note that in BigQuery 'location' is required and not optional. If not, here is an example for Snowflake and BigQuery profiles:
1
# example profiles.yml file
2
3
## SNOWFLAKE ##
4
## profile name, replace 'my_snowflake_profile' with a name of your choice ##
5
my_snowflake_profile:
6
outputs:
7
default:
8
type: snowflake
9
account: [account id]
10
11
## User/password auth, other options (Keypair/SSO) require other configs ##
12
user: [username]
13
password: [password]
14
15
role: [user role]
16
database: [database name]
17
warehouse: [warehouse name]
18
## Schema is used as filter by default, you can use the 'ignore schema'
19
## option to see cross schema lineage
20
schema: [schema name]
21
## OPTIONAL - if you want to create the lineage based on queries from more
22
## than the last 7 days or you have 10k or more queries in the history pulled during
23
## the requested dates range, add this parameter (NOTE: account_usage requires more permissions, see note below).
24
query_history_source: account_usage
25
26
## BIGQUERY ##
27
## profile name, replace 'my_bigquery_profile' with a name of your choice ##
28
my_bigquery_profile:
29
outputs:
30
default:
31
type: bigquery
32
33
## Service account auth, other options require other configs ##
34
method: service-account
35
keyfile: [full path to your keyfile]
36
37
project: [project id]
38
## Dataset is used as filter by default, you can use the 'ignore schema'
39
## option to see cross dataset lineage (only in the same location)
40
dataset: [dataset name]
41
threads: 1
42
## Location is mandatory, can be one of US or EU, or a regional location
43
location: [dataset location]
44
priority: interactive
Copied!
Note that the credentials provided in the profiles file need to have the relevant permissions in order for Elementary to work.
For details, please refer to:
Generate lineage graph
1
# Creates a lineage graph from queries executed between 7 days ago and current time,
2
# using the my_snowflake profile from ~/profiles_dir/profiles.yml
3
4
edl -d ~/profiles_dir -p my_snowflake
Copied!
All the options for the CLI operation are also available by simply running:
1
edl --help
Copied!
Last modified 3mo ago
Copy link