Introduction

Elementary was built out of the need to effortlessly and immediately gain visibility into the data stack, starting with tracing the actual upstream & downstream dependencies in the data warehouse, without any implementation efforts, security risks or compromises on accuracy.
Data lineage made simple, reliable, and automated.

It is a new open-source tool that focuses on simple setup and immediate visualization, no complex deployments or code changes are required. It consists of a CLI that analyzes logs and metadata and generates a web interface. 
⭐ If you like what we are building, support us with a ​
​ ⭐
(Some of the features are under construction 🚧
 )
Demo
Watch a video demo, or try our new live demo.
Features
Lineage visualization: Visual map of data flow and dependencies in the data warehouse. 
Dataset status: Present data about freshness and volume on the lineage graph.
Accuracy: Reflects the actual state in the DWH based on logs.
Plug-and-play: No need for code changes.
​Graph filters: Filter the graph by dataset, dates, direction, and depth. 
You can impact our next features in this roadmap by voting 👍 to issues and opening new ones.
Our plan is to eventually build a data observability platform that is open, transparent and powered by the community. A solution that data teams could easily integrate into their workflows, detect data incidents and prevent them from even happening in the first place.
We are working on Dataset monitoring, Lineage history, Column level lineage, Full lineage, and more.
Quick start
1
pip install elementary-lineage
2
​
3
# The tool is named edl (Elementary Data Lineage),
4
# run it to validate the installation:
5
edl --help
Copied!
Now we need a connection file in a simple YAML called profiles.yml. 
Here is a template to create a Snowflake \ BigQuery one.
Go to our quickstart page for further instructions on how to create it. 
If you use dbt, you can start right away by running this command with the path to your profiles.yml and the relevant profile name (note that in BigQuery location is required):
1
edl -d ~/profiles_dir -p <profile_name>
Copied!
Community & Support
For additional information and help, you can use one of these channels:
​
​Slack (live chat with the team, feature requests, community support, discussions, etc.)
​GitHub (bug reports, contributions)
 👍
 Roadmap (Vote for features and add your inputs)
​Twitter(updates on new releases and stuff)
 📧
 Email us (about whatever)
Integrations
​Snowflake​
​
​BigQuery 
 
Redshift 
 
Ask us for integrations on Slack or as a Github issue.
License
Elementary is licensed under Apache License 2.0. See the LICENSE file for licensing information.

Quickstart

Last modified 26d ago

Copy link

Contents