Exploring Data Lineage. Data lineage is broadly understood as the lifecycle that spans from data's origin to where it moves over time across the data estate. It is used for different kinds of backwards-looking scenarios such as troubleshooting, tracing route cause, and data pipelines, and debugging. Lineage is also used for data quality analysis, compliance, and 'what if' scenarios, often referred to as impact analysis. 

Lineage is represented visually to show how it moves from source to destination, including how the data was transformed. Given the complexity of most enterprise data environments, these views can be hard to understand without doing some consolidation or masking of peripheral data points. Microsoft Purview data catalog will connect with other data processing, storage, and analytics systems to extract lineage information. The information is combined to represent a generic scenario specific lineage experience in the catalog. 

A data estate may include systems doing data extraction, transformation, analytics, and visualization systems. Each of these systems captures rich, static, and operational metadata that describes the state and quality of the data within the system's boundary. The goal of lineage in the data catalog is to extract the movement, transformation, and operational metadata from each data system at the lowest grain possible. Lineage is represented as a graph. Typically, it contains source and target entities in data storage systems that are connected by a process invoked by a compute system. Data systems connect to the data catalog to generate and report a unique object referencing the physical object of the underlying data system such as SQL stored procedures or notebooks. 

High fidelity lineage with additional metadata like ownership is captured to show the lineage in a human readable format from source and target entities. Lineage is a critical feature of the Microsoft Purview data catalog to support quality, trust, and audit scenarios. The goal of a data catalog is to build a robust framework where all the data systems within our environment can naturally connect and report lineage. Once the metadata is available, the data catalog can bring together the metadata provided by data systems to power data governance use cases.