Data lineage and observability: choose the evidence your investigation lacks

A sales dashboard can refresh on time and still give the finance team the wrong answer. It can also show the right calculation on yesterday’s data. Those failures look similar to the reader of the report, but they demand different evidence from the data platform lead.

Data lineage helps trace origins, transformations and downstream dependencies. Data observability brings signals about the behaviour of data and its pipelines, such as freshness, volume, execution failures and departures from expected patterns. Products can combine these capabilities. The buying decision should turn on which investigation your team cannot complete, rather than which label a supplier puts on the platform.

Follow the incorrect number

Consider a hypothetical revenue dashboard that includes cancelled orders after the business has changed its reporting rule to exclude them. Every scheduled job completes. The data arrives within its expected window, and the totals follow a familiar pattern. Operational monitoring may have no reason to alert: the system is reliably carrying out the old instruction.

Lineage can help the investigator work backwards from the dashboard to the table, query and contributing sources. That narrows the search for where cancelled orders entered the measure. It does not decide whether the old or new rule is authoritative. Someone accountable for the measure must establish the intended meaning, including when the change took effect.

The distinction matters commercially. Buying more anomaly detection will not necessarily resolve recurring disagreements between finance and sales over which orders count. Conversely, a definitions workshop will not reveal why a scheduled data transfer stopped. Diagnose the missing evidence before adding another platform.

Read the boundaries of the lineage graph

Databricks Unity Catalog documents automatic lineage for supported queries, including column-level relationships. Its documentation also specifies limits: column lineage requires source and target references by table name rather than file path, and user-defined functions can obscure column mappings. External assets require registration and configured relationships.

These details should shape an evaluation. If a consequential calculation sits inside a function, ask the supplier to show the evidence available for that exact transformation. If a report depends on an external system, establish how its relationships are captured and maintained. A continuous-looking diagram should not be treated as proof that every important step was observed automatically.

Permissions also affect what an investigator can see. Databricks describes lineage views that mask objects for which the viewer lacks the required access. Test the investigation using the role of the person who will actually perform it. An administrator’s complete view does not establish that an on-call analyst can trace the same dependency.

Record unsupported or manually maintained links explicitly. That makes it possible to decide whether a gap needs another integration, an engineering inspection or an accountable manual process. Hidden gaps make both change-impact analysis and incident diagnosis harder to trust.

Then follow the late result

In a second hypothetical case, the revenue definition is correct but an upstream feed has stopped arriving. The dependency graph shows where the dashboard’s data comes from. It may help identify affected consumers, yet the graph alone does not establish when the last usable records arrived or which run failed.

A freshness check supplies a different kind of evidence. dbt’s freshness configuration supports warning and error thresholds based on the age of the most recent data. Its documentation allows a loaded timestamp field or query and, where supported, warehouse metadata. The choice matters: the timestamp being checked needs to represent the freshness requirement the business actually cares about.

A recent load timestamp could coexist with stale business events if a delayed batch was only just ingested. Ask whether the requirement concerns arrival time, the time of the underlying event or the period covered by a completed batch. Test the required meaning, and avoid treating one healthy timestamp as a guarantee that the dataset is complete.

Now combine the evidence. Operational signals identify the missed delivery or failed run. Lineage identifies downstream reports and services potentially affected. The data owner decides whether a stale result can remain available with a clear qualification, whether an update must be held back and what reconciliation is required after recovery.

Buy around a repeatable investigation

Take one incorrect result and one delayed feed through the proposed tooling with your own representative transformations. Ask the team to locate the contributing sources, inspect the relevant execution, identify the impact and reach the person authorised to decide what correct means. Include a path through an external tool or an awkward transformation that matters in production.

Judge the result by the investigation it supports: what became observable, what remains inferred, which access restrictions intervened and how much engineering work was required. This reveals the running cost behind an attractive demonstration.

Where the evidence leads to competing definitions, continue with the business-definition problem that dashboards cannot fix. Lineage and observability can make the disagreement easier to locate. Resolving it still requires a decision about the data the business intends to use.