Data Quality SLAs: What to Promise, What to Measure, What to Page On

A data contract tells a consumer what a field is. It does not tell them whether it showed up this morning. That gap is where most of the operational pain in a data platform actually lives: the schema was valid, the pipeline reported success, the dashboard rendered, and the number was six hours stale because an upstream source was late and nothing in the chain treated that as a failure. This is the post about the runtime half — the promise you make about an asset's condition over time, how you measure it honestly, and the much harder question of what you are actually willing to wake someone up for.

A Contract Says What It Is; an SLA Says Whether It Showed Up

The two artifacts get conflated constantly, and the conflation is expensive because they fail differently and are owned differently. A data contract is structural and mostly static: types, semantics, allowed values, what counts as a breaking change. It is violated by a change. An SLA is operational and continuous: is this asset current, complete, and plausible right now. It is violated by a Tuesday.

You need both, and you need them separated, because the response paths differ. A contract breach is a code-review problem — you catch it before merge, you negotiate, you version. An SLA breach is an incident — someone is consuming bad data right now and the clock is running. Teams that fold quality into the contract document end up with a document describing a healthy state nobody is monitoring, which is a specification, not a service level.

That is the filter I would apply to every proposed SLA before it goes on the board. If the honest answer to "what happens when this is missed" is "we would look at it eventually," the promise is fine as a monitored metric and dishonest as an SLA. Publishing agreements nobody will act on trains consumers to discount all of them, including the three that were real.

The Four Dimensions Worth Promising

You can measure dozens of things about a table. Four of them carry nearly all the operational value, and each fails in a characteristically quiet way — quiet meaning the pipeline reports success while the consumer gets something wrong.

The first three are cheap and deterministic; you can implement them against warehouse metadata without buying anything. The fourth is where tooling starts to earn its cost, and also where false positives are born, because "the distribution moved" and "the business changed" produce identical signals. That dimension needs a human who knows the domain attached to it, which makes it the most expensive one to run well and the one most likely to be quietly abandoned six months in.

Detection Is Cheap. Paging Is Expensive.

The instinct when standing up monitoring is to wire every check to an alert channel. What happens next is predictable: the channel fills, signal-to-noise collapses, and within a quarter people mute it. A muted alert channel is functionally identical to no monitoring while still costing every hour spent maintaining the checks — the worst position available, because it also manufactures the belief that the estate is covered.

The fix is to separate detection from notification. Detect broadly, because running checks is cheap and the history is genuinely useful for tuning later. Then route by consequence, not by confidence:

Page a human only when a failing asset has a named consumer whose immediate cost is real — a regulatory report, a customer-facing surface, a downstream job that will propagate the damage before morning. Open a ticket for assets that matter but survive until business hours. Log it silently for everything else, and review the accumulated log on a cadence to find the checks that should be promoted or deleted.

Making that routing defensible requires knowing who is actually downstream of a failing asset, which is exactly what lineage is for. Without it, severity gets assigned by whoever set the check up, based on how important they believe their table is — and everyone believes their table is important. With column-level lineage, an alert can arrive already carrying the affected downstream assets and their owners, which turns the severity question into a lookup instead of an argument. It also tells the responder what to go quarantine or caveat, which is often more urgent than the root cause.

Measure What the Consumer Feels, Not What the Tool Emits

Most quality dashboards report on the monitoring system rather than on the data. Checks configured, checks passing, percentage green — all of which can improve while consumer experience gets worse, because the easiest way to raise a pass rate is to loosen thresholds or add checks to healthy tables.

The measures that resist that gaming are defined from the consumer's side:

Time-to-detection. How long an issue existed before anything noticed. This is the number that tells you whether monitoring is doing its job, and the only honest way to calculate it is retrospectively, from incidents you eventually found by other means — a consumer complaint, a reconciliation, an audit. If everything you know about is something a check caught, you have no idea what your detection latency is.

Time-to-resolution, split at the handoff. Detection to acknowledgement, then acknowledgement to fix. Splitting it matters because the halves have different causes: a slow first half is a routing and ownership problem, a slow second half is a tooling and complexity problem, and a single blended number hides which one you have.

Consumer-discovered issue rate. The share of incidents where a person downstream noticed before the platform did. This is the most uncomfortable metric to publish and the most honest one, because it directly measures whether the program has bought anyone anything. It should trend down. If it does not, coverage growth is not helping.

None of these require a benchmark to be useful — they are read against their own trend, on your own estate. Anyone quoting industry-standard targets for them is selling something; the values depend so heavily on estate size, domain, and consumer tolerance that a cross-company comparison carries close to zero information.

When Observability Becomes a Second Job

Every check is a small permanent liability. It has a threshold that drifts as the business changes, an owner who may leave, and a failure mode where it fires on a legitimate change and someone has to decide whether to retune it or ignore it. Multiply that across an estate-wide rollout and you have manufactured a maintenance workload that competes with the pipeline work it was supposed to protect.

This is the same failure that shows up in month three of a stewardship rollout, and for the same structural reason: the launch is scoped and funded, the ongoing labor is not. The tell is a growing population of checks that fire regularly and are routinely dismissed. Those are not monitoring — they are a queue of unmade decisions, and each one erodes trust in the checks that are real.

Two disciplines keep it survivable. First, deletion is maintenance: a check that has produced only dismissals over a meaningful window should be removed rather than tuned, and removing it should require no approval ceremony. Second, treat threshold tuning as recurring capacity rather than project work — it is the same ongoing-labor line that gets systematically understated when modeling the TCO of a governance or observability platform, where the check catalog in the demo is rarely the expensive part.

Start With One Table Someone Would Escalate About

Estate-wide coverage as a first milestone is how these programs stall — long build, no incident visibly prevented, budget questioned at the next planning cycle. The sequencing that works starts from consequence: find the asset where a failure would produce a real escalation with a named person on the other end, and write one honest SLA for it. Freshness, usually, because it is the cheapest to measure and the most frequently violated.

Then wire the response before adding a second check. An SLA with a threshold and no owner is a metric; an SLA with an owner and a defined consequence is infrastructure. Once one exists and has survived a real breach, expansion is a template exercise rather than an argument — and it will naturally grow toward the assets that carry load, which is the same reason intake-side quality work outperforms downstream cleanup, and the same reason a classification effort should follow sensitivity rather than the table list.

One caveat on scope. SLAs protect the pipeline up to the point where data stops moving and starts being interpreted. A table can be perfectly fresh, complete, and valid while two teams still report different revenue, because they defined the metric differently in two BI tools. That is a distinct problem with a distinct fix — the semantic layer — and no amount of quality monitoring on the underlying table will surface it.