Data Deletion Is a Reachability Problem, Not a Storage One

Q: What are the derived copies most deletion pipelines miss?

The ones that were never columns. A retrieval or analytics stack typically holds extracted text, document chunks, embeddings in a vector index, a keyword index, a cache of previous answers, conversation logs that quoted the content, scheduled exports, BI extracts, and any training or fine-tuning dataset assembled from the source. None of these are edges in a warehouse lineage graph, because the graph models table-to-table transformations and these are artifacts produced by applications. They have to be enumerated deliberately, once, as a standing register, because nothing in the governance stack will discover them for you.

Sep 9, 2026 · Data Governance

There is a symmetry problem sitting underneath every governed data estate, and almost nobody designs for it.

Every layer in the stack answers a question about data you are keeping. Classification labels what a field contains. Lineage records where it travelled. Contracts govern how it is allowed to change. Quality SLAs promise it will be fresh and complete. Access control decides who may read it. Eleven posts on this site describe that estate, and every one of them assumes the data continues to exist.

Then a deletion request arrives. Not a schema change, not an incident — an obligation to make a specific person's data stop existing, everywhere, and to be able to say so. And the entire apparatus turns out to point the wrong way.

Quick answer: deletion is not a storage operation and it is not a policy document. It is a reachability traversal — the same walk your impact analysis already does, run for a different reason and at a different resolution. Your lineage graph is the closest thing you own to a deletion engine, and it will still not finish the job, because it stops at two boundaries: below the table, where the file outlives the row, and beyond the table, where embeddings, indexes, caches and extracts are copies that were never columns. Neither boundary produces an error when you cross it, which is why the ticket closes anyway.

The Ticket Closes Before the Work Does

The failure has a recognizable shape. A request comes in. Someone runs a delete against the system of record. The row disappears from every query anyone thinks to run. The ticket is marked complete, and the compliance report counts it.

Nothing about that sequence is negligent. It is exactly what the tooling encourages, because the tooling reports on the query surface and erasure is a claim about the storage surface. The gap between those two is invisible by construction: a partially completed deletion emits no error, produces no failed test, and is indistinguishable in the ticketing system from a complete one. It is discovered later, by accident, when somebody queries a copy nobody remembered.

a delete statement changes what a query returns. erasure is a claim about what exists. in a modern data platform those are deliberately different things, and the difference is a feature you paid for

Boundary One: Below the Table

Start with the layer directly under the row, because it is documented, vendor-acknowledged, and still routinely missed.

Modern table formats are built on immutable files plus a transaction log. A delete does not rewrite history; it writes a new version that excludes the row. That is the mechanism that makes time travel and rollback possible, and it is the mechanism that makes a delete insufficient. Databricks states it plainly in its own GDPR guidance: “By default, Delta Lake retains table history, including deleted records, for 30 days, and makes it available for time travel and rollbacks.” And, on the layer under that: “even if previous versions of the data are removed, the data is still retained in cloud storage.” The operation that actually removes it is VACUUM, which the same page describes as removing files no longer referenced by the table and older than a retention threshold, permanently deleting the data.

So there is a documented window — thirty days by default, longer if someone extended retention for a good operational reason — in which the subject is absent from every query and completely present on disk. Anyone who can read a prior table version, or the storage bucket, has the record.

Delta is just the legible example. The same structure exists in every estate through less documented paths:

Where the row survives	Why it is missed
Table history / time travel	It is a feature, enabled by default, and the retention window is set by whoever tuned it for rollback safety — not by whoever owns erasure.
Unreferenced storage files	Removing a version dereferences the file; it does not delete it. A separate maintenance operation does, on its own schedule.
Snapshots and backups	Owned by platform or infrastructure, not by data governance. Their retention was designed against data loss, which is the opposite requirement.
Replicas and lower environments	Dev and staging are commonly seeded from a production copy. The deletion runs in prod; the copy is untouched and will outlive several requests.
The restore path	The one nobody models: a deletion completes correctly, and six weeks later an unrelated incident restores a backup taken before it. The subject is back and no process notices.

That last row is the one worth sitting with. Every other item on the list is a task you can schedule. The restore path is a reintroduction path, and it means erasure completeness is not a state you reach once — it is an invariant something has to keep re-checking.

Boundary Two: Beyond the Table

The second boundary is newer, worse, and almost entirely absent from the governance literature: the copies that were never columns.

A lineage graph models transformations between structured assets. Column feeds column, table feeds table. That is the correct model for the warehouse, and it is why lineage is such a good starting point for a deletion traversal. But a retrieval or ML stack produces artifacts that are not the output of a SQL transformation and therefore never become edges in that graph: extracted text, document chunks, embeddings in a vector index, a keyword index, a cache of prior answers, conversation logs that quoted the content, an export someone scheduled two years ago, a training or fine-tuning dataset assembled from a snapshot.

Each is a genuine copy of the data. None is discoverable by the machinery you built to trace it.

The embedding case is the sharpest, because it looks handled and is not. Most vector stores implement deletion as a metadata flag: the query skips the entry, and the index structure is left intact because rebuilding it is expensive. A June 2026 preprint, “Ghost Vectors: Soft-Deleted Embeddings Remain Reconstructible in HNSW Vector Databases”, tested exactly this across ChromaDB, FAISS and Weaviate. Their finding is that soft-deleted vectors are still physically present in the on-disk index and recoverable by reading it directly — the authors describe extraction running “entirely from Python scripts that read the on-disk index files directly, never touching the database query API.”

What came back out is the part that should change how you scope a deletion. From a biographical dataset they reconstructed 25.5% of exact person names and 46.4% of geographic locations. From a synthetic medical dataset, 100% of patient age and gender markers. From facial embeddings, 99% top-1 identity recovery. The paper also proposes a mitigation, an epoch key rotation scheme it reports reduces recovery to 0% at roughly 0.005 ms per record — so this is a solvable problem, not an unsolvable one. It is simply not solved by the delete call you already made.

I have not reproduced any of this myself, and I want to be exact about that: those numbers are the authors', on their datasets and their configurations, and I am citing a preprint. What survives regardless of the exact percentages is the structural claim, which the same paper states in one line — soft delete is implemented as a metadata operation, not a physical one. That is an architecture fact, and it is enough to invalidate “we deleted it from the vector store” as a completed step.

Fine-tuned weights are the same problem with no clean fix at all. If the subject's data was in a training run, deleting the source dataset does not remove its influence from the model. Machine unlearning is an active research area, not a runbook step, and the practical consequence is that the decision that mattered was made before training, not after the request arrived — which is an argument for treating training-set assembly as a governed, classification-gated pipeline rather than a data science convenience. This is the same boundary governing AI agents' data access and RAG governance run into from the other side.

The Real Gap: Schema Granularity vs Value Reachability

Underneath both boundaries is one mismatch, and naming it is most of the fix.

Impact analysis asks: if I change this column, what breaks? That is a question about schema, and a column-level lineage graph answers it well. Deletion asks: where did this person's values end up? That is a question about reachability at the value level, and the same graph answers it only approximately — because an edge tells you data could flow along that path, not that this subject's rows did, and not whether the destination retained them after a filter, an aggregation, or a join that dropped them.

	Impact analysis	Deletion traversal
Question	What breaks if this column changes?	Where did this subject's values reach?
Granularity	Schema — column and table	Value — rows belonging to one subject
Good enough answer	A superset. Over-warning is cheap.	A superset is not enough — you must confirm each destination, and a miss is the whole failure.
Terminates at	The last modeled asset — correct, because unmodeled things don't break	The last modeled asset — wrong, because unmodeled copies still hold the data
Failure signal	Something breaks loudly, later	Nothing happens at all, ever, until discovery

Read the last two rows together and you have the whole argument. The traversal terminating at the last modeled asset is correct behaviour for impact analysis and a silent, total failure for deletion — and because the failure signal is nothing happening, the two are indistinguishable from inside the tool.

What the Vendor Answer Leaves Out

The governance vendors are not wrong about this, they are just brief. Collibra's lineage explainer puts the requirement accurately: “If a consumer requests that their data be deleted under CCPA, the organization must be able to trace that individual's data from the entry point through every downstream system, backup and analytical model to ensure it is completely eradicated.”

That sentence is correct and it is doing an enormous amount of work. It asserts a traversal through every downstream system, backup and analytical model — three categories, two of which the lineage graph does not contain. And when I fetched the page to check, it offers no operational detail on running that traversal, and does not mention embeddings, vector stores or training data anywhere. Neither does the compliance-side literature, which is largely about writing a retention schedule and naming a policy owner — useful, and a different document from the one an engineer needs on the day.

So the two available genres are “here is the VACUUM command” and “here is a retention policy template.” The engineering question in between — what is the actual traversal, and where does it stop — is the one nobody publishes, and it is the one that determines whether the work got done.

A Traversal You Can Actually Run

None of this needs a new platform. It needs the layers you already have, pointed backwards, plus one register that has to be written by hand because nothing will generate it for you.

Step	Which layer answers it	What it gives you
1. Scope — which fields are in scope for this subject?	Classification	The identifier columns and the sensitive attributes. If classification is incomplete, every later step inherits the gap — this is the load-bearing dependency.
2. Candidates — where could those fields have reached?	Lineage	The list of destinations to check. Treat it as candidates, never as proof of completeness.
3. The register — what copies exist that lineage cannot see?	Nothing. Write it by hand.	Indexes, embeddings, caches, exports, extracts, training sets, lower environments, backups. A standing list, maintained deliberately.
4. Obligation — who has to act, and who confirms?	Data contracts	Deletion as a versioned event consumers must acknowledge, rather than a broadcast nobody answers. The contract already names the owner and the consumer set.
5. Proof — did it finish, and by when?	Quality SLAs	A completeness check that runs after the window and asserts absence at every registered destination, with a time-to-completion and an alert on non-confirmation.
6. Re-check — did anything reintroduce it?	The same SLA, on a schedule	Coverage for the restore path. Erasure is an invariant to hold, not a task to close.

Step 3 is the one that carries the risk, and it is the only one with no tool behind it. It is also cheap: an afternoon with the teams that run the retrieval stack, the BI estate, and the export jobs will produce a register that is ninety percent right and immediately more useful than the graph, because it contains precisely the destinations the graph is structurally incapable of holding.

Step 5 is what converts this from a policy into something with a failure signal. Absent a measured completeness check, a deletion's only evidence is that somebody closed a ticket — and as with metric definitions and intake quality, the thing that is not measured is the thing that silently drifts.

Start With One Subject, End to End

The sequencing that works here is the one that works for the rest of this cluster: take a single real request and run it to genuine completion, including the parts nobody usually does. Not a policy program — one subject, every destination, with the vector index and the backups and the lower environments actually checked and the completeness assertion actually written.

One subject fully traversed will find the copies. It will also produce the register in step 3 as a by-product, which is the artifact that makes the next hundred requests mechanical. A comprehensive deletion policy written before that exercise is a document about a system nobody has mapped.

Working on the governance layer for real?

I run a free community for people building real data and AI pipelines — warehouse modeling, governance, and AI-augmented workflows. No course, no paywall, just practitioners shipping.

Join the free AI community →

Frequently Asked Questions

Why does deleting a row not satisfy a right-to-erasure request?

Because a delete statement changes what a query returns, and erasure is a claim about what exists. In a modern table format the two are deliberately different. Databricks documents that Delta Lake by default retains table history, including deleted records, for 30 days so that time travel and rollback work, and that even once previous versions are removed the underlying data is still retained in cloud storage until a VACUUM removes the unreferenced files. Everything in between is a state where the row is invisible to consumers and fully present on disk. The same gap exists in most estates through snapshots, backups, replicas, and dev environments restored from production copies.

Can I use data lineage to run a deletion request?

Partly, and the part it misses is the part that gets you in trouble. A lineage graph is built at schema granularity: it records that a column fed another column. A deletion request is a question at value granularity: where did the rows belonging to this one person end up. The graph is the right shape at the wrong resolution, so it yields candidate destinations rather than an answer. It is still the best starting point most organizations have, because the alternative is asking teams what they remember. Use it as the list of places to check, not as proof that checking is finished.

Does deleting a record remove it from a vector database?

Usually not physically. Most vector indexes implement deletion as a metadata flag so queries skip the entry while the index structure stays intact. A 2026 preprint studying three HNSW implementations — ChromaDB, FAISS and Weaviate — found soft-deleted vectors remain recoverable by reading the raw index files at the storage layer without touching the query API, reporting 25.5% of exact person names and 46.4% of geographic locations reconstructed from a biographical dataset and 99% top-1 identity recovery from facial embeddings. Treat an embedding as a derived copy needing its own deletion path and its own proof.

What are the derived copies most deletion pipelines miss?

The ones that were never columns: extracted text, document chunks, embeddings, a keyword index, a cache of previous answers, conversation logs that quoted the content, scheduled exports, BI extracts, and any training or fine-tuning dataset assembled from the source. None are edges in a warehouse lineage graph, because that graph models table-to-table transformations and these are artifacts produced by applications. They have to be enumerated deliberately, as a standing register, because nothing in the governance stack will discover them for you.

Who is accountable for completing a deletion across systems?

The same producer who is accountable for the data existing. Deletion works best as an obligation attached to the data contract rather than a ticket routed to whoever is free, because the contract already names an owner, a consumer set, and a change protocol. That makes the deletion a versioned event consumers must acknowledge, rather than a broadcast nobody confirms — and gives the completeness question an owner per hop, instead of one central team trying to prove a negative about systems it does not operate.

How do you prove a deletion actually completed?

By making completeness a measured property with a deadline rather than a closed ticket. The practical form is the one already used for freshness: a check that runs after the deletion window and asserts the subject is absent from every registered destination, with a defined time to completion and an alert when a destination fails to confirm. This matters because the failure mode is silent by construction — a partially completed deletion produces no error, looks identical to a completed one in the ticketing system, and surfaces only when someone queries the wrong copy.

What would make this wrong

If your estate has no derived-representation layer, boundary two does not apply to you. A warehouse with BI on top and no retrieval stack, no embeddings and no fine-tuning has a deletion problem that is genuinely mostly storage-and-backups — and for that estate the vendor answer is close to sufficient. Half this argument is contingent on an AI layer existing.
The vector numbers are one preprint that I have not reproduced. I am relying on the authors' reported results across three implementations, and specific percentages may not generalize to other index types, other configurations, or systems that rebuild on delete. The structural claim — soft delete as a metadata operation — is what I would defend; the percentages are theirs, not mine.
If your platform enforces short history retention and aggressive file cleanup by default, boundary one narrows to a few days. Configuration matters more than architecture here. An estate that vacuums daily and holds no long backups has a materially smaller window than the thirty-day default I used as the example.
Value-level lineage may stop being the missing piece. If column-level lineage tooling matures into row- or subject-level provenance, the traversal I describe as manual becomes a query, and the hand-written register in step 3 becomes the only remaining gap rather than the main one.
If nobody will fund the completeness check, this reframe changes nothing. Steps 1 through 4 are analysis; step 5 is the only one that creates a failure signal. An organization that adopts the traversal and skips the SLA has bought a better mental model and exactly the same silent failure it had before.
If deletions in your context are rare and manual, a register plus a checklist may beat all of this. The traversal framing earns its cost at volume. Three requests a year do not justify a governed deletion pipeline, and saying so is more honest than pretending every estate needs one.