AI Data Residency: Where Your Prompts and Embeddings Actually Live

When a prompt leaves the building it becomes four separate artifacts with four different retention policies, four different storage locations, and four different answers to the question of who can compel access, and most enterprise AI programs have inventoried exactly one of them. The prompt and its retrieved context go one place. The completion goes another. The embeddings generated to make retrieval work sit in a vector store that nobody classified as sensitive because it holds numbers rather than text. And the logs and traces, which are the artifacts teams forget entirely, persist longest of all, frequently in a different region under a different vendor's retention schedule. A regional endpoint addresses the first of these and is routinely presented as addressing all four. This piece separates the artifacts, explains what in-region processing does and does not guarantee, walks the deployment ladder from self-hosted weights to default multi-tenant, and gives a decision framework organized by data class rather than by vendor promise.

Key Takeaways

• Four artifacts, not one. Prompts and retrieved context, completions, embeddings, and logs and traces each have their own location and retention policy. Logs are the most commonly overlooked and typically the longest-lived.
• Inference in region does not mean storage in region. A regional endpoint guarantees where the computation happens. Abuse-monitoring retention, safety review, and support tooling are separate policies that frequently route elsewhere unless contractually excluded.
• Zero retention is a contract term, not an architecture. It is enforceable and worth having, and it describes a commitment about vendor behavior rather than a technical impossibility of retention.
• Embeddings are derived data that can leak their source. Inversion research has repeatedly recovered substantial portions of original text from embedding vectors, which makes a vector store in the wrong jurisdiction a residency problem rather than a latency one.
• In-region processing does not by itself resolve a jurisdictional demand. Where a provider is incorporated and which entity controls the keys matter alongside where the servers sit, which is why a local subsidiary contract is a starting point rather than an answer.

The Four Artifacts

Data residency for AI systems is the question of which physical and legal jurisdiction holds each artifact a model interaction produces, and the answer differs per artifact rather than per vendor. Treating an AI deployment as a single data flow is the error that produces surprises during audit.

The retrieved context deserves particular attention because it inverts the usual sensitivity assumption. The user's typed question may be innocuous. What the retrieval layer attaches to it, pulled from internal document stores, can be the most regulated material the organization holds, and it travels in the same payload. An access control model that governs the document store while ignoring what retrieval injects into prompts is not governing the sensitive path.

Logs are the artifact that causes the most audit findings, for a mundane reason. Request and response bodies get logged during development to debug quality problems, the logging is never removed, and the observability platform holds a complete transcript of every model interaction under a retention period chosen for operational convenience. The telemetry cost pressures that push organizations toward sampling and tiering, discussed in the observability cost problem, are the same pressures that determine how long that transcript survives and where it is copied.

What a Regional Endpoint Guarantees

Providers offer regional deployment, and the guarantee is genuine but narrower than the marketing implies.

What it does guarantee: the inference computation occurs on infrastructure in the named region, and the request is routed there rather than to a default location.

What it does not automatically guarantee: that no copy of the request is retained elsewhere for abuse monitoring or safety review, that support staff accessing a troubleshooting ticket are located in region, that the model weights themselves were trained or are served from region, or that the vendor's own logging and telemetry stay in region.

The abuse-monitoring exception is the one that surprises organizations most often. Major providers retain prompts and completions for a defined window for safety review, and this is a separate policy from inference residency. It is usually waivable by agreement, frequently requiring an application or a specific contract tier, and it is not waived by default merely because a regional endpoint was selected.

Zero retention is a contract, and that is fine

A zero data retention agreement commits the provider not to store prompts or completions beyond the duration of the request. These agreements are real, enforceable, and worth negotiating.

The honest framing is that this is a promise about vendor behavior rather than a technical guarantee. The request still traverses the vendor's infrastructure, still exists in memory on their hardware, and still passes through whatever systems sit in the path. The control is contractual and auditable, not physical. Organizations whose threat model includes the provider itself, or a legal authority compelling the provider, need a different rung on the ladder rather than a stronger clause.

Where a technical rather than contractual guarantee is required, confidential computing is the relevant mechanism, using hardware-enforced enclaves so that data remains encrypted during processing and is inaccessible to the infrastructure operator. The approach and its limits are covered in confidential computing and hardware enclaves. It is the only option on the list that changes the answer from a promise to a property, and availability for large model inference remains limited.

The Legal Reality, Without the Hype

Three points cover what executives need, and each is regularly overstated in both directions.

Transfer frameworks govern movement, not access. Mechanisms such as adequacy decisions, standard contractual clauses, and the EU to US data privacy framework establish lawful bases for moving personal data across borders. They are necessary and they address a specific legal question. They do not address who can later compel production of the data.

Extraterritorial reach is why a local subsidiary is not a complete answer. Legislation in several jurisdictions, most prominently the US CLOUD Act, allows authorities to compel a provider subject to their jurisdiction to produce data under its control regardless of where that data is physically stored. A contract with a European subsidiary of a US-headquartered provider does not by itself remove the parent's exposure. This is the specific gap that sovereign cloud offerings are built to address.

In-region processing does not resolve a jurisdictional demand. The operative questions are who controls the infrastructure, who holds the encryption keys, and which legal entity would receive the order. Servers in Frankfurt operated by a company subject to a foreign production order are a partial control. This is a reason to be precise about what the deployment achieves rather than a reason to avoid regional endpoints.

Sovereign cloud offerings are where this gets genuinely mixed. Some are substantive: locally incorporated operating entities, local personnel with exclusive administrative access, customer-held keys, and a legal structure specifically designed so the foreign parent cannot be compelled to produce customer data. Others are a regional deployment with a sovereignty label and no change to the control chain. The distinguishing question is simple to ask and revealing: which legal entity holds the keys and the administrative access, and can the parent company compel that entity to act.

The Deployment Ladder

The ladder is the decision, and the common mistake is applying one rung to everything. An organization that puts all traffic on self-hosted weights pays a capability penalty on the ninety percent of requests that involve no sensitive data. An organization that puts everything on a default endpoint has made a decision about its most sensitive data without noticing. Routing by data class is what makes the ladder useful, and doing that centrally rather than per application is the architectural reason to run an AI gateway, which is the layer that can enforce the routing rule instead of leaving it to each team's configuration.

The Embedding Problem

Embeddings are not anonymized data. This is the single most common technical misconception in AI residency planning, and it has a direct governance consequence.

An embedding is a numeric vector derived from source text, and it looks like nothing: several hundred floating point numbers with no readable content. That appearance drives a reasonable-seeming conclusion, which is that a vector database holds derived mathematical representations rather than the underlying sensitive documents, so it can be hosted wherever latency and cost suggest.

Embedding inversion research has undermined that conclusion repeatedly. Published work has demonstrated recovery of significant portions of original text from embedding vectors alone, in some settings reconstructing a majority of the source content exactly, particularly where the attacker can query the same embedding model. The practical reading is that an embedding should be classified at or near the sensitivity of its source document, not at the sensitivity of a random number.

Three consequences follow. The vector store is in scope for residency and must sit in the same jurisdiction as the documents it was derived from. The embedding model endpoint is a data flow, since generating the embedding means sending the source text somewhere. And deleting a source document does not delete its embedding, which makes the vector store an independent target for retention and erasure obligations. Teams that understand how vector search and embeddings work operationally often still have not extended their data classification to cover the index, which is where the gap appears in audit.

A Decision Framework by Data Class

Residency decisions should be made once per data class and enforced centrally, rather than per project and negotiated repeatedly.

Public and non-sensitive. Marketing copy, published documentation, general research. Default endpoints are appropriate. Governance effort spent here is effort not spent where it matters.

Internal but unregulated. Meeting notes, internal wikis, code that is not itself a trade secret. A regional commercial endpoint with a zero retention agreement and logging that excludes request bodies covers this adequately.

Regulated personal data. Customer records, health information, financial detail. Regional endpoint at minimum, with a documented transfer basis, zero retention in writing, abuse monitoring explicitly waived, logging that captures metadata and not content, and a vector store in the same jurisdiction. Single-tenant deployment where the regulator or the customer contract expects it.

Restricted and sovereign. Data under explicit localization mandates, classified material, or anything where the threat model includes a foreign government demand. Self-hosted or genuinely sovereign single-tenant only, with customer-held keys. Contractual assurances do not reach this class.

Two implementation notes make the framework survive contact with reality. Classification has to happen at retrieval time rather than only at document ingestion, because the sensitivity of a request is determined by what retrieval attached to it. And the routing decision belongs in infrastructure rather than in policy documentation, because a rule that depends on every engineer reading the policy will be violated by the first team under deadline pressure. The broader operating model for making these decisions once and applying them consistently is the subject of AI governance policy.

Frequently Asked Questions

Where does prompt data actually go when using a commercial LLM API?

The prompt travels to the provider's inference infrastructure in whichever region the endpoint specifies, is processed, and returns a completion. Separately from that, it may be retained for a defined window for abuse monitoring and safety review unless that retention is contractually waived, it may appear in the customer's own application logs and observability platform with a much longer retention period, and any embeddings generated from related documents persist in a vector store whose location was chosen independently. The prompt path is usually the best-governed of these and the least durable.

Does a regional endpoint mean data never leaves the region?

No. It means inference computation happens in that region. Abuse-monitoring retention, safety review processes, support staff access during troubleshooting, and the provider's internal telemetry are governed by separate policies that may route elsewhere unless specifically addressed in the contract. Regional endpoints are a genuine and useful control, and treating them as a complete residency guarantee is the most common error in this area.

Are embeddings considered personal data?

Where the source text was personal data, the safe position is yes. Embedding inversion research has repeatedly demonstrated recovery of substantial portions of original text from vectors alone, so the assumption that embeddings are anonymized derived data does not hold. The practical consequence is that vector stores belong in the same jurisdiction as their source documents and are in scope for deletion and retention obligations independently of those documents.

What does a zero data retention agreement actually guarantee?

It commits the provider contractually not to store prompts or completions beyond the request duration, which typically means abuse-monitoring retention is waived. It is a real and enforceable control worth negotiating. It is a commitment about provider behavior rather than a technical impossibility, so it does not help against a threat model that includes the provider or a legal authority compelling the provider. Those require self-hosting, genuine sovereign arrangements, or confidential computing.

Does hosting in Europe protect data from foreign legal demands?

Only partly. Physical location is one factor; the others are which legal entity operates the infrastructure, who holds the encryption keys, and whether a parent company in another jurisdiction can be compelled to produce data under its control. Legislation with extraterritorial reach, notably the US CLOUD Act, is designed to follow control rather than geography. Genuine sovereign offerings address this through local incorporation, local exclusive administrative access, and customer-held keys, and offerings that change only the server location do not.

The Bottom Line

The useful diagnostic is not whether an organization has chosen a regional endpoint. It is whether anyone can produce, for a given AI application, the location and retention period of all four artifacts. In most enterprises the prompt path has an answer, the log path has an answer nobody likes once they look, and the embedding store has no answer at all because it was never classified as holding sensitive data.

That is a tractable problem, and it is tractable in a specific order. Inventory the four artifacts per application. Classify data once and route by class rather than by project. Push the routing decision into a gateway so it is enforced rather than documented. Negotiate retention terms in writing instead of inferring them from a region selector. And extend data classification to the vector store, which is the artifact most likely to be sitting in the wrong jurisdiction holding a recoverable copy of the documents the entire program was designed to protect.