Multi-Tenant RAG Isolation
A legal-research platform serves hundreds of rival firms - design retrieval so no firm's documents can ever surface in another firm's answers, and prove that isolation to a CISO.
Last updated:
A legal-research platform serves hundreds of rival law firms, and no firm's documents may ever surface in another firm's answers. The retrieval design is ordinary; the isolation design is not. What matters is that separation is structural rather than a filter somebody has to remember to add, because the question a CISO asks is never whether you filter by tenant - it is which single mistake could cross tenants, and what stops it.
“Isolation has to be structural, not a WHERE clause you remember to add. Bind tenant identity once from the signed token, make a query without a namespace impossible to express, then assert on every returned chunk and treat a mismatch as a security incident.”
Clarifying Questions (Ask These First ~5 min)
| Question | Why it matters |
|---|---|
| How many tenants, and what is the size distribution? | Hundreds with a long tail of tiny firms rules out one index each on cost alone |
| Is there shared content such as case law and statutes? | Almost always yes, which makes this a shared corpus plus private corpora |
| Do any tenants require their own encryption keys? | Customer-managed keys effectively force physical separation for those tenants |
| Is any model ever trained or tuned on tenant data? | The second isolation boundary, and the one people forget |
| What is the contractual position after a breach? | Determines how much you invest in proof rather than in assurance |
| Any data-residency requirements? | Region sharding sits on top of tenant isolation, not instead of it |
| Who can read logs, traces and error payloads? | Chunk text in an observability tool is a cross-tenant leak path |
Architecture (Draw This)
Say this as you draw it
Key Components (30-second pitch each)
- 1Tenant identity comes from the token - Never a parameter the client can set, never inferred from document content. One place resolves it into a request-scoped context and everything downstream reads it from there, so a new code path cannot invent its own idea of which tenant is asking.
- 2Namespace bound at client construction - The retrieval client is constructed with the namespace already attached, so issuing a query without one is impossible to express rather than merely wrong. Structural impossibility is what survives a new engineer adding a feature in a hurry.
- 3Defence in depth: filter, then assert - Retrieval filters by tenant, and assembly independently re-checks every returned chunk. A mismatch is dropped, alarmed and paged rather than logged. The assertion layer is the thing you show a CISO, because it proves you do not trust your own filter.
- 4Caches are the classic leak path - Embedding caches, prompt caches, semantic answer caches and CDN layers all need tenant_id in the key. A semantic cache returning another firm's answer because the question was similar is precisely the nightmare scenario, and it is invisible in testing with one tenant.
- 5Encryption and key scope drive the topology - Per-tenant customer-managed keys mean that tenant's data cannot share an index with anyone else's. That is why key requirements are a clarifying question - they decide the isolation model rather than being configured on top of it.
- 6Evidence, not assurances - Automated cross-tenant tests on every build, a canary document per tenant that must never appear elsewhere, per-tenant audit logs the tenant can read directly, and a current penetration-test report. This is the actual answer to prove it.
What Separates a Strong Answer
- 1Bind tenant context early - Establishing tenant context at the start of the request flow ensures no part of the process can operate without it, preventing accidental cross-tenant operations.
- 2Assert tenant identity rigorously - Implementing an assertion layer that checks tenant identity at assembly catches any discrepancies, providing a robust defense against data leaks.
- 3Use tenant_id in cache keys - Incorporating tenant_id in cache keys prevents data leakage through cache reuse, a common oversight that can lead to significant security breaches.
- 4Provide tenant-readable audits - Offering per-tenant audit logs that tenants can access builds trust and allows for independent verification of data handling practices.
Isolation Models
| Model | Strengths | Costs |
|---|---|---|
| Index per tenant | Strongest blast-radius story; per-tenant keys and residency become trivial | Hundreds of indexes to operate, and cost per tiny tenant is brutal |
| Shared index, namespace per tenant | One system to run, strong isolation when the namespace is bound structurally | Relies on the vector database honouring namespaces; noisy neighbours share nodes |
| Shared index, metadata filter only | Cheapest and simplest to build | One missing filter is a breach, and it is the hardest model to defend in a security review |
| Shared read-only corpus + per-tenant index | Matches the real data model; no tenant data on the shared side | Two retrieval paths to merge and rank coherently |
Leak Paths and Their Controls
| Path | Control |
|---|---|
| A query issued without a tenant filter | Namespace bound at client construction; a filterless query is rejected by the wrapper |
| Semantic or prompt cache reuse | tenant_id in every cache key, including embedding and CDN layers |
| Fine-tuning or evaluation on pooled data | Contractual prohibition plus a technical gate - tenant data never enters a shared training set |
| Logs, traces and error messages carrying chunks | Redaction at the emitter; identifiers and tenant labels only |
| Support and admin tooling with a global view | Break-glass access, per-tenant approval, fully audited and time-boxed |
| Backups, exports and restores | Scoped per tenant, and restore tested per tenant rather than globally |
3 Biggest Risks
- 1A new code path that forgets the namespace - Every feature added under deadline is a chance to query without a tenant scope. Mitigated structurally - the only client that exists is already bound - plus a CI test that attempts an unscoped query and fails the build if it succeeds.
- 2Cross-tenant cache reuse - Caches are added for performance late in a project, by someone who was not in the isolation design discussion. Mitigated by making tenant_id a mandatory component of the cache key type itself, so an unscoped key does not compile.
- 3The shared corpus quietly accumulating tenant content - Someone promotes a useful tenant-contributed document into the shared corpus to improve answers. Mitigated by making the shared corpus write path a separate, reviewed pipeline with provenance on every document.
Google Stack: Vertex AI Vector Search with per-tenant namespaces or dedicated indexes → CMEK per tenant where contracts require it → IAM conditions carrying tenant context → Gemini → Cloud Logging with redaction and per-tenant log buckets
Azure Stack: Azure AI Search (index per tenant, or a shared index with security filters driven by Entra ID claims) → customer-managed keys per tenant → Azure OpenAI → Microsoft Purview for audit and retention → Front Door with tenant-scoped cache keys
AWS Stack: OpenSearch Serverless collections per tenant, or Bedrock Knowledge Bases with metadata filters → KMS key per tenant → IAM session tags carrying tenant_id so the credential itself is scoped → Bedrock Claude → CloudTrail per-tenant audit trails
Other Options: Pinecone namespaces, Qdrant collections, or Weaviate native multi-tenancy with per-tenant shards. All three make the tenant a first-class object in the API rather than a filter value, which is exactly the property that makes isolation structural.
Frequently asked questions
Hundreds of tenants - index per tenant, or namespaces?
Namespaces by default, index per tenant for those whose contracts demand their own keys or residency. Design for both on day one: the tenant record carries its isolation mode and the retrieval client resolves it. Retrofitting a second mode later is a rewrite of every read path.
How do you actually prove isolation to a CISO?
Four artefacts: an architecture document showing exactly where tenant_id is bound; the assertion layer with its alarm history; a CI suite that attempts cross-tenant retrieval on every build; and a per-tenant canary document monitored to confirm it never appears elsewhere. Plus direct access to their own audit log.
Can all tenants share one embedding model?
Yes. The model is code, not data - it sees tenant text at inference only and retains nothing. Fine-tuning is where that stops being true, which is why pooled fine-tuning is prohibited outright rather than merely discouraged.
What about noisy neighbours?
Per-tenant rate limits and quotas, with a separate pool for the largest tenants. Correctness isolation and performance isolation are different problems: solve the first structurally and the second with quotas, and never let a performance fix weaken the first.
How do you support sharing between two tenants - co-counsel on a matter?
Model it as an explicit grant object with a scope and an expiry, never by copying documents into the other tenant. The grant is evaluated at retrieval time and appears in both tenants' audit logs, so the sharing is revocable and visible on both sides.
What happens when a tenant leaves?
Deletion is contractual and includes derived data: the namespace is dropped, embeddings and caches purged, backups expired on a stated schedule, and a certificate of deletion issued. Vectors derived from their documents are their data and must be covered explicitly.