Every enterprise RAG project I've worked on hits the same wall: the first security review. The architecture looks clean — documents in a vector store, a retrieval step, an LLM generating answers. But when the security team asks "how do you ensure User A can't retrieve chunks from documents User B has no access to?", most implementations don't have a good answer. Here's how I build RAG systems that do.

Why Most RAG Implementations Fail Security Review

The naive approach to document permissions in RAG is to apply access control at the document level before ingestion — only index documents a user has access to. This works for small, static deployments but breaks down immediately in enterprise environments where:

  • The same vector index serves multiple users with different permission sets
  • Document permissions change after ingestion (a file moves from Public to Confidential)
  • A single document contains sections with different classification levels
  • The index is shared across multiple projects or business units

Rebuilding per-user indexes is prohibitively expensive at scale. Re-indexing on permission changes is operationally fragile. What you need is chunk-level security filtering at query time.

The Architecture: Chunk-Level RBAC

The pattern I use treats security metadata as a first-class citizen of every chunk in the vector store. When a document is ingested, each chunk carries a security envelope alongside its embedding:

{
  "id": "doc-1234-chunk-042",
  "content": "Q3 revenue projections show...",
  "embedding": [...],
  "security": {
    "project_ids": ["proj-finance-q3"],
    "classification": "confidential",
    "allowed_roles": ["finance-analyst", "senior-manager"],
    "owner_tenant": "nmg-finance"
  }
}

At query time, the retrieval step applies a pre-filter using the calling user's resolved permissions before semantic search runs. On Azure AI Search, this looks like a filter expression passed alongside the vector query:

filter = "security/project_ids/any(p: p eq 'proj-finance-q3')
          and security/classification in ('public', 'internal', 'confidential')"
The critical insight: filtering happens before the ANN search, not after. Post-filtering after retrieving the top-K reduces your effective candidate pool and degrades recall. Most managed vector stores now support pre-filtering — verify this before choosing your stack.

Permission Resolution at the API Boundary

The retrieval filter is only as trustworthy as the permission resolver that builds it. I enforce this at the API gateway layer, never trusting the client to pass its own permissions:

  • Identity is verified via Azure AD / Entra ID token validation at the API gateway
  • Permissions are resolved from a permissions service (not from the request payload)
  • The filter is constructed server-side and injected into every retrieval call
  • The filter is logged alongside the query for audit trail purposes

Any architecture where the client can influence its own retrieval filter is not enterprise-ready.

The Trade-off: Security vs. Retrieval Quality

Pre-filtering reduces the candidate pool, which reduces retrieval recall. If a user only has access to 5% of the indexed corpus, the retrieval quality for that user will be lower than a user with broad access — fewer semantically similar chunks are in scope.

Mitigations I use in practice:

  • Increase top-K on filtered queries — retrieve more candidates to compensate for the smaller pool
  • Tune reranking per permission scope — a cross-encoder reranker after pre-filtering recovers significant recall
  • Chunk overlap strategy — larger chunk overlap ensures semantically rich context doesn't split across permission boundaries
  • Hybrid search — combining keyword (BM25) with vector search improves recall when the vector pool is constrained

Handling Permission Changes After Ingestion

Documents change classification. Projects get decommissioned. Users leave the organisation. The vector store must stay in sync.

I implement this through an event-driven permission sync: a change event from the document management system triggers a targeted metadata update on the affected chunks — not a full re-index. Azure AI Search supports in-place metadata updates via the Merge or MergeOrUpload action, which updates the security fields without touching the embedding. This keeps the operation cheap and near-real-time.

The Audit Trail

Security teams also need to answer: "what chunks were retrieved to generate this response, and who had access to them at that point in time?" I log three things for every RAG call:

  1. The user identity and resolved permission set used to build the filter
  2. The IDs of the chunks retrieved (pre and post reranking)
  3. The security metadata snapshot of each retrieved chunk at retrieval time

This is stored in a structured log (Azure Monitor / Log Analytics in my Azure deployments) and forms the basis for compliance reporting, data lineage queries, and incident investigation.

Security-compliant RAG is not dramatically more complex than unsecured RAG — it's mostly discipline in where you apply controls and ensuring they run server-side. The architectures that fail security review are usually the ones that bolt on access control as an afterthought rather than designing the security envelope into the chunk schema from day one.

If you're designing an enterprise RAG system and want to talk through the specifics of your access model, get in touch.