The Files Your AI Can Read

AI tools that know your business aren't smarter. They're better at finding the right answer at the right time. Here's how that works, and where it goes wrong.

The Difference Between Knowing and Looking It Up

There is a version of AI that knows only what it was trained on. Ask it about your company's refund policy and it will make something up, confidently. Ask it about a document you uploaded last week and it will have no idea what you're talking about. This is the AI most people first encountered: capable in general, useless for anything specific.

And then there is the version that can look things up for itself.

Retrieval-Augmented Generation (RAG) is the technology behind it. It is why AI tools can now answer questions about your specific organization, your specific documents, your specific data. It is what separates a general-purpose chatbot from something that knows your world.

Understanding how it works matters, not because you need to build it, but because you are almost certainly already using it, and the security implications are significant.

How RAG Works

The name is technical but the concept is simple. Think of the difference between two employees:

Employee A went through training three months ago and has been answering questions from memory ever since. They are confident. They are sometimes wrong. They cannot tell you anything that wasn't covered in their onboarding.

Employee B has the same training, but before answering any question, they quickly check the company handbook, the relevant policy document, or the latest internal memo. Their answers are grounded in current information.

RAG makes AI more like Employee B.

Here is what happens under the hood: when you ask a RAG-powered AI a question, the system does two things before generating a response. First, it searches a connected knowledge base (your documents, your database, your files) for content relevant to your question. Second, it hands that retrieved content to the AI model along with your question, and the model uses both to generate its answer.

The model is not smarter. It just has better information to work with in the moment.

Why RAG Changes What AI Can Do For You

RAG is what makes enterprise AI useful in the first place. Without it, every AI deployment is a generalist that knows nothing about your organization. With it, AI can answer questions about your policies, summarize your internal documents, help onboard new employees, surface relevant precedents, and work with your own data rather than guessing.

The business case is compelling. The adoption has been rapid. Most major AI platforms, including Microsoft Copilot, Google Workspace AI, Salesforce Einstein, and many others, use some form of RAG to ground their responses in your organizational data.

If you use any of these tools, you are using RAG.

Where the Security Risk Lives

RAG introduces three security concerns your organization should be thinking about.

Access control carries over, or it doesn't

When an AI system retrieves documents to answer a question, the question is: does it respect the permissions that already exist on those documents?

If a junior employee asks a RAG-powered AI about executive compensation and the AI retrieves and summarizes a document that employee should never have been able to access. That is a real exposure. Not a hack. Not a breach. Just the AI doing its job without appropriate guardrails.

The fix is ensuring your RAG system enforces the same access controls as your underlying document system. Not all of them do by default.

Garbage in, garbage out, and now it's authoritative

A RAG system is only as reliable as the knowledge base it retrieves from. Outdated policies, incorrect information, or deliberately poisoned documents will produce confident, well-formatted answers that are wrong.

This matters especially in regulated industries. An AI that retrieves and cites an outdated compliance policy is not helpful. It is a liability.

What goes into the knowledge base

When you enable RAG for an AI tool, you are connecting it to a set of documents or data. The question worth asking: does everyone who authorized that connection understand exactly what is in scope? Sensitive employee data, unreleased financial information, legal documents: if they are in the connected knowledge base, they are potentially retrievable.

Safe Harbor: Three Things You Can Do This Week

• Find out if the AI tools you use are RAG-powered. Look for features described as "connected to your documents," "grounded in your data," or "integrated with your knowledge base." If those phrases appear, RAG is involved.
• Ask who controls the knowledge base. For any RAG-powered tool your organization uses, find out what documents are connected, who has access to add or remove content, and how often it is reviewed for accuracy.
• Test it with a sensitive question. Ask the AI a question about something you know should be restricted: a document only certain people should see, a policy that hasn't been published yet. The answer will tell you a lot about whether access controls are working.

Next week: Not all "open" AI is as open as it sounds. When a model gets called "open source", what you actually get is often something narrower, and the gap between open source and open weights changes what you can trust, run, and build on.