RAG Not Finding Answers? Five Knowledge Base Problems
This is the most common failure after a RAG knowledge base goes live, and the hardest to track down, because it throws no error. The upload shows success, the admin panel is green, and the chat still cannot find the answer. Below are five causes we traced on production systems, ordered from the outside in.
1. The disk filled up and the vector store went read-only
This is the outermost cause and the one people skip. Something else on the machine fills the disk, the vector database switches itself to read-only, and new content never gets written. The upload screen still says success. The symptom: old content is found, nothing added this week is.
Check the disk level before anything else. It saves hours. On a self-hosted setup, set a disk alert at 80%, before this happens, not after.
2. The document is published, but the knowledge base is not attached to the app
These are two separate switches, and it is easy to flip only one. The document itself has to be in a published state, and the knowledge base has to be inside that app's retrieval scope. Do only the first, and the admin panel is all green while the content is never searched.
3. After a few turns, query rewriting pulls the question off course
Watch for this in any app that rewrites the user's question before searching. A few turns into a conversation, the rewriting step gets pulled by the earlier history and turns the user's words into a different question. Retrieval then misses. The typical symptom: the first turn finds the answer, and the same question asked a few turns later finds nothing.
Log the rewritten query, run real multi-turn conversations through it and read the logs. The drift is obvious on sight.
4. Chunking cut the answer in half
A complete answer spans two chunks. Retrieval hits the first half, the second half never comes back, and the model answers half the question. This one is the sneakiest, because the system does answer, just incompletely, and the business side concludes the model is not smart enough.
The fix is to chunk by meaning and by heading, so a complete entry is never split. For FAQ content the most stable form is one question and one answer per section. After changing how you chunk, re-import the content. Changing the setting without re-running the import does nothing.
5. Old and new versions sit in the knowledge base together
The old document was never taken down, the new policy went in beside it, and retrieval returns both. Which one the model follows is a coin toss. This is worse than not finding anything: the answer looks correct, and the customer receives a policy that no longer applies.
The fix is not on the retrieval side. It is on the content side. Every piece of knowledge needs a clear status, live or retired, and an update has to retire the old version, not upload a second copy beside it.
Why RAG fails quietly
All five causes share one trait: every dashboard says things are fine. Upload status, indexing status and app status are each green on their own, while the path from a real question to the right paragraph is broken somewhere between them. That is why checking the admin panel tells you nothing. The only test that counts is asking the question the way a user asks it and reading what came back.
The same pattern shows up beyond retrieval. When a lookup fails and the model receives an empty result, it reads "nothing found" as a confirmed fact and answers with confidence. We cover that and other production failures in why AI agents fail in production.
Make acceptance a fixed three-step check
- Ask a real question inside the app and confirm the new content is in the retrieved results. Do not trust the check mark in the knowledge base admin.
- Print the retrieved text and read it. Confirm it hit the complete passage, not half of it.
- Every time you add knowledge, re-run questions the system already answered correctly, and confirm the new content did not push the right answers out.
These three steps take ten minutes. They save you from finding out two weeks after launch that customers have been receiving an outdated policy the whole time.
Where this matters most
In customer service, a knowledge base gap turns straight into a wrong answer to a customer. On the live system in our customer service case study, the AI searches the knowledge base for product facts, published policies, troubleshooting steps and video guides on every email before drafting a reply, so retrieval quality decides reply quality. If your knowledge base is live and still not answering, the free plan maps where it breaks in two working days.