Direct answer: A business retrieval-augmented generation (RAG) system is ready for controlled use when it retrieves the right evidence, respects each user’s document permissions, signals uncertainty and can be monitored and corrected. Test those properties with representative business questions before widening access.
Why a good demo is not enough
A demo usually uses clean questions and a small document set. Production users ask incomplete questions, use local terminology and expect answers from material that changes. A credible evaluation therefore needs a versioned test set drawn from actual work: questions with known supporting documents, questions the system should refuse, ambiguous requests and requests from users with different access rights. Keep personal or confidential examples out of shared test fixtures unless the testing environment is approved for them.
Define answer quality in terms of evidence
Separate retrieval quality from answer quality. For retrieval, ask whether the right source appears in the candidate set and whether obsolete or irrelevant versions are excluded. For answers, check whether every important statement is supported by retrieved text, whether the system identifies a missing answer and whether its citations point to the passage that supports the claim. A fluent answer can still be wrong; a useful review records the source passage and the reason a response passed or failed. The OWASP guidance for LLM applications highlights risks around prompt injection and sensitive information disclosure. Include adversarial documents and untrusted instructions in the test set, and verify that retrieved content cannot override the assistant’s rules.
Make permissions part of retrieval
Decide where authorization is enforced before selecting a vector database. The safest design applies the same access decision to search results that the source system applies to the underlying document. Check group changes, revoked access, document deletion, shared folders and cached answers. A user should not receive a passage merely because another user could retrieve it. Test with two accounts that have overlapping but different access; include a document that was once allowed and is now revoked. Log the document identifiers used to form a response without logging private document content unnecessarily.
Set a release gate
Build a small evaluation matrix: task type, source freshness, expected evidence, permitted audience and failure severity. Review a representative sample manually, then rerun the same cases after prompt, index, embedding or model changes. Do not collapse every failure into a single average score. A permission leak and a harmless formatting error have different consequences. Define a rollback path for a bad index or connector update, and make the owner of each source system responsible for its access rules.
Monitor the system after launch
Track failed retrievals, unanswered questions, stale citations, permission denials, latency and user corrections. Review incidents with the exact index and prompt versions used at the time. The NIST AI Risk Management Framework provides a structure for mapping, measuring and managing AI risks; use it to assign owners and review intervals rather than treating a launch checklist as permanent assurance. Start with a bounded audience, inspect the early questions and expand only after recurring failures have an owner and a repair route.
A practical first sprint
Choose one business workflow and one approved document collection. Write 30–50 representative questions, including questions that should return no answer and questions that differ only by user permissions. Agree on evidence and access pass criteria, run a baseline, fix the largest failure class and repeat. For a build plan that connects evaluation to delivery, explore our AI development service and the AI MVP guide.
Frequently asked questions
How do we know whether a RAG answer is trustworthy?
Check that the cited passage supports the answer, that the passage is current and that the requesting user was permitted to see it. A citation alone is not proof of correctness.
Should every business document be indexed?
No. Start with an approved collection that has an owner, a retention rule and clear access controls. Expand only when the evaluation shows that the additional source improves the target workflow.
What should happen when no reliable source is found?
The assistant should say that it cannot answer from the available evidence and provide an appropriate route to a human or source system. It should not invent a definitive answer.