Direct answer: Audit an AI prototype for production by testing the complete user journey, evaluating realistic cases, checking data boundaries and security, and setting release gates for reliability, cost, monitoring and ownership.
An AI prototype proves that a promising interaction is possible. A production product has to work for real users, with real data, under normal operating conditions. The audit should therefore answer a more demanding question: can the team deliver the intended outcome repeatedly, safely and at an acceptable cost?
If the concept still needs product definition, start with AI development and our AI MVP development guide. The audit below turns a demonstration into a decision about what to build next.
1. Test the complete user journey
Identify the user, the task they need to complete and the point at which the AI contributes value. Walk through the journey from input to action, including sign-in, permissions, human review, error handling and support. If the prototype only shows a polished answer, it has not yet demonstrated the surrounding product workflow.
Write acceptance criteria for the task rather than for the model alone. A successful result might require a user to find the correct document, understand the answer, check its source and complete the next step without leaving the product.
2. Build an evaluation set from realistic work
Collect examples that represent common requests, difficult cases, missing information and deliberately misleading inputs. Define what a good result looks like and what failure requires escalation. Record the current baseline so prompt, model and retrieval changes can be compared against the same cases.
The NIST AI Risk Management Framework provides a useful structure for governing, mapping, measuring and managing AI risks. The audit should turn that structure into concrete tests for this product, with an owner for each important failure mode.
3. Check data boundaries and security
Trace what the system can read, what it can write and whose authority it uses. Test whether one user’s information can appear in another user’s answer. Check how instructions embedded in retrieved material are handled, how secrets are stored and how output is constrained before it triggers an action. The OWASP guidance for large language model applications is a practical starting point for threat review.
For retrieval-based products, evaluate permissions at retrieval time, not only in the interface. For agentic workflows, require explicit limits on tools, approvals and irreversible actions.
4. Verify the operating model
- Reliability: Define timeouts, retries and a fallback when the model or an integration is unavailable.
- Cost: Measure the cost of representative tasks, including retrieval, model calls, review and support.
- Observability: Log enough to diagnose failures without collecting unnecessary personal or confidential data.
- Ownership: Name the people who approve changes, monitor quality and respond to incidents.
Decide what to do next
The outcome of an audit is a scoped decision. Some prototypes need a narrower first release; others need better data permissions, stronger evaluation or a redesigned workflow. Record the gaps, their impact, the evidence required to close them and the release gate for each. That gives the team a path from a convincing demonstration to a product it can operate.
Frequently asked questions
What makes an AI prototype ready for production?
It is ready when the complete user journey meets defined acceptance criteria, realistic evaluations expose manageable failure modes, access controls work, and the team can operate and support the system at an acceptable cost.
When should the evaluation be repeated?
Repeat the relevant tests after meaningful changes to the model, prompts, retrieval data, permissions, tools or user workflow, and use the same baseline cases to detect regressions.
