Accuracy is not one number
A model can produce fluent answers and still fail the product. It may use the wrong customer record, call the correct tool with an invalid argument, omit a required caveat, or succeed at a task that should have been escalated.
Evaluation should therefore follow the workflow. Measure the decisions and artifacts the application depends on, not only the perceived quality of a final paragraph.
Build the test set from risk
A compact initial evaluation set should include more than happy paths.
- Representative successful cases taken from the intended workflow.
- Boundary inputs that are valid but easy to misinterpret.
- Missing, stale, conflicting, and unauthorized context.
- Tool failures, timeouts, retries, and partially completed actions.
- Cases where the only correct behavior is to ask, stop, or escalate.
Evaluate components and outcomes
Component checks catch structured-output errors, invalid tool arguments, retrieval mistakes, and policy breaches. End-to-end checks ask whether the user reached the correct business outcome. A system needs both because a run can pass every local check and still produce the wrong overall result.
The evaluation set should become part of release discipline. Model, prompt, retrieval, tool, and policy changes all rerun the same important cases before production traffic becomes the test suite.
Observability completes the loop
Pre-release examples cannot predict every real input. Production traces should reveal where users correct the system, which tools fail, where latency accumulates, and when costs change. Those observations become new evaluation cases, turning incidents and near-misses into durable coverage.