Learn
What is agent evaluation?
Agent evaluation measures whether an agent role performs acceptably — with test cases, rubrics, regression runs, and production sampling — before and after skill changes. It turns 'the demo looked fine' into pass/fail evidence.
In depth
LLM outputs are stochastic. Agents add tools, memory, and multi-step failure modes. Evaluation is how teams ship safely: a fixed set of inputs with expected properties (not always exact text), scored by rules or judges, run on every skill change.
Layers: unit evals — single-step tool calls; scenario evals — end-to-end task with mock connectors; production sampling — human review of live runs; red team — adversarial prompts and injection attempts.
Good eval sets include edge cases you already got wrong once — those become permanent regression tests. Track acceptance rate, correction categories, and escalation rate over time.
Neuro OS roles carry eval sets in git alongside skills — change the procedure, run the eval, then promote to production.
Examples
- Support role — 50 tickets with expected category and draft quality rubric
- Legal first reader — citations must match source paragraphs
- Red team — injection strings in retrieved docs must not exfiltrate secrets
- CI job — skill PR fails if acceptance rate drops below threshold
- Weekly sample — human scores 20 live runs
Related terms
FAQ
Eval vs monitoring?
Eval is proactive regression testing. Monitoring is reactive observation of production.
How many test cases?
Start with 10–30 real failures you have seen; grow with every incident.
Run governed agent roles on a company OS — not only definitions in a glossary.