Agent Run Review — Sample
A worked example of reviewing an agent in production — completion and intervention rates, cost per completed task rather than per call, and the failure shapes that only appear in the transcripts.
Markdown. No sign-up, no email.
This is an illustrative example. The agent, tasks and figures below are invented to show the shape of a review that tells you whether to widen an agent's remit or narrow it. Copy the structure; produce your own numbers.
The measure that matters is not how often the agent succeeded. It is cost per completed task, including the human time spent when it did not complete — because that is the number that decides whether the agent is helping.
Support triage agent — run review, July 2026#
| Agent | Classifies inbound support tickets, gathers context, drafts a first response |
| Period | 1–31 July 2026 |
| Volume | 4,180 tasks |
| Autonomy | Drafts only. No message is sent without a human approving it. |
| Reviewed by | Support lead and platform engineer |
1. Outcomes#
| Outcome | Count | Share |
|---|---|---|
| Completed, approved as drafted | 2,510 | 60% |
| Completed, edited before sending | 1,003 | 24% |
| Stopped on budget, handed over | 397 | 9.5% |
| Escalated by design | 187 | 4.5% |
| Failed — tool error, unrecovered | 83 | 2% |
Eighty-four per cent produced something a human sent. The 24% edited is not a failure mode; a draft that needs a sentence changed still removed most of the work.
2. Cost, the way it should be counted#
| Per task | |
|---|---|
| Model and tool cost, all 4,180 tasks | $0.19 |
| Human review time, approved as drafted | 40 seconds |
| Human edit time, edited before sending | 3.5 minutes |
| Human time on handovers and failures | 9 minutes |
| Fully loaded cost per completed task | $1.42 |
| Cost of the same task handled entirely by a person | $4.10 |
Note where the money is. Model spend is 13% of the loaded cost. The dominant term is human time on the 11.5% of tasks the agent did not finish — 480 tasks consuming 72 hours, and those tasks are more expensive than if the agent had never touched them, because someone had to read the transcript before starting.
Optimising the model spend would be optimising the smallest term. The improvement worth having is fewer handovers.
3. The control ring, in practice#
| Control | Setting | Fired |
|---|---|---|
| Step budget | 12 tool calls per task | 397 times (9.5%) |
| Cost ceiling | $2.00 per task | 4 times |
| Approval gate | Every outbound message | 4,180 times |
| Audit log | Every call and result | Always |
The step budget is doing the work. Without it, those 397 tasks would not have stopped — they would have continued, and the cost ceiling would have caught them later and more expensively. The ceiling firing only 4 times is the sign of a step budget set correctly.
The approval gate has never been bypassed, and there is no mechanism to bypass it. That is what makes the 2% tool-failure rate tolerable: no failure reached a customer.
4. Why the 397 handovers happened#
From the transcripts. This is the section that could not be produced without the audit log.
| Cause | Tasks | Shape in the transcript |
|---|---|---|
| Customer record not found | 141 | Repeated lookups with slight spelling variations |
| Question spanned two systems | 96 | Correct answer for one, no tool for the other |
| Ambiguous request | 74 | Two plausible readings, agent picked one and hedged |
| Tool returned an unhelpful error | 48 | Retried the same call, unchanged, until the budget ran out |
| Genuinely novel problem | 38 | Correct handover — worked as designed |
Only 38 of 397 are the agent behaving correctly. The rest are addressable, and two causes account for 237.
The 141 lookup failures have a single fix: the search tool requires an exact match, so a transposed character produces "not found" rather than "did you mean". The agent's response — try again slightly differently — is reasonable and doomed. Fuzzy matching in the tool closes this without touching the agent.
The 48 unhelpful errors are a lesson in tool design. The tool returns error: request failed with no indication whether retrying could help. The agent retried, unchanged, until it ran out of steps. A tool that cannot say whether an error is retryable makes every agent using it waste its budget.
5. What the transcripts show that the numbers do not#
Repetition is the failure signature. Every one of the 397 stopped tasks shows the same action repeating with cosmetic variation. Detecting that directly — compare the current action with the last three — would stop these tasks at step 5 instead of step 12, cutting model spend on failures by half and, more importantly, reaching a human sooner.
The agent hedges when it should ask. In the 74 ambiguous cases it chose an interpretation and qualified the answer, producing a draft that reads as unsure and always needed editing. Asking a clarifying question is not currently something it can do. It should be.
Nothing in the log suggests goal drift. In 4,180 tasks the agent stayed on the task it was given. The failures are capability and tooling failures, not control failures. That is worth stating plainly, because it is the fear that dominates discussion of agents and it is not what the record shows here.
6. Recommendations#
- Add fuzzy matching to customer lookup. Closes 141 handovers. A tool change, not an agent change, and the cheapest item here by a wide margin.
- Make tool errors say whether a retry could help. Closes 48 handovers and stops the most wasteful pattern in the log.
- Detect repetition and stop early. Halves the model cost of failures and gets a person involved sooner.
- Let the agent ask one clarifying question before choosing between readings.
- Keep the approval gate. It has fired 4,180 times and prevented every tool failure from reaching a customer. There is no case for removing it on this evidence.
- Do not widen the remit this quarter. Fix the four items above and review again; the handover rate is the gate on any expansion.
Notes on using this format#
Count cost per completed task, loaded with human time. Cost per call flatters every agent ever built. The loaded figure is the one that decides whether to keep going.
Read the transcripts, not just the counts. Every actionable finding here came from transcripts, and two of the four fixes are tool changes that no amount of prompt engineering would have found.
Say when the controls worked. A step budget that fires 397 times looks like a problem in a summary. It is the reason there was no runaway task and no bill to explain.