# Agent Run Review — Sample

**This is an illustrative example.** The agent, tasks and figures below are invented to show
the shape of a review that tells you whether to widen an agent's remit or narrow it. Copy the
structure; produce your own numbers.

The measure that matters is not how often the agent succeeded. It is **cost per completed
task, including the human time spent when it did not complete** — because that is the number
that decides whether the agent is helping.

---

## Support triage agent — run review, July 2026

| | |
|---|---|
| Agent | Classifies inbound support tickets, gathers context, drafts a first response |
| Period | 1–31 July 2026 |
| Volume | 4,180 tasks |
| Autonomy | Drafts only. No message is sent without a human approving it. |
| Reviewed by | Support lead and platform engineer |

## 1. Outcomes

| Outcome | Count | Share |
|---|---|---|
| Completed, approved as drafted | 2,510 | 60% |
| Completed, edited before sending | 1,003 | 24% |
| Stopped on budget, handed over | 397 | 9.5% |
| Escalated by design | 187 | 4.5% |
| Failed — tool error, unrecovered | 83 | 2% |

**Eighty-four per cent produced something a human sent.** The 24% edited is not a failure
mode; a draft that needs a sentence changed still removed most of the work.

## 2. Cost, the way it should be counted

| | Per task |
|---|---|
| Model and tool cost, all 4,180 tasks | $0.19 |
| Human review time, approved as drafted | 40 seconds |
| Human edit time, edited before sending | 3.5 minutes |
| Human time on handovers and failures | 9 minutes |
| **Fully loaded cost per completed task** | **$1.42** |
| Cost of the same task handled entirely by a person | $4.10 |

Note where the money is. **Model spend is 13% of the loaded cost.** The dominant term is human
time on the 11.5% of tasks the agent did not finish — 480 tasks consuming 72 hours, and those
tasks are more expensive than if the agent had never touched them, because someone had to read
the transcript before starting.

Optimising the model spend would be optimising the smallest term. The improvement worth having
is fewer handovers.

## 3. The control ring, in practice

| Control | Setting | Fired |
|---|---|---|
| Step budget | 12 tool calls per task | 397 times (9.5%) |
| Cost ceiling | $2.00 per task | 4 times |
| Approval gate | Every outbound message | 4,180 times |
| Audit log | Every call and result | Always |

**The step budget is doing the work.** Without it, those 397 tasks would not have stopped —
they would have continued, and the cost ceiling would have caught them later and more
expensively. The ceiling firing only 4 times is the sign of a step budget set correctly.

The approval gate has never been bypassed, and there is no mechanism to bypass it. That is
what makes the 2% tool-failure rate tolerable: no failure reached a customer.

## 4. Why the 397 handovers happened

From the transcripts. This is the section that could not be produced without the audit log.

| Cause | Tasks | Shape in the transcript |
|---|---|---|
| Customer record not found | 141 | Repeated lookups with slight spelling variations |
| Question spanned two systems | 96 | Correct answer for one, no tool for the other |
| Ambiguous request | 74 | Two plausible readings, agent picked one and hedged |
| Tool returned an unhelpful error | 48 | Retried the same call, unchanged, until the budget ran out |
| Genuinely novel problem | 38 | Correct handover — worked as designed |

**Only 38 of 397 are the agent behaving correctly.** The rest are addressable, and two causes
account for 237.

The 141 lookup failures have a single fix: the search tool requires an exact match, so a
transposed character produces "not found" rather than "did you mean". The agent's response —
try again slightly differently — is reasonable and doomed. Fuzzy matching in the tool closes
this without touching the agent.

The 48 unhelpful errors are a lesson in tool design. The tool returns `error: request failed`
with no indication whether retrying could help. The agent retried, unchanged, until it ran out
of steps. **A tool that cannot say whether an error is retryable makes every agent using it
waste its budget.**

## 5. What the transcripts show that the numbers do not

**Repetition is the failure signature.** Every one of the 397 stopped tasks shows the same
action repeating with cosmetic variation. Detecting that directly — compare the current action
with the last three — would stop these tasks at step 5 instead of step 12, cutting model spend
on failures by half and, more importantly, reaching a human sooner.

**The agent hedges when it should ask.** In the 74 ambiguous cases it chose an interpretation
and qualified the answer, producing a draft that reads as unsure and always needed editing.
Asking a clarifying question is not currently something it can do. It should be.

**Nothing in the log suggests goal drift.** In 4,180 tasks the agent stayed on the task it was
given. The failures are capability and tooling failures, not control failures. That is worth
stating plainly, because it is the fear that dominates discussion of agents and it is not what
the record shows here.

## 6. Recommendations

1. **Add fuzzy matching to customer lookup.** Closes 141 handovers. A tool change, not an
   agent change, and the cheapest item here by a wide margin.
2. **Make tool errors say whether a retry could help.** Closes 48 handovers and stops the
   most wasteful pattern in the log.
3. **Detect repetition and stop early.** Halves the model cost of failures and gets a person
   involved sooner.
4. **Let the agent ask one clarifying question** before choosing between readings.
5. **Keep the approval gate.** It has fired 4,180 times and prevented every tool failure from
   reaching a customer. There is no case for removing it on this evidence.
6. **Do not widen the remit this quarter.** Fix the four items above and review again; the
   handover rate is the gate on any expansion.

---

## Notes on using this format

**Count cost per completed task, loaded with human time.** Cost per call flatters every agent
ever built. The loaded figure is the one that decides whether to keep going.

**Read the transcripts, not just the counts.** Every actionable finding here came from
transcripts, and two of the four fixes are tool changes that no amount of prompt engineering
would have found.

**Say when the controls worked.** A step budget that fires 397 times looks like a problem in a
summary. It is the reason there was no runaway task and no bill to explain.