Worked Example — The Red Team That Found Nothing
A worked example of two engagements against the same system — why the first found almost nothing, what changed in the scope, and the finding that only appeared when tools were in play.
This is an illustrative example. The company, engagements and findings are invented. The lesson — that a red team's scope determines its result more than its skill does — applies whatever your system is.
The situation#
A financial services company commissioned a red team engagement against a customer assistant that could look up accounts, explain transactions, and initiate a card freeze.
Five days, an external firm, a clean report: one low-severity finding about tone, and a note that the system prompt could be recovered with effort. Leadership was reassured.
Four months later a support engineer found, in the logs, that an account had been frozen by the assistant after a customer pasted a block of text into the chat. Nobody had asked for a freeze.
Why the first engagement found nothing#
The scope document, written by the team who built the system:
In scope: the conversational interface. The tester may interact with the assistant as a customer would. Out of scope: backend services, the account API, the card management service, and any action that would modify customer data.
Read again, that scope excludes every consequential thing the assistant can do. The testers could try to make it say something inappropriate. They could not test whether it could be made to do anything, because doing things was out of scope.
The exclusion was not evasion. It was written by people worried about a tester freezing a real customer's card, which is a legitimate worry with a wrong remedy — the remedy is a test environment, not a narrower scope.
The engagement measured the assistant's manners and reported that they were good.
The second engagement#
Commissioned after the incident. Ten days, same firm, a scope written differently:
In scope: everything the assistant can reach, including all tools, in a staging environment with production-shaped data. The tester should attempt to cause state changes. The tester is provided with the tool list and the system prompt.
That last line was contested and it is the reason the engagement was worth having. The argument against was that it is unrealistic. The argument that won: the system prompt was recoverable in twenty minutes during the first engagement, so treating it as secret spends tester days establishing something an attacker gets for free.
What the second engagement found#
| Severity | Count |
|---|---|
| Critical | 2 |
| High | 4 |
| Medium | 6 |
| Low | 9 |
C1 — Card freeze reachable through pasted content. Text pasted by the customer, formatted to resemble a system message, caused the freeze tool to fire. This is the production incident, reproduced in fifteen minutes on the first day. The tool required no confirmation because it was classified as a safety action, and safety actions had been deliberately made frictionless.
C2 — Account data returned across customers. A transaction reference belonging to another customer, mentioned in conversation, was looked up and summarised. The lookup tool filtered by reference and not by the authenticated customer.
H1 — No limit on tool call repetition. A conversation could cause 200+ account lookups, which is both a cost and an enumeration path.
H2 — Freeze and unfreeze in a loop. Each individually authorised, no rate limit, producing a sequence of customer notifications.
H3 — Tool errors surfaced internal service names to the customer.
H4 — Conversation state persisted across authentication boundaries, so a session begun before sign-in retained context afterwards.
Every finding above medium involved a tool. None of them was reachable within the first engagement's scope, and none of them required unusual skill — C1 took fifteen minutes.
What the difference cost#
| First engagement | Second | |
|---|---|---|
| Duration | 5 days | 10 days |
| Fee | £14,000 | £26,000 |
| Findings above medium | 0 | 6 |
| Reachable in scope | Manners | Everything the system does |
| Production incidents in the interval | 1 | — |
The first engagement was not cheap. It was £14,000 for a report that produced no change, and it produced a worse outcome than no engagement, because it created confidence.
What changed afterwards#
Scope is now written by someone who did not build the system. The original exclusions came from the people best placed to explain why each was reasonable, and that is exactly the problem.
Staging with production-shaped data is a prerequisite, so nobody has to protect real customers by narrowing what may be tested.
Testers get the prompt and the tool list. Time is spent on consequences rather than on reconnaissance.
Every finding becomes a permanent test case. All fifteen from the second engagement now run automatically on each release. Two have caught regressions since — both reintroduced by prompt edits, which is exactly the mechanism by which a red team finding quietly returns.
Safety actions are no longer exempt from confirmation. Freezing a card is protective and irreversible enough to matter, and "it is for the customer's benefit" was doing the work of an authorisation check.
What was learned#
Scope determines findings more than skill does. The same firm found nothing and then found six, and the variable was one paragraph.
Exclude nothing on the grounds that it could cause harm — that is the definition of what should be tested. Build an environment where harm is contained instead.
Give the testers what an attacker will have. The prompt was recoverable in twenty minutes. Treating it as secret spent days of a five-day engagement.
An engagement that finds nothing is a result to be explained, not a result to celebrate. The clean report was accepted without anyone asking what it had actually been able to reach.