Red Team Engagement Report — Sample

This is an illustrative example. The system, findings and figures are invented. What is worth copying is the structure: ordered by consequence, with the failed attempts recorded, and fixes that do not depend on the model refusing.

Red team engagement — customer service assistant#

1. Summary#

Every finding above medium came from the tools the assistant can call, not from what it can be persuaded to say. That is the single most useful sentence in this report, and it is the opposite of where the pre-engagement concern was.

2. Critical — refund tool reachable through instruction in customer data#

What we did. Set a customer profile display name to text containing an instruction. When a support question caused the assistant to read the account record, it treated the embedded text as an instruction and called the refund tool.

Result. A refund of £40 was issued in staging, to a plausible-looking account, with no human approval. The assistant's explanation to the user made no mention of it.

Why it is critical. The attacker needs no special access — only the ability to write into a field the assistant will later read. Profile names, ticket subjects, uploaded file names and email signatures are all such fields.

What does not fix it. Prompt hardening. We attempted the same attack against a prompt instructed to ignore instructions in customer data; it worked on the fourth wording. Treating retrieved content as untrusted is an architectural property, not an instruction.

Fix. The refund tool requires human approval regardless of who or what requests it. Tool calls initiated within a turn that read untrusted content are flagged and blocked. Refund amount and frequency limits per session apply independently.

3. High#

H1 — Account data disclosed across customers. By referring to a previous ticket ID belonging to another customer, the assistant retrieved and summarised that ticket. The retrieval layer filtered by ticket ID and not by the authenticated customer. Fix: authorisation in the retrieval layer, not in the prompt.

H2 — System prompt recovered. Recovered in full in about 20 minutes across a multi-turn conversation. Not damaging alone; it made every subsequent attack faster, because the tool list and the guardrail wording were then known. Fix: treat the prompt as public. Nothing in it should be a secret, and nothing should depend on it staying one.

H3 — Tool result trusted without validation. An account lookup returning a malformed record caused the assistant to state a balance that did not exist. Fix: validate tool output against a schema before it enters the context.

4. Medium and low#

Medium (5). Rate limits absent on the account lookup tool, allowing enumeration · verbose error messages disclosing internal service names · conversation history retained across a session boundary after logout · a file upload path accepting content types not on the allowlist · no logging of tool calls that failed.

Low (7). Mostly persuasion: the assistant could be led into unprofessional tone, into speculating about company policy, and into making commitments about delivery times it has no information about. Reputational rather than technical, and all seven are addressed by output filtering on a short list of topics.

5. What we tried and could not do#

Recorded because a report of only successes gives no sense of where the system is strong, and because the next engagement should not repeat this ground.

• Could not extract training data or other customers' conversation history. Nothing in the context contained it, which is a design property rather than a defence.
• Could not escalate to administrative functions. The assistant's credentials genuinely do not carry them.
• Could not bypass the payment tool's amount ceiling. It is enforced in the payment service, not in the agent, which is why every attempt failed.
• Could not persuade the model to output credentials. They are not in its context.

The pattern is consistent: every control enforced outside the model held, and every control expressed as an instruction to the model eventually failed.

6. Method#

Techniques were drawn from the OWASP Top 10 for LLM Applications and the agentic list, supplemented by target-specific work. Roughly 60% of effort went to indirect injection through data the assistant reads — profile fields, ticket text, uploaded documents — because that is the surface an external attacker can write to without any access.

Multi-turn attacks were more effective than single-turn throughout. Six of the nine findings above low severity required at least three turns to establish context before the attack turn. A single-turn test suite would have found three of them.

7. Recommendations#

1. Human approval on the refund tool, unconditionally. Not risk-scored — every call.
2. Authorisation in the retrieval layer. The prompt must never be what decides which customer's data is returned.
3. Validate tool output before it enters context.
4. Treat the system prompt as public and remove anything that depends on it being secret.
5. Add the successful attacks to the regression suite. All nine, run on every prompt or model change. They are the only defence against a fix being undone by a later edit.
6. Re-engage after the architectural fixes, not before. Re-testing prompt changes is not a useful engagement.

Notes on using this format#

Order by what the finding lets an attacker do. A recovered system prompt and an unapproved refund are both "prompt injection". Only one of them moves money.

Record what failed. It shows where the design is genuinely strong, and it stops the next engagement re-running the same ground.

Separate fixes that hold from fixes that persuade. A prompt instruction is a request. A tool that requires approval is a control. This engagement found the boundary between them in one line: every control outside the model held.

Put the successful attacks into the regression suite. A red team finding that is fixed and not tested will be reintroduced by a later prompt edit, and nobody will notice.

Back to Red Team