AI Security — Frequently Asked Questions
Answers on securing AI systems — prompt injection and why it cannot be filtered away, permission-filtered retrieval, guardrail models, what automated red teaming can and cannot tell you, and where to start.
We have not built a red-teaming tool. See AI Security for why the obvious version is not worth shipping.
The threats#
What is prompt injection, concretely?#
Content your system reads that carries instructions aimed at the model — in a web page, a document, a support ticket, a tool response. Everything arrives as one stream of text, and the boundary between "what my operator told me" and "what I just read" is not enforced by anything.
The severity depends entirely on architecture. In a chatbot the result is a bad answer. In a system holding tools it is an action taken on someone else's instruction.
What is indirect injection?#
The harder form: the attack is not in the user's message but in a document the system retrieves later. Testing user input misses it completely, because the user did nothing wrong.
Any system that reads content it did not author is exposed to this.
Can we filter injection out?#
Not reliably. The attacker also writes text, and any filter is something they can write around. Filtering raises the bar; it is not a boundary.
Design so a successful injection has limited effect: narrow tool scope, confirmation on irreversible actions, and permissions enforced outside the model.
What is the highest-severity issue in practice?#
Retrieval that is not filtered by the requesting user's permissions. That turns an assistant into a search engine over everything the organisation holds, exposed through a chat box.
It is a data breach rather than a quality bug, and it is one of the most common real findings.
Is model output a risk to our own systems?#
Yes, and it is frequently overlooked because the output came from your own system. Text rendered into a page, passed to a shell, or used to build a query is untrusted input.
This is an old vulnerability class in new clothing. Escape it, validate it, never pass it unchecked.
Defences#
What actually reduces risk?#
Architecture, not scanning:
Sort tools by reversibility and require confirmation for the irreversible ones. Do not let content the system reads directly trigger a consequential tool. Filter retrieval by the asking user's entitlements at query time. Give the system its own scoped, revocable credentials. Enforce spend and rate caps outside the system. Treat output as untrusted.
Each is a design decision, cheap early and expensive to retrofit.
Are guardrail models enough?#
They raise the bar and they are not a boundary. A guardrail model is itself a model — it can be manipulated and it fails silently.
Use one as a layer, never as the control that permits a risky capability.
Should we run a bug bounty?#
Not until the architectural controls are in place, or you are paying market rate for findings a checklist would have caught.
When you do, write the scope carefully. AI systems attract a long tail of reports along the lines of "I persuaded it to say something embarrassing", which are real but rarely rank as vulnerabilities. Define upfront what counts — data exposure, privilege escalation, unauthorised actions — and what does not, or triage capacity disappears into the tail.
Testing#
Why not just buy an automated red-teaming tool?#
Generating adversarial inputs is easy. Judging whether the system did something wrong is the hard part, and it usually requires another model — one with the same weaknesses, manipulable by the same content, producing confident verdicts either way.
A tool reporting "0 successful attacks" is indistinguishable from one failing to recognise successful attacks. False assurance is worse than no tool.
So what can be automated usefully?#
Anything with an objective answer:
Known-attack regression suites — once an attack works, encode it and re-run it forever, no judge required. Permission tests: does user A's query ever retrieve content only user B should see. Refusal consistency across runs. And output handling, which is ordinary application security testing.
Automate what has a correct answer; use humans for what needs judgement.
How do we test our own system without buying anything?#
Run the four checks above, add every real attack you encounter to a permanent regression suite, and give one person a focused day trying to make the system misbehave.
That day usually outperforms a scanner, because a curious human explores paths a generator does not.
Getting started#
Where should a team start?#
Permission-filtered retrieval if you have a system over internal documents — highest severity, clearest fix. Then narrow the tool scope of anything that acts. Then caps enforced outside the model.
Those three remove most of the realistic risk, and none requires a product.
Are the published risk lists useful?#
Yes, as a checklist of categories rather than a strategy. Note that agent-specific risks — goal hijacking, tool misuse, memory and context poisoning — are now treated separately from the older model-focused lists. If you run agents, the agent list is the more relevant one.
What is the most common mistake?#
Treating AI security as a content-filtering problem. Most of the severe findings are access control and blast radius — who can reach what, and what the system is permitted to do — which are ordinary security questions applied to an unusual component.