AI Testing and Assurance
Independent evaluation of an AI system, producing a held-out evaluation set you keep, an adversarial pass, and a confidence score that states what it does not cover.
Testing a system that gives a different answer every time breaks the contract every other kind of testing relies on. A pass rate is not a guarantee, a golden output is not an assertion, and a green suite says nothing about the case nobody wrote.
This engagement gives you an independent read on whether an AI system is safe to leave running, and leaves you with the apparatus to keep checking.
What we test#
| Dimension | What it means |
|---|---|
| Correctness by category | Common cases, edge cases, refusals and adversarial, each with its own floor. A blended score hides the category that collapsed |
| Refusal quality | Whether it still declines what it should. This degrades silently when a model is tuned to be helpful |
| Adversarial resistance | Prompt injection, goal hijacking, tool misuse, data exfiltration through outputs |
| Drift | Whether output distributions have moved since the last accepted release |
| Cost per successful outcome | Not per call. A cheap answer that fails twice costs more than the better one |
| Autonomy boundary | What it does unsupervised, and whether that has widened by accident |
What you receive#
- A held-out evaluation set you own, versioned, structured by category. This is the deliverable with the longest life, and most organisations do not have one.
- A findings report with each issue reproduced, not described.
- An adversarial pass, with the inputs that worked.
- A confidence score with its reasons, including an explicit list of what is not covered and why.
The rule about the score#
A number without its reasons is something people learn to nod at. Ours always ships with the section naming what was untested, because that is the only part that changes what anyone does before shipping.
94% does not mean ship. It means here is what we know, and here is the shape of what we do not. A person at your organisation decides whether that is acceptable for this release, this week, with this customer.
Independence#
We do not test what we built. If we implemented the system, someone else should assure it, and we will say so rather than take the work.
A supplier who both builds and certifies has no independent element in the arrangement, which is exactly the property assurance is supposed to provide.
What we will not do#
- Give a pass because the suite was green. The interesting question is always what the suite does not cover.
- Test a learned system with deterministic methods and call the result assurance.
- Withhold a finding because it is inconvenient to the timeline.
How we work, in public#
The AI Testing Center publishes the methods in full, and our own QA function is documented including its procedures and the confidence report template.
Get in touch with what the system does and what a wrong answer would cost.