Autonomous Test Generation — Frequently Asked Questions
What generated tests are actually worth, why the judge is the hard part, calibrating LLM-as-judge, and the boring methods that hold up when testing systems that vary between runs.
We publish the AI Testing Center methodology and have not built a tool. See AI Testing for why.
Generated tests#
Can AI write our tests?#
Quickly and in volume. The question is what they are worth, and the answer depends on what the model had to work from.
Given only the implementation, it has nowhere to learn intent except the code itself — so it describes current behaviour, bugs included, and the resulting suite locks in whatever the code does today. That is actively unhelpful: it raises the cost of fixing the bug, because now a test fails when you correct it.
So generation is useless?#
No — it is useful where intent is available. Given a specification, generating the cases, particularly boundaries and error paths humans reliably skip, is genuinely valuable.
The distinction is whether the model is inferring intent from code (weak) or working from a stated requirement (strong).
How do we sanity-check a generated test?#
Break the code it covers and confirm the test fails. A surprising share do not — they assert nothing meaningful, or they exercise a mock rather than the behaviour.
Make this a rule for generated tests specifically, because unlike hand-written ones nobody watched them fail on the way to passing. That accidental verification step is missing.
The judge problem#
Why can't we just automate evaluation of AI outputs?#
Because you need something to decide whether an output is acceptable, and for non-deterministic systems that is usually another model.
That judge has the same weaknesses as the system under test: manipulable by the same content, inconsistent between runs, and confident either way. A suite whose oracle is unreliable does not fail loudly — it reports a pass rate that means less than it appears to.
Is LLM-as-judge useless then?#
No, but it is an instrument requiring calibration, and almost nobody calibrates it.
Measure the judge against human labels first — a few hundred outputs, labelled by people, agreement measured. If it agrees 70% of the time, every number it produces afterwards carries that error.
How should a judge be used well?#
Narrow questions, not overall quality. "Does this answer contain a claim unsupported by the provided sources?" is answerable. "Is this a good answer?" is not.
Prefer relative comparison — which of two versions is better — over absolute scores. And keep a human-reviewed set that never changes, so you can detect the judge drifting when its model is updated underneath you.
What works instead#
What are the reliable methods?#
Unglamorous ones:
Property-based assertions — the output cites only supplied sources, is valid JSON matching a schema, refuses when the corpus cannot answer. Deterministic and cheap.
A fixed task set, re-run, tracking the pass rate over time. The direction matters more than the absolute number.
Regression on real failures: every bad output anyone reports becomes a permanent case. This is the highest-value habit on the list and the easiest to skip, because adding the case feels like paperwork once the bug is fixed.
Component measurement — for retrieval systems, hit rate independent of generation.
Why measure components separately?#
Because an end-to-end score hides the ceiling. If retrieval succeeds 60% of the time, the system cannot exceed 60% however good the generation, and no prompt work changes that.
Teams routinely spend weeks on prompts for a retrieval problem because they only measured the whole.
How many test cases do we need?#
Start with twenty real ones with known answers. Twenty representative cases beat two hundred invented ones, because invented cases reflect what you imagined rather than what users do.
Grow the set from reported failures.
How do we report results honestly?#
Pass rate with the number of runs. A single run of a non-deterministic system is an anecdote. "18 of 20, over three runs" is a statement someone can act on; "90%" is not.
Buying#
What should we look for in a testing tool?#
Whether it publishes its judge's measured agreement with human labels. Whether its output distinguishes "tested and held" from "we found nothing". Whether deterministic property checks are the foundation and model judgement is a labelled layer above.
A tool that reports a system is correct is overclaiming. The honest report says what was covered and what was not.
Why did you publish methodology instead of a tool?#
Because the methodology is the scarce part. Teams struggle here not because they lack a test runner but because they do not know what a defensible evaluation looks like — and a tool answering the wrong question confidently makes that worse.