QA: KPIs
Measures for a QA function judged on defects prevented rather than tests written, including escaped defect rate, requirement defect catch and confidence calibration.
Markdown. No sign-up, no email.
Test count is now meaningless. An agent can generate ten thousand cases before lunch, and a suite measured by size grows until it is too slow to run and too noisy to read.
These measure whether defects were prevented, caught, or escaped.
The six that matter#
| Measure | Definition | Target | How it gets gamed |
|---|---|---|---|
| Escaped defect rate | Defects found in production per release | Falling | Reclassifying production defects as change requests |
| Requirement defect catch | Defects caught at requirement stage, before code | Above 20% of all | Not recording requirement rejections as catches |
| Coverage of changed code | Share of each change exercised by a test | Above 85% | Chasing line coverage with assertion-free tests |
| Confidence calibration | Predicted confidence against what actually happened | Within 5% | Never scoring the score |
| Suite runtime | Wall clock for the full run | Under 15 min | Removing slow tests rather than fixing them |
| Flake rate | Tests failing intermittently without a code change | Under 1% | Retrying until green, which hides the flake and the bug behind it |
Confidence calibration is the keystone. If releases scored at 94% fail as often as ones scored at 70%, the number is decoration. Score the score every month: take the last twenty releases, compare predicted confidence with what actually went wrong, and correct the model rather than the memory of it.
Flake rate above 1% destroys the whole function. Once a team believes a red test might be lying, every red test becomes a negotiation, and the suite stops being a gate.
Two counter-metrics#
| Counter-metric | Catches |
|---|---|
| Defects found per test written | Collapsing while suite size grows means volume without value |
| Time from defect found to defect fixed | A QA function excellent at finding and irrelevant because nothing gets fixed |
Defect analytics, reviewed monthly#
| Question | Action if bad |
|---|---|
| Which modules hold the most defects? | Those get mandatory human review, not just agent review |
| Which defects recurred? | A missing regression test. Add it, and ask why it was missing |
| What is the top root cause this month? | Fix the cause, not the instances |
| Which escaped defects had no test at all? | The honest coverage gap, and the most useful list here |
What is deliberately not measured#
- Number of tests. Free to produce, so it measures nothing.
- Number of defects found. Rewards finding over preventing, and a fall could mean better code or worse testing.
- Test pass rate. Approaches 100% naturally and says nothing about what is untested.
- Automation percentage. A count of tooling, not of quality.
Cadence#
| Per release | Confidence score with reasons, coverage of changed code, suite result |
| Weekly | Flake rate, defect-to-fix time |
| Monthly | Escaped defects, calibration check, defect pattern analysis |
| Quarterly | Suite health: what is slow, what is flaky, what has never failed and may be asserting nothing |
The quarterly hunt for tests that have never failed is worth more than it sounds. A test that has never once gone red is either guarding something that cannot break or is not actually asserting anything, and both cost runtime and attention on every run.