QA: AI Agents
Requirements checking, test generation, defect analysis and the release confidence score, with the reasons behind it, plus the calls that stay with a person.
Markdown. No sign-up, no email.
Most QA effort has historically gone into producing tests and running them. Both are now cheap. What is still expensive, and still decides whether a release is safe, is knowing what has not been covered and being willing to say so before a date.
Requirements testing, before any code exists#
The cheapest defect to fix is one caught in a sentence. An agent reads the requirements and returns three things:
| Check | Example finding |
|---|---|
| Completeness | "Users can reset their password" does not say what happens to active sessions |
| Contradiction | Section 2 says exports are admin-only, section 7 has a manager exporting |
| Testability | "The system should be fast" cannot be tested. "95th percentile under 400ms" can |
A requirement that cannot be tested cannot be accepted. Sending it back at this stage costs an email; discovering it during acceptance costs a release.
Test generation#
The agent produces, from the requirement and the diff:
- Functional cases covering the stated behaviour
- Regression cases around what the change touches
- Edge cases: empty, maximum, boundary, concurrent, the second call, the retry
- API cases: contract, status codes, malformed payloads, auth failures
- Security cases: injection, broken access control, and whatever the current risk list names
Edge cases are the honest reason to automate this. They are not intellectually difficult; they are the ones a tired person skips at 5pm on a Thursday, and machines are not tired.
Automated defect analysis#
Individual defects get fixed. Patterns across defects are where the money is, and nobody has time to see them by hand.
| Pattern | What it means |
|---|---|
| Recurring defects | The same fault re-entering. Usually a missing regression test, occasionally a bad merge habit |
| High-risk modules | Where defects concentrate. Predicts where the next one lands better than any estimate |
| Regression patterns | Which changes break which areas, historically |
| Root causes | Grouped by cause rather than symptom, which is what makes the list actionable |
Release confidence#
The agent produces a single number, and the number is the least important part of it.
Release Confidence: 94%
- Requirements coverage: 100% (18 of 18 have at least one test)
- Regression suite: pass, 0 failures in 1,240 cases
- Edge coverage on changed modules: 88%, two boundary conditions untested
- Security scan: clean
- Untested: the payment retry path, because staging has no sandbox for it
- Highest-risk module touched: billing, 4 defects in the last 90 days
A confidence score without its reasons is a number people learn to nod at. The value is in the fifth line, the one naming what is not covered, because that is the only part that changes what anyone does before shipping.
94% does not mean ship. It means: here is what we know and here is the shape of what we do not. A person decides whether that is acceptable for this release, this week, with this customer.
What stays with a person#
- The release decision. Always. The score informs it and never makes it.
- Accepting untested risk, with a name against it and a date.
- Signing off a requirement as complete and testable.
- Deciding a defect is acceptable to ship, and telling Support before a customer finds it.
- Waiving a test. Recorded, with a reason, because a waiver nobody sees becomes a habit.
The failure this is built against#
A green pipeline and a confident number, on a release where the untested path is the one that matters. Everything above is arranged so the untested path is stated in the same breath as the score, rather than being the thing the score quietly excluded.