QA: SOPs
Six procedures from requirement review through test generation, the release confidence gate, defect pattern analysis, flake management and the escaped-defect postmortem.
Markdown. No sign-up, no email.
SOP 1: Requirement review#
Run: on every requirement, before estimation.
The agent checks completeness, contradiction and testability. A human reads the findings and either sends the requirement back or accepts it with the gaps recorded.
A requirement that cannot be tested is not accepted. The sentence "the system should be responsive" costs nothing to reject now and costs a release to discover later, when two people disagree about what it promised and both are reading the same words.
SOP 2: Test generation and review#
Run: with every change, in the same pass as the code.
- The agent generates functional, regression, edge, API and security cases from the requirement and the diff.
- A human reviews what is missing, not what was produced. Reviewing generated tests one by one is the slow half; asking what they failed to cover is the valuable half.
- Anything the agent flags as untestable in the current environment is recorded as a known gap rather than quietly dropped.
SOP 3: Release confidence gate#
Run: before every release.
The score ships with its reasons, including the untested list. Then a person decides.
| Score | Normal handling |
|---|---|
| Above 90% | Release, with the untested list read aloud rather than skimmed |
| 75 to 90% | Release only with a named accepter for each untested item |
| Below 75% | Do not release without an explicit, recorded exception |
These are defaults, not authority. A 96% release touching payments may still need a human test pass, and an 80% release of a static content change may be entirely fine. The score informs the decision; it never replaces the person making it.
SOP 4: Defect pattern analysis#
Run: monthly.
Group by root cause rather than symptom, then act on the top three:
- Recurring defect means a missing regression test. Write it, then ask why it was missing.
- A high-defect module gets mandatory human review on every change until the rate falls.
- A regression pattern between two areas means they are coupled in a way the design does not admit. That is an architecture finding, and it goes to Engineering.
SOP 5: Flake management#
Run: weekly.
A flaky test is worse than no test: it teaches the team that red might mean nothing.
- Quarantine it the day it is spotted, out of the gating suite and onto a list.
- Fix or delete within two weeks. Quarantine is not a destination.
- Never resolve a flake by adding a retry. A retry hides both the flake and the real race condition underneath it, and the race condition is the one that reaches production.
SOP 6: Escaped-defect postmortem#
Run: on every production defect.
| Question | Purpose |
|---|---|
| Was there a test for this? | If no, that is the coverage gap, and it is the useful answer |
| If yes, why did it pass? | A wrong assertion is a worse problem than a missing test |
| Which stage should have caught it? | Requirement, unit, integration, security, or manual |
| What is the test that would have caught it? | Write it before closing the postmortem |
An escaped defect that does not produce a test will escape again, and the second occurrence is always more expensive, because the first one was survivable.
Escalation#
| Situation | Goes to |
|---|---|
| Confidence below 75% and the date will not move | CEO. This is a risk decision, not a QA one |
| The same defect escaping twice | Engineering, as an architecture question rather than a bug |
| Flake rate above 3% | Stop feature work on that suite until it is under control |
| A security case failing | Security immediately, and the release stops |