Software Testing: Proving It Works Before Customers Find Out It Doesn't

You cannot test everything. Any non-trivial system has more possible states than you could exercise in a lifetime, so every test strategy is a decision about what not to test.

Making that decision deliberately, rather than by running out of time, is most of the skill.

Deciding what deserves testing#

Rank by two things: how likely is this to break, and what does it cost if it does.

High on both — payments, permissions, anything calculating money, data migration — deserves exhaustive treatment. Low on both — a rarely used cosmetic setting — deserves a glance.

This sounds obvious and is routinely ignored in favour of testing what is easy to test. A suite with four hundred tests on a formatting utility and none on the billing calculation is common, and it provides false assurance.

Write the ranking down. It makes the trade-off explicit and defensible when someone asks why an area was not covered.

The levels, and what each is for#

The common mistake is too many end-to-end tests. They are slow, they fail for environmental reasons, and a failure tells you something is broken somewhere. Teams stop trusting them, then stop reading them, then stop running them.

A workable balance: many fast unit tests over logic that matters, a moderate number of integration tests over the seams, and a small number of end-to-end tests covering only the journeys that must never break.

Coverage is a diagnostic, not a target#

Coverage measures which lines executed during tests. It does not measure whether the assertions were meaningful.

// 100% coverage. Tests nothing.
test("calculates total", () => {
  calculateTotal(items);        // called, never checked
});

Mandate a coverage number and you get tests written to satisfy the number. Use it instead as a question generator: this critical module is at 20% — is that deliberate? That question is useful. The target is not.

The cases people forget#

Most defects hide in a small number of predictable places:

• Empty — zero rows, no history, first-time user, blank field
• One — the singular/plural boundary, and code that assumed a list
• Boundaries — minimum, maximum, and one either side. Off-by-one errors live exactly here
• Wrong type — text in a number field, an unexpected null
• Very large — the customer with 40,000 orders
• Duplicate submission — double-clicked, or retried after a timeout
• Concurrency — two people editing the same record
• Interruption — connection lost mid-operation. Does it leave consistent state?
• Time — month end, year end, leap day, daylight-saving change, a different time zone
• Permissions — what each role must NOT be able to see or do

Running an existing feature through that list usually finds something.

Testing what is not deterministic#

Increasingly relevant, because AI features break the usual contract: the same input can produce different valid output, so asserting exact equality does not work.

What does work:

Test properties instead of strings. Does the JSON parse? Are required fields present? Do the line items sum to the stated total? Is there no personal data in the output? These hold across all valid phrasings.

Score qualities against a short rubric. Faithfulness, tone, completeness — three or four dimensions, scored. Long rubrics score inconsistently.

Measure a pass rate, not a single result. Run each case several times. A prompt that passes every time at temperature 0 and 60% of the time in production is a system you do not yet understand.

Our Prompt Testing guide covers building this as an automated suite.

Test data#

The quiet cause of most "worked in test, failed in production" incidents.

If test data is smaller, cleaner or differently shaped than production, you tested a different system. Realistic volume matters as much as realistic values — pagination, timeouts and query performance only fail at scale.

Use masked production data where you can do so lawfully, or generate synthetic data that matches production's shape and distribution. And make it repeatable: a test that passes or fails depending on the state left by the previous run is worse than no test, because it trains people to ignore failures.

What to automate, and what not to#

Automate what is repeated and objective: regression, calculations, API contracts, permission rules.

Do not automate what is rarely done or genuinely subjective. Exploratory testing — a person using the system with intent to break it — consistently finds classes of problems automation does not, because automation only checks what someone thought of in advance.

The correct split is not "automate everything". It is: automate the repetitive checking so people have time for the thinking.

FAQ#

How much testing is enough?#

Enough that you can change the code and quickly learn whether you broke something. If people are afraid to refactor, too little. If the suite is so slow it gets skipped, you have the wrong mix.

Should developers or testers write tests?#

Both, at different levels. Developers write unit tests as part of writing code. Testers bring an adversarial perspective — how might this break, what did nobody consider — which is a genuinely different skill, not a junior version of development.

What is regression testing?#

Re-running tests to confirm a change did not break something that previously worked. It is the main reason automated tests exist, and the reason every fixed bug should become a permanent test.

Why do our tests pass but production breaks?#

Usually test data that does not resemble production, environment differences, or tests asserting implementation details rather than behaviour. Check data volume and shape first — it is the most common cause.

Is manual testing obsolete?#

No. Automation checks what you already thought of. Exploratory testing finds what you did not, and it is the only way to assess whether something is usable rather than merely correct.

How do we test a system we did not build?#

Start from the outside: what does it promise to do, and what breaks the business if it stops? Write end-to-end tests for those few journeys first. They give you a safety net before you understand the internals.

What should we do when a bug reaches production?#

Reproduce it in a failing test, fix it, keep the test forever. Then ask which control should have caught it — testing, review, or monitoring — and strengthen that one. The bug is information about your process, not just about the code.

What else is coming for Testing