The Project That Should Not Have Used AI — A Worked Example
A document classification project costed against the alternative nobody had priced — what the rules-based version achieved, what the model added, and the arithmetic that decided it.
This is an illustrative example. The scenario and figures are composed to show the method, not drawn from a named client engagement.
A finance team receives around 4,000 documents a month into a shared mailbox — invoices, remittance advices, statements, purchase orders, and a long tail of correspondence. Someone opens each one and routes it to the right queue.
The request: "Can AI classify these automatically?"
The answer was yes. It was also the wrong thing to build, and the reason is arithmetic.
What the rules-based baseline achieved#
Before scoping the model, one afternoon was spent on the question the readiness checklist asks first: has a deterministic solution been considered and ruled out?
A sample of 500 documents was examined by hand. The findings:
| Signal | Documents it correctly routed |
|---|---|
| Sender address matches a known supplier | 61% |
| Subject line contains a document type keyword | 18% |
| Attachment filename pattern | 7% |
| Total covered by simple rules | 86% |
| Requiring judgement | 14% |
Three rules, roughly a day and a half of work, no training data, no model, no ongoing cost. It routed 86% of the mailbox correctly and could explain every decision.
What the model would have added#
The proposal was a classifier over document text. Estimated performance from a pilot on the same 500 documents: 94%.
Eight percentage points better. On 4,000 documents a month that is 320 additional documents routed automatically — around 16 per working day.
The arithmetic that decided it#
| Rules | Model | |
|---|---|---|
| Build effort | 1.5 days | ~6 weeks (pipeline, labelling, evaluation, integration) |
| Labelling effort | none | ~3,000 documents |
| Running cost | negligible | per-document inference |
| Explains its decisions | yes | not without extra work |
| Fails visibly | yes — no rule matched | no — confidently wrong |
| Maintenance | edit a rule | monitor drift, retrain, keep a test set |
| Documents automated | 3,440/month | 3,760/month |
The model's incremental benefit was 16 documents a day. Handling those manually took one person about twelve minutes daily.
Against that: six weeks of build, a labelling exercise, an ongoing inference cost, and a permanent monitoring obligation — a system that needs someone to notice when it degrades, forever.
The part that changed the decision#
The 14% requiring judgement was not evenly distributed. It was mostly the long tail of correspondence — disputes, queries, anything unusual.
Those are precisely the documents where misrouting is expensive. A misrouted invoice is a minor delay; a misrouted dispute sits in the wrong queue while a customer waits.
So the model would have been most confident on the documents the rules already handled, and least reliable on the ones where errors actually cost something. Automating that 14% was not a benefit — it was moving the risk out of sight.
What was built#
The three rules, plus an explicit "needs a human" queue for anything unmatched. The queue averages around 25 documents a day and is the only thing a person opens.
Total effort: under two days. The finance team got 86% of the benefit in 3% of the time.
When the answer would have been different#
This is not an argument against AI classification. Change three things and the decision flips:
- Volume. At 40,000 documents a month, 16 per day becomes 160, and the arithmetic changes.
- Rule coverage. If simple signals had covered 40% rather than 86%, there would be a real gap.
- Judgement genuinely required. If routing depended on reading the content rather than the sender, no rule would work.
The point is that nobody knew which situation they were in until the baseline was measured. It took one afternoon.
What transfers#
Measure the trivial baseline before scoping the model. It is usually a day's work and it frequently ends the project — which is a good outcome, arrived at cheaply.
Express the benefit in units of work, not percentage points. "Eight points better" sounds compelling; "sixteen documents a day" invites the right comparison.
Check where the errors land. A system that is accurate on the easy cases and unreliable on the consequential ones is worse than its headline number suggests.
See AI in business and the readiness checklist, whose first section exists because of projects shaped like this one.