Worked Example — Fast in the Demo, Slow on Launch Day
A worked example of an AI feature that met its latency target and felt broken anyway — the difference between total time and time to first token, and the retry storm nobody had modelled.
This is an illustrative example. The product, launch and figures are invented. The gap it describes — a system that meets its stated performance target and is experienced as broken — comes from measuring the wrong thing carefully.
The situation#
A B2B analytics product launched an assistant that answers questions about the customer's own data. It had a performance requirement, it was tested against it, and it passed.
95th percentile response time under 10 seconds.
Load tested at three times expected peak. Passed at 8.9 seconds.
On launch day the feature was described by customers as unusable. The support queue filled within two hours. Measured latency was 8.4 seconds at the 95th percentile — better than the test.
What was wrong with the requirement#
The demo, which everyone had seen and approved, streamed its answer. Text began appearing in about 400 milliseconds and continued for eight seconds. It felt fast.
Production did not stream. The response was assembled server-side and returned complete. The same eight seconds, spent staring at a spinner.
Total response time was identical. The experience was not comparable.
| Demo | Launch | |
|---|---|---|
| Time to first visible output | 0.4 s | 8.4 s |
| Total time | 8.6 s | 8.4 s |
| Users who abandoned before completion | 3% | 41% |
| Support tickets in the first day | — | 180 |
Forty-one per cent abandoned, and most of them retried. Which produced the second problem.
The retry storm#
Abandoned requests were not cancelled. The server continued generating an answer nobody would see, and continued to be charged for it.
| First day | |
|---|---|
| Requests initiated | 41,000 |
| Requests abandoned before completion | 16,800 |
| Abandoned requests still generated and billed | 16,800 |
| Manual retries | 11,200 |
| Share of spend on answers nobody saw | ~38% |
Peak concurrency was roughly 1.7 times what the load test had modelled, entirely because of retries the model had not included. The load test generated a fixed request rate; real users respond to slowness by adding load.
What the load test had missed#
Three things, each ordinary.
It measured total time, because that was the requirement. Time to first token was never recorded, so the difference between the demo and production was invisible to it.
It did not model user behaviour. A fixed request rate is a well-behaved load generator and a poor model of people, who retry, refresh and open second tabs.
It never tested abandonment. No test client ever disconnected mid-request, so nothing revealed that the server kept working, or that nothing cancelled the upstream call.
The fix#
Streaming, first and most important. Time to first token dropped to 600 milliseconds. Total time unchanged. Abandonment fell from 41% to 4% with no improvement in the number the requirement had specified.
Cancellation on disconnect. When the client goes away, the upstream call is cancelled. Removed the entire abandoned-generation spend.
A per-user in-flight limit. A second request from the same user replaces the first rather than joining it. Retry storms became arithmetically impossible.
Progressive status for slow queries. Queries needing a large scan now show what they are doing — "reading 2.4 million rows" — rather than a spinner. Perceived speed improved on queries that got no faster.
The revised requirement#
The original requirement was replaced, because it had been met while the feature failed.
Time to first visible output under 1 second at the 95th percentile. Total response under 20 seconds at the 95th percentile, with visible progress throughout. No request continues after the client disconnects.
Note that the total-time budget was loosened from 10 seconds to 20. Once output is visible immediately and progress is shown, total time matters far less than it appeared to — and the looser budget removed pressure to degrade answer quality for speed.
The result#
| Launch | Two weeks later | |
|---|---|---|
| Time to first output, p95 | 8.4 s | 0.6 s |
| Total time, p95 | 8.4 s | 9.1 s |
| Abandonment | 41% | 4% |
| Spend on unseen answers | 38% | ~0% |
| Support tickets per day | 180 | 6 |
| Peak concurrency | 1.7× modelled | 0.9× modelled |
Total time got slightly worse and every outcome improved.
What was learned#
Measure time to first output, not just total time. They are different requirements and only one of them is what a person experiences.
Model users, not request rates. Real load includes retries, refreshes and abandonment, and each of them adds load precisely when the system is least able to take it.
Test disconnection. Nothing in the load test ever went away mid-request, so the most expensive behaviour in the system was never exercised.
A demo can set an expectation the production path cannot meet. The demo streamed. Nobody recorded that as a requirement because nobody noticed it was a difference.