AI Performance and Load Report — Sample
A worked example of load-testing an AI feature — percentile latency rather than averages, cost as a first-class result, and the queue behaviour that decides what happens at the peak.
Markdown. No sign-up, no email.
This is an illustrative example. The system and every figure are invented. The method transfers: for AI features, latency and cost are the same test, and the average is the least useful number in it.
Load test — document summarisation service#
| System | Summarises uploaded documents; user waits for the result |
| Expected load | 40 requests/minute typical, 180 at the daily peak |
| Requirement | 95th percentile under 8 seconds; no request over 30 seconds |
| Tested | 2026-07-22, against the release candidate |
| Inputs | 2,000 real documents, sampled to match the production length distribution |
1. Result#
Passes at typical load. Fails at peak. At 180 requests a minute the 95th percentile is 19.4 seconds and the slowest request is 74 seconds. Two changes bring peak within requirement; both are queue behaviour rather than model choice.
2. Latency by load#
| Requests/min | Median | 90th | 95th | 99th | Max |
|---|---|---|---|---|---|
| 40 | 2.1 s | 4.0 s | 4.8 s | 7.2 s | 11 s |
| 80 | 2.3 s | 4.6 s | 5.9 s | 10.1 s | 18 s |
| 120 | 2.8 s | 7.1 s | 9.8 s | 17.4 s | 31 s |
| 180 | 4.2 s | 14.2 s | 19.4 s | 41.0 s | 74 s |
| 240 | 9.8 s | 38.0 s | 52.0 s | 98.0 s | timeout |
The median is reassuring and irrelevant. At peak it is 4.2 seconds while one request in twenty waits 19 and one in a hundred waits 41. Reporting an average here — 6.1 seconds — would describe an experience nobody has.
Degradation is not linear. Between 120 and 180 the 95th percentile roughly doubles for a 50% increase in load, which is the signature of a queue rather than of the model being slow.
3. Where the time goes at peak#
| Stage | Median | 95th |
|---|---|---|
| Upload and extract | 0.4 s | 1.1 s |
| Waiting for a provider connection | 1.1 s | 14.8 s |
| Model generation | 2.4 s | 3.2 s |
| Post-process and store | 0.3 s | 0.4 s |
Model generation is nearly flat across all load levels. The provider is not the bottleneck. The connection pool is: it holds 20 concurrent connections, and at 180 requests a minute with a 2.4-second generation the arithmetic requires closer to 8 — except that long documents hold connections far longer, and a handful of them block everything behind them.
This is head-of-line blocking, and it explains the whole shape of section 2.
4. Cost under load#
Reported here rather than in a separate document, because for AI features they are one result.
| Requests/min | Cost/hour | Cost per request |
|---|---|---|
| 40 | $9.60 | $0.0040 |
| 180 | $43.20 | $0.0040 |
| 180 with retries | $51.80 | $0.0048 |
Cost per request is flat, as expected. The retry row is the finding: at peak, 18% of requests that time out client-side are retried by the browser, and the original request completes anyway and is charged. Roughly 20% of peak spend buys results nobody sees.
5. What happens past the limit#
Tested deliberately, because the answer is a design decision that is usually made by accident.
At 240 requests a minute the service does not degrade gracefully — it queues without bound until requests time out. Users see a spinner for 90 seconds and then an error, having waited the entire time for nothing.
The queue has no depth limit and no shed policy. A system that fails at the 91st second is worse than one that says no at the first, because it consumes the user's time and the provider's tokens to deliver a failure.
6. The two changes#
Bound the queue and reject early. With a depth limit set from the requirement, a request that cannot be served within 8 seconds is refused immediately with a clear message. Modelled against the same trace: 95th percentile at peak falls to 6.9 seconds, with 4% of peak requests refused outright.
That is the trade the requirement implies, and it should be stated as such: at peak, 4% of users are told to try again in a moment, so that 96% get an answer in under 7 seconds. The alternative is everyone waiting 19.
Separate long documents into their own pool. Documents over 20 pages are 6% of volume and consume 34% of connection time. Given their own connections and their own expectation — an emailed result rather than a wait — head-of-line blocking disappears for the other 94%.
With both: 95th percentile at peak 4.4 seconds, no request over 12 seconds, retry waste eliminated.
7. What was not tested#
- Sustained peak. The peak test ran 20 minutes. Behaviour over hours is unknown.
- Provider-side rate limiting. We did not reach it. It exists and we do not know where.
- Cold start after a deployment, under load.
- Degraded provider latency. Everything here assumes the provider performs as it did on the day.
The last is the largest gap. Provider latency is outside our control and moves without notice, and none of these results say what happens when generation takes 6 seconds instead of 2.4.
Notes on using this format#
Report percentiles, never averages. The average at peak was 6.1 seconds and described nobody's experience. The 95th and the maximum are what users actually feel.
Break latency down by stage. Model generation was flat throughout; the entire problem was in a connection pool. A total-latency number would have prompted a change of model, which would have fixed nothing.
Put cost in the same report. The retry finding — a fifth of peak spend on results nobody receives — is invisible if load and cost are tested separately.
Test past the limit on purpose. How a system behaves when overloaded is a design decision. Untested, it is made by whichever default was in the library.