# AI Performance Testing Checklist

**System:** _______________  **Date:** _______  **Tested by:** _______________

> You do not control the bottleneck. The model runs on someone else's infrastructure, behind a rate
> limit, with latency you cannot tune. That changes what you measure and what you can fix.

## 1. Targets set before testing

- [ ] Target set on **p95**, not the average
- [ ] Time-to-first-token target set (if streaming)
- [ ] Total completion target set
- [ ] Expected concurrency stated
- [ ] **Cost ceiling per request stated**
- [ ] Behaviour at capacity decided in advance: queue, degrade, or shed

## 2. Measure

- [ ] Time to first token
- [ ] Total completion time
- [ ] **p50 / p95 / p99** — not the mean
- [ ] Throughput at target concurrency
- [ ] **Rate-limit rejections** — your real ceiling
- [ ] **Cost per completed request**, including retries
- [ ] Tokens per request
- [ ] Queue depth and wait time

## 3. Tests

- [ ] **Baseline** — single request, repeated. Best case and its variance
- [ ] **Sustained load** — expected concurrency, sustained period
- [ ] **Spike** — sudden burst. Queue, shed or fail?
- [ ] **Long input** — near the context limit
- [ ] **Degraded backend** — provider slow or erroring. The failure you will actually get
- [ ] **Cost under load** — retries multiply spend invisibly

## 4. AI-specific failure modes checked

- [ ] Rate limits, not CPU, confirmed as the constraint
- [ ] **Retry amplification** — a timeout triggers a retry and you pay for both
- [ ] Context growth impact on latency and cost
- [ ] Cold start (self-hosted models)
- [ ] **Streaming masking degradation** — time-to-first-token flat while total time worsens

## 5. Levers verified

- [ ] Caching in place for repeated or near-identical requests
- [ ] Prompt length tracked as a metric
- [ ] Number of retrieved chunks tuned
- [ ] Cheaper model evaluated for the same task
- [ ] Streaming used where the interface allows
- [ ] Explicit queue with position indicator, not an unbounded wait
- [ ] Deliberate degradation path defined

## 6. Targets by interaction type

Set the target from what the interaction actually needs, then measure against it.

| Interaction | Reasonable target | Yours |
|---|---|---|
| Interactive chat | First token < 1s, total < 5s | |
| Form assist / autocomplete | Total < 2s | |
| Document processing | Minutes acceptable, progress required | |
| Batch / overnight | Throughput matters, latency does not | |

## 7. Agent-specific

Complete if the system takes actions rather than answering.

- [ ] End-to-end completion time measured, not per-call latency
- [ ] **Steps per task recorded** — step count multiplies everything downstream
- [ ] Cost per *completed* task, not per call
- [ ] Step count regression treated as a performance regression even when the task still succeeds
- [ ] Behaviour measured when a tool is slow rather than failing

## 8. What you can actually change

You do not control the model, so verify each lever is available:

- [ ] Caching — the largest single lever at volume
- [ ] Prompt length reduction
- [ ] Fewer retrieved chunks
- [ ] A cheaper model that still passes the evaluation set
- [ ] Streaming, where the interface allows
- [ ] Explicit queue with a position indicator
- [ ] Deliberate degradation path rather than a timeout

## 9. Results

| Measure | Target | Measured | Pass |
|---|---|---|---|
| Time to first token (p95) | | | |
| Total completion (p95) | | | |
| Throughput at target concurrency | | | |
| Rate-limit rejection rate | | | |
| Cost per completed request | | | |
| Steps per task (agents) | | | |

## 10. Findings

| # | Finding | Impact | Action | Owner |
|---|---|---|---|---|
| | | | | |

## Sign-off

| | Name | Date |
|---|---|---|
| Tested by | | |
| Accepted by | | |