AI Strategy and Transformation: KPIs
The six measures this function is judged on, the two counter-metrics that catch what they hide, and how each one gets gamed.
Markdown. No sign-up, no email.
The measurement problem in this function is specific. Almost every metric an AI transformation team naturally reaches for, pilots launched, agents deployed, adoption percentage, can be grown without the company gaining anything. They measure activity, and activity is the thing a transformation programme produces most easily when it is failing.
Everything below measures either an outcome or a latency. Each entry states how it is gamed, because a metric published without its failure mode gets gamed by default.
The six that matter#
| Measure | Definition | Target | How it gets gamed |
|---|---|---|---|
| Realised benefit vs forecast | Post-change measure minus baseline, as a percentage of the forecast benefit | 70 to 130% | Forecasting low so every programme overdelivers. Track forecast accuracy in both directions |
| Cycle time on the changed process | Median end-to-end elapsed time, same definition before and after | Down 30%+ | Re-drawing the process boundary so the slow part falls outside it |
| Rework rate after automation | Share of automated outputs corrected, reversed or redone downstream | Under 5% | Not counting corrections made by the person who received the output |
| Candidate-to-live latency | Days from a candidate being identified to the change running in production | Under 60 | Marking things live while they run on one desk in shadow mode |
| Reversal rate | Share of changes withdrawn within 90 days | 5 to 15% | Reporting zero. Zero means nothing ambitious was attempted, not that everything worked |
| Cost of AI per changed process | Inference, licences and engineering time attributed to that change | Falling per unit | Leaving inference spend in a central bucket nobody attributes |
The reversal-rate target is a band, not a ceiling. A function reporting zero reversals over a year is either not attempting anything difficult or is not admitting when something failed. Both are worse than a handful of honest withdrawals.
Two counter-metrics#
A counter-metric exists to catch the damage a headline metric hides.
| Counter-metric | Catches |
|---|---|
| Downstream escalation volume | Automation that hits its cycle-time target by pushing exceptions onto another team |
| Staff time reallocated, and to what | "Hours saved" that quietly became hours spent supervising the automation |
The second is the one most often skipped. A process that took four hours of work and now takes twenty minutes of work plus three hours of checking has not saved anything; it has changed the job title of the person doing it.
What is deliberately not measured#
- Number of pilots. Grows on its own and correlates with nothing.
- Agents deployed. A count of tools, not of results.
- Adoption percentage. People can be made to log in. It says nothing about whether the work got better.
- Model accuracy in isolation. A model at 94% inside a process that routes 40% of cases to a human anyway has an unclear effect on the outcome. Measure the process.
Reporting cadence#
| Monthly | Candidate-to-live latency, cost per changed process |
| Quarterly | Realised benefit vs forecast, cycle time, rework, reversals, both counter-metrics |
| Annually | Portfolio review: which changes still hold their benefit two years on |
The annual line matters more than it looks. Benefit realisation is usually measured once, at 90 days, and never revisited, so a saving that decayed after six months stays on the books as a success indefinitely.
Where the numbers come from#
Data and Analytics owns the measurement, not this function. That separation is deliberate: a team that measures its own programme's success has an obvious conflict, and the first thing an auditor looks for is whether the two roles sit in the same place. This function defines what to measure and sets the baseline; the number itself is produced elsewhere and published whichever way it lands.