KPIs to measure the business impact of AI
Which indicators actually track an artificial intelligence programme, which ones are pure vanity, and how to build a dashboard that survives a board review.
An AI programme without clear indicators ends up being discussed through anecdotes. And anecdotes lose against any budget cut.
What follows is the minimum set that supports a serious conversation on the subject.
The four indicators for a system
For any implementation, these four are enough to know whether it works:
Coverage. What share of cases actually flows through the system. If it's low, you have an adoption or scope problem, not a technology one.
Human intervention rate. How many outputs need correcting. It should fall during the first months and then stabilize. If it doesn't fall, the use case may have been the wrong choice.
Time or cost per case. Measured against the baseline captured before implementation. This is the indicator that translates directly into money.
Output quality. Measured however the case demands: satisfaction, error rate, case reopens, accuracy per field. Without this one, savings may be paid for with worse service.
The three indicators for the programme
At organizational level, beyond individual systems:
Cases in production versus cases in testing. An organization with twelve pilots and none in production has an execution problem. The ratio between those numbers is the best diagnostic of programme health.
Time from idea to production. If the first case took nine months and the third took three, the organization is building capability. If they all take the same, every project starts from zero.
Cumulative savings or revenue, against total programme investment. This is the number that goes to the board.
The vanity indicators
Three metrics that get reported often and say nothing:
Number of queries. Measures activity, not value. An assistant can field thousands of queries without changing a single decision.
Number of licensed users. Assigned licences aren't adoption. The useful indicator is recurring usage: how many people come back the following week.
Model accuracy in lab tests. A model with excellent accuracy on test data can perform poorly in live operations. What matters is performance on real cases in production.
How to build the baseline
This is the step most often skipped, and the one that makes everything else possible. Before implementing, measure the current process for two to four weeks:
- Volume per period.
- Average time per case.
- Error and rework rate.
- Hourly cost of the role involved.
- Current satisfaction level, where applicable.
That measurement takes a few hours of work and determines whether the project can be evaluated at all.
A one-page dashboard
What to bring to a monthly review:
| What | How it looks |
|---|---|
| Coverage | Percentage, with trend |
| Human intervention | Percentage, with trend |
| Time per case | Current vs baseline |
| Quality | Whichever indicator applies |
| Cumulative saving | Amount, against investment |
| Incidents | Count and type |
Six rows. If the dashboard needs more than a page, it's probably measuring the tool rather than the business.
The indicator nobody measures and should
Which decisions changed. It isn't quantitative, which is why it gets left out, but it's the most revealing: did anyone do something differently because of what the system showed?
If the answer is no, the system is one more report, however sophisticated. If the answer is yes, there's value even when the other indicators don't show it yet.
Frequently asked questions
What are the minimum KPIs for an AI project?
Four: coverage, human intervention rate, time or cost per case against the baseline, and output quality. Those four tell you whether the system works and whether it's being used.
Why isn't the number of queries a useful metric?
Because it measures activity, not value. A system can receive thousands of queries without changing a single decision or saving any time. Measure outcomes, not usage.
How often should indicators be reviewed?
Weekly during the first two months, while tuning is still happening, and monthly afterwards. Quarterly reviews are for deciding about the programme, not about the system.