Measuring AI ROI without vanity metrics

Why so many ROI claims fall apart

Finance leaders have heard the same story many times: a pilot saved a large number of hours, the case was approved, and a year later nobody can find the savings in the accounts. The reasons are consistent. The baseline was not recorded. The hours saved were not redeployed or removed. The costs of running and supervising the system were left out. And the metric chosen was the easiest to collect, not the most relevant.

Choose one outcome the business already tracks

Start with a measure that existed before AI: cycle time, cost per case, error rate, revenue per salesperson, days sales outstanding, customer wait time. Using an existing measure avoids an argument about definitions and lets you compare with history. Choose one primary outcome and at most two supporting ones. More than that, and the temptation to pick the favorable number grows.

Fix the baseline first

Before launch, record how the workflow performs: volume, time per unit, cost per unit, error rate, and the variation across weeks. If you can, keep a comparison group, such as a team or region that continues the old way for a period. Without a baseline, every improvement is an opinion.

Count the whole cost

Net cost includes model usage, hosting, integration, evaluation, monitoring, review time, support, training and the share of engineering and operations time devoted to the system. It also includes the cost of fixing errors that reach customers. Leaving these out is the most common way to overstate the benefit. The cost-per-unit method in unit economics of AI products is a useful companion.

A worked example

Notice that the saving is smaller than the headline of $28,000 from handling cost alone. It is real because it is measured against a baseline and includes the cost of running the system.

Metrics that mislead

Metric	Why it misleads	A better substitute
Hours saved	Hours may not be redeployed	Cost per unit, or capacity used on new work
Prompts or sessions	Measures activity, not value	Share of work completed without rework
User sign-ups	Curiosity is not adoption	Weekly use by the intended team
Model accuracy alone	May not reflect real inputs	Error rate in production, with impact
Pilot results	Pilot teams are unusually engaged	Results after wider rollout

Be honest about attribution

Other things change at the same time as an AI rollout: staffing, volume, pricing, seasonality. Where you can, use a comparison group or a before-and-after window long enough to smooth out noise. Where you cannot, say so, and present a range rather than a single figure. A cautious number with a clear method is more persuasive than a large one nobody can reproduce.

A word on timing

Measure at the right interval. Too soon, and the figure reflects learning costs and launch noise. Too late, and the original reasons for the project are forgotten. We suggest agreeing at the start on a first checkpoint, a second at a longer interval, and a date to decide whether to continue, widen or stop. Put the dates in the plan, not in anyone's memory.

Non-financial value

Not everything useful shows up in the accounts. A system may lower the risk of a costly mistake, make a hard job more bearable, or let a small team serve a customer faster. Record these benefits, describe how you assessed them, and keep them separate from the financial result. Mixing them in tends to weaken trust in both.

Common errors

Teams often measure a pilot against a baseline taken in a different season, forget to count the time spent on exceptions, or credit the system with improvements that a staffing change produced. A second reader, ideally from finance, should challenge the method before the result is shared. If the number survives that review, it is far more likely to survive a board meeting.

What to ask

Before approving an AI investment, ask five questions. What is the baseline, and who recorded it? Which single outcome will tell us it worked? What is the all-in monthly cost? What would we expect to see by the end of the first quarter if it is working? And what is our rule for stopping if it is not? Teams that can answer these are usually ready to learn from the result, good or bad. Our approach describes how we think about evidence alongside operating teams.

This article is for general information only and is not investment, legal, tax or accounting advice.