How to measure returns on AI

In the beginning there was AI, and bosses everywhere lost their minds and said you must all use this technology no matter whether it is useful and no matter how much it costs. And they introduced AI leaderboards and came up with stupid words like “tokenmaxxing”. And lo, usage did indeed rise. And then the bosses remembered some very basic concepts like “budgets” and realised that this might not be such a great idea after all. And then a different question rang through the boardrooms, and this question was about returns, and it was harder to answer.

This potted Genesis of AI adoption is not entirely fair. As long as organisations are trying to encourage usage, it can still make sense to look at things like employees’ token consumption. “I would expect software engineers to spend more money than a finance analyst,” says the boss of one software firm. “But anyone within finance, it shouldn’t be zero.”

But usage (also known as input) measures don’t tell you whether any value is being created. And even when AI is producing benefits, points out Mika Ruokonen of LUT University in Finland, lots of them accrue quietly to individuals in the form of scattered time savings. If an extra hour or two of an employee’s time has been freed up to shop on Vinted, someone has benefited (mainly Vinted) but it isn’t your company (unless you work at Vinted).

Shifting the focus to returns on AI investment means concentrating more on outcomes-based measures, such as team productivity or customer satisfaction. But these require careful thought. Scientists, for example, are using AI to produce more papers (yay!) of dubious merit (boo!). A recent study by Tulsi Suchak of the University of Surrey and her co-authors found that papers derived from a health and nutrition data set ticked along at an average of four a year between 2014 and 2021; in the first nine months of 2024, there were 190.

So outcomes-based measures have to be designed to capture both positive and negative effects of AI. At Google, for example, engineers have long been measured on three dimensions. Speed is one: how long does a code review take, say. Ease is another: how much friction is there in the system for things like onboarding new developers. The last, critically, is quality: “I don’t want to just go fast and make it easy to ship terrible software,” says Richard Seroter of Google Cloud.

Complicating matters further is the fact that it often takes a while for the benefits of AI to show up. People have to spend time learning how to use the technology. Increased output can create bottlenecks elsewhere. A paper published last year by Kristina McElheran of the University of Toronto and her co-authors looked at AI adoption among American manufacturing firms, and confirmed the existence of a “J-curve” in which productivity dips before it leads to an improvement. This effect is especially pronounced at established firms and seems to be partly explained by changes to management practices that had previously helped firms to keep tabs on performance. Any kind of calculation about projected returns has to take this initial period of disruption into account.

Bosses also have to take a view on something much more fundamental: what kind of return are they interested in? One way to show financial returns on AI is to seize on time savings as an excuse to cut headcount. For some jobs and in some circumstances, that might make sense. But chainsaws and employee morale go together about as well as salt and slugs. And people who are no longer there cannot be redeployed in more productive ways. A better option is to calculate the amount of money that AI saves on expanding headcount. (Working out how much extra revenue can be attributed to the technology is even more of an art; at the very least, you need good baseline data and a culture of A/B testing to isolate its effects.)

On top of everything, there is wild uncertainty about where AI is heading. That is an argument for a third set of measures, to go alongside ones on inputs and outcomes. These measures, which Mr Seroter calls “organisation-based”, are more geared towards capturing how well change is being managed. They could include data on levels of employee satisfaction with the way AI has been implemented, or progress towards building in-house expertise in areas that AI will not automate. A one-eyed focus on usage ignores the importance of returns. A rigid focus on returns ignores the need to keep learning. ■