Experimentation vs Production Economics

Suppose an AI pilot handles 90 percent of its cases correctly, and the workflow it is meant to join needs 99 percent. How much more work is that? A common first answer is about ten percent more: nine points on a scale of a hundred. Count the errors instead. In 10,000 cases, a system that is right 90 percent of the time makes 1,000 errors. One that is right 99 percent of the time makes 100. The production bar is not nine points higher. It is a tenfold cut in errors.

Engineers who run large systems know this curve well. Google’s site reliability engineers put it plainly: cost does not rise in a straight line as reliability improves, and one more increment of reliability may cost a hundred times more than the one before¹. The same is true of quality. That is one reason a pilot’s cost is such a poor guide to what production will cost. Data, Integration and Operational Costs ended with another: a pilot can prove an idea with clean sample data, one connection and a team willing to check every output by hand. Production can do none of those things.

The core idea

An experiment and a production system are different investments with different objectives, and they should be judged by different numbers. An experiment buys information. Its job is to reach a good decision, quickly and at a sensible cost: does the idea work, will people use it, is the value large enough? A production system buys reliable delivery. Its job is to produce business outcomes at the required quality, for as long as it runs, at an acceptable complete cost.

Two mistakes follow from confusing them, and they point in opposite directions. The first is to run an experiment as if it were already production, spending on robustness before anyone knows the idea is worth it. The second is to approve production on an experiment’s numbers, as if the pilot’s invoice were a sample of the real bill. This module measures the real bill as total cost of ownership: build plus run plus operate plus change, over the system’s life, as Understanding AI Total Cost of Ownership defined it. An experiment’s cost is only the first few lines of that sum.

An experiment is priced by the decision it informs

How much should an experiment cost? The useful answer comes from decision analysis. In 1966 Ronald Howard showed that the probabilities and the economics of a decision can be considered together, so that reducing an uncertainty can be given a numerical value². Information is worth what it would change about the decision, and no more.

An illustrative worked example makes the idea concrete; its figures are invented. Suppose a finance team is asked to approve 250,000 to test an AI tool that matches supplier invoices. If the tool works, the production system gains 3 million over its life, net of all costs. If it does not, the organization loses 2 million. Finance puts the chance of failure at 30 percent.

Without a test, the expected value of going ahead is 0.7 times 3 million, minus 0.3 times 2 million: 2.1 million minus 600,000, or 1.5 million. It is worth doing, and three times in ten it loses 2 million. Now imagine a test that reliably showed in advance which case you were in. You would go only when the tool works, so the expected value rises to 2.1 million. The difference, 600,000, is the most any test of this decision can be worth. It equals the chance of failure times the loss it would let you avoid. A real experiment is sometimes wrong, so it is worth less than that ceiling, but the ceiling sets the scale. Here the 250,000 is under the ceiling, so it can be justified, but only if the test is quick and aimed at the riskiest assumption, such as the match rate on live invoices.

Three practical rules follow. First, budget an experiment against the decision it informs, not against zero. Spending 250,000 to protect a 2 million loss can be a bargain; the same 250,000 to protect a 200,000 loss is not. Second, design the experiment to answer the question that would change the decision, usually the assumption with the most money resting on it, rather than to produce a demonstration. Third, speed has value. A test that takes six months delays the decision and the gain with it, so time to decision is an economic measure in its own right.

The example also shows why an experiment that ends in a stop can be a success: in the 30 percent case, stopping is precisely what it was paid to make possible. The portfolio discipline that turns this into practice, with stop rules written before the money moves, belongs to Prioritizing the AI Portfolio in Module 09.

What a pilot proves, and what it does not

As What Does AI Value Actually Mean? showed, a pilot proves feasibility; scale and return are separate questions. For the production case the practical question is narrower: which of the pilot’s numbers can be carried across, and which must be measured again?

Quality measured on the pilot’s cases is real evidence, but those cases were often chosen, cleaned or both, and the Thai screening clinics in the previous chapter showed what live data does to a threshold. Time saved is real for the tasks and people tested; it says less about the tasks and people who were not. Enthusiasm is the weakest number of all, because pilot users are usually volunteers, enjoy close support from the team and are using something new.

Cost per request is the number most often carried across, and it should not be. A pilot rarely pays for integration, security, on-call support or monitoring, and many pilots contain a hidden worker: a team member who checks outputs by hand before anyone sees them. That person is not in the pilot’s budget, but the work they do is real, and in production it must either be priced as an exception queue or engineered away.

The pilot is a small share of the bill

The gap between a working prototype and a working system is well documented. In a widely cited 2015 paper, Google engineers observed that the model code is only a small fraction of a real machine learning system; the infrastructure around it is “vast and complex”³. Many experiments never cross the line: IDC research conducted with Lenovo, a hardware vendor, found that for every 33 AI proofs of concept a company launched, only four reached production⁴, and Gartner predicted in 2024 that at least 30 percent of generative AI projects would be abandoned after proof of concept, with escalating cost among its reasons⁵. A 2022 survey of published deployment case studies adds that problems arise at every stage, from data management to monitoring⁶.

The illustrative figure shows the shape. A three-month pilot costs 200,000: mostly build, with a little cloud spend and some of the team’s time. Production costs 1.8 million in its first year, as integration, security, monitoring and support are built, and 1.1 million in each year after that, as the system runs, is operated and absorbs changes. Over three years that is 4 million. The pilot was 5 percent of it. None of the families were hidden; Understanding AI Total Cost of Ownership and Data, Integration and Operational Costs priced each of them. They simply arrive at the production gate, and nothing in the pilot’s invoice shows them coming.

The production bar is a cost decision

Quality is one bar production must clear. Availability is another, and it follows the same arithmetic. A system at 99 percent availability may be down for about 88 hours a year, which is acceptable for an internal report and unacceptable for a workflow that stops when the system does. Each extra nine cuts the permitted downtime tenfold, to about 9 hours at 99.9 percent, and requires more redundancy, more monitoring and someone on call. That is the non-linear cost curve the site reliability engineers describe¹. A pilot that is down for an afternoon, by contrast, inconveniences a few volunteers.

The executive point is that the bar should be set by the workflow, not by engineering pride. What does one error cost the business, and what does one hour down cost? The expected-downtime test in Data, Integration and Operational Costs gives the method for the second question. Without an agreed bar, a production estimate is meaningless, because the same system can cost several times more at one bar than at another.

Two ways to get it wrong

The first error is to overbuild the experiment. A team builds a production-grade platform, with high availability, full security review and polished interfaces, for an idea nobody has yet shown to be valuable. Money and months are committed before the main uncertainty is reduced, and the time to decision stretches. Worse, the size of the investment makes stopping harder, which is exactly the escalation of commitment that Prioritizing the AI Portfolio warns against.

The second error is to underbuild production. A successful prototype is promoted with its shortcuts intact: the manual steps, the hand-checking, the single server, the absence of monitoring or anyone on call. Sculley and his colleagues called these shortcuts technical debt, and the interest is paid later as incidents, rework and hurried re-engineering³. Shortcuts are rational in an experiment because many experiments end; they are expensive in production because production systems do not.

A few things are not optional at either stage. Security for real customer data, and the law on personal data, apply from the first day an experiment touches them.

Rebuild the case at the gate

At the production gate, the pilot’s case should not be scaled up. It should be rebuilt, because almost every number in it was measured in conditions production will not have.

Four things belong in the rebuilt case. Full cost of ownership, including the hidden worker priced as an exception queue or removed by engineering. Adoption among the people who must use the system, rather than the volunteers who chose to. Scenarios for volume: what happens if usage is three times the forecast, and what happens if only a fifth of the intended users adopt it? And an agreed bar and an owner: the quality and availability the workflow needs, and a named person accountable for the outcome after launch. How those scenarios turn into cost per customer and break-even is the subject of the next chapter.

The case does not stop being an estimate on launch day. The FinOps discipline treats comparing forecast cost with actual cost, per unit of business value, as a continuing practice⁷⁸. Usage, prices and behavior all drift, so the production case should be re-run with real figures each quarter. How to take a system across the line in stages, region by region, is covered in From AI Pilot to Production to Scale.

Story: Sweden’s screening trial and England’s decision

Breast screening programs in Sweden and England both rely on double reading: two specialists read every mammogram. By 2021 several retrospective studies, which ran AI over archives of past mammograms, had shown promising results, but no randomized trial had tested AI inside a working program⁹. That is the gap between a pilot and production. An archive is chosen data with known answers. A screening program is thousands of women a week, real radiologists, and outcomes that take years to show.

Sweden ran the test inside its national program. From April 2021 women at four screening sites in the southwest of the country were randomly assigned either to standard double reading or to AI-supported screening, in which an AI system scored each examination, sent most of them to one radiologist and the highest-risk ones to two, and marked suspicious areas. The first safety analysis, of 80,033 women, was published in August 2023. It found a similar cancer detection rate, no rise in false positives, and 44 percent less reading work. The main question, how many cancers would surface between screening rounds, needed two more years of follow-up⁹.

Suppose you run England’s program at the end of 2024. It carries out about 2.1 million screens a year, each read by two specialists¹⁰. Do you (A) adopt AI as one of the two readers nationally, on the strength of Sweden’s numbers; (B) wait for the Swedish follow-up and for others to move first; or © pay for a large trial in English screening conditions before deciding?

In February 2025 the UK government chose C. It put 11 million pounds, through the National Institute for Health and Care Research, into the EDITH trial: nearly 700,000 women at 30 NHS sites, testing whether AI lets one specialist safely do the screening work that takes two today¹⁰.

Sweden’s full results came out over the following year. Among the 105,934 women randomized, the AI group found 6.4 cancers per 1,000 women screened against 5.0, without significantly more recalls or false positives, and needed 61,248 screen readings against 109,692, a 44 percent cut¹¹. The two figures measure different things: the 29 percent is cancers found at the screen itself, and the 44 percent is the number of human readings, not radiologists’ total time. In January 2026 the main result followed: cancers surfacing between screens were no more common with AI, 1.55 against 1.76 per 1,000, and sensitivity rose from 73.8 to 80.5 percent with the same specificity¹². Option B would have bought that certainty, but only about Sweden.

Why not A? Because few pilot numbers carry over unchanged, even from a well-run randomized trial. The Swedish trial screened women aged 40 to 80, mostly every 18 to 24 months, with one AI product at four sites⁹. England invites women aged 50 to 71 every three years¹³, so cancers have longer to grow between rounds, and the result that matters most, cancers missed between screens, could come out differently. That reasoning is this chapter’s, not the government’s; its stated case was that a successful trial could free up hundreds of radiologists¹⁰.

The economics of option C follow the chapter’s rule: price the experiment by the decision it informs. A 2026 model of the UK program, built on prospective trial evidence, estimated that replacing one human reader with AI would cut lifetime costs by about 31 pounds for every woman invited while slightly improving outcomes¹⁴. It is a model, not a measurement, which is exactly why the measurement is worth buying. And if the trial finds that AI does not hold up in English conditions, the stop will be a success: it will have prevented a national change that would not have paid.

What this means for leaders

Treat the two investments differently and ask different questions of each. For an experiment, ask what decision it informs, how much money rests on that decision, and how soon the answer will come; a large experiment protecting a larger decision can be good value, and a stop can be a success. For production, ask for a rebuilt case: full cost of ownership, adoption among the real users, volume scenarios, an agreed quality and availability bar, and an owner. Never let the pilot’s invoice stand in for the production budget, and never let a pilot be built as if it were already production.

Check yourself

1. A cheap pilot is good evidence that production will be cheap.MythFact
2. The most an experiment can be worth is the expected loss it lets you avoid.MythFact
3. Moving from 90 to 99 percent quality is roughly a ten percent improvement.MythFact
4. Experiments should be built to production standards so they can scale quickly.MythFact
5. An experiment that ends in a decision to stop can be an economic success.MythFact
6. The production case should be rebuilt from full TCO, real adoption and volume scenarios, not scaled up from the pilot.MythFact

Reflection: one pilot, two budgets

What comes next

A rebuilt production case still has to answer the question finance will ask first: what does each customer, each transaction or each outcome cost, and does that number improve or worsen as use grows? England’s screening decision turns on exactly that: what each screen costs to read, and how value and cost behave across a whole national program. Measuring it properly is the subject of AI Unit Economics and Economics at Scale.