The AI Risk Landscape

Pages like the illustrative one below are a familiar sight in steering-committee packs. A team has built an AI system. It agrees with experienced staff 98 percent of the time on a test set of 5,000 past cases. Strictly, that is an agreement rate with past human decisions, not proven accuracy, but the page calls it accuracy. The status light is green, and the team would like a signature before Monday’s go-live.

Look at what the page does not say. It does not say what the system does with its answers, who acts on them, or what happens in the two percent of cases where it is wrong. It does not say how quickly anyone would notice an error, how many customers one error could reach, or whether the result can be undone. Every number describes the model. None describes the business.

That gap is not hypothetical. In 2018 Zillow, the US property website, began buying homes directly, using its own pricing models to decide what to offer. In the third quarter of 2021 the company bought 9,680 homes, sold 3,032, and wrote down the value of its inventory by 304 million dollars because it had paid more than it now expected to sell for. In November it announced it would close the business and cut about a quarter of its workforce. The shareholder letter was candid: the company had been unable to forecast home prices “by much more than we modeled as possible”¹.

A forecast that misses is a model problem. A forecast that commits the company’s capital to thousands of homes in a single quarter, in a market moving faster than the model could follow, is a business problem. The difference between the two is the subject of this chapter.

Risk is a property of the use

The National Institute of Standards and Technology defines risk as a composite of how likely an event is and how severe its consequences are. It adds a point that matters more for leaders: AI systems are socio-technical. Their risks come from the technology together with how it is used, who operates it and the setting it is used in².

That gives a working definition for an executive. AI risk is the potential business impact when an AI-enabled system produces, amplifies or acts on an outcome that is incorrect, unsafe, unauthorized or inappropriate. The error rate is one input. The consequence of the error is the other, and only the business can judge it.

Both questions matter. The mistake is to answer the first and assume you have answered the second. The goal is not zero risk, which no useful system and no human workforce achieves. It is risk that is understood, owned and controlled in proportion to the value at stake.

Risk can enter at every layer

A common mistake is to treat the model as the whole system. In practice the model is one layer of six, and risk can enter at any of them.

At the bottom, data decides what the system knows; a model built on past prices can miss a market that moves faster than it ever has, as Zillow found. The model is the next layer. The application wraps the model in a product, and a good model in a badly designed product is still a bad system. Tools and integration decide what the system can read and change, and every connection widens the surface: researchers have shown, in controlled tests, that instructions hidden in a web page or an email can hijack an AI assistant that reads them, a problem the chapter Security and AI Attacks covers³. The workflow decides where the output lands, and a correct output in the wrong process is still a bad outcome. At the top sit people and the organization: who trusts the output, who checks it and who is allowed to stop it.

The top layer is easy to underrate. One of the largest reviews of AI risk taxonomies, the MIT AI Risk Repository, catalogues 1,725 risks from 74 frameworks. Its authors found that human decisions cause nearly as many of those risks (38 percent) as AI systems themselves (42 percent); the rest have other or unclear causes. Both shares count risks named in the frameworks, not incidents or losses⁴. More than forty years ago, the psychologist Lisanne Bainbridge described the same irony in industrial automation: the more a system is automated, the harder the remaining human job of supervising it becomes⁵. Why oversight fails, and how to make it real, is the subject of Human Oversight and AI Incidents.

The map: six families of risk

There is no single authoritative list of AI risks, and you do not need one. NIST’s profile for generative AI names twelve risks that the technology creates or makes worse, from confabulation and data privacy to information security and the integrity of the supply chain⁶. The OWASP Top 10 for LLM applications ranks the security risks, led by prompt injection⁷. For an executive, six families cover the ground, and each has its own chapter in this module. The six are this course’s grouping, and most of NIST’s twelve fall into them: confabulation into accuracy and reliability, harmful bias or homogenization into bias, data privacy into privacy, information security into security, intellectual property and value chain and component integration into the IP and providers family, and human-AI configuration into operations and oversight. The rest, such as harmful or dangerous content, information integrity and environmental impacts, are content and societal harms that cut across the families.

The families are a map, not a filing system, because real failures rarely stay in one box. The Dutch childcare benefits scandal is a well-documented example. For years the Tax Administration used an automated system that designated benefit applications as risky, and nationality was one of its indicators. It had also kept data on the dual nationality of 1.4 million people long after the law required it to be deleted. In December 2021 the Dutch data protection authority fined the administration 2.75 million euros for discriminatory and unlawful processing⁸. By then the wider scandal had already brought down the government, which resigned in January 2021⁹. One system touched bias, privacy, oversight and law at the same time, and the harm reached thousands of families before anyone with authority stopped it.

The first cluster: wrong, trusted, acted on

The family many leaders meet first is accuracy and reliability. It has three members worth telling apart. Accuracy is whether the output is correct. Hallucination, which NIST calls confabulation, is confidently stated content that nothing supports⁶. Reliability is whether the system behaves consistently enough for the process it serves. The next chapter treats all three in depth; here it is enough to see how they become business risk.

The chain explains why the same error rate can be trivial or serious. Break any link, by checking the output, withholding trust or keeping the action reversible, and the risk shrinks. Remove every check, and a small error rate flows straight into the world.

The same capability, three different risks

Because risk lives in the use, the same capability can carry very different risk in different places. Consider one capability, reading documents and summarizing them, in three illustrative uses.

Regulators reason the same way. The EU AI Act does not regulate models by how accurate they are; it classifies systems by what they are used for. Using AI to price life and health insurance for individuals is listed as high-risk, as is using it to decide eligibility for public benefits, which is exactly the kind of decision at the heart of the Dutch case¹⁰. How the law’s tiers fit together with your own risk appetite is covered in the module synthesis, Understanding AI Risk.

Four multipliers: undo, reach, notice, speed

The classic starting point, likelihood times impact, is useful and incomplete. For AI, four further questions multiply or shrink the impact of any error. The table shows three illustrative actions.

Reversibility asks whether the result can be undone. A draft can be deleted; a payment, a signed contract or a deleted record mostly cannot. Reach asks how many people, systems or decisions one error can touch before it is stopped: one reader, one customer or the whole book of business. Detectability asks whether anyone would notice. Some errors announce themselves; others, such as a customer who is overpaid, generate no complaint at all.

Speed is the fourth, and automation changes it most. A person makes one mistake at a time. A system connected to automation can repeat one mistake thousands of times before anyone looks, as the story below shows with refunds.

Autonomy ties the four together. The more the system can do on its own, the fewer people stand between an error and its consequence. This module uses one ladder for that, with five rungs from generate through recommend, plan and execute to operate autonomously; Agentic AI and Autonomous Actions explains it. When several multipliers stack up, irreversible, far-reaching, hard to notice and fast, the controls must be strongest.

Story: one assistant, two go-live choices

The following is an illustrative composite, not a documented case. Return to the dashboard that opened this chapter and suppose it belongs to a hotel group with resorts along one coast. A hurricane has forced several of them to close in peak season, and about 12,000 guests are owed refunds and compensation for cancelled stays in the next three weeks. The guest-services team has built an assistant that reads bookings, guest messages and receipts and proposes a refund. On 5,000 past refund cases, its proposal matched an experienced agent’s 98 percent of the time.

Two choices reach the head of guest services on the same Friday.

Choice A is the team’s proposal: pay every refund under 5,000 automatically from the first day, because guests whose holidays a storm has ruined want their money and experienced agents are scarce. It is fast. But two percent of 12,000 is about 240 wrong refunds in three weeks, and every one of them is paid before a person looks at it. The underpayments will surface slowly, as complaints and perhaps as complaints to the consumer regulator. The overpayments will not surface at all, because nobody complains about being paid too much, and recovering money from guests after a storm is both hard and damaging to the brand. A published threshold also tells anyone inclined to inflate a request exactly where to aim.

Choice B uses the same assistant differently. Only refunds under 200 are paid automatically. Suppose, for illustration, that a third of the 12,000 claims fall below that line: about 4,000 automatic payments, of which 2 percent, about 80, will be wrong, each by less than 200. Each day a reviewer re-checks a random 5 percent of the automatic payments, about 200 over the three weeks, which is enough to see whether the error rate is holding. The other 8,000 or so refunds arrive as drafts that agents approve in batches, which still saves most of their time. One named person can switch automation off. As the samples build evidence, the automatic band widens, perhaps to 500.

Choice B is slower in its first days and costs agent hours. That is its price, and it should be stated openly. What it buys is visible in the multipliers: errors are caught while they can still be undone, reach is limited to a small band of refunds, and the daily sample makes the silent errors visible. The dashboard was identical for both choices. The risk was not, because the risk was never in the 98 percent. It was in what the business allowed the system to do with it.

What this means for leaders

The landscape yields four habits. First, ask the business question every time: what happens when it is wrong? A go-live page that answers only the model question is incomplete, however good its numbers. Second, look at the whole system: data, application, integrations, workflow and people, not only the model. Third, use the map to check which families a use case touches, knowing that many touch more than one. Fourth, size the multipliers, and put the strongest controls where errors are irreversible, far-reaching, quiet and fast. What you then do with a risk, whether you accept, reduce, transfer or avoid it, is the subject of the module synthesis.

Check yourself

1. AI risk basically means hallucination.MythFact
2. The same model can carry very different risk in two different workflows.MythFact
3. A system that is 98 percent accurate is safe to automate fully.MythFact
4. Human decisions cause a large share of AI risks.MythFact
5. The EU AI Act classifies AI systems mainly by how accurate they are.MythFact
6. Errors that nobody complains about carry no risk.MythFact

Reflection: read your own dashboard

What comes next

The map shows where AI risk lives. The family you are likely to meet first is accuracy: systems that are fluent, confident and sometimes wrong. The next chapter, Accuracy, Hallucination and Reliability, explains why that happens and how to design systems that detect, contain and recover from it.