Human Oversight and AI Incidents

In a study published in 2023, a team at University Hospital Cologne asked 27 radiologists to read 50 mammograms and grade each one on the standard breast-cancer risk scale. Beside every image sat a suggestion presented as coming from an AI system. In fact the researchers controlled the suggestions, and for some of the images they made them deliberately wrong. Every reader was a trained professional, free to ignore the machine. How far would you expect their accuracy to fall when the suggestion was wrong?

When the suggestion was right, inexperienced readers graded about 80 percent of mammograms correctly. When it was wrong, they graded 19.8 percent correctly. Moderately experienced readers fell from 81.3 to 24.8 percent. Even the most experienced group, with an average of more than 15 years in the specialty, fell from 82.3 to 45.5 percent¹. Nobody forced them to agree. The human was in the loop on every image, and accuracy still fell below half in every group.

Presence is not control

Many AI governance documents promise a “human in the loop”. The Cologne study shows why that phrase settles nothing. A person can sit at the decisive point of a process, sign every output and still exercise almost no independent judgment. Human oversight is meaningful only when four conditions hold. The person has the authority to reject, override or stop the system without having to justify it upward first. They have the information to judge: the inputs, the evidence behind the output and some sense of how confident the system is. They have the time to look, which a queue of several hundred cases before lunch does not allow. And they have the tools to act: a way to pause the system that works without the system’s cooperation.

The law has started to say the same thing. As of October 2026, the EU AI Act (Art. 14(4)) requires providers to design high-risk systems, with Annex III duties applying from 2 December 2027, so that the people assigned to oversee them can understand the system’s capacities and limits, interpret its output, decide to disregard, override or reverse it, and interrupt it through a stop button or a similar procedure. The same article asks that those people remain aware of “automation bias”: the tendency to rely automatically, or too much, on the system’s output². The NIST AI Risk Management Framework, the US government’s voluntary framework for AI risk, puts it in operational terms: mechanisms must be in place, and responsibilities assigned and understood, to supersede, disengage or deactivate an AI system whose outcomes are inconsistent with its intended use³.

Automation bias: why reviewers stop reviewing

The psychologists Linda Skitka, Kathleen Mosier and Mark Burdick gave the problem its name in a series of experiments in the late 1990s. They defined automation bias as using an automated aid as a shortcut in place of vigilant checking, and they found two kinds of error⁴. In an error of omission, people miss a problem because the aid did not flag it. In an error of commission, people follow the aid’s instruction even when other, valid information in front of them says it is wrong. The radiologists in Cologne made errors of commission. The detectives in the story later in this chapter made both.

The bias feeds on success. A system that is right most of the time teaches its reviewers that checking is wasted effort, and the rare wrong answer arrives when attention is lowest. This is the human side of the irony Lisanne Bainbridge described in 1983, which Operational and Workforce Risk covered as a loss of skill⁵. Here the concern is narrower and more immediate: the skilled person is present and capable, and still does not challenge.

The evidence points to one remedy. Skitka’s team found that people told they would be held accountable for their decision accuracy showed less automation bias⁶. Accountability for the outcome, not for clearing the queue, changes how people look. The Cologne study also found inexperienced readers the most exposed, so oversight left to the most junior person in the queue is the weakest kind¹. Showing the evidence behind an output, not only the answer, and telling reviewers when the model has changed, help too.

Where the human stands on the ladder

Agentic AI and Autonomous Actions set out the module’s one autonomy ladder: generate, recommend, plan, execute and operate autonomously. Oversight takes a different form on each rung, and the classic terms map onto it cleanly.

On the first three rungs the human is in the loop: nothing changes in the world until a person decides or approves. That is where automation bias does its damage, because the formal control is strong and the real one can be empty. On the fourth and fifth rungs the human is on the loop: the system acts within limits while people monitor it, handle the exceptions it escalates and can stop it. There the risk is the omission error. Nobody looks at the dashboard, the alert threshold is wrong, or the person who can stop the system cannot be found.

Out of the loop, with no human watching at all, is not a rung. It should be a deliberate decision, reserved for tightly bounded tasks with small and reversible consequences, and someone should be able to say why it is acceptable. More often it is where a system ends up by drift, when in-the-loop reviewers stop reviewing and on-the-loop monitors stop watching.

Testing whether oversight is real

An executive cannot watch reviewers at work, but can ask for evidence. The question is not whether a human is in the process. It is whether that human ever changes the outcome, and whether the organization would know if they stopped.

The override rate is the first number to ask for, and it has to be interpreted. A rate near zero can mean the system is excellent or that nobody is looking; a rate that does not move when the model is updated suggests the latter. Time per case is the second, because no one exercises judgment in a few seconds. A strong test is to seed known errors into the queue now and then and see how many are caught. The last is the stop drill: whether the person named as able to pause the system can actually do it, today, without calling the vendor.

When an error becomes an incident

Not every AI mistake is an incident. A draft with a wrong figure that its author corrects before sending is an error: fix it, log it and watch for patterns. Treating every error as an incident floods leadership. Treating real incidents as errors hides harm. The OECD offers a definition that organizations can adapt. An AI incident is an event in which the development, use or malfunction of an AI system directly or indirectly leads to harm: to people’s health, to critical infrastructure, to rights or legal obligations, or to property, communities or the environment. An AI hazard is an event that could plausibly lead to one⁷. The hazard category matters, because a near miss caught by luck is the cheapest lesson an organization will get.

Public incidents are rising: the Stanford AI Index counts 362 in 2025, up from 233 in 2024⁸, though that measures visibility as much as frequency. Inside an organization, the definition and the severity levels should be set before the first incident, by the people who will have to act on them, together with the triage questions: what happened, who and how many are affected, which systems, data and actions are involved, and whether it is still happening.

The incident lifecycle

Once something is an incident, the response is a sequence of distinct stages, and each fails in its own way.

Detection should never depend on one channel. Monitoring, audits, the security team, employees and customers all count, and an organization whose customers are its first alarm has a detection problem. Containment stops the harm from growing: pausing the model, disabling a tool, revoking access, switching to a manual fallback or requiring human approval for a class of action. It depends on the stop mechanism the oversight design put in place. Recovery returns the business to a safe state, which can mean reversing actions, reprocessing work and reviewing every decision the system made while it was wrong. Review asks why the system was allowed to act and why nobody caught it, not only what the model did. It needs evidence that has to be preserved early: the model version, inputs, outputs, tool calls, human actions and a timeline. Learning turns findings into changed controls and tracks them to completion. NIST’s framework asks that incidents be communicated to the people affected and that the processes for tracking, responding to and recovering from them be followed and documented³.

None of this works if it is invented on the day. The owners, the fallback and the stop mechanism should be named in advance and rehearsed in a tabletop exercise in which the AI fails on a bad evening. How the business keeps running without the system was covered in Operational and Workforce Risk; here the point is that containment depends on it.

Some incidents start a legal clock

An AI incident is often also a data breach, a cybersecurity incident or an operational disruption, and several regimes set deadlines that run from awareness, not from the end of the investigation. One failure can start more than one clock.

The practical consequence is a decision right. The incident plan has to say who decides, within hours, whether a clock has started, and that person has to be part of the first call, not the third.

Story: a Thursday in Farmington Hills

On a Thursday afternoon in January 2020, Robert Williams was at his desk at an automotive supply company when the city’s police department called and told him to come to the station to be arrested. He thought it was a prank. An hour later, as he pulled into his driveway in Farmington Hills, a police car blocked him in. Two officers handcuffed him on his front lawn in front of his wife and two young daughters. He was held overnight. Around noon on Friday, two detectives sat him down and showed him a still from a store surveillance video of a heavyset man in a red cap. He held the picture next to his face and told them it was not him. In his recollection, one detective said to the other, “I guess the computer got it wrong.” He was not released until that evening, 30 hours after his arrest⁹.

The theft had happened in October 2018: five watches worth 3,800 dollars. In March 2019 an examiner at the Michigan State Police ran a still from the video against a database of about 49 million photos, and Williams’s driver’s license photo came back among the possible matches. The report sent to Detroit police said in bold capitals that it was only an investigative lead and not probable cause for arrest. The detectives put his photo into a six-picture lineup and showed it to a loss-prevention contractor who had seen only the same video. She picked him. That was the basis for the warrant⁹.

Every safeguard a policy writer would list was present. A trained examiner reviewed the matches. The system’s own output warned against relying on it. A witness identified the suspect. A detective applied for a warrant, and a prosecutor charged. None of these steps was independent of the algorithm. The lineup was built around its candidate, and the witness had seen only the same video the algorithm had searched. Each person treated the previous step as the check. That is an error of commission repeated down a chain.

What happened next is the incident-response lesson. The department’s public answer was that it did not make arrests based solely on facial recognition⁹. In February 2023 its officers arrested Porcha Woodruff, eight months pregnant, for a carjacking, after another facial recognition match. She was held for about 11 hours, and the case was dismissed about a month later¹⁰. Only in June 2024, settling Williams’s lawsuit, did the department accept rules that a learning process might have produced years earlier: no arrest based solely on a facial recognition result or on a lineup that directly follows one, independent evidence first, training on the technology’s risks, an audit of cases back to 2017 and court oversight for four years. The ACLU of Michigan counts three known wrongful arrests in Detroit involving the technology; all three people were Black¹¹. The oversight failed for everyone, and the organization did not learn until it was made to.

What this means for leaders

Treat “a human reviews it” as a claim to be tested, not a control to be counted. For every AI system that matters, know which rung it is on and therefore whether the risk is rubber-stamping or not watching. Ask for the override rate, the time per case and the results of a seeded-error test, and make reviewers accountable for the quality of outcomes rather than the speed of the queue. Make sure someone can stop the system tonight without the system’s help, and that they have practiced. Agree the incident definition, the severity levels and the owners before the first incident, and put the person who decides on legal notification into the first hour. After an incident, insist on a review that changes a control, and check that the change happened.

Check yourself

1. Requiring a human to approve every AI output is enough to prevent harmful decisions.MythFact
2. Experienced professionals are immune to automation bias.MythFact
3. Making reviewers accountable for the accuracy of outcomes can reduce automation bias.MythFact
4. A near-zero override rate proves that an AI system is accurate.MythFact
5. Containment and recovery are separate stages of incident response.MythFact
6. Notification to regulators can wait until the investigation is complete.MythFact

Reflection: fail one system tonight

What comes next

This chapter completes the module’s tour of individual risks: accuracy, bias, privacy, security, intellectual property, third parties, operations and the workforce, agents, and now oversight and incidents. The last chapter of the module, Module 06 Synthesis — Understanding AI Risk, brings them together into one executive judgment: which AI risks are acceptable, which need controls, and which should stop a deployment.