Operational and Workforce Risk

At the end of 2021, four endoscopy centers in Poland introduced an AI system that flags possible polyps during colonoscopy. It was designed to help doctors find more of them. Then researchers looked at something else: how well the same doctors performed when they worked without it. Before the AI arrived, their adenoma detection rate in unassisted procedures was 28.4 percent. A few months after it arrived, in procedures done without the AI, the rate was 22.4 percent, a fall of six points, or about a fifth¹.

These were not trainees. They were experienced specialists, each with more than two thousand colonoscopies behind them. The study is observational, so it cannot prove the AI caused the fall, and other centers may see different results. But it is among the first real-world measurements of a risk that any organization adopting AI may carry: the tool helps, and the people beside it can quietly get worse at the job the tool is helping with.

The risk that grows while the numbers improve

Most AI risks in this module are about what the system does: a wrong answer, a biased score, a leaked document, a failed provider. Operational and workforce risk is about what the system does to the organization around it. It is the risk that AI changes how work is done in ways that weaken the organization’s ability to perform, check and recover, even while every productivity measure improves.

That is what makes it dangerous. Throughput rises, queues shrink and unit costs fall, and all of that can be true while the capability underneath erodes. The usual dashboards do not show it, because they measure the normal day. This risk only becomes visible on an abnormal one: an outage, an unusual case, a crisis, a regulator’s question that nobody left in the building can answer.

Whether a reviewer actually challenges the AI’s output is the subject of Human Oversight and AI Incidents; this chapter is about the capability an organization needs to keep, and how it loses it.

The ironies of automation

A classic explanation of this risk was written more than forty years ago, about process plants rather than chatbots. In 1983 the engineering psychologist Lisanne Bainbridge published a short paper called Ironies of Automation². Her argument was that automation does not remove the human from the system. It changes the human’s job, and often makes it harder.

When a machine takes over the routine work, the person is left with two tasks: watching the machine and taking over when it fails. Both are difficult. People are poor at monitoring something that is almost always right. And taking over requires exactly the skill that the automation has stopped them practicing. The person is expected to step in on the rarest, hardest cases with the least recent experience. Bainbridge’s conclusion is the line every executive sponsor of AI should know: “it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training”².

This is the automation paradox. Automation makes normal operations easier and abnormal ones harder. Generative AI can extend the paradox from control rooms into offices, because it can now do much of the routine part of drafting, analysis, review and diagnosis, which is where professionals used to keep their skills sharp. How far office skills actually erode is still being measured; the endoscopy study is an early signal, not a settled result. The generative AI profile of the US National Institute of Standards and Technology lists over-reliance and the way humans and AI are configured to work together among the distinct risks of the technology, not as a side effect of model quality³.

Three ways capability erodes

Capability does not disappear in one way. It erodes through three different mechanisms, and each needs a different response.

Deskilling is what the endoscopy study measured: experienced people losing some of a skill they used to exercise every day. It is the direct form of Bainbridge’s irony, and it matters most where the human is the backstop for the AI.

Never-skilling is slower, harder to measure and may prove more serious. Experts were made by years of routine work: the simple cases, the first drafts, the reconciliations and the basic diagnoses. If AI does that work, juniors lose the ladder that experts climbed. As AI and the Future of Work showed, early-career employment in the occupations most exposed to AI is already falling behind trend⁴. Whatever that means for jobs, it raises an operational question for every employer: who will be the expert who checks the AI in ten years, and where will that person have learned?

Offloaded judgment is the quietest of the three. People still do the task, but they stop thinking it through. A survey of 319 knowledge workers by researchers at Carnegie Mellon University and Microsoft Research found that the more confident people were in generative AI, the less critical thinking they reported applying; the more confident they were in their own ability, the more they applied⁵. The finding is self-reported, so it is a warning rather than a measurement. It is consistent with what managers describe: work that looks finished and has not been understood.

Erosion is a design choice, not a law

None of this means AI must make people worse. In a large study of customer-support agents, access to an AI assistant raised issues resolved per hour by 15 percent on average, and on the days the system was down the agents who had used it still performed better than before it arrived⁶. The AI had taught them. The skill stayed with the person.

Set that beside the endoscopy result and the lesson is clear. The same class of technology can build human capability or replace the practice that sustains it. The difference lies in how the work is designed around it.

The practical design choices are not exotic. Let people attempt some cases before seeing the AI’s answer. Prefer tools that show their reasoning and sources rather than only a conclusion. Keep a deliberate share of hard cases for people who are still learning, with a senior beside them. Schedule unaided practice for skills that the organization must keep, the way safety-critical industries schedule drills. And measure capability directly: how many people can do the critical task without the tool, not only how fast the team works with it.

Where expertise concentrates

The second operational risk is concentration. AI can often handle explicit knowledge: procedures, policies, routine cases. What remains with people is tacit expertise, the judgment that is hard to write down because it lives in experience.

As AI absorbs the routine work, that tacit expertise concentrates in fewer people, who are often the most experienced and closest to retirement. When they leave, the organization may lose the ability to check its own AI-enabled process or to run it when the AI cannot. AI also creates a new kind of key person: the one engineer who understands how the prompts, the integrations and the data feeds fit together. An organization that removes a dependency on a few clerks and creates a dependency on one integrator has not reduced its operational risk. It has moved it.

Degraded mode: the AI-down test

Every critical process needs an answer to a simple question: what happens when the AI is not there? Providers have outages. Model updates change behavior. A security incident can force a system offline. As Model and Third-Party Risk showed, a single provider outage can stop many teams at once. The question here is not how to restore the provider. It is whether your own people can keep the work going while it is gone.

The AI-down test asks three things. Can the process continue without the AI? For how long? At what capacity?

Not every process needs a manual route. Some can simply wait. The failure is not choosing to accept a pause. It is discovering on the day of the outage that a critical process has become impossible to run without the AI, and that nobody decided this. A fallback that exists only on paper is not a fallback. The procedure has to be written down, the people have to have done it recently, and the organization has to have run it with the AI switched off. How to detect, contain and learn from the incident itself is the subject of Human Oversight and AI Incidents.

When the workforce says no

The last operational dependency is the one most often left off the risk register: the consent of the people doing the work. AI changes roles, monitoring, pay and job security, and a workforce that does not trust how those changes are being made can stop operations faster than any outage.

Strikes are the visible end of this. In 2023 the Writers Guild of America struck for 148 days with AI among its central issues, and the agreement that ended it says AI cannot write or rewrite literary material⁷⁸. In October 2024 dockworkers at 36 US ports struck for three days with automation unsettled⁹.

Few organizations will face a strike over an AI pilot. They will face the quieter version: people who route around the new tool, who stop sharing what they know with a system they believe will replace them, or who leave. The remedy is not a better announcement, which AI and the Future of Work covers. It is involving the people and their representatives early, being specific about what changes, and planning the formal consultation that the law already requires in many countries.

Story: Toyota puts people back on the line

A well-documented example of an organization noticing that automation had eroded its capability, and deliberately rebuilding it, comes from before generative AI, which is part of its value. The lesson did not depend on the technology.

In the decade before 2010, Toyota grew fast. In February 2010, facing recalls of millions of vehicles in the United States, its president Akio Toyoda told the US Congress: “We pursued growth over the speed at which we were able to develop our people and our organization”¹¹. The recalls had their own causes, which were not automation. But Toyoda’s diagnosis, that people development had fallen behind, shaped what came next.

Toyoda asked Mitsuru Kawai, a veteran of half a century on Toyota’s shop floors, to restore craftsmanship in the company’s plants¹². Over about three years, Toyota introduced around 100 manual-intensive workspaces across its Japanese factories. At its oldest plant, Honsha, people took over from robots in forging crankshafts, and the improvements they then made eliminated about 10 percent of the material waste in that process¹³. The aim was not nostalgia. It was the learning culture of the Toyota Production System, in which people who understand a process by hand are the ones who find ways to improve it. Kawai put it directly: “To be the master of the machine, you have to have the knowledge and the skills to teach the machine”¹².

The parallel for AI is close, though not exact: Toyota’s robots did not draft or decide, while AI now does part of the professional work itself. The organizations that keep improving their AI-enabled processes are likely to be the ones whose people still understand the work well enough to see where the AI is wrong, where it could be better and what to do when it is gone. Toyota kept the robots. It also kept the people who could teach them.

What this means for leaders

Operational and workforce risk does not argue for slowing down AI. It argues for adopting it in a way that leaves the organization more capable, not merely more efficient. Four habits do most of the work.

First, name the capabilities you must keep: the skills needed to check the AI, to handle the exceptions and to run the process when the AI is gone. Second, design the workflow to build skill, with unaided practice, visible reasoning and hard cases kept for learners. Third, run the AI-down test on every critical process, and treat an untested fallback as no fallback. Fourth, treat the workforce’s trust as a dependency, consulting early and planning the legal consultation before the pilot.

Check yourself

1. If an AI system works well, the people beside it need less training.MythFact
2. Experienced specialists can lose skill after working with AI.MythFact
3. Working with AI always makes people worse at the task.MythFact
4. When AI does the routine work, juniors can lose the practice that turned earlier juniors into experts.MythFact
5. Every process that uses AI needs a manual fallback.MythFact
6. Workforce consultation can be left until the AI system is ready to launch.MythFact

Reflection: the expert you could not replace

What comes next

This chapter looked at AI that changes how people work while a person still stands behind every outcome. The next step is AI that acts on its own: planning, using tools and making changes inside enterprise systems. The next chapter, Agentic AI and Autonomous Actions, examines what changes when AI moves from preparing the work to doing it.