Multimodal and Agentic AI

In June 2025 the research firm Gartner published two forecasts about AI agents in the same press release. By 2028, it expected a third of enterprise software applications to include agentic AI, up from less than 1 percent in 2024. And by the end of 2027, it expected more than 40 percent of agentic AI projects to be canceled, because of escalating costs, unclear business value or inadequate risk controls¹.

The same release put a number on the noise. Many vendors, Gartner said, were rebranding existing assistants, chatbots and robotic process automation as agents, a practice it called “agent washing”. Of the thousands of vendors claiming agentic products, it estimated that only about 130 were real¹.

These are an analyst firm’s forecasts from mid-2025, not measurements, and both target dates are still ahead, so they should be read as informed opinion. Both can be true at once, and that is the useful thing about them. The capability is real and spreading fast. So is the gap between what is sold as an agent and what can safely be given work. Closing that gap is a leadership task, not a technical one, and it starts with being clear about what has actually changed.

Seeing and doing become one system

Multimodal AI showed that a model can read images, audio, video and documents as well as text. From AI Assistants to AI Agents showed what turns a model into an agent: a goal, a plan, tools and a loop that checks results. Neither idea needs repeating here. What is new is that the two are fusing.

Until recently, perception and action were separate products. One system described a photograph; another, scripted in advance, moved data between applications. In 2026 a single system can look at a screen and click on it, listen to a caller and answer, read a scan and send the alert. Perception is no longer only how the system understands a case. It is how the system finds its way around the software and the people it works with.

That merger changes the executive questions. Whether a system can do a task is increasingly settled, as Where AI Is Going Next showed, and the reliability, cost and autonomy it does it with are the dials to watch. Three new questions sit on top of those dials. Through which door does the system enter your software? What is it allowed to see and hear, and for how long is that kept? And at which point does a named person still sign? The rest of this chapter takes them in turn.

The screen becomes a door

The clearest sign of the merger is the computer-using agent. In October 2024 one AI lab released a model that operates a computer the way a person does: it looks at a screenshot, decides what to do, moves the cursor, clicks and types. The lab itself called the capability experimental and “at times cumbersome and error-prone”². In January 2025 a second lab released a similar agent that reads screenshots and works through the buttons, menus and text fields people see³.

The attraction for an enterprise is obvious. Much of the work that never got automated sits in old applications with no programming interface, in supplier portals, or in screens stitched together by people copying from one window into another. An agent that can work the screen reaches all of that without an integration project.

The cost is equally plain. Screen-working agents still fail often on ordinary computer tasks, roughly one in three on the benchmark described in Where AI Is Going Next⁴. They are slow, because every step needs a fresh look at the screen. And they read whatever the screen shows, including text planted to mislead them. The same system card that describes the method lists prompt injection from web content as an added risk, and it relies on asking the user to confirm before critical actions³. How such attacks work, and why limiting what a system can reach is the main defense, is the subject of Security and AI Attacks⁵.

Two doors into your systems

Every agent that does work in your organization enters your software through one of two doors. The back door is a connector: a programming interface or tool that the system’s builders have deliberately exposed, with defined actions and permissions. The front door is the screen: the agent uses the same interface a person would.

The choice is not technical housekeeping. It decides how far an agent can reach, and therefore how much harm a mistake can do. A connector exposes a short list of actions, such as “create a draft order” or “look up a part”, and each one can be permitted, limited and logged. A screen exposes everything the logged-in account can see and touch.

That suggests a simple rule for leaders. Use connectors for anything consequential or high volume. Allow screen work for the gaps, where no connector exists, the stakes are low and someone is watching, and treat each such use as a signal that a connector is worth building. The controls themselves, from least privilege to approval and reversibility, belong to Agentic AI and Autonomous Actions; the door decides how many of them you will need.

The plumbing is being standardized

The back door is getting easier to build, because the connections are being standardized. In November 2024 one lab published the Model Context Protocol, an open standard for connecting AI systems to data sources and tools in place of one-off integrations⁶. In April 2025 another introduced Agent2Agent, an open protocol for agents from different builders to discover each other’s capabilities and coordinate tasks⁷.

Both standards moved quickly to neutral homes. In June 2025 Agent2Agent became a Linux Foundation project, founded with seven large technology companies and supported by more than 100⁷. In December 2025 the Model Context Protocol moved to a new foundation under the Linux Foundation, co-founded by rival AI labs and backed by the largest cloud providers, and reported about 10,000 active servers one year after launch⁸.

For an executive, the meaning is practical. The cost of connecting an agent to a new system is falling, so connectors will multiply, many of them added by teams or suppliers without a central decision. Each one is a new door. Treat connectors as you treat AI systems in AI Inventory and Risk Classification: listed, owned and reviewed. A connector someone installed from a public directory is part of your supply chain, and it can carry instructions as well as data.

From prompted to ambient

The third shift is in when the system works. Most AI so far has been prompted: a person asks, the system answers, and nothing happens between requests. Multimodal agents increasingly work ambiently. They listen to a meeting or a call, watch a camera feed or a queue of scans, and act when something they perceive matches a trigger.

Ambient systems are often where the value is, because nobody has to remember to ask, and the delay between an event and a response shrinks. But the governance questions change character. A prompted system sees what someone chose to give it. An ambient one sees everyone in range: colleagues on the call, customers in the shop, patients in the room. The questions become who is being recorded, whether they know, what is kept, and who may look at it later.

Law already reaches some of these choices.

Read the label, test the behavior

Gartner’s estimate that only about 130 agentic vendors are real is a forecast-firm judgment, not a census¹. The pattern behind it is easy to check for yourself. From AI Assistants to AI Agents gave the test for whether a system plans, uses tools and acts. With multimodal agents, add three demands to any vendor or internal team.

First, a log of actions taken on real cases, not a demonstration. Second, for each action, the door it used: a connector with defined permissions, or the screen. Third, the points at which a person approved, corrected or stopped the system, and how often. A product that cannot produce these is, for your purposes, an assistant, whatever its name. That is not a reason to reject it. It is a reason to price, govern and measure it as an assistant.

What a satisfactory answer looks like is already documented. Since May 2025 the software platform GitHub has offered a coding agent that takes a task, changes code and proposes the change for review; it is named here as evidence of a design, not as a recommendation¹¹. The agent works through the platform’s own interfaces, not a screen, and can push only to a single branch created for it. It cannot approve or merge its own pull request, and by default the automated checks on its code do not run until a person with write access approves them. Every commit it makes links to the session log of what it did, and administrators receive audit-log events¹². The door, the action log and the human signature are part of the product. That is the bar to hold other agents to.

Story: the stroke alert that reached three hospitals

The clearest real example of perception joined to action is older than the current wave of agents, and smaller than most agent pitches. That is what makes it instructive.

A comprehensive stroke center in upstate New York treats patients with large-vessel strokes, which need a clot removed as fast as possible. It receives them from 14 hospitals within about 75 miles, only three of which belong to its own network¹³.

Before. A patient arrived at a local hospital and had a brain scan. Software processed the perfusion scan and e-mailed results to a list of providers, for every scan whether or not it showed an occlusion. A radiologist reviewed the images and called the care team; the team then arranged the transfer. The alerts were many, and they were easy to ignore.

After. In 2021 the network added software that reads the CT angiogram, flags a suspected large-vessel occlusion, and pushes the alert and the images to the phones of both the local doctors and the thrombectomy team at the center, with secure messaging between them. The system perceives, and it takes one small action: it tells the right people, with the evidence attached. Every treatment decision stays with physicians¹³.

The results, across 262 patients who had a clot removed, were real and specific. The time from arrival at the center to the start of the procedure fell from about 95 to about 80 minutes. Transfers from the three network hospitals fell from 157 to 120 minutes. But transfer time from all referring hospitals barely moved, from 162 to 159 minutes: the eleven hospitals outside the network were not on the system¹³.

Two further details matter as much as the gains. The first is what the software caught, which takes two numbers to state. For the occlusions it was designed to find, in the terminal internal carotid artery and the first segment of the middle cerebral artery, it detected 64 of 73, or 88 percent. Across all 111 patients treated after it was introduced, it flagged 69, or 62 percent, because many patients had occlusions outside its design scope, mostly in the carotid artery in the neck, in smaller branches of the middle cerebral artery or at the back of the brain. Those patients still reached treatment through the existing radiologist pathway, which the network kept. The second detail is that the study measured workflow times, not patient outcomes¹³.

Read as a trajectory, the case contains the whole chapter. The value came from joining perception to a modest, well-chosen action, not from autonomy. The benefit stopped exactly where the connections stopped. And the human pathway was kept, because perception was good but not complete.

What this means for leaders

The capability will keep improving along the dials described in Where AI Is Going Next. The decisions that stay with you are about design. Choose workflows where the evidence arrives as images, sound or screens and the delay is in getting it to the right person or system, because that is where joining perception to action pays first. Prefer connectors to screens for anything that matters, and register every connector. Decide what an ambient system may watch and keep before it is switched on, not after the first complaint. And keep the human pathway running in parallel until the record shows the system catches what people catch.

Check yourself

1. Multimodal and agentic AI are still separate product categories.MythFact
2. An agent that operates the screen is safer than one that uses a connector, because it does only what a person could do.MythFact
3. Within about a year, the main agent connection standards moved to neutral, multi-company governance.MythFact
4. Gartner forecast both rapid growth in agentic software and the cancellation of over 40 percent of agentic projects.MythFact
5. Ambient systems raise mainly technical questions, not legal ones.MythFact
6. In the stroke network, the alert software made transfer faster from every referring hospital.MythFact

Reflection: the evidence that waits

What comes next

Everything in this chapter still happens inside software, even when the input is a camera and the output is a phone alert. The next step takes the same pattern of perceiving and acting into machines that move. The next chapter, AI + Robotics, looks at what changes when an action is physical and cannot be undone with a click.