Multimodal AI

Here is a support ticket, invented for illustration but typical of what many business-software companies receive.

An assistant that can read only text learns almost nothing from it. Everything useful sits in the attachments. The screenshot shows the error code. The screen recording shows the three clicks that trigger it. The voice note says which report fails, and that it started after Tuesday’s update. A support engineer would open all three before deciding anything.

A great deal of what organizations know arrives like this: as photos, scans, recordings, diagrams and video, not as tidy text. The models that read the ticket’s text, described in Large Language Models, now increasingly read the attachments too. The question for a leader is not whether that is impressive. It is when the attachment is worth reading, and what reading it costs.

A modality is a choice, not an upgrade

A modality is a type, or channel, of information: text, images, audio, video, and other signals such as sensor readings, telemetry and location. A unimodal system works with one of them; a photo goes in and a label comes out. A multimodal system can process, combine or generate more than one. The research field treats combining modalities as its own problem, not as a sum of single-modality skills: the system has to represent each input, line them up with each other and fuse them into one judgment¹.

There are two ways to get the choice wrong, and organizations make both.

The first mistake is to keep everything as text and lose what the attachments hold. The second is to make everything multimodal because it sounds more advanced, and then pay for it in cost, error and exposure. The rest of this chapter builds the case for the middle position: add a modality when it holds information that changes the decision.

In, out, or both

Multimodality runs in two directions, and product announcements often blur them. What a system can read and what it can produce are separate capabilities.

Text in and text out is where most people first met generative AI. Media in and text out is understanding: a chart, a scanned contract or a call recording goes in, and an answer comes out. Text in and media out is generation: a written brief becomes an image, a voice or a short clip. Both at once is a system that hears a question, looks at a shared screen and answers aloud.

A model that is strong in one quadrant may be weak or absent in another. Reading images does not imply creating them, and transcribing speech does not imply replying in a natural voice. How models create new images and sound is the subject of the next chapter, How Generative AI Works. This chapter concentrates on understanding, because that is where many of today’s business uses, and many of the risks, sit.

How a model takes in a picture or a sound

You do not need the engineering, but one idea removes much of the mystery. As Tokens, Context and Embeddings showed, a language model reads text as tokens and works with them as numbers. Multimodal models extend the same trick to other inputs.

An influential 2021 paper showed that an image can be cut into a grid of small patches and fed to a transformer as if each patch were a word; its title was An Image is Worth 16x16 Words². In the same year a second team trained a model on 400 million images paired with their captions from the internet, so that a picture and a sentence describing it land close together in one shared space. Without seeing any of the 1.28 million labeled training images of the standard ImageNet benchmark, it matched the accuracy of a classic model trained on all of them³. That shared space is why you can now search a photo library by describing what you want, or ask a question about a chart.

Two consequences follow for leaders. First, a modality is not “seen” or “heard”; it is converted into pieces and numbers, and anything lost in that conversion is lost to the model. Second, every piece counts toward the model’s context and its bill, which is why images, audio and video are priced the way they are.

One model or a team of specialists

There are two broad ways to build a multimodal application. The first is a pipeline of specialists: a speech model turns a call into text, a vision model reads the image, a language model writes the answer, and a coordinating layer joins the results. The second is one unified model that takes every input directly.

Voice assistants show the trade-off. When one widely used assistant replaced its three-model voice chain with a single unified model, its maker reported that average response time fell from 2.8 or 5.4 seconds to 320 milliseconds, and that the chain had lost tone, speakers and background noise [@openai-2024-hello-gpt-4o; @openai-2024-gpt-4o-system-card]. Every hand-off in a pipeline adds time and drops information.

That does not make the unified model the default answer. Pipelines have real strengths. Each part can be tested, tuned or replaced on its own; a narrow speech or vision model is often faster and cheaper at high volume; and the text in the middle is a record a person can inspect. Where the words carry the information, as in most meeting notes, a transcript followed by a language model may be all you need. Where tone, timing, layout or the relationship between inputs matters, translating everything into text first throws away the evidence.

What each modality costs

Images, audio and video are not priced like text, because they become far more pieces. One major provider’s published counting rules, as of October 2026, make the scale concrete⁴.

At those rates a 45-minute recorded call is about 86,400 tokens as audio and about 710,100 as video, about 8.2 times as many. A token count is not a price, though. On some of the same provider’s models audio costs more per token than video, which narrows the gap in money to about two and a half times⁵; other providers count and charge differently, and prices fall over time. Treat the ratio as an order of magnitude, and price your own volume on the current rate card. The leadership question is not “can it watch the recording?” but “is the picture worth several times the cost of the sound?”

New ways to be wrong

Every added modality brings its own failures, and none of them is the failure of a person. Models invent detail in text, as Large Language Models showed⁶; seeing and hearing more does not cure that. It adds new forms of it.

Images. In 2024 researchers gave four leading vision-language models seven tasks a child can do: do two circles overlap, how many times do two lines cross, which letter is circled. The models averaged 58 percent; the best reached 78 percent. When the shapes were spaced further apart, accuracy rose to nearly 100 percent⁷. The lesson is not that the models are useless. It is that they fail on fine spatial detail, exactly the kind found in engineering drawings, charts and crowded screenshots, and that they answer confidently either way.

Speech. A FAccT 2024 study ran more than 13,000 recorded clips through a widely used speech-recognition model. About 1 percent of transcriptions contained whole phrases or sentences that nobody had said, and 38 percent of those inventions were harmful, such as violent language or false associations. They were more frequent for speakers with long pauses, as in aphasia⁸. One percent sounds small until the transcript becomes a medical note, a complaint record or the input to the next model in a pipeline.

Video. Video is not just many images. It adds sequence, timing and sound, and the model must work out what happened and in what order. On a 2025 benchmark of 900 videos, the best commercial model answered 82 percent of questions about clips under two minutes correctly, but 67 percent for videos of 30 to 60 minutes⁹.

Two practical rules follow. Test on your own real inputs, blurred, cropped, noisy and long, never only on the clean examples in a demo. And decide before launch what happens when the model is unsure: it hands over to a person.

Privacy rides along

Text that a person types usually contains what they chose to write. A screenshot contains whatever was on the screen: other customers’ names, salaries, contract values. A photo of a site shows faces and number plates. A call recording carries a voice, which can identify a person, and a video call carries a face and often a home. Every modality you add widens what the system ingests, stores and sends to its provider, which is why masking, retention and where the processing happens belong in the design, not in a review after launch.

Does the modality earn its place?

All of this reduces to one question, asked before any money is spent: does another input hold information that the text does not?

If no other input holds missing information, stay with text: it is cheaper, faster and easier to test. If one does, find the slice of cases where it matters, test on real examples from your own operation, and pilot with a person checking the output. Alongside the tree sit two questions that never go away: what in this input is sensitive, and what does each interaction cost at real volume? The decisive input is not always where a demonstration suggests, as the story shows.

Story: the demo that looked like live video

In December 2023 Google launched its Gemini model with a video called “Hands-on with Gemini”. In it, a person draws, gestures and plays games at a table while the model seems to watch and comment aloud as things happen. The description under the video noted that, for the purposes of the demo, latency had been reduced and the model’s outputs shortened for brevity¹⁰. A post on Google’s developer blog, published the same day, explained the prompting behind the video: the team gave the model still frames from the footage, together with written prompts¹¹. Google told Bloomberg that the video had been made by using still image frames from the footage and prompting via text, as TechCrunch reported the next day¹².

The difference lay in where the information lived. In the video, the model recognizes a game of rock, paper, scissors from silent hand gestures. In the written version, it was shown still images of the gestures and asked what the person was doing, with the hint that it was a game¹². Part of what made the answer right was in the text, not in the pictures. And a handful of chosen frames is not video. It carries none of the sequence, timing and sound that, as the cost and failure sections showed, are where video’s accuracy falls and its token count climbs. Gemini’s co-lead responded that all the user prompts and outputs in the video were real, shortened for brevity, and that the video illustrated what multimodal experiences built with the model could look like and was made to inspire developers¹⁰. Nothing in the record says the model could not do what the video showed. The record says the video did not show how it was done.

For a leader the case is a template for reading any multimodal demonstration. Ask what actually went in: continuous video or chosen frames, sound or a transcript, and whether a written prompt carried part of the answer. Ask who chose the inputs, and how long each answer took. Then run the same task on your own real, messy inputs, live, at the speed and volume your operation needs. The question that settled this post-mortem is the question of the whole chapter: where does the information live?

What this means for leaders

Multimodal AI lets systems use a far richer picture of the business than text alone, and the most promising uses are those where a text-only process visibly loses information today: scanned documents with stamps and handwriting, photos from the field, screens that show what a customer actually did. These are capabilities, not proven results: whether one pays off in your process has to be shown there, in a pilot measured on your own inputs. The discipline is to treat each modality as a line item with a benefit, a cost, a failure profile and a privacy footprint, and to approve it on those terms. Approving “multimodal” as a label, because the system “sees and hears like a person”, is how organizations end up paying for video to transcribe speech.

Check yourself

1. A model that can read images can also create them.MythFact
2. Video is just a longer series of images.MythFact
3. Multimodal models cannot invent details, because they can see or hear the evidence.MythFact
4. An hour of video can count as several times the tokens of the same hour’s audio.MythFact
5. A pipeline of specialist models is always worse than one multimodal model.MythFact
6. Adding a modality can make a system costlier and riskier without improving its answers.MythFact

Reflection: where does your information live?

What comes next

This chapter kept one distinction throughout: understanding many kinds of information is different from generating them. Models that read a screenshot or a recording are also, increasingly, models that write the reply, draw the image or speak the answer. The next chapter, How Generative AI Works, explains how they create something new, and why the same request can produce a different answer every time.