Security and AI Attacks

Picture an ordinary business email arriving in an employee’s inbox. It reads like routine correspondence, addressed to the person who receives it. Nobody needs to open it. Some days later the employee asks the company’s AI assistant a question on a related topic, and the assistant, searching the mailbox for useful context, reads that email too. Woven into its wording are instructions. The assistant follows them: it gathers sensitive material from the files and messages it can reach, and tucks it into a link that quietly delivers it to a server the attacker controls.

This is not a thought experiment. Researchers at a security start-up reported exactly this chain to Microsoft in January 2025, in Microsoft 365 Copilot. They called it EchoLeak. Microsoft rated it 9.3 out of 10 for severity, fixed it on its own servers, and found no evidence that anyone had used it maliciously¹². The researchers described it as the first real-world “zero-click” prompt injection in a production AI system. To get past the assistant’s injection filter, the email addressed its instructions to the human reader rather than to the AI³.

Notice what did not happen. Nobody stole a password. No malware ran. The model was not broken; it did what it was built to do, which is read content and act helpfully on it. Privacy and Confidential Data looked at information reaching the wrong place by accident. The harder case is someone making it happen on purpose.

Language is now an attack surface

Traditional software keeps a hard line between code and data. A customer’s address is stored and displayed; it is never run as a command. Security teams spent decades building and defending that line. AI systems blur it. Instructions and content arrive through the same channel, language, and the model has no reliable way to tell which words are its owner’s instructions and which are simply material it was asked to read.

That matters because modern AI systems do not only talk. They search document stores, read email, call tools, update records and send messages. So the security question changes. It is no longer only whether the model is safe. It is what the whole system can reach and do if someone steers it.

Securing only the model protects one box of four. Identity, permissions, the content the system reads, its tools, its outputs and its actions all belong inside the security boundary. The security community’s reference list of risks for language-model applications puts prompt injection in first place, and its list reads like a tour of that chain: sensitive information disclosure, poisoning, improper handling of outputs, excessive agency and runaway consumption⁴. The US government’s generative AI risk profile treats information security as one of a dozen core risks for the technology, including attacks by injection and poisoning⁵.

Four failures that look alike

Four terms get confused in board discussions, and the confusion leads to the wrong fix.

Direct prompt injection is the simplest: the attacker types hostile instructions straight into the system, hoping to bypass its rules, reveal its hidden configuration or trigger a tool. Indirect prompt injection is often the version that matters more for enterprises, because the attacker never talks to your AI at all. The instruction waits inside content the AI will later read: a web page, a support ticket, a contract, a supplier’s PDF, an email. In 2023, Kai Greshake and colleagues showed this working against real, deployed assistants, including a search engine’s chat feature, and argued that retrieval turns every document an AI can read into a possible source of commands⁶. EchoLeak was the same idea, two years later, inside a corporate mailbox.

A jailbreak is a prompt designed to get around the model’s own safety training. It is a reason to keep controls outside the model, not a separate security program. Hallucination, finally, is not an attack. Nobody is manipulating anything; the model simply produces something wrong, as Accuracy, Hallucination and Reliability showed. The distinction matters because the fixes differ. Better grounding reduces hallucination. It does nothing against an instruction someone planted on purpose.

Attacks on the data and the model

Prompt injection gets the headlines, but it is one family among several. The US National Institute of Standards and Technology keeps a taxonomy of attacks on machine learning. It sorts them into evasion, poisoning and privacy attacks, with model extraction among the privacy attacks, and adds direct and indirect prompt injection for generative systems⁷. Three of these have been demonstrated convincingly enough that executives should know them by name. Most of the evidence below comes from research demonstrations; public records of these attacks exploited in the wild are thinner, which is a reason to design ahead rather than a reason to relax.

Poisoning plants tainted material in the data a model learns from, so that it misbehaves later, often long after the plant. A common assumption was that this would need a meaningful share of a huge training set. In October 2025, researchers from Anthropic, the UK AI Security Institute and the Alan Turing Institute found otherwise: as few as 250 malicious documents created a hidden “backdoor” in models ranging from 600 million to 13 billion parameters, whatever their size⁸. The backdoor was deliberately harmless and the work a controlled experiment, but the lesson about scale stands. The same logic applies to the knowledge bases and feedback data your own systems learn from.

Evasion crafts inputs that fool a model at the moment of decision. In a well-known demonstration, a few stickers made a road-sign classifier read a stop sign as a speed limit sign in most video frames from a moving vehicle⁹. In business the target is more often a fraud screen, a content filter or a document check.

Model extraction uses ordinary queries to recover what a model has learned: in 2024 a research team did so for under 20 US dollars against two production language models¹⁰. A model that cost millions to build can leak through its own front door.

The other three families are quieter. Data leakage happens when the system reveals what it can reach, which is why Privacy and Confidential Data insisted that retrieval respect the rights people already have. Unsafe output is generated code, queries or emails passed into other systems without checks, which turns the AI into one link in an attack chain. Runaway cost is a flood of long requests or an agent stuck in a loop, which turns a security incident into a financial one⁴. Weaknesses in the models, libraries and services you buy are a supply chain question, and Model and Third-Party Risk takes them up.

The dangerous combination

Not every AI system is equally exposed. In June 2025 the developer Simon Willison gave a useful test a memorable name, the lethal trifecta. Any system that combines access to private data, exposure to untrusted content and the ability to communicate externally can be tricked into sending that data to an attacker¹¹. EchoLeak had all three: a mailbox full of private material, an inbox open to anyone on the internet, and a way to reach an outside server.

In October 2025 Meta’s security team turned the idea into a design rule, the Agents Rule of Two. In any one session, an agent should have at most two of three properties: it processes untrustworthy input, it has access to sensitive systems or private data, or it can change state or communicate externally. If a task truly needs all three, the agent should not operate autonomously and needs supervision, such as human approval¹².

The power of the rule is that it does not depend on catching the hostile sentence. An assistant that reads outside email and holds private data, but cannot send anything anywhere, can be fooled without being able to deliver the prize. Security teams call everything a manipulated system could access, change, send or trigger its blast radius. The Rule of Two is a way to keep it small by design. Agentic AI and Autonomous Actions, later in this module, takes up the ladder of autonomy itself and how to widen it on evidence.

Why no filter will save you

Your security teams will recognize the pattern. It looks like SQL injection, the attack in which someone types a database command into a web form and the system runs it. The comparison holds in three ways: untrusted input becomes a command, the defense starts by treating all outside input as hostile, and nobody mistakes it for a user being creative.

Here is where the analogy breaks, and it breaks in the place that matters most. SQL injection has a structural fix. Parameterized queries send commands and data down separate channels, so the database never confuses the two. Language models have no equivalent. Everything they read arrives as one stream of words. In December 2025 the UK’s National Cyber Security Centre warned that treating the two attacks as alike is dangerous, because prompt injection may never be mitigated the way SQL injection was, and advised designing systems to limit the impact instead¹³. The same month, a leading model provider wrote that prompt injection is “unlikely to ever be fully ‘solved’”¹⁴.

The evidence on defenses points the same way. In October 2025, researchers from three major AI developers tested twelve published defenses against jailbreaks and prompt injection. Most had reported near-zero attack success when they were introduced, measured against fixed sets of known attacks. Against attackers who adapted to each defense, success rates rose above 90 percent for most, and human red-teamers got through every one¹⁵.

None of this means filters are useless. They raise the cost of an attack and stop the lazy ones. It means a filter cannot be the reason you believe a system is safe. So with SQL injection we eliminate the risk; with prompt injection we contain it. We cannot eliminate it.

Contain it: controls outside the model

If some manipulation will succeed, the design question becomes how to make sure one success does not become an incident. The answer is layers, each enforced outside the model, so that a fooled model meets a wall it cannot talk its way past.

The foundation is identity and least privilege. Each AI system runs under its own identity, not a shared, powerful service account, and gets only the data, tools and permissions its task needs, not the full rights of the employee it helps. The boundary is enforced by the systems around the model. An AI that says it is acting for an employee has not been authorized by saying so.

Validation comes next: checking what goes in, what comes out and every tool call against an allowlist, so that generated code is tested before it runs and an outgoing message is checked before it leaves. Human approval belongs on high-impact or irreversible actions, such as payments, deletions, legal messages and account changes. It is a real control only if the approver can see what they are approving. Lisanne Bainbridge’s classic paper on the ironies of automation warned in 1983 that people asked to watch over automated systems are poorly placed to catch the rare failure¹⁶. Monitoring and red teaming mean logging what the system actually did and paying people to break it before attackers do. At the top sits the emergency stop: the ability to halt an agent, disable its tools, revoke its credentials and block its traffic within minutes, owned by someone who can act without a meeting.

One more point belongs here, because it is a common mistake in boardrooms. A model provider secures its model service and infrastructure. Your prompts, data, permissions, tools and business logic remain yours. Security is shared, and widely used frameworks, such as NIST’s AI Risk Management Framework, treat security and resilience as properties of the whole system and its operators¹⁷.

Story: the web form that steered a sales agent

Salesforce’s Agentforce puts AI agents to work inside a company’s customer-relationship system. One ordinary feature of that system is Web-to-Lead: a public form on a company’s website where a prospect leaves a name, a company and a free-text description, which becomes a lead record. Anyone on the internet can fill it in.

In July 2025 Noma’s researchers showed what that description field, which accepts up to 42,000 characters, could carry. They submitted a lead whose description held instructions written to look like part of a normal request. When an employee later asked the agent an everyday question about the new lead, the agent followed the planted instructions as well: it gathered other CRM records and packed them into the address of an image, so that loading the image would send the data to an outside server¹⁹.

The page would normally refuse to load an image from an unknown site. Here it did not, because the server sat on a domain still listed among Salesforce’s trusted addresses, although its registration had lapsed. The researchers bought it for about 5 US dollars¹⁹. The researchers themselves rated the chain critical, 9.4 out of 10 on the CVSS severity scale (the 9.3 for EchoLeak was Microsoft’s own rating), and reported it on 28 July 2025 [@noma-forcedleak-2025; @thehackernews-forcedleak-2025]. Measured against the Rule of Two, the agent had all three properties in one session: untrusted input from a form anyone could fill in, sensitive customer data, and a way to send that data out.

The fix sat outside the model. From 8 September 2025 Salesforce enforced its Trusted URLs allowlist for Agentforce and Einstein AI, so that agent output could no longer be sent to untrusted addresses, and it re-secured the expired domain [@noma-forcedleak-2025; @thehackernews-forcedleak-2025]. In its words, the services behind Agentforce “will enforce the Trusted URL allowlist to ensure no malicious links are called or generated through potential prompt injection”²⁰. The research was published on 25 September²¹.

Two lessons outlast the patch. A planted lead may still fool the model, but fooling it no longer delivers the prize, because a control the model cannot talk its way past has cut the outbound leg of the dangerous combination. And that control needed upkeep: an allowlist entry for a domain nobody was renewing had quietly become the attacker’s way out. A wall outside the model protects only as long as someone owns it. Customers had homework too; the researchers advised auditing existing lead data for suspicious submissions¹⁹.

What this means for leaders

Four lessons follow. Treat every AI system that reads outside content as exposed, because any document, email or web page can carry instructions. Ask for blast radius, not cleverness. Put the controls outside the model, where a fooled model cannot talk its way past them. And distrust any claim that prompt injection is solved: ask how a defense performed against adaptive attackers and human red-teamers, not against a fixed list of known attacks.

Check yourself

1. An attacker must interact with your AI system to inject instructions into it.MythFact
2. A secure model means a secure AI application.MythFact
3. An assistant that only reads data cannot cause a security incident.MythFact
4. Prompt injection can be engineered away, the way parameterized queries fixed SQL injection.MythFact
5. A few hundred poisoned documents can backdoor a language model, whatever its size.MythFact
6. An agent that reads untrusted email and can send messages, but holds no sensitive data, fits the Rule of Two.MythFact

Reflection: find the dangerous combination

What comes next

This chapter looked at how AI systems can be manipulated or compromised by someone acting on purpose. The next chapter, Intellectual Property and Copyright, turns to a different exposure: what happens to ownership, licensing and confidentiality when AI creates, transforms and reuses content.