Privacy and Confidential Data

Before reading on, make a guess. Of all the data that employees copy, paste or upload into AI tools at work, what share is sensitive: source code, customer records, financial or HR information? One in twenty? One in ten?

Cyberhaven, a data-security company, tracks how data moves inside its customers’ organizations. In its 2025 report it found that 34.8 percent of the corporate data employees shared with AI tools was sensitive, up from 10.7 percent two years earlier¹. Its 2026 report put the figure at 39.7 percent of all data movements into AI tools, and found that the average employee entered sensitive data into an AI tool about once every three days².

These are one vendor’s measurements of its own customers, not a random sample of the economy, and the 2026 figure counts data movements rather than volume. Treat the exact numbers with care. The direction is harder to argue with. Employees are not mainly typing trivia into AI tools. They are giving them the information that does the work, and the work runs on sensitive information.

Follow the information

Many leaders, asked about AI privacy, ask whether the tool is secure or approved. Those are good questions, and they are not enough. The more useful question is where the information goes.

Information enters as a prompt, often with far more context attached than the user typed. The model processes it. An answer comes out, and the answer, the prompt and the documents behind them are often kept in logs and conversation history. Then the answer reaches people and other systems: a colleague, an email, a ticket, another vendor’s tool.

Each step is a place where personal or confidential information can reach someone it should not. That is why the US National Institute of Standards and Technology lists “privacy-enhanced” among the characteristics of trustworthy AI³, and why its profile for generative AI names data privacy as one of the risks that is new or worse with this technology, including the leakage and unauthorized disclosure of personal information⁴. Approval of a platform gives you stronger contracts and better controls. It does not tell you what data went in, why, who can see the answer, where it is stored or how long it stays.

Two risks that overlap

Privacy and confidentiality are often used as if they meant the same thing. They overlap, and the difference matters for who owns the problem.

Privacy is about people. It concerns how information relating to an identifiable person, a customer, an employee, a reader or a patient, is collected, used, shared, kept and protected. Confidentiality is about access. It concerns information that must not reach unauthorized people or organizations, whether or not anyone is named in it.

An attendee list from a public event is hardly a secret, but it is personal data and privacy rules apply. Unreleased strategy or source code contains no personal data and can still do great commercial damage if it leaks. An HR investigation file is both. Legal categories differ by country, so classify by sensitivity, business impact and legal requirement, and let the privacy team and the security team each own their half.

Use only the data the task needs

The cheapest privacy control is a question asked early: does this use actually need this data?

Start from everything the model could technically use, which is almost always far more than the task requires. Narrow it to data with a defined, legitimate purpose. Customer support records do not become training data or marketing analytics simply because they are convenient. Narrow again to the fields the task needs. An assistant that answers questions about order status needs the order status, not the full customer profile, payment details and contact history. Then mask or remove what is left over before it is sent.

Some information never belongs in an ordinary prompt: passwords, private keys, access tokens and database credentials. People paste them by accident, usually while asking for help with a broken script. Secret scanning, redaction and data loss prevention tools exist to catch exactly this, and they are worth their modest cost.

European law makes this principle a duty rather than a preference. The GDPR requires a lawful basis and a defined purpose for processing personal data, and data protection by design and by default⁵. The sensible default anywhere is minimum data, minimum access and minimum retention, expanded only when the business case justifies it.

The model sees more than the prompt

When an employee types a short question, they see one line. The model may receive a great deal more.

Modern assistants attach chat history, retrieved documents, the user’s profile, hidden system instructions, database results and the outputs of other tools. None of it was typed, and most of it the user never sees. A question that looks small can open the whole file room. The user may not know that happened. The organization still needs to know what entered the path.

Agents raise the stakes, because an agent can not only see information but send it onward, by email, into tickets or into other systems. The question grows from “what can it see?” to “where can it send it, and what can it do?” How to bound an agent’s permissions is the subject of Agentic AI and Autonomous Actions, later in this module.

Retrieval must respect the rights people already have

Many enterprise assistants work by retrieval: the user asks, the system searches a body of documents, and the model answers from what it finds⁶. This design reduces how much data is sent and keeps answers current. It also creates a common privacy failure in enterprise AI.

If the search index ignores the permissions on the original files, the assistant can faithfully quote a document the person asking was never allowed to open. Nothing was invented. The model did its job well. The failure is one of authorization, and the security community now lists it among the top risks for these applications, under sensitive information disclosure and weaknesses in the stores that hold documents for retrieval⁷.

Notice what encryption does here: nothing. Encryption protects data in storage and in transit from outsiders. It does not decide whether the right person is asking for the right purpose. Authorization does. As Data Strategy for AI put it in Module 04, permissions have to travel with the data; retrieval is where that promise is tested.

Inference, logs and training are three questions

Leaders often ask a provider one question: “Do you train on our data?” It matters, but it is one of three.

Inference is data passing through the model to produce an answer. Ask where that processing happens, because location can differ by region, service and plan. Logs and history are stored records of prompts, answers, documents and tool calls. They help with debugging and audit, and they are also a new store of sensitive data. Ask who can read them, how long they are kept and whether they can be deleted. Training is data shaping the model itself. Whether your data is used depends on the specific service, configuration and contract, and one provider’s practice tells you nothing about another’s.

Training deserves care because it is hard to undo. Europe’s data protection authorities have said that a model trained on personal data cannot automatically be treated as anonymous; it is anonymous only if it is very unlikely that the people in its training data can be identified or their data extracted through queries⁹. Once personal data shapes a model, deleting the original file does not take it out.

The chain is also longer than one vendor. Cloud hosting, search indexes, monitoring tools and plugins can each hold copies. Verify each answer against current documentation and the contract, and assume that deleting data in one place does not remove every copy.

Shadow AI is a design problem, not only a discipline problem

Employees adopt AI faster than policy can keep up. When the approved route is slow or missing, they use whatever is closest: a personal account, a free tool, a phone. That is shadow AI. Cyberhaven found that about a third of ChatGPT use at work in its customer base ran through personal accounts².

A 2026 breach-cost study, as summarized by a law firm, reported shadow AI incidents in 43 percent of breached organizations, up from 20 percent a year earlier¹⁰.

There are two tempting responses, and both fail. A blanket ban feels safe, but people who need to get work done move to tools and devices where the organization sees nothing. No rules at all means data leaks quietly into whatever tool is nearest. What works is in between: approved tools that are genuinely easy to use, clear rules about which data may go into which tool for which purpose, and training that explains why. When the safe route is also the fast route, shadow AI shrinks. How to keep an inventory of the AI systems in use is covered in AI Inventory and Risk Classification, in Module 07.

Personal data brings legal duties, in Europe and the United States

When the data path carries personal data, the law follows it. The duties are different on each side of the Atlantic, and an executive should know the shape of both.

In the European Union, the GDPR has applied since 25 May 2018. For AI on personal data, four duties matter most. You need a lawful basis and a defined purpose. You must tell people how their data is used, including in AI processing. You must build data protection into the design and the defaults. And you must carry out a data protection impact assessment, a DPIA, before processing that is likely to be high risk. The UK regulator’s guidance, written for the UK GDPR, which mirrors the EU text here, is blunt: in the vast majority of cases, using AI on personal data will trigger the legal requirement for a DPIA¹¹. Where an AI makes a decision with legal or similarly significant effects on a person without meaningful human involvement, Article 22 adds a further right. A leak of personal data through an assistant can also be a reportable breach, with clocks that Human Oversight and AI Incidents covers later in this module.

The United States has no comprehensive federal privacy statute. About 20 states have their own laws, and California’s is the reference point. Its updated regulations require a documented risk assessment for processing that poses significant privacy risk, a duty that applies to new high-risk processing from 1 January 2026. From 1 January 2027, a business that uses automated decision-making technology to make a significant decision about a California resident, in lending, housing, education, employment or healthcare, must give notice before use, offer an opt-out unless an exception applies, and answer access requests¹².

Story: the AI connector that crossed company lines

On 1 May 2025 Asana launched a server built on the Model Context Protocol, an open standard for connecting AI assistants to business software. Through it, a customer’s AI assistant could read that customer’s tasks, projects and comments in Asana and answer questions about them¹³.

On 4 June the company found a logic flaw in the server. Under certain conditions, information from one customer’s Asana domain could be returned to MCP users in other organizations. Depending on how the connector was used, that could include task details, project metadata, team details, comments and uploaded files. A spokesperson told BleepingComputer that roughly 1,000 customers were affected¹⁴. Asana said the cause was a bug, not a hack, and there was no indication that anyone had exploited it or actually seen another organization’s data¹³.

The feature was a month old and in demand. Its owners could keep it running while engineers patched it, fix it quietly, or take it down and tell customers. Pause here and choose.

Asana took the server offline from 5 June, fixed the code, contacted affected organizations directly and reset every connection, so each customer had to reconnect before using it again. The service came back on 17 June¹³. Administrators were advised to review MCP access logs and the answers their assistants had generated, looking for anything that seemed to come from another organization¹⁴.

Three lessons carry beyond one company. The model did nothing wrong: it answered faithfully from what the connector handed it, and the failure lay in the access check, exactly where the permission tree above puts it. The exposure did not end with the fix, because anything that crossed would sit in another company’s assistant answers and history until someone found it. And for every customer, the first question was simply whether any employee had connected an assistant at all. Only an organization that knows its data paths can answer that in an afternoon.

What this means for leaders

Privacy in AI is not a property of the vendor or of the approval stamp. It is a property of the path, and the path is designed by your organization. Four habits make the difference. Ask for the data path before the demo. Make minimum data, defined purpose, minimum access and controlled retention the default, and require a reason to expand any of them. Insist that retrieval inherit people’s existing access, and that this is tested with real cases. And make the approved route easy enough that shadow AI loses its appeal.

Check yourself

1. An approved enterprise AI platform means confidential data is safe.MythFact
2. Encryption solves privacy.MythFact
3. An AI assistant can leak information while answering faithfully.MythFact
4. If the provider does not train on our data, there is no privacy risk.MythFact
5. The United States has a comprehensive federal privacy law that covers AI.MythFact
6. Regulators expect most AI uses of personal data to need a data protection impact assessment.MythFact

Reflection: trace one path

What comes next

This chapter looked at information reaching the wrong place through design gaps and everyday behavior. The next chapter, Security and AI Attacks, asks the harder question: what happens when someone deliberately tries to manipulate an AI system, or uses it as a new route into the organization?