Centralized vs Federated AI

Ask a leadership team how it organizes AI and you will usually hear one word: centralized, or federated, or hybrid. When McKinsey asked the question element by element in the global survey it published in March 2025, the answers split. For risk and compliance, and for data governance, organizations often use a fully centralized model, such as a center of excellence. For technical talent and for the adoption of AI solutions, respondents most often reported a hybrid or partially centralized model, with some resources held centrally and others spread across functions and business units. Companies with annual revenues below 500 million US dollars were more likely to centralize these elements fully¹.

Those organizations are not undecided. They are answering a better question than the one most leadership teams argue about, and the rest of this chapter is about how to ask it well.

The question is not “who owns AI?”

The argument usually starts as a choice between two camps. Business-unit leaders want to build their own AI because they understand their customers, their workflows and their economics better than any central team. Technology and finance leaders want one central team because, without one, the enterprise pays for the same capability three times, with three sets of controls.

Both camps are right about something, because AI is not one thing. It is a set of shared foundations, such as model access, identity, security standards and evaluation methods; a portfolio of use cases; dozens of redesigned workflows; and, for some firms, a handful of AI-enabled products. Each has a different natural owner. A single label forces all of them into the same shape.

The choice also matters more than it looks, because structure leaks into systems. In 1968 the computer scientist Melvin Conway observed that organizations which design systems are constrained to produce designs that copy their own communication structures². Draw the AI organization as three separate divisions and you will tend to get three separate AI estates. Draw it as one central queue and you will get systems shaped by the queue.

The previous chapter, Build vs Buy vs Partner, decided which AI capabilities the enterprise should own. This one decides where, inside the enterprise, each of them should live.

What a center buys, and what it costs

A centralized model puts most AI capability and most AI decisions in one enterprise function. It runs the platform, the common tools, the standards and the shared engineering, and business units consume what it provides. The gains are real. A capability is built once and reused many times. Security, architecture and evaluation stay consistent. Scarce specialists sit together, learn from each other and are easier to recruit. Shared infrastructure costs less.

Michael Porter’s analysis of shared activities across business units explains why those gains are never free. Sharing an activity brings costs of coordination, of compromise and of inflexibility³. Coordination is the meetings, the queue and the prioritization fights. Compromise is subtler: a shared activity has to be performed one way for everyone, even where that way is not the best for any single unit. Inflexibility is the price of changing a shared service that many units depend on.

Corporate centers also tend to overrate their own value. Michael Goold and Andrew Campbell found that executives seeking synergy across business units are prone to a synergy bias, overestimating its benefits and underestimating its costs, and a parenting bias, believing that units will cooperate only if the center compels them to⁴. In AI, those biases produce a familiar failure: a central team that owns every project, forms a long queue and ends up understanding the technology better than the business it serves.

What federation buys, and how islands form

A federated model gives business units substantial ownership of their AI: their own product teams, data people, engineers and priorities, with light enterprise coordination at most. It brings what a center struggles to match. Teams sit next to the customer and the workflow, decisions are made close to the problem, the unit owns the result, and different units can try different approaches at the same time.

The failure mode is just as recognizable. Unit A builds a platform, unit B builds another and unit C buys a third. Uber’s engineers described exactly this pattern before they built a shared machine-learning platform: separate teams were building bespoke one-off systems to put models into production, and there was no established path for doing so⁵. The bill for such AI islands arrives as duplicated spending, inconsistent security, fragmented data and a growing list of vendors.

The deeper cost is lost compounding. Marco Iansiti and Karim Lakhani argue that firms get the most from AI when data and models sit on a shared, integrated foundation rather than in applications owned by separate units, because every new use then builds on the last⁶. On islands, nothing learned in one unit can be reused in the next.

Hub, spokes and the gray area

The most useful evidence on drawing the line comes from Tim Fountaine, Brian McCarthy and Tamim Saleh of McKinsey, writing in Harvard Business Review in 2019. They found that none of the three models, central hub, business-unit spokes or a hybrid of the two, is always better at getting AI to scale. Two large financial institutions they worked with chose opposite designs, one central and one decentralized, and both built AI at the top of their industry⁷.

What the authors did find is that a few responsibilities belong at the hub almost regardless of model, and a few belong in the spokes. The rest sits in a gray area, where the firm’s own situation decides.

Notice what sits in the spokes: almost everything to do with adoption. The center can build a tool, but only the business can redesign the work around it, change the incentives and measure whether anything improved. Notice too that the hub’s list is short. The authors warn against building companywide systems and standards in one sweep before the business cases exist, and report organizations that spent hundreds of millions of dollars on upfront data projects only to abandon them midway⁷.

For the gray area, two questions place most capabilities. How much does this capability gain from scale and reuse? And how much does it depend on domain context?

High scale and low context points to the center: model access, identity, security standards and the evaluation method gain nothing from being built twice. Low scale and high context points to the business: domain workflows, product requirements and use-case priorities need people who live inside the work. High on both calls for a shared core, built once and shaped locally; a search service across the firm’s documents is a common example. Low on both is usually something to buy, or not to do at all.

A central platform is not central decision-making

One distinction removes much of the argument. The enterprise can own the foundations and still leave the decisions about value with the business.

Uber’s answer to its islands shows the pattern at work. Its engineers built Michelangelo, an internal machine-learning-as-a-service platform covering data management, training, evaluation, deployment, prediction and monitoring. By 2017 dozens of teams were using it, with about 10,000 features in a shared feature store⁵. The engineers’ account reports adoption and scale, not business results, so it shows that the split can be run, not that it pays. The platform was central. The models, and the product decisions they served, stayed with the teams that used it.

That last role is what a good AI center of excellence should be. It spreads patterns, training and methods, and it makes itself less necessary over time. It goes wrong when it starts owning every project, becomes the queue and measures technology delivered rather than business outcomes. What the shared platform should actually contain is the subject of the next chapter.

Strategy and maturity move the line

Where the line falls should follow strategy, not fashion. “Our peers have a central AI team” is not a reason. As What Is an Enterprise AI Strategy? argued, the test of a strategy is whether its actions reinforce one another⁹, and the operating split is one of those actions. Three common AI ambitions pull it in different directions.

If the ambition is productivity across the whole workforce, a stronger center makes sense for the platform, purchasing, security and enablement, while business units own how their teams adopt the tools. If the ambition is differentiated AI products, product teams need real autonomy over customer problems, experiments and outcomes, and the center supplies the platform, model access and specialist help. If AI brings heavy regulatory exposure, the center owns risk standards, validation and audit, and business units still own their use cases and their customers.

The line also moves with the organization; how to assess that maturity is the subject of The AI Maturity Model in Module 09. Fountaine and colleagues name three factors for the gray area: the maturity of AI capabilities, the complexity of the business model and the pace of innovation required. Early in the journey it often makes sense to concentrate specialists in a hub and deploy them to the units; as methods standardize, the same experts can work as well, or better, inside the business. Complex, multi-business firms tend to keep pools of experts at the hub, and firms that must innovate fast often keep more strategy and capability-building there too⁷. The survey result that smaller firms centralize more fits the same logic: with few specialists, sharing them is the only way to cover every unit¹.

Good ideas should be able to travel in the other direction. A use case usually starts as a local experiment; if it works and other units need it, it becomes a reusable pattern; if many units depend on it, it becomes a service the center runs. Funding, specialists and decision rights then have to follow the line, which later chapters of this module take up.

Story: the day the split ran backwards

The following is an illustrative composite, not a single company. It is built from patterns common in organizations whose specialist departments adopted generative AI quickly.

The head of digital at an auction house starts her day knowing the shape of the problem but not its size. The house has three specialist departments: fine art, jewelry and wine. Two years ago it created a central team to own all AI. One platform, one set of controls, one place to go. It was a sensible intention.

At half past eight the head of fine art is at her door. His request for a cataloging tool has sat in the central queue for four months, and the catalog for the autumn sale is due. At eleven she learns that the jewelry department tired of waiting and bought its own valuation assistant. Nobody checked where consignors’ details and photographs go when specialists upload them. At two in the afternoon the wine department proudly shows her a third tool. It does almost exactly what the other two do, and none of the three can use the house’s shared provenance and catalog archive, the record of past sales that is the house’s real advantage.

At half past five she writes one line in her notebook: we centralized the projects and federated the platforms. The center had become a bottleneck and the departments had become islands. The house was paying the costs of both models and getting the benefits of neither.

The redraw follows from the chapter’s two questions. The approved tools, the rules for consignor data, the vendor contracts and access to the provenance archive gain from scale and carry house-wide risk, so they move to the center as a self-service platform. The queue goes back to the departments as their own backlog of use cases, because only the wine specialists know which cataloging work matters to wine buyers. The fix was not a new title or a bigger central team. It was treating “who owns AI” as many jobs instead of one.

The infrastructure manager, not the timetable office

A national rail network is a useful picture of the healthy split. The infrastructure manager runs what no single train company should run alone: the track gauge, the signals and the timetable slots. Inside those rules, each operator chooses its own trains, fares and service. A network office that also wrote every operator’s timetable would be the bottleneck. Operators that each laid their own gauge would run well on their own lines, but no train could cross to anyone else’s. Those are the islands. An AI center should be the infrastructure manager, not the timetable office for every operator. The picture has two limits: track changes over decades while shared AI services must change every few months, and a good local experiment should graduate into a shared service, which an operator’s fares never do.

What this means for leaders

The first discipline is to refuse the label. When a leadership team debates whether the company is centralized or federated, it is arguing about a slogan. The productive conversation lists the main AI capabilities and decisions and places each one, using the hub-and-spoke evidence for the clear cases and the scale and context questions for the gray area.

The second is to separate the platform from the decision. Much of the conflict between central teams and business units comes from treating ownership of the infrastructure as ownership of the use case. A center that runs shared services well, and hands the backlog back to the business, removes both the queue and the motive to build islands.

The third is to watch for the two symptoms. A long queue at the center and complaints about distance from the business mean the line is drawn too far toward the center. Several units buying or building the same capability means it is drawn too far toward the edge. When both appear at once, as at the auction house, the line is drawn in the wrong places, not merely too far one way.

The fourth is to redraw on purpose. The right split for a firm with few specialists and an early portfolio is not the right split three years later. Review it as the strategy and the organization’s maturity change.

Check yourself

1. Most organizations run every element of AI under the same model.MythFact
2. An enterprise can run a central AI platform and still let business units decide where AI creates value.MythFact
3. Research shows that a central AI hub consistently outperforms decentralized models.MythFact
4. Sharing an AI capability across business units has costs as well as benefits.MythFact
5. Hybrid is always the right answer.MythFact
6. Once drawn, the line between the center and the business should stay fixed.MythFact

Reflection: find the backwards split

What comes next

Several of the capabilities that belong at the center, such as model access, document search and quality testing, point to the same thing: a shared platform that business units can use on their own. What should that platform provide, and how do you keep it from becoming expensive infrastructure that nobody uses? The next chapter, AI Platform Strategy, takes up that question.