The gap between what you think your agents are doing and what they're actually doing is the number one reason AI projects get shut down. And it’s not because of technology. It’s your own security and legal teams pulling the plug once they realize nobody can answer basic questions about what’s running in their environment.
A few weeks ago I asked a room full of engineers a simple question: who has an AI agent running in production right now? Most hands went up.
Then I asked a follow-up: keep your hand up if you know exactly what that agent is doing at this very second. What data is it touching? Who is it talking to?
Almost every hand dropped.
Here’s how I’ve come to think about it: one agent is a science experiment. A fleet of agents is a liability. Most teams are somewhere in between, and too few IT departments are asking the right questions.
What agent sprawl is and how it happens
Agent sprawl starts with one agent. Then five. Then ten. Marketing spins one up to help with content. Finance has one triaging expense reports. Ops has one running internal workflows. Individually, every one of those was a reasonable call. Nobody sat down and decided to build something ungovernable. It just accumulated, one useful agent at a time…
Retool surveyed 307 CTOs, CIOs, and CISOs in 2026 and found only 5% were very confident they had full visibility into what was running in their production environments. That was about AI-generated tools. Agent fleets are arriving on top of that.
The agents themselves aren’t the problem. Yet, smart teams end up with systems nobody can answer for not because they got careless, but because something shifts the moment teams go from one agent to many. The problem with agent sprawl is the accountability—who owns it when something goes wrong.
You can’t hold the agents accountable. The developer who built it, maybe? But at the end of the day, your company is accountable for the systems you run, every. single. time.
How governing a multi-agent system differs from governing one agent
Single agents vs. multiple agents (or agent fleets)
A single agent is simple. It does one job under one set of permissions. You can hold the whole thing in your head, and if something breaks, you know exactly where to look and exactly who’s accountable, because there's only one thing to point at.
Now, a fleet is multiple agents working at the same time, talking to each other, delegating tasks between themselves. The moment you cross from one agent to many, you’re not managing an agent anymore. You’re managing a system, and systems behave in ways their individual parts never did on their own.
Managing agent fleets vs. single agents
Teams build fleets for specialization. Instead of one generalist agent trying to do everything, you build specialists—one that researches, one that checks, one that drafts, one that approves—and each gets genuinely good at its one job. An orchestrator coordinates them, breaking a complex task into pieces and routing each piece to the agent best suited for it.
That specialization and orchestration is where the trouble starts. The same features that make fleets powerful are what make them hard to govern. It’s the same architecture, viewed from your security team’s side of the table. Delegation is permissions passing between agents. Autonomy is decisions happening without a human in the room. Specialization is more agents, more connections, more surface area.
And that complexity compounds. Two agents means one relationship to track. Five means ten. Ten agents means forty-five possible connections between them, and that’s before you count the databases, APIs, and third-party tools each one touches. By the time a problem surfaces, you didn’t just build ten agents. You built a network, and nobody’s watching the network.
The three places multi-agent governance actually breaks
In production, multi-agent governance breakdowns are almost always the result of one of three failures:
1. Permission bleed
An agent gets access to a database to do one specific job, which is fine. But now it holds that access for everything it does afterward, and the moment it delegates a task to another agent, permissions move to a place nobody signed off on. Nobody grants this on purpose, it just leaks.
2. The black box
Your agent makes a decision. An auditor asks why. If your answer is a pile of logs scattered across four different tools, you have archaeology, not an audit trail. “Give me a couple weeks to dig” isn’t an answer auditors accept.
3. Escalation gaps
Everyone agrees a human needs to be in the loop for high-stakes decisions: approving a large payment, touching sensitive customer data. But almost nobody has actually built that loop. Does the agent know when to stop? Who does it escalate to? What happens while it waits? If you can’t answer that, your agents are either making calls they shouldn’t or stalling silently.
What a governed multi-agent system looks like at 3am
Here’s a system I built for a familiar scenario: the 3 am on-call nightmare, where one exhausted engineer sifts through Datadog, PagerDuty, Slack, and GitHub trying to find the source of an incident. Instead of one person doing all of that, four agents split the work. An orchestrator sits at the top deciding who does what, and three specialists work underneath it—one investigates, one analyzes, one communicates. Each has its own scope, its own tools, its own permissions. There’s no shared credentials or borrowed context.
Say a bad deploy pushes latency on the checkout API from 420 milliseconds to 1,840. PagerDuty fires an alert. In most companies, that’s the moment a human wakes up. Here, the orchestrator wakes up instead.
The audit trail answers the black box problem.
Every tool call, every input, every output is timestamped and on record. If an auditor asks next month why an incident unfolded the way it did, there’s one log to open, not four systems to reconstruct from memory. The orchestrator delegated in a specific order. It didn’t skip a step or call the wrong agent first, and that sequence is recorded both across the system and inside each agent individually.
Permission scoping answers permission bleed.
The investigator reads metrics and events. It can’t touch Slack. It can’t touch Linear. The coordinator posts updates and files postmortems, but it can’t query Datadog. Permissions don't drift between agents, because each one only holds what it needs, for as long as it needs it.
The human approval gate answers the escalation gap.
Thirty seconds after the incident fires, on-call gets one structured message: a hypothesis (the deploy caused the regression), a confidence level, and a recommended action (roll back). But the agent doesn’t roll back. It proposes. There’s a button, and a person has to click it. The action that could take down production if it's wrong is the one action that requires a human hand.
Why governance has to be inherited, not rebuilt for every agent
A centralized system makes governance something every new agent inherits, rather than something you rebuild from scratch each time someone spins one up. Look back at what just happened in that walkthrough. Every governance feature—the permission scoping, the approval gate, the audit trail—wasn’t built agent by agent. It came with the platform.
The risk of non-centralized agent governance
The alternative is the stitched-together stack: agents living in one tool, workflows in another, permissions somewhere else entirely, apps in a fourth. Every seam between those tools is a place governance can fall through, and every new agent means rebuilding the guardrails again from zero.
A centralized platform flips that. A new agent gets the same framework automatically. And your security team approves the platform once, not every single agent every single time, which is the actual difference between security acting as your blocker and security acting as your ally.
A five-question governance checklist to run this week
Ask these questions about your agent setup to understand your baseline for managing agents in production:
- Does every agent have defined data boundaries?
- Is there a human approval gate for high-stakes actions?
- Can you produce a full audit trail on demand?
- Do your agents inherit your existing permission system, or did you build a second security model just for them?
- Is governance built in from day one, or bolted on after the fact?
If you can answer yes to all five, then your security team becomes your biggest champion instead of your biggest blocker. If you can’t yet, you now know exactly where to start and which of the three failures to fix first.
