The Friday before launch, one malformed API call from an AI agent can provision the wrong business service, and it will do so with total confidence and no hesitation. The fix is not better agents; the fix is a checklist that treats permissions, approvals, and rollback as features of your infrastructure, not afterthoughts of your prompt.
That scene is playing out in a specific way right now. A startup I'll call Northwind, which sells to logistics companies, had an agent tasked with standing up a staging environment for a client demo. The agent had broad enough credentials to talk to the company's internal service catalog, which is common when teams want agents to "just get things done." On the Friday before a Monday launch, the agent interpreted a stale configuration file as the source of truth and provisioned a full production-grade database cluster, complete with external egress and a public endpoint, instead of the isolated sandbox it was supposed to create. It did exactly what it was asked, using exactly the wrong input.
The mechanism here is not new, and it is worth understanding precisely. In 2017, the British Airways passenger service system failed, grounding flights at Heathrow and Gatwick for days. The official investigation, reported by the BBC and others, traced the direct cause to a single engineer who, working on a backup power system, followed a procedure and then manually switched off a UPS, cutting power to the entire data center. But the deeper cause was that the system, which had been built up over decades, had accumulated so many interconnections that nobody could reliably predict what a single, apparently routine action would do. The engineer was confident. The procedure was documented. The blast radius was not.
That is the exact shape of the problem with agent provisioning. The agent is the engineer; the API is the UPS; the interconnected catalog of services is the decades of accumulated infrastructure. The agent is not reckless. It is following a documented procedure against a system whose true dependencies nobody has mapped in a way machines can respect.
So the checklist. It is short, and it is for the humans who deploy agents, because the agent will not write its own guardrails.
Permissions that match the task, not the role
The agent should never carry credentials that can do more than the narrowest version of its job. If it needs to create a sandbox, it should not have permission to create a production cluster. This sounds obvious. In practice, most agent failures happen because teams grant broad role-based access and then rely on the prompt to keep the agent inside the lines. Prompts are suggestions. Permissions are enforcement. The Friday-before-launch version of this is checking every credential the agent actually uses, not the ones you think it has.
Approvals that sit in the path of destructive change
The second control is a required human approval for any action that creates, modifies, or deletes a resource the agent did not ask to touch. The approval step must be in the request path itself, not a log entry the human reviews on Monday. A simple rule: if a provisioning call changes the blast radius of the system, it pauses until a named person confirms it. That one step, a few seconds of friction, would have stopped Northwind's database cluster before it existed.
Rollback that restores a known state
Rollback is the control that gets skipped most often because it feels like planning for failure. But the restore has to be tested on the same cadence as the provisioning itself. The Northwind team did not have a rollback for a production cluster because they had never created one; the runbook only covered sandboxes. The better practice is to define, for every service an agent can provision, the exact inverse operation that returns the system to its prior documented state, and to rehearse it quarterly, not because you expect a Friday disaster, but because the first time you exercise a rollback should not be the first time you need it.
Northwind got lucky. The Monday demo went to a client who was a week from signing; the public database endpoint was caught by a security scan before anyone outside the building saw it. The company spent that weekend writing the checklist above from scratch, under pressure, after the fact.
That is the backward way to do it. Read the British Airways report and you will see the same pattern: a single actor, an apparently reasonable action, a system too interconnected to predict. The engineer survived; the systems did not. Your agent will not be the last one to misinterpret a stale config on a Friday. Make sure the infrastructure, not the prompt, is what catches it.
Comments
No comments yet.