Governing AI agents: autonomy is a control decision, not a feature

The moment an AI system can take an action rather than produce an answer, the governance question changes completely. A wrong answer is a wrong answer. A wrong action sends an email, moves money, changes a record or deletes a resource - and it does so at machine speed and volume.

Autonomy is a spectrum, and it should be an explicit choice

1. Suggest - the agent proposes, a human executes. Lowest risk, most friction.
2. Execute with confirmation - the agent prepares the action and a human approves the specifics.
3. Execute with notification - it acts, and a human is told after.
4. Fully autonomous - it acts within defined limits and nobody is routinely told.

Most organisations drift up this ladder without deciding to. Each step should be a recorded decision tied to the consequence of the action, not a default that emerged because confirmation prompts were annoying.

Blast radius is the primary control

Since you cannot guarantee the agent behaves correctly, constrain what incorrect behaviour can reach:

• The agent's permissions are the user's permissions. Enforce authorisation in the tool layer against the end user's identity, never in the prompt. An instruction saying "only access data this user may see" is not access control.
• Narrow, typed tools. A tool that fetches an order by ID is bounded; one that runs arbitrary queries is not.
• Rate and budget limits per action type, so a loop is contained rather than unbounded.
• Irreversibility gates. Anything destructive, financial or outbound requires confirmation regardless of autonomy level.
• Segregated sessions - an agent that has processed untrusted external content should not then hold privileged capability.

The design question worth asking out loud: if this agent were fully controlled by an attacker for one hour, what is the worst it could do? If that answer is unacceptable, no amount of prompt engineering fixes it - the permissions have to change.

Accountability stays with a person

An agent cannot be accountable. Every agent needs a named human owner responsible for what it does, and that ownership must survive the owner changing role - an ownerless agent still acting in production is the AI equivalent of an orphaned service account.

Where an agent acts on customers, the organisation is answerable for the outcome regardless of autonomy. "The model decided" is not a defence to a regulator, a court or a customer.

Logging that supports investigation

Record, per action: which agent, acting for which user, what it was asked, what it retrieved, which tools it called with what parameters, and what changed. Without the tool-call detail you cannot reconstruct why an action happened, and reconstructing intent after the fact is the whole problem in an agent incident.

A stop mechanism

You need the ability to disable an agent immediately, and to identify and reverse what it did in a time window. Build both before granting write access - they are almost impossible to retrofit under incident pressure.

Key takeaways

• Autonomy level should be a recorded decision, not a drift.
• Constrain blast radius - assume the agent can be influenced.
• Enforce authorisation in the tool layer, never in the prompt.
• Build the stop-and-reverse mechanism before granting write access.