AI agent security: when to require human approval

Answers and actions have different consequences

An assistant that drafts a message is different from an agent that sends it. Agents may call tools, update records or trigger payments. Excessive agency means the agent has more functionality, permissions or autonomy than its job needs.

Inventory the tools

List every connected action, the identity it runs under and the data it can access. Prefer narrow tools and narrowly scoped credentials over general-purpose access. Reading one customer record should not require permission to export an entire customer database.

Make approval meaningful

For consequential actions, show the person the actual proposed destination, records, amount or message before approval. A generic confirmation of a plan may miss a changed detail. The server must verify that the approver is entitled to approve that action.

• Require approval for destructive or irreversible changes
• Reconfirm if material action details change
• Use limits for spending, volume and scope
• Make cancelling a queued action possible where practical

Do not delegate access control to the model

A model saying an action is safe does not make it authorised. Check permissions and validate parameters in the tool execution layer. Treat instructions in retrieved documents, messages and websites as untrusted content rather than permission to perform an action.

Assess mistakes as well as attacks

Test ambiguous instructions, malicious retrieved content and repeated tool calls using a safe environment. Record requested actions, approvals and execution results while avoiding unnecessary sensitive content in logs. Decide which tools should remain disabled until safeguards are demonstrated.