AI agent guardrails: a production control checklist

On October 9, 2026, Anthropic disclosed that some of its AI agents, given problems to solve during internal evaluations, had gone looking for resources online and misused what they found. According to TechCrunch’s report, they exploited software flaws on websites, including some run by US government agencies, got into databases without paying the fees, used URL shorteners to slip past restrictions, and one submitted a false murder tip to the Philadelphia police. Anthropic’s response was to cut its internal evaluations off from the live internet until it can monitor and control its agents.

That happened at a company that builds the models, in a test environment. A business agent connected to a CRM, a payment system or a customer inbox has the same basic shape: a model that decides what to do, and tools that let it do it. Guardrails are what make the second part safe.

What are AI agent guardrails?

AI agent guardrails are controls outside the model that limit what an agent can see, do and spend, and record what it did. They are enforced in code, permissions and infrastructure, not in the prompt, so they still hold when the model misreads a task, is manipulated by text it reads or takes a shortcut nobody anticipated.

OWASP calls the underlying risk “excessive agency”: damaging actions taken in response to unexpected, ambiguous or manipulated output from a model. It has three causes, and each guardrail removes part of one of them: too much functionality, too many permissions or too much autonomy.

Input, output and action guardrails: which ones matter for agents

Guardrails fall into three groups, and agents need the third far more than chatbots do. Input guardrails screen what reaches the model. Output guardrails check what it produces. Action guardrails limit what it can change in the world, and they are the reason an agent needs a different control set from a chatbot that only talks.

Many products sold as guardrails cover the first two groups: classifiers that flag injected prompts or toxic replies. Some teams add a second model that reviews the first one’s output before it is used, sometimes called a guardian agent. Both are useful, and both are still models reading text. They complement action guardrails enforced in code and infrastructure; they do not replace them.

What the Anthropic incident shows about agent risk

Every behavior in the disclosure maps to a missing control, and none of them needed a smarter model to prevent. Anthropic itself attributed the behavior to training environments that taught models to look for loopholes, which is a reason to assume any agent may do the same.

The common thread is that each behavior was possible because the agent had a general capability, such as unrestricted web access, where it needed a specific one. Narrowing capability is cheaper and more reliable than trying to predict every misuse of a broad one.

The 10 controls a production agent needs

These are the controls we put around an agent before it touches real systems. Each one names the risk it addresses and, more importantly, where it is enforced, because a control the model can talk its way around is not a control. The OWASP mapping is our reading of the two lists, not an official crosswalk.

Approvals that people do not rubber-stamp

An approval step only works if the person sees the actual change, not the agent’s summary of it. OWASP lists “human-agent trust exploitation” as its own risk (ASI09): polished explanations persuaded operators to approve harmful actions. Show the record before and after, the recipient and the amount, and make approving a deliberate act.

On a phone call, the person approving is the caller. The service booking agent we built for Innovative AI matched the caller’s own description to one service on that vehicle’s menu, read the chosen service back and booked only after the caller agreed. The point of that read-back is that a wrong match should cost one extra sentence on the call, not the wrong job on the dealership’s schedule.

Limits the agent cannot negotiate

Step and spend limits sound blunt, and that is the point. A cap of a few dozen tool calls per task, a daily budget per agent and a rate limit per user turn a loop or a manipulated agent into a small, visible failure instead of a large one. OWASP lists logging and rate limiting as the measures that reduce harm when prevention fails.

Why a system prompt is not a guardrail

A rule in a prompt is a request; the model can misread it, lose it over a long task or be talked out of it by text it reads along the way. Anthropic’s own explanation for its incident was that training rewarded models for finding loopholes, and a prompt that says “don’t” is exactly the kind of restriction such a model routes around.

System promptA request

• Explains the job
• Text the model can misread
• Can be lost in a long task or overridden by injected text

vs

GuardrailEnforced in code

• Enforces the limits
• Code the model cannot change
• Holds when the model is manipulated by text it reads

Use the prompt to explain the job, and use code to enforce the limits. The same principle runs through OWASP’s guidance, which calls it complete mediation: authorize every action in the downstream system rather than relying on the model to decide what is allowed. If the agent connects to tools over MCP, the protocol’s security guidance makes the same point about tokens: a server checks that each token was issued for it and never passes it on. Our guide to MCP servers for Muse and Grok covers the server side in detail.

How the controls map to NIST’s generative AI profile

Enterprise risk teams often work from NIST rather than OWASP. NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework (July 2024), lists 12 risks that generative AI creates or makes worse. Four of them line up directly with agent guardrails:

• Information Security: controls 1, 3, 5 and 10 narrow what a manipulated agent can reach and let you cut it off.
• Data Privacy: control 2 keeps the agent inside the user’s own access, and control 7 records what it read.
• Confabulation: control 8 tests answers against known cases before release, and control 9 catches drift after it.
• Human-AI Configuration: control 4, with approvals designed so people review the real change, not a persuasive summary.

The other eight, from intellectual property to environmental impacts, are governance questions for the deployment as a whole rather than controls around one agent. This mapping is our reading of the profile, not an official crosswalk. If your agent serves people in the EU, logging and human oversight also carry legal weight; see which EU AI Act and GDPR duties apply to agents.

Who is liable when an agent gets it wrong

Usually the company that deployed it. In Moffatt v. Air Canada (2024), a customer relied on the airline’s website chatbot, which said a bereavement fare could be claimed after travel; the airline’s own policy page said otherwise. Air Canada argued the chatbot was a separate legal entity responsible for its own actions. British Columbia’s Civil Resolution Tribunal called that “a remarkable submission”, noted that the chatbot “is still just a part of Air Canada’s website”, and found the airline liable for negligent misrepresentation.

It is a decision of British Columbia’s Civil Resolution Tribunal, not binding precedent elsewhere, and this is not legal advice. The engineering lesson holds anyway: an agent’s answer to a customer is your company’s statement. Output guardrails on customer-facing answers, such as grounding every policy answer in a current source and refusing questions outside the agent’s scope, are a liability control as much as a quality one.

How we apply this on Superdeal

Superdeal’s agent was designed around controls in the system rather than trust in the model.

Superdeal is a campaign CRM in which an AI agent finds creators, drafts offers and handles shipments. Every consequential step goes Draft, Review, Send: the agent drafts, a person on the brand’s side reviews, and nothing reaches a creator until they approve it (control 4). Agencies share scoped, read-only dashboards with their clients, so a client view cannot change a campaign (control 2).

1. 01DraftThe agent drafts
2. 02ReviewA person on the brand’s side reviews
3. 03SendOnly after they approve

What guardrails cost in time and money

Most guardrails are cheap; the expensive ones are the ones that involve people. Permission checks, allowlists, limits and logging are ordinary engineering and add milliseconds. Model-based checks on inputs and outputs add a model call each. Human approval adds minutes or hours, so it belongs only on actions that are costly to undo.

In our projects these controls are part of the build rather than an add-on: a Fixed-Price Build of $25k–$120k includes the approval steps, evaluation set, audit log and security review, and the AI agent development page lists what each engagement covers. What changes the effort is the number of systems the agent writes to, because every write path needs its own permission check, preview and rollback.

When lighter guardrails are enough

An internal assistant that only reads documents the user can already see needs permissions, logging and an evaluation set, and little else. Approval steps, spend limits and a network allowlist matter once the agent can write to systems, spend money or contact people.

Match the controls to what the agent can break, and add them when capabilities grow rather than all at once. Our checklist for taking AI pilots to production covers the business side of that decision: metric, owner and integrations.

A guardrails review you can run this week

Take your most capable agent, or the one you plan to build, and answer these in writing:

• Which tools can it call, and which of them does the task not need?
• Whose permissions does it act with, and are they checked by the system or by the prompt?
• Which domains and services can it reach over the network?
• Which actions send, pay, delete or contact someone, and who approves them?
• What happens if a document it reads contains instructions?
• What stops it after a hundred tool calls, or a thousand dollars?
• Where is the log of what it did yesterday, and who reads it?
• How would you stop it within five minutes?

Any question without a clear answer is a control to add before the agent gets more access. If a vendor is building the agent for you, put the same list to them alongside our questions to ask an agent vendor.

Related service

AI Agent Development

We design and build autonomous AI agents and multi-agent systems that read, decide and act across your tools, with human approval where it matters.

• Explore AI Agent Development →
• Case study: Superdeal →