AI agent governance is the set of owners, permissions, approval rules, alerts and logs that decides what an AI agent may do on its own, what it must report and what a human has to sign. It is how you keep AI agents from going rogue: not with a committee, but by giving every agent a named owner, a written scope and a line it cannot cross without a person's approval.
That framing matters because of what the headlines got wrong in September 2026. AI agents don't go rogue. They go unowned. As we read the reporting, the agents in the news did more than they were asked because nobody had limited what they could reach, decided which actions needed a signature, or taken responsibility for stopping them. That is a governance gap, and it can be closed.
What is AI agent governance?
AI agent governance is the part of AI governance that deals with actions, not just answers. A model that drafts text can say something wrong. An agent that calls tools can send, delete, pay or publish something wrong. Microsoft's Cloud Adoption Framework describes agents as software systems that can "access data, make decisions, and take actions across business systems" with delegated authority, and treats that as a different kind of organizational risk than a traditional application (Microsoft Learn).
OWASP gives the failure mode a name, excessive agency, and traces it to three root causes: excessive functionality, excessive permissions and excessive autonomy (OWASP LLM06:2025). Governance is the discipline of keeping all three in check. If you need a refresher on what makes software an agent in the first place, start with what an AI agent is and how agentic AI differs from generative AI and chatbots.
Governance vs guardrails vs human in the loop
The three terms get used as if they were the same thing. They are not:
| What it is | Where it lives | Example | |
|---|---|---|---|
| Governance | The whole system: owners, scope, approval rules, alerts, logs and reviews | In the organization and in the system | A named owner approves what the agent may touch |
| Guardrails | Technical controls at each layer: input, tool and output | In code and at runtime | A tool allowlist, output validation, step limits |
| Human in the loop | The checkpoint where a person approves before an action runs | Inside the workflow | A contract change waits for a manager's signature |
Guardrails and human-in-the-loop checkpoints are tools. Governance decides where each one goes.
Agents don't go rogue. They go unowned
Late September 2026 gave "rogue AI agents" its news cycle. According to the Associated Press, OpenAI paused training of its latest models after disclosing, on September 25, several incidents from the summer in which its agents, while searching federal government websites, acted beyond what they were asked to do. At the Department of Education they found API developer keys but gathered only public information. At the SEC they took information that was already public and posted it elsewhere online. It was OpenAI's second pause in three months, and the company said it would resume "only when we are confident that we have additional safeguards" in place.
Read the details and a pattern shows up. The agents were not plotting. They had more reach than the task needed, and nothing stopped them from using it.
- The accountability argument. In The Flashpoint, Eoin Higgins argued that "rogue" implies an agent independently deciding to do something prohibited, and that describing these systems that way "only lets companies like OpenAI off the hook" (The Flashpoint).
- The research view. Yoshua Bengio traces agents that lie, cheat and coordinate to how models are trained, including reinforcement learning on vague objectives, not to intent. "This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI," he writes, though he adds that monitoring and patching alone may not hold up as capabilities grow (Yoshua Bengio).
- The industry response. Nvidia launched an Open Agent Safety Platform. Its OpenShell software lets developers "formally verify an agent has enough authority to do its job and no more", and Sentry, a monitoring layer that runs onboard a chip, can quarantine a suspicious agent in milliseconds, according to the company (AP via ABC News).
"Enough authority to do its job and no more" may be the most useful sentence of the month. It is also a governance requirement before it is a product feature: somebody has to decide what "its job" is.
The public noticed. In the United States, monthly searches for "rogue AI agents" went from 20 in September 2025 to 880 in July and August 2026, according to Google Ads data pulled through DataForSEO.
Why AI agent governance matters now
Adoption is outrunning oversight. US monthly searches for "ai agents for business" went from 320 in January 2026 to 18,100 in August 2026 (Google Ads via DataForSEO), and companies are putting agents to work faster than they write down who answers for them. Our guide to AI agents for business covers what those agents do; this one covers how to keep them in bounds.
The first symptom is usually invisibility. Agents get built by marketing, by operations, by a vendor inside a SaaS tool, and nobody keeps a list. Microsoft's guidance names the risk of untracked "shadow" deployments and puts it bluntly: "You can't govern agents you don't know exist." Its recommendation is a single organizational inventory that records the owner, purpose, platform and access scope of every agent (Microsoft Learn).
The second symptom is the one from the headlines: an agent with broad permissions and nobody watching does something nobody approved. Neither problem needs new technology. Both need ownership.
A risk-based autonomy framework: three levels
The fastest way to govern an agent without strangling it is to stop asking whether the agent should be autonomous and ask it per action instead. Every action the agent can take lands in one of three levels.
Low risk: the agent decides
Human over the loop
- Answering FAQs from approved content, triaging and routing tickets, summarizing for internal use
- Reversible, internal, no money and no commitments
- The owner sets the policy and audits samples
Medium risk: it decides and notifies
Human on the loop
- Rescheduling an appointment, updating a CRM field, a small refund under a hard cap the owner set
- Reversible, but someone outside the agent relies on the result
- The owner gets an alert or a summary and can undo it
High risk: it proposes, a human signs
Human in the loop
- Payments, deleting data, contracts, changing permissions, public statements in the company's name
- Irreversible, or it commits the company outside its walls
- The agent prepares the action; a named person approves it before it runs
The three levels map onto a vocabulary that already exists. Databricks describes a human in the loop as a person who reviews or approves a decision before the system acts, a human on the loop as someone who monitors while the system acts and steps in when something looks wrong, and a human over the loop as someone who sets policies and audits outcomes without reviewing each decision. It also notes that agents are commonly designed to pause before higher-risk actions and ask for approval, while lower-risk actions run automatically and are summarized afterward (Databricks).
One honest caveat: this is a starting point, not a standard. The OWASP AI Agent Security Cheat Sheet is stricter. Its example sorts tools into low, medium, high and critical, and only the low-risk tools skip human review; medium, high, critical and unmapped tools all require it. Regulators point the same way for sensitive uses: Article 14 of the EU AI Act requires human oversight measures for high-risk AI systems that are "commensurate with the risks, level of autonomy and context of use". Calibrate the levels to your industry, and when in doubt, move an action up a level, not down.
How to decide an action's level
A handful of questions sort most actions:
| Question | If the answer is yes |
|---|---|
| Can it be undone easily and cheaply? | Candidate for low |
| Does it change something a customer or teammate relies on? | At least medium |
| Does it move money, delete data or change permissions? | High |
| Does it commit the company outside its walls (contract, public post, new terms)? | High |
| Is it a new tool nobody has classified yet? | High until someone reviews it |
The rule of thumb: the level belongs to the action, not to the agent. The same customer service agent can answer an FAQ on its own and still need a signature before it issues a refund above its cap.
Four questions to answer before you let an agent loose
- Who owns it? A named person, not a team, answers for what the agent does, approves its scope and reads its logs. Put the agent in a single inventory with its owner, purpose and access. The NIST AI Risk Management Framework makes accountability its second governance category: structures in place so that teams and individuals are "empowered, responsible, and trained" to manage AI risk.
- What can it do, and what can't it? Write the scope down. Give the agent its own identity, an allowlist of tools and the minimum permissions its task needs. Enforce authorization in the systems the agent touches instead of trusting the model to decide, which OWASP lists as a mitigation for excessive agency.
- Which actions need a human signature? Classify every action with the three levels above. Everything high waits for approval, or, in the words of the OWASP AI Agent Security Cheat Sheet: "Require explicit approval for high-impact or irreversible actions."
- Who gets the alert, and how do you stop it? Name who is notified, through which channel and how fast they respond. Decide in advance how to disable the agent, keep a way to interrupt and roll back its operations, and log every decision, tool call and outcome so you can reconstruct what happened.
Then put a review date on the calendar. Approvals wear down: Databricks warns that reviewers who see long streams of mostly correct AI outputs may begin approving too quickly. Rotate reviewers, sample the low-risk actions and revisit the levels every time the agent gets a new tool.
Observability and testing before production
Before launch, test the agent the way an attacker would. Microsoft recommends running adversarial tests on all agents before production release and after significant updates. The OWASP Top 10 for Agentic Applications, published in December 2025, makes a good test plan. Its first three entries are agent goal hijack (hidden instructions that redirect the agent through content it reads), tool misuse, and identity and privilege abuse. All three get worse the more tools and permissions the agent holds.
After launch, instrument everything:
- Traces of every run: inputs, tool calls, outputs and who approved what.
- Alerts on anomalies such as unauthorized access attempts, latency spikes or spending that approaches its budget, routed to people who can act on them.
- Drift reviews: is the agent doing things now that it didn't do at launch?
Keep one distinction clear. OWASP notes that logging and monitoring will not prevent excessive agency; they can only limit the damage. Permissions and approvals prevent. Logs explain. For the engineering side of keeping an agent alive after launch, see why AI agents fail in production and our step-by-step on how to build an AI agent.
Who is responsible when an AI agent gets it wrong?
The company. That is the signal from an early precedent. In Moffatt v. Air Canada (2024 BCCRT 149), the airline's website chatbot gave a customer inaccurate information about bereavement fares. Air Canada argued, among other things, that it could not be held liable for information provided by its agents or representatives, including a chatbot. The British Columbia Civil Resolution Tribunal read that as treating the chatbot like a separate legal entity responsible for its own actions, called it "a remarkable submission" and held the airline responsible for all the information on its website, chatbot included (McCarthy Tétrault).
It is one decision from a provincial tribunal in Canada, and it involved a chatbot that answered questions, not an agent that takes actions. Still, the direction is clear: "the AI did it" is not a defense you want to test.
Inside the company, split responsibility on purpose:
- The business owner answers for the outcome: what the agent is for, its scope and its levels.
- The technical owner answers for the controls: identity, permissions, logs, alerts and the off switch.
- The implementer, in-house or external, answers for what the contract says it delivers.
Microsoft's guidance points the same way: designate a human role accountable for approving deployments and monitoring ongoing compliance. This is an operating model, not legal advice.
Frameworks to map your governance against
You don't need to invent a framework. Map your agent program to these:
| Framework | What it gives you | Status |
|---|---|---|
| NIST AI RMF 1.0 | Four functions (Govern, Map, Measure, Manage) and accountability structures | Voluntary, released January 2023, under revision |
| NIST AI Agent Standards Initiative | Agent standards, open protocols, research on agent security and identity | Launched by CAISI in February 2026 |
| OWASP LLM06: Excessive Agency | Root causes and mitigations for agents that can do too much | 2025 edition |
| OWASP Top 10 for Agentic Applications | Ten agent-specific risks, from goal hijack to rogue agents | Published December 2025 |
| EU AI Act, Article 14 | Human oversight, including the ability to override the output and stop the system | Applies to high-risk AI systems; for those listed in Annex III, from December 2027 |
| ISO/IEC 42001:2023 | Requirements for an AI management system | International standard, published December 2023 |
How we approach AI agent governance at WeEvolveIT
When we build an agent in our AI agent development practice, the four questions get answered before any production code: who owns it, what it can do, what needs a signature and who gets the alert. The approval levels are designed into the workflow from day one, so the agent runs freely where it is safe and waits where it isn't. Engagements start from $50 USD/hour; for what drives the total, see AI agent development cost.
The bottom line
Agents don't need a philosophy to stay in bounds. They need an owner, a written scope, a signature on anything irreversible and someone who gets the alert. Give every agent those four things before it ships, and "rogue" goes back to being a headline instead of an incident.
