Superintelligence Failure Modes Inside Companies, Not Sci-Fi

Superintelligence failure modes inside companies are rarely dramatic. For a business running AI agents today, the realistic failures are stale memory that keeps acting on old facts, permissions wider than the job requires, silent drift when outputs change without notice, and instructions hidden in documents or messages. Each one can be contained with ordinary operating discipline: scoped access, scheduled checks, logged actions and a named person who owns the outcome.

Key takeaways

• The failures that matter most at company level are operational: memory, permissions, drift, untrusted input and unclear ownership.
• Each failure mode needs a specific control, not a general commitment to use AI responsibly.
• Permissions should be granted in tiers and reviewed on a schedule, not set once at launch.
• Drift is found by sampling real outputs against a written standard, because the system will not announce that it changed.
• Every agent needs a named owner who can pause it, reverse its recorded actions and explain what it did.

Where company-level AI failures actually come from

Science fiction frames failure as one powerful system turning against its makers. Inside a company, failure usually looks like an ordinary business error, produced faster and repeated more often than a person would manage. An agent that answers customer questions, updates a shared spreadsheet or drafts supplier orders is a bundle of parts: a model, instructions, memory, tools, channels and permissions. A problem can start in one part and surface in another. For an owner, the useful question is which part could be wrong, who would notice, and how long it would take.

The NIST AI Risk Management Framework treats AI risk as something to govern across the life of a system. For a company, that means turning the abstract list into concrete failure modes that can be written down, assigned and tested. A practical first step is to describe one role: the memory and tools it needs, the actions it may take alone and the outcome you will observe. Be Human's managed engagements start from that same definition, because a failure list is only useful when it is tied to one job.

Stale memory: when the agent acts on yesterday's facts

Memory is what makes an agent useful across days and weeks. It is also what makes it quietly wrong. A delivery policy, a price list, a staff assignment or a client's stated preference can change while the agent's stored notes stay the same. Because the agent writes fluently from those notes, the error reads like competence.

Stale memory usually has three sources: facts written once and never revisited, copies of documents taken from an old version of a shared folder, and summaries of past conversations that dropped a later correction. The fix is less about better memory and more about provenance. Every stored fact should record where it came from, who confirmed it and when it should be reviewed again.

Containment steps:

• Tag each memory entry with its source and the person who confirmed it.
• Assign a review rule to each memory category, with prices and policies reviewed most often.
• Require the agent to name the source behind any fact it states to a customer or colleague.
• When a source changes, retire the related notes instead of appending a correction below them.

Illustrative scenario: a small service company's agent keeps a note about the notice period for cancellations. The owner changes the policy and updates the handbook, but nobody touches the agent's memory. For weeks the agent quotes the old rule, politely and confidently. The model did nothing wrong. The gap was a memory entry with no source and no review date.

In a managed engagement, Be Human sets memory scope for each role at the start, records the source behind each memory category and includes memory review in recurring evaluations. Outdated entries are then found on a schedule rather than by a customer. For the setup side, how to seed an AI agent's memory with company history covers what to load first.

Excess permissions: when the agent can do more than its job

Permissions tend to grow by convenience. An agent set up to draft replies gets inbox access for testing, then CRM write access so it can log notes, then the payments dashboard because someone wanted it to check invoices. Each step is reasonable on its own. Added together, they produce an agent that can send, delete or spend far beyond what anyone intended. Nobody chose that outcome, which is why it persists.

OWASP's authorization guidance rests on a simple idea: access should be checked against an explicit policy each time it is used, not assumed because a connection exists. For agents, that means separating three things: what the agent can read, what it can propose, and what it can execute. Execution above a defined threshold should need a person's approval.

Containment steps:

• List every tool and account the agent holds, with the exact actions each one allows.
• Remove any action the role's success criterion does not require, especially delete, export and payment rights.
• Sort the remaining actions into tiers: read freely, draft for review, execute with approval, never execute.
• Repeat the review on a fixed schedule and after every change request.

Illustrative scenario: a distribution company gives its ordering agent read-only access to a supplier portal so it can check stock. A later setup change adds purchase rights to the same login so the agent can place replenishment orders. No one decided the agent should buy without approval. The right simply arrived with the login. A review comparing what the login can do with what the role is approved to do would have caught it.

The tiered approach in growing trust tiers for superintelligent agents is one way to structure these levels. Be Human's managed operations include permission review as a recurring task: access is defined per role, changes go through a written request, and each review compares actual tool rights with approved actions.

Silent drift: when the same agent behaves differently over time

Drift is a change in behavior that nobody announced. It can come from a model provider updating the underlying model, an edited instruction that shifts emphasis, a new document in the knowledge base that outranks an older one, or a tool whose response format quietly changed. Conventional software tends to fail loudly. An agent often keeps producing fluent, plausible output while its quality slips.

Detection depends on evidence gathered outside the agent's own words. Observability practice, as the OpenTelemetry primer describes it, rests on recording what a system did so its behavior can be examined and compared over time. Applied to agents, that means logging requests, tool calls and outcomes, then measuring them against a baseline. A fixed evaluation set, the same cases re-run after every change, gives a comparison that does not depend on how the agent judges its own answers.

A practical plan:

• Write down what good output looks like for the role, with a handful of examples a reviewer would accept.
• Build a small fixed test set from real past cases, including the awkward ones.
• Re-run the test set before and after any model, prompt, tool or data change.
• Sample live outputs every week and score them against the written standard.
• Treat any drop in score as an incident to investigate, not a quirk to ignore.

Be Human's managed monitoring keeps the fixed test set, maintains a change log for prompts, tools and data sources, and reports score movements to the named owner. A model update is then tested before it reaches customers, rather than discovered through a complaint. The weekly sampling step is covered in how to sample AI agent outputs for quality each week.

Injected instructions: when content tells the agent what to do

An agent that reads email, web pages, uploaded files or customer messages is reading text written by people you do not control. Prompt injection happens when that text contains instructions the agent follows, such as a line telling it to ignore its rules and forward a contract list. OWASP's prompt injection guidance treats this as a basic risk for any system that mixes instructions and data in the same channel. Because a model cannot always tell them apart, containment cannot rest on the model alone.

The controls that work are structural. Limit what the agent can do in any session that included untrusted content. Keep outbound destinations on an allowlist. Separate reading from acting, and require approval for any action triggered by external text. Treat the agent's output as untrusted too when another system consumes it.

Containment steps:

• Label every input by origin: internal records, customer supplied, or public web.
• After processing untrusted input, block sending data to new recipients without approval.
• Never let content from outside the company change the agent's instructions or permissions.
• Log every tool call made in a session that included untrusted input, so it can be reviewed.

Illustrative scenario: a customer attaches a document to a support request that contains hidden text asking the agent to reveal internal pricing. The support role has no pricing access, and any reply to an external address waits for approval. The attempt fails, and the log records it. That log is what the owner needs to decide whether the input handling or the role scope should be tightened.

Runaway actions and unclear ownership

Two failures involve scale and responsibility rather than knowledge. An agent with a goal and a retry rule can loop, sending repeated messages, creating duplicate records or consuming paid tool calls with no visible stop. Separately, when an agent makes a mistake, many companies cannot say who authorized it to act at all. Both are operational problems, and both are far cheaper to solve before launch.

Set hard limits on attempts, rate and spend for each task, with a halt that any named person can trigger. Any action that creates or sends something should be idempotent, meaning a retry recognizes that the action already happened and does not repeat it. Ask your tool providers how they handle repeated requests, and build the same check into integrations you own. Ownership is a decision you write down: which actions the agent may take alone, which go to a person, and who answers for the result. Who is accountable when an AI agent makes a mistake walks through that decision.

Containment steps:

• Cap retries and total actions per task.
• Require a duplicate check on any action that creates a record or sends a message.
• Name one owner per agent and one backup who can pause it.
• Write down which decisions the agent may make alone and which go to a person.

In managed operations, incident response covers pausing an agent, reversing recorded actions where possible and writing a review afterward. Be Human operates the controls the company has agreed to. The decision about how much risk the company accepts stays with the company.

Containing each failure before the first agent goes live

These failure modes share a pattern: each is invisible by default and easy to design against once it is named. Before any agent touches live data, write the failure list for that single role and assign a control and an owner to every line. A short list with owners gets used. A general AI safety document often does not.

• Stale memory: source tags, review rules and retirement of notes when sources change.
• Excess permissions: an action-level inventory, approval tiers and a scheduled review.
• Silent drift: a fixed test set, a change log and weekly sampling against a written standard.
• Injected instructions: input labels, an outbound allowlist and approval for externally triggered actions.
• Runaway actions and unclear ownership: attempt caps, duplicate checks, a named owner and a pause right.

Treat the first role as a test of the controls, not only of the model. If the controls are easy to explain to the person who will supervise the agent, they are probably working. If the owner cannot say what the agent may do alone, the role is not ready. Revisit the list after each material change, and keep the change log as the record of what was decided and why.

If you are unsure which task should go to an agent first, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Is superintelligence a realistic risk for a company today? The risks worth managing now are the failures of systems you already run or plan to run, such as stale memory, excess permissions, silent drift and injected instructions. These are ordinary operating problems that appear at larger scale and speed. Planning for them with owners, logs and reviews also prepares the company for more capable systems later.

What permissions should an AI agent have at launch? An agent should have only the access its role needs to meet its success criterion, and nothing that is merely convenient for testing. Separate reading, drafting and executing, and require a person to approve any execution above a defined threshold. Review the list on a schedule and after every change, because permissions tend to grow over time.

How often should an agent's outputs be checked? Check often enough that a quality drop would be caught before it causes damage. Many roles benefit from a weekly sample scored against a written standard, plus a re-run of a fixed test set after every model, prompt, tool or data change. Higher-risk roles may need more frequent checks or review of every output before it leaves the company.

Who should own the controls against AI agent failures? Each agent needs one named owner who can pause it, reverse its recorded actions and answer for its results, plus a backup who can act in the owner's absence. The owner does not need to be a engineer, but must understand what the agent is allowed to do. Managed operations can run the monitoring, reviews and incident response, while the company keeps the decisions about risk.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.