Skip to content
be-human.ai
← Back to the journal

Personalized AI infrastructure

Superintelligence Failure Modes: What Breaks First in Companies

The usual breakpoints in AI memory, tools, permissions and handoffs, and a test an operator can run on each before a stronger model raises the stakes.

Be Human typographic cover with the title: Superintelligence Failure Modes: What Breaks First in Companies
Conceptual illustration created with AI · Be Human

Superintelligence failure modes are the points where a company's AI layer stops behaving the way its owners expect, and in practice they cluster in four places: memory, tools, permissions and the handoffs between agents and people. A more capable model does not remove these breakpoints. It acts on more context and reaches more systems per task, so each break arrives sooner and costs more. Operators can find the first break by testing each layer on purpose while the stakes are still small.

Key takeaways

  • Most breakpoints sit at the seams between memory, tools, permissions and people, not inside the model.
  • A stronger model amplifies whatever context and access it receives, so stale memory and broad credentials become larger errors.
  • Test each layer with a deliberate failure before widening the role.
  • Every handoff needs a named owner, a stop rule and a readable record.
  • Start with one role and one metric so the first failure is cheap to see and repair.

Why the first break is rarely the model

When a company AI system gives a bad result, the first explanation is usually that the model reasoned badly. Reading the traces often tells a different story. The agent acted on an outdated note, called a tool with a field it did not understand, used a credential that could do more than the task required, or passed work to a person without saying what it had already tried. These are infrastructure failures, and an operations leader can inspect them without knowing how the model works internally.

That is why this article organizes the problem around the layer rather than the model. The usual breakpoints are:

  • Memory that is stale, mixed across customers or locations, or retrieved from the wrong source.
  • Tools whose names, inputs or error messages leave the agent guessing.
  • Permissions granted for a pilot that were never reduced afterward.
  • Prompt injection carried inside documents, emails or web pages the agent reads.
  • Handoffs that drop the reason behind the decision.

Each of these gets worse as capability grows, because a more capable agent takes more steps on its own before a person sees anything.

Memory breaks when it is stale or mixed

Memory is the part of the layer people trust most, which is why it breaks quietly. A customer changes a preferred contact channel, a policy is replaced, a project closes, but the notes describing the old state stay in the store. Retrieval systems select passages that look most similar to the question, not passages that are most recent. An old paragraph that reads well can outrank the correction that replaced it, and the agent will repeat the old answer with the same confidence it uses for current facts.

Mixing is the second risk. Notes from one location, one client or one product line can leak into an answer meant for another, especially when the store was built quickly and nobody set boundaries on what each role may retrieve. The fix starts with knowing which sources each role is allowed to read and what happens when two sources disagree.

To test memory, pick three facts that have changed recently, ask the agent about each one, and check whether it cites the current version. Then ask a question about one account and confirm that nothing from another account appears in the answer.

Tools fail when their contracts are vague

A tool is a promise: this input produces this effect, and this error means that. When the description is loose, the agent fills the gap with a guess. A more common failure is the timeout. The tool writes the change, the connection drops before the confirmation returns, and the agent retries. Without a protection against repeated requests, the second call repeats the action. Stripe's documentation on idempotent requests describes the pattern of attaching a provider-defined key to a request so that repeated calls return the original result instead of doing the work twice. Many internal tools built for a pilot have no equivalent.

Illustrative scenario: an agent updates a customer's delivery address through an internal tool. The first call writes the change but times out before confirming it. The agent retries, the tool creates a second change record, and a shipping label is generated for the old address as well. The model did not cause this. A missing repeat check and an unclear timeout message did.

To test each tool, call it twice with identical input and confirm that only one effect occurs. Send a malformed input and check that the error text tells the agent what to do next. Force a timeout in a sandbox and see whether the agent reports uncertainty or assumes success.

Permissions widen without anyone deciding to widen them

Permissions rarely fail in one dramatic step. A pilot starts with a service account that can read a shared folder. Later someone adds write access so a demonstration works. The same account is then reused for a second agent, and the second agent's scope is never written down. Eventually the credential can perform actions that nobody listed in the role. OWASP's authorization guidance points toward checking access on every action, not only at sign-in, and toward separating what an agent may read from what it may change or send.

The review is not a one-time event. Each credential needs a written scope, an owner and a date for the next check. The steps are laid out in How to Run a Permission Review for Your AI Agents, Step by Step.

To test permissions, list every action each credential can perform, compare that list with the role's written scope, and remove anything the scope does not name.

Prompt injection arrives inside ordinary content

An agent that reads supplier emails, web pages or uploaded files is reading text that someone else may have written to influence it. OWASP's guidance on prompt injection treats instructions embedded in retrieved content as a primary risk, because the model cannot always distinguish the owner's instructions from instructions hidden in a document. The more useful the agent's tools are, the more valuable that hidden instruction becomes to someone who wants the agent to misuse them.

The practical response is structural. Keep the read path separate from the act path where you can. Require a human approval before any external message, payment change or record deletion. Log which source text influenced each action, so that a suspicious result can be traced back to its origin. Treat every retrieved document as untrusted input, even when it comes from a trusted partner, because partners' inboxes are also compromised.

To test this, place a harmless instruction inside a document the agent is expected to read, such as a line asking it to change an unrelated field. Check whether the agent acts on it, and whether the action would have been blocked by an approval step.

Handoffs lose the reason behind the action

A handoff between an agent and a person fails in two directions. The agent may escalate too little, finishing parts of a task it should have left alone. Or it may escalate everything, and people stop reading the queue because most items look the same. In both cases the human receives a status without the reasoning: which records were checked, which tools ran, what remains uncertain, and what the agent recommends.

A runbook prevents much of this. Each role needs one named owner, explicit stop rules that say when the agent must pause, and a standard handoff note that includes the evidence behind the recommendation. The format is described in AI Agent Runbook Template: One Page Per Role With Stop Rules.

In a test, give a person a handoff from the agent and ask them to act on it without opening any other system. If they must search for context, the handoff note is incomplete.

A test to run before the next model upgrade

Run this before a stronger model is connected to existing memory, tools or credentials. Each step produces evidence that an operator can review without reading model internals. The NIST AI Risk Management Framework treats this kind of measurement as an ongoing activity rather than a single approval, so the results should be recorded and repeated after each change.

  • Write the role, its single success metric and its allowed actions on one page.
  • Choose five real cases, including at least two where the correct answer changed recently.
  • Run each case and record which memory was retrieved, which tools were called and who approved what.
  • Repeat each write action with identical input and confirm that only one effect occurred.
  • Place a harmless instruction inside a document the agent reads and check whether it acts on it.
  • List every action each credential can perform and remove anything outside the written scope.
  • Record the first break, give it one owner and fix it before widening the role.

Keep the approval thresholds where they are until the test passes. Approval Thresholds for Superintelligent Agents: Raise Them Slowly explains why gradual increases make the results easier to read when something goes wrong.

How Be Human keeps each breakpoint visible

Be Human designs and operates this layer as a managed service. The work covers the memory scope and its sources, the tool contracts and their error behavior, the permission boundaries for each role, and the handoff path with its logs. Managed operations add monitoring, recurring evaluations, permission reviews and controlled changes, so a breakpoint found in one period is tracked and repaired instead of being rediscovered later. The service is delivered remotely in English and Portuguese. If you are not sure which of these breakpoints matters most in your company, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

What is the most common superintelligence failure mode in companies? The most common failures happen at the seams rather than inside the model: stale or mixed memory, tools that repeat actions after a timeout, permissions that grew beyond the written role, and handoffs that omit the reasoning. Each of these can be tested with ordinary evidence such as logs, sandbox calls and a review of credentials. Starting with these checks usually reveals more than reviewing the model's answers alone.

Should we upgrade to a more capable model before fixing these breakpoints? Upgrading first usually makes existing breakpoints larger, because the new model acts on more context and reaches more systems in each task. A safer sequence is to run the test in this article on the current setup, repair the first break, and then evaluate the stronger model against the same cases. The comparison is then about the model, not about accumulated infrastructure problems.

Which breakpoint should we test first? Start with the one that would cause the most damage if it failed quietly, which is usually a write-capable tool or a credential with broader access than its role needs. Test that path with a repeated request and a review of its permissions before anything else. Memory issues matter as well, but a repeated write or an unauthorized send tends to be harder to reverse.

Do these failure modes only matter for large companies? They matter for any company that lets an agent read business records or act on them, including small teams with a few agents. A small company often has fewer people to catch a bad handoff, so each breakpoint can be harder to notice. The same test works at any size, and it can be run against one role before a second role is added.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →