Skip to content
be-human.ai
← Back to the journal

Personalized AI infrastructure

Superintelligence Stack Maturity: Five Checks for Your Company

Five checks show whether your memory, tools, permissions and observability can compound as AI models improve, and which gap to close first.

Be Human typographic cover with the title: Superintelligence Stack Maturity: Five Checks for Your Company
Conceptual illustration created with AI · Be Human

Company superintelligence maturity is the degree to which your memory, tools, permissions and observability are organized well enough that better models make your operations more capable, not just more impressive in a demo. You can test it with five concrete checks, each answered by evidence rather than opinion. A company that passes them can upgrade models without rebuilding its context, while a company that fails them mostly gets faster answers without knowing whether those answers are right.

Key takeaways

  • Maturity is measured by what your systems retain, what they can reach, what they are allowed to do, and what they can show afterward, not by the model name on the invoice.
  • Memory should be owned, scoped and reviewable before any agent is connected to it.
  • Every tool needs a written purpose, a named owner, a clear split between reading and acting, and a defined failure behavior.
  • Permissions must match the approval boundary of each role, and they must be reviewed on a schedule.
  • Observability is the evidence trail that tells you whether a model change actually helped.

Why maturity matters before the next model arrives

Models change faster than most operations do. A stack that consists mainly of prompts, copied documents and manual follow-up gets little from a better model, because the limit was never raw reasoning. The limits are usually context, access and feedback: the agent does not know the current facts, it cannot reach the right system, or nobody can tell when it made a bad call. Improvement compounds only where the surrounding layer keeps the company's knowledge in a usable form, exposes tools with clear limits, records what happened, and measures the result.

The five checks below are meant for a business owner or operations leader who needs to decide where to invest next. None of them requires you to understand model internals. Each one asks for a artifact you can inspect: a list, a log, a permission table or a test set. If you cannot produce that artifact quickly, the gap is real, and it will not close by itself when the model improves. For a broader view of the components involved, see Superintelligence Stack Explained: Models, Harness, Memory, Permissions.

Check one: Is your memory owned, scoped and current?

Memory here means the business facts an agent relies on: customer history, approved terms, process steps, exceptions, and the reasons behind past decisions. Most companies have this knowledge, but it lives in inboxes, personal notes and the heads of a few people. An agent that reads from that scattered material will sound confident and be wrong at the edges.

Ask four questions. Who owns each source of memory, and who corrects it when it is wrong? Does each role see only the memory it needs? When was each source last checked against reality? And when the agent does not know something, does it say so, or does it guess?

A practical test is to write ten questions a new employee would ask in the first week, and see whether the agent answers from current sources and cites them. Failures in this test are usually not model failures. They are missing owners, stale documents or sources that contradict each other.

  • Each memory source has a named owner and a review date.
  • Each agent role reads only the sources its job requires.
  • Answers can be traced back to a specific document or record.
  • The agent states uncertainty instead of filling gaps.

Check two: Are your tools described well enough to be used safely?

A tool is any connection that lets an agent act on a system, such as a CRM, a calendar, a shared inbox or a ticket queue. Maturity here means every tool has a stated purpose, documented inputs, a clear distinction between reading and writing, and a behavior when the system is unavailable. The [Model Context Protocol introduction](https://modelcontextprotocol.io/docs/getting-started/intro) describes one common way to expose such tools in a consistent format, which helps when you later want to change the model or the host application.

Vague tools are dangerous for a specific reason. A more capable model will use an ambiguous tool with more confidence, and therefore with more consequences. Name each tool for the action it performs, limit what it returns to what the task needs, and write one sentence explaining when it should not be used. If a tool cannot be explained in that way, it is not ready for an agent.

Check three: Do permissions match your approval boundaries?

Permissions and approval boundaries are related but not identical. A permission defines what the system will allow. An approval boundary defines what the business wants a person to confirm. Mature companies write both down, action by action: reading, drafting, sending, spending, changing a record and deleting. For each action they name who may approve it and under what conditions.

The security principle of least privilege, described in the [OWASP authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html), is the right starting point. Grant the smallest set of actions that completes the role, and add more only when a documented need appears. The common failure is inheritance: an agent receives broad access for a pilot, the pilot becomes production, and nobody revisits the scope.

Illustrative scenario: a small service company uses an agent to prepare quotes for a sales lead. The agent was approved to draft quotes for a manager to review. After an integration update, the same connection also allowed the agent to send quotes directly to clients. Nobody decided that, but nobody had checked it either. The fix was to separate the draft permission from the send permission and to add the send check to the quarterly review. For the review steps, see Quarterly AI Agent Permission Review: Steps and Evidence.

Check four: Can you see what the agent did and why?

Observability is the ability to reconstruct an outcome after the fact. For an agent, that means a record of the request, the sources consulted, the tools called, the approvals requested and granted, the errors returned and the final output. The [OpenTelemetry observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) explains the general vocabulary of traces, metrics and logs, and the [OpenTelemetry semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/) describe how model calls can be recorded consistently.

The practical test is simple. Pick one decision the agent made last month and ask whether a reviewer who was not involved can understand what happened without asking the engineer who built it. If the answer is no, you cannot evaluate a model change, you cannot investigate an incident, and you cannot explain the agent's behavior to an auditor or a client. Logging sensitive content also needs care, so define what is stored, for how long and who can read it.

Check five: Can you measure whether a change made things better?

The last check is about improvement itself. A mature company keeps a fixed set of realistic tasks with expected outcomes, runs that set before and after any change to the model, prompt, tools or memory, and tracks the result. The [OpenAI evals guide](https://platform.openai.com/docs/guides/evals) and Anthropic's note on [evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) both describe this discipline in practical terms. The test set should include ordinary cases and the awkward ones, such as incomplete records and conflicting instructions.

Measure cost as well as quality. A cheaper model that fails more often may cost more per completed task once review time is counted. Keep one simple measure, such as cost and review minutes per completed task, and compare it over time. Also confirm that you can restore the agent's configuration and memory if something breaks; AI Agent Backup and Restore: What to Save and How to Test It covers what to save.

A practical order for the first pass

You do not need to fix everything before starting. Work through the checks in this order, because each one depends on the one before it.

  • Write the list of memory sources, their owners and their review dates.
  • Describe each tool in one sentence of purpose and one sentence of limits.
  • Map every action to a permission and an approval rule, then remove anything nobody can justify.
  • Turn on a readable record of requests, tool calls, approvals and errors.
  • Build a small test set from real cases and run it before the next change.
  • Repeat the whole list at a fixed interval, and record what changed.

Be Human designs and operates this layer as a managed service. A first engagement defines one useful role, the memory and tools that role needs, its approval boundaries and one observable success criterion. Managed operations then cover monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements, so these five checks are repeated on a schedule instead of being done once and forgotten.

If you want to find the first task worth handing to an agent, take the free Be Human AI diagnostic.

Frequently asked questions

Do we need to pass all five checks before starting with an AI agent? Not necessarily, but you should know which checks you fail and accept the risk explicitly. A narrow, read-only pilot can start with partial memory and limited tools, provided its permissions and logs are in place. Broader actions should wait until the approval boundaries and the test set are written down.

Is superintelligence maturity the same as buying the most advanced model? No. A strong model placed on unowned memory, vague tools and unclear permissions will produce results that are hard to trust and hard to improve. Maturity is about the surrounding layer, which is what allows any model, including a future one, to add value over time.

How often should we repeat these checks? Repeat the permission review and the test set at a fixed interval, and run the full set of checks whenever a model, a major tool or a responsibility changes. Most companies find that the first full pass is the slowest, and later passes are shorter because the artifacts already exist.

Who in the company should own these checks? One named person should own the overall result, usually an operations leader, with clear owners for each memory source and each tool. Technical staff or an implementation partner can run the work, but the business owner must be able to read the evidence and approve changes to permissions and boundaries.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →