Skip to content
be-human.ai
← Back to the journal

Personalized AI infrastructure

What a Superintelligence Company Accumulates Over Time: Five Assets

Five assets a company AI layer builds over time: clean memory, test fixtures, permission maps, runbooks and eval sets, and how to keep each one owned.

Be Human typographic cover with the title: What a Superintelligence Company Accumulates Over Time: Five Assets
Conceptual illustration created with AI · Be Human

A company layer of AI agents compounds when it accumulates reusable assets, not when it collects more prompts. The five assets that matter most are clean memory, test fixtures, permission maps, runbooks and evaluation sets. Each one makes the next agent faster to scope, safer to run and easier to check, and the value stays with the company only when it owns them outright.

Key takeaways

  • Compounding comes from artifacts the company can reuse: memory, fixtures, permission maps, runbooks and eval sets.
  • Clean memory has named sources, owners and review dates, so an agent does not answer from stale or unapproved records.
  • Fixtures and eval sets turn 'it seems better' into a repeatable check that runs before a change reaches production.
  • Permission maps and runbooks define what an agent may do, when it must ask a person, and how it is stopped.
  • Ownership means the company holds the files, configurations and exports, with a named internal owner for each asset.

Why assets compound where prompts do not

A prompt is written for one task and usually lives inside one conversation or one tool setting. When the process changes, the prompt goes stale, and nobody remembers why a sentence was phrased the way it was. An asset is different: it is a file, record or configuration that another person or agent can read, test and reuse. Each new agent that starts from these artifacts begins further ahead than the one before it.

Model improvements from outside your company help, but they arrive on the vendor's schedule and apply to everyone. Your assets are the part that only you can build, because they encode your policies, your customers, your exceptions and the decisions your managers already made. Anthropic's engineering guidance on context engineering describes context as a finite resource that has to be curated rather than stuffed, which gives a useful test for every asset here: does it help the agent decide, or does it add noise?

Clean memory: what the agent is allowed to know

Memory is the set of facts an agent can retrieve when it works. Clean memory means every fact has a source document or system of record, a named owner, a date of last review and a rule for when it expires. A folder of exported chat logs is not memory in this sense. It is a pile that a retrieval step will search, and whatever it returns first will shape the answer.

The practical steps are short: - Name the source system for each category of knowledge, such as the policy document, the price file owner or the client record system. - Record who approves changes to each category and how a change is made. - Set a review date for every category and remove or flag anything that cannot be verified. - Separate personal and client data from general operating knowledge, and limit access by role.

Illustrative scenario: a regional service company keeps two versions of its warranty terms, one from a supplier and one from its own policy. The agent answered from whichever file the search returned first, so answers varied by wording. Cleaning memory meant naming the approved version, marking the other as superseded and recording who signs off on warranty language. The agent's answers became checkable against one document, and staff could see which source each answer used. For the sequence that prepares records like these before any agent touches them, see data foundations for superintelligence.

Test fixtures and eval sets: how you prove a change worked

A fixture is a fixed set of realistic inputs, often with the expected result or an acceptable range. An eval set is the collection of fixtures plus the scoring method, run after every change to prompts, tools, memory or permissions. Without them, improvement is a feeling, and a feeling cannot be handed to the next person.

Agents are harder to test than fixed software because the same request can take different paths. Anthropic's guidance on evals for AI agents recommends grading outcomes and running more than one trial, since a single successful run can hide inconsistency. Good fixtures also include uncomfortable cases: an incomplete request, two records that disagree, a request outside the agent's scope, and an instruction that tries to override a rule. Those cases are where failures appear, and they are the cases a new employee will eventually meet.

Build the first set from real work rather than invented examples: - Collect requests from the last few months that the team handled well, and mark the correct outcome for each. - Add the requests that caused rework, escalations or complaints, with the outcome that should have happened. - Write the scoring rule in plain language before anyone runs the agent, so the result is not argued afterward. - Store the set in a location the company controls and version it like any other business file.

Be Human's managed approach treats fixtures and eval sets as part of the first engagement and runs them again during recurring evaluations, so a configuration change is compared against the same cases rather than judged from memory.

Permission maps and runbooks: what the agent may do and when it stops

A permission map lists each action an agent can take, the system it touches, the data it can read or write, and the approval required before it acts. It is the company's answer to the question of what happens if something goes wrong at two in the afternoon. Keep it as a plain document that a manager can read in ten minutes, not a set of settings that only an administrator understands. OWASP's authorization guidance favors denying access by default and checking permissions on each request, and that principle belongs in the map as well as in the software.

A runbook is the companion document: one page per role, stating the purpose, the normal path, the conditions that require a human, and the stop rules that pause the agent. Stop rules matter most. They should name concrete triggers, such as a refund above a set threshold, a customer message that mentions legal action, or a tool that returns errors twice in a row. The one-page format is described in the runbook template guide, and the review cadence for the permission map is covered in the quarterly permission review guide.

Keeping the five assets owned by the company

Ownership is the part most companies skip, and it decides whether the assets compound for you or for someone else. An asset is owned when the company can read it, copy it, change it and move it without asking permission from the party that built it. A memory store that only the vendor can export is not owned, even if the answers look good.

Check these points before you accept any build: - Every asset has a named internal owner who can approve changes. - Files, configurations and eval cases are stored where the company can read them. - At least two people inside the company hold administrative access to the accounts that host them. - A permission review is scheduled, and its evidence is kept with the map. - Memory, runbooks and eval sets can be exported in a readable format on request. - The company knows what would happen to each asset if the working relationship ended.

Observability belongs in this list too. The OpenTelemetry observability primer explains logs, traces and metrics as the signals that let teams reconstruct what a system did. For agents, the equivalent record shows which memory was retrieved, which tool was called and where a human approved or stopped an action. Without that record, an incident review becomes guesswork and the lessons stay with whoever was on call. The broader question of dependence on one provider is treated in how to avoid AI vendor lock-in.

In Be Human's managed approach, memory categories, fixtures, eval sets, permission maps and runbooks are documented as part of the operating layer, and changes to permissions follow a written request and approval path. The aim is that the company can see and hold each artifact, and that monitoring, evaluations and incident response add to the same record over time rather than starting over.

Where to start with the five assets

Do not build all five at once. Start with the one role you intend to hand to an agent first, then build only the assets that role needs. A customer support role needs clean policy memory, a fixture set of real tickets, a permission map for the systems it reads and writes, and a runbook with stop rules. An internal reporting role needs fewer of them but still needs a named owner for every source.

A workable order for most teams: - Choose one role with a single measurable outcome, using the guidance in choosing your first agent. - Clean the memory that role will search, and name an owner for each source. - Write fixtures from real past work, including the difficult cases. - Draft the permission map and runbook, then review them with the person who carries the consequences. - Run the evaluation before launch and keep the results with the other artifacts.

If you are not sure which task deserves the first agent or which of these assets it would need, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

Does a company need all five assets before its first agent? No. Start with the assets required by one role and one measurable outcome. The others can be added as the agent's scope grows, provided each new asset has an owner and a review date from the start.

What is the difference between memory and chat history? Chat history records what was said, while memory is the curated set of facts an agent may retrieve, each with a source and an owner. Raw logs can feed a review, but they should not be searched directly as if they were approved knowledge. Clean memory is smaller and more deliberate.

Who should own the eval set? The business owner of the outcome should own it, with operations or technical staff maintaining the mechanics. The owner decides what a correct result looks like, which is a business judgment, and the technical team keeps the runs and storage working. Ownership should be written down so a change in staff does not leave the set orphaned.

Can these assets move if we change providers? They can, if they are stored in readable formats the company controls and documented well enough for another team to rebuild the configuration. Assets kept only inside one vendor's console are hard to move. Ask for exports and document the dependencies before you commit, not after you need to leave.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →