Skip to content
be-human.ai
← Back to the journal

Personalized AI infrastructure

Choosing Your First Superintelligent Agent: One Role, One Metric

A decision test for picking your first superintelligent agent: one role, the memory it needs, and one observable metric you can check every month.

Be Human typographic cover with the title: Choosing Your First Superintelligent Agent: One Role, One Metric
Conceptual illustration created with AI · Be Human

The first superintelligent agent for a company should perform one role that a named person can describe in a sentence, draw on memory the company already controls, and report one success metric that someone can check without trusting the vendor. If a proposal fails any of those three tests, it is a demonstration rather than a first agent. The test below fits into one planning meeting and produces a decision you can defend.

Key takeaways

  • Choose one role with a named business owner, a clear input and a clear output.
  • Give the agent only the memory that role needs, and record where each piece comes from and who can change it.
  • Pick one observable metric measured from records you already keep, with a baseline taken before the agent runs.
  • Write approval boundaries before the first run, so people decide in advance what the agent may send, change or spend.
  • Treat the first agent as a test of your operating layer, and keep its scope small enough that a person can review every change.

Why one role is the right unit of decision

Many first AI projects stall at the same point: the mandate is broad. A phrase like being the operations assistant sounds useful, but nobody can say what it reads, what it may write, who checks its work, or whether it is getting better. A role is narrower. It has a job title in plain words, a list of inputs, a list of outputs, and a person who accepts or rejects what comes out.

One role also limits the cost of a mistake. If one agent handles booking confirmations for one team, a wrong answer is visible, reversible and easy to trace. If one agent handles pricing, contracts and complaints at the same time, the same error can spread across several commitments before anyone notices. The first agent should be the smallest useful unit, not the largest imaginable one.

A broader ambition still matters, but it belongs in the roadmap rather than in the first scope. The myths that derail company AI plans often begin with a first project that tries to do everything at once.

The five-question role test

Run each candidate role through the same questions. A role that fails two or more should wait for a later stage.

  • Does a named person already do this work, and would they accept the agent's output as a draft they review?
  • Does the work recur often enough that one metric would show movement within a normal operating cycle?
  • Are the inputs in systems the company already owns and can export, such as a shared inbox, a booking calendar or a document folder?
  • Can a wrong output be caught before it reaches a customer, a payment, a signed document or a regulated record?
  • If the role underperforms in a contained way, would the company learn something specific about its own process?

Roles that usually fail this test include company-wide strategy, pricing decisions, and any task where an error becomes irreversible before a person reviews it. These may be real future uses. They are poor first choices because the failure is expensive and the success criterion is vague. A role about internal triage, document preparation or scheduling usually passes, because the output is easy to inspect and the process already has an owner.

Decide what memory the role needs, and nothing more

Memory is the part of the agent that makes its answers specific to your company: your policies, past decisions, customer history and current status. It is also the part most likely to be over-provisioned. Give the first role a short list of memory sources, and for each one write down where it lives, who owns it, and whether the agent may only read it or may also write to it.

Retrieval is the usual starting point. The agent looks up current documents when it needs them, so the company can update or remove a document without retraining anything. The AWS guidance comparing retrieval and fine-tuning explains the trade-off clearly, and it is worth reading before anyone proposes changing a model's weights for a first role.

Be realistic about the security side too. Any text the agent reads from outside, such as a customer email or a supplier attachment, may contain instructions written to steer it. The OWASP guidance on prompt injection prevention describes why such inputs must be handled as data rather than commands. For a first role, the practical rule is simple: the agent never gains a new permission because a document told it to.

A memory register for the first role can be a single table kept by the business owner, with these fields:

  • Source name and location, such as a policy folder or a nightly export from the booking system
  • Owner, meaning the person who answers when the content is wrong
  • Review rule, meaning how often the content is checked and by whom
  • Access level, either read only or read and write
  • Exclusions, meaning the categories the agent must not use, such as personal data the role does not need

If a memory source cannot be assigned an owner, it is not ready for the agent, and the project should fix that first.

The memory an agent holds is a set of decisions about what it may know, and each one should have a name attached.

Choose one metric you can observe

A metric is observable when someone outside the project can read it from a record the company already keeps, without asking the agent or the vendor to summarize it. Customers are happier is not observable. The share of scheduling replies sent without a staff edit, read from the sent folder, is observable. The difference is the whole point of the test.

Use this sequence to settle the metric before the first run:

  • Write the outcome in one sentence that a staff member would recognize as their own goal.
  • Name the record where the outcome appears, and the person who can access it.
  • Measure the current process for a short, fixed period before the agent runs, so you have a baseline to compare against.
  • Define what counts as a pass and what counts as a failure, including the edge cases the team already argues about.
  • Track the agent's own activity, such as the number of drafts it produced, separately, and do not count it as success.

The last step matters most. Activity measures show that something happened. Outcome measures show whether it helped. Evaluation practice for agents, as described in OpenAI's guide to evals and Anthropic's notes on evaluating AI agents, relies on a fixed set of test cases scored repeatedly rather than on impressions from a demo. A small company can apply the same discipline by keeping a set of past cases from the real process and rerunning them after every change to the prompt, the memory or the tools.

Set the approval boundary before the first run

Approval boundaries describe what the agent may do alone, what requires a person to approve, and what it must never do. Write them before the first run. After a good demonstration, the temptation is to loosen them quickly, and that is the moment they are most likely to drift. Raise them slowly, once the metric shows the role behaving predictably. The guide to approval thresholds for superintelligent agents covers how to change them with evidence rather than enthusiasm.

Illustrative scenario, not a client case: a services company wants an agent to draft replies to scheduling requests that arrive in a shared inbox. The agent may read the booking calendar and the policy folder, and it may create drafts. It may send a reply only when the request matches a standard appointment type and contains no question about price, cancellation terms or a complaint. Anything else goes to the coordinator with a short summary. The success metric is the number of drafts a coordinator must rewrite before sending, reviewed each week against the baseline taken before the pilot started.

In that scenario, the boundary is easy to check because it is written down. A one-page runbook with stop rules, such as the AI agent runbook template, lets a reviewer see in one place what the role may do, what it must escalate and when the agent should be paused.

How Be Human handles the first role

Be Human's managed approach begins with the same decision. A first engagement defines one useful role, the memory and tools it needs, the approval boundaries, and an observable success criterion. Managed operations can then include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements, so the metric keeps being checked after launch rather than only at the demonstration. Service is remote and available in English and Portuguese.

To find the first task worth handing to an agent in your company, take the free Be Human AI diagnostic.

Frequently asked questions

Should the first agent talk directly to customers? Usually not in the first role, because customer-facing errors become visible immediately and make the approval boundary harder to keep narrow. Start with internal drafting or triage, where a person reviews output before anything leaves the company. Move toward direct contact only after the metric has been stable and the boundary has been reviewed.

How many metrics should we track for the first agent? Track one primary metric that decides whether the role continues, plus a few guardrail checks that catch harm, such as errors reaching customers or access to sources outside the agreed list. The guardrails are what trigger a pause, while the primary metric is what justifies keeping the role. Adding many more measures usually turns the review into a debate about which chart matters.

Do we need a data project before we can choose a role? You need enough reasonably current records to measure a baseline and to confirm each memory source, but you do not need a company-wide data program first. Pick a process whose records already exist and whose owner can explain them. If the records are messy, the first project can include a small cleanup of that one process.

What if the metric improves but the team does not trust the agent? Treat that as a finding rather than a failure of the metric. Ask what people are checking that the metric does not capture, and add that check to the review. Adoption depends heavily on how the work is introduced, which is covered in the guide on getting your team to actually use an AI agent, and the approval boundary should stay unchanged until the trust gap is understood.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →