Org Design for Superintelligent AI: The Roles Humans Keep

Organization design for superintelligent AI means deciding in advance which decisions, outcomes and relationships stay with named people while agents take on more of the work. The roles that remain human are accountability, judgment on contested trade-offs, trust-based relationships, authority over agent boundaries, and stewardship of company context. Define those first, and the rest of the structure can change as models improve.

Key takeaways

• Draw the organization around outcomes and decisions, not tasks, because agents will keep absorbing tasks.
• Every agent needs a named human who answers for its results, and a separate person who controls what it may access and do.
• Keep judgment and relationships human at the moments that carry consequence: commitments, disputes, exceptions and turning points.
• Sort each decision into keep, share or delegate with a simple guide, and revisit the sorting on a schedule as trust is earned.
• You do not need to predict future model capability. You need clear owners, approval boundaries and evidence that they work.

Why design roles before the capability arrives?

Most organization charts describe who performs tasks. As agents take over drafting, reconciling, researching, scheduling and monitoring, that way of drawing the chart stops working. A person who used to produce a weekly report may now own only one thing: whether the report can be trusted. If the chart does not say so, nobody owns it.

Waiting for a more capable model does not remove the problem. It enlarges it. A more capable agent can act across more systems, faster, and its mistakes can be harder to spot. That raises the value of clear human roles rather than lowering it.

The practical consequence is that you do not have to forecast what future systems will do. You have to decide what your company will not hand over without a named person attached. That is a design choice you can make now, and it holds up whichever way the technology moves. For the wider preparation work around it, see how to prepare your company for superintelligent AI today.

Which roles stay human, and why?

Five roles are worth naming explicitly, because each one tends to disappear quietly when work is automated.

• Accountability owner: a named person who answers for an outcome to customers, regulators, colleagues and the board. An agent can perform a task. It cannot be held to account for it.
• Judgment holder: the person who decides when the right answer depends on values, precedent or trade-offs rather than facts. Pricing exceptions, hiring choices and how to treat a long-standing customer in difficulty belong here.
• Relationship lead: the person the other party trusts. Customers, suppliers, partners and staff often need to know that someone who can be held to a promise is on the other end, especially in disputes, renewals and bad news.
• Boundary setter: the person who decides what each agent may read, write, send and spend, and who can narrow that at any time.
• Context steward: the person who keeps the policies, exceptions, definitions and source documents the agent relies on correct. An agent working from stale context fails confidently.

A sixth function runs across all of them: challenge. Someone has to sample the agent's work, disagree with it and report what they find. Without that, the other roles run on assumption.

Notice that none of these roles is about doing the work faster. They exist to keep the work answerable. That is why they grow in importance as agent capability grows, and why they are worth staffing deliberately rather than inheriting by accident.

What does accountability look like when an agent does the work?

A useful test: if an agent's output causes harm at the worst possible moment, whose phone rings, and what are they allowed to do about it? If the answer is unclear, or if it is three different people, the design is unfinished.

Three jobs are often merged into one and should be separated on paper.

• Outcome owner: answers for the result of the process the agent works in, such as accurate invoices or timely customer notices. This is usually an existing operations or functional leader.
• Agent operator: watches how the agent behaves day to day, reviews exceptions and raises problems early.
• Boundary owner: controls permissions and approval rules, and signs off any change to them.

Keeping the outcome owner and the boundary owner apart matters. The person who benefits most from speed should not be the only one able to widen the agent's access. In a small company one person may wear two hats. Write that down, and name a deputy so an absence does not leave a gap.

External references help here. The NIST AI Risk Management Framework and the OECD AI Principles both treat accountability and human oversight as central to trustworthy AI, which gives you vocabulary that boards, auditors and insurers already recognise. The OWASP authorization guidance adds a practical rule, least privilege with deny by default, and it applies to agents exactly as it applies to a new employee. At the top, someone also has to sponsor the whole effort, and the executive sponsor role has its own requirements.

How do you decide what to keep, share or delegate?

Not every decision needs a person, and a person on every decision defeats the purpose. Work through these questions in order for each decision an agent might touch. The first yes sets the rule.

• Does it commit money, a legal obligation or a promise to a customer beyond a preset limit? Keep: a human approves before it happens.
• Does it affect someone's livelihood, health, safety or standing? Keep: a human decides, and the agent prepares the file.
• Does the right answer depend on company values or precedent rather than facts? Keep with the judgment holder.
• Is the relationship at a turning point, such as a complaint, renewal, negotiation or bad news? Share: the relationship lead leads, and the agent supplies the briefing.
• Is the action hard to reverse, or does it rely on text from an outside party that could be misleading? Share: the agent proposes and a human confirms.
• Is it reversible, low in value and well covered by clear rules? Delegate: the agent acts and logs, and a human samples the results.

Anthropic's guidance on building effective agents recommends starting with the simplest solution that works and adding complexity only when it is needed. Apply the same habit to delegation. Start a decision in share, move it to delegate as evidence accumulates, and move it back when the evidence turns.

Illustrative scenario: a regional equipment distributor uses an agent that watches inbound shipments and drafts customer notices when a delay appears. The head of customer operations is the outcome owner for timely communication. Routine notices are delegated: they are reversible and low in consequence, so the agent sends them under a set rule and logs each one. When a delay hits a long-standing customer with a contract deadline, the rule routes the case to the account lead. That person calls the customer, decides whether to offer a substitution and asks finance for a credit if one is warranted. The agent prepares the briefing: order history, alternatives, open commitments. The coordinators who once chased status updates now review exceptions each morning, and they tell the context steward when a rule misses a case. Nobody's job is shipment tracking any more, and nobody is unsure who answers for the customer.

How do you redraw roles step by step?

Start with one function rather than the whole company. A function where an agent is already in use or planned gives you real decisions to sort.

• Step one: list the outcomes and decisions in that function, not the tasks.
• Step two: mark each decision keep, share or delegate using the guide above, and write the reason in one sentence.
• Step three: name the outcome owner, agent operator and boundary owner for each agent, plus a deputy for each.
• Step four: write the approval boundaries. State what the agent may do alone, what needs a person, and what is never allowed.
• Step five: define one observable success criterion and one observable warning sign per agent, such as exceptions that keep rising or reviewers who have stopped rejecting anything.
• Step six: set a review rhythm. Permissions drift, and a quarterly permission review keeps them honest.
• Step seven: route changes through a visible process, so that widening an agent's authority is a decision with a name on it. How to request and approve changes to a managed AI agent covers the mechanics.

Two cautions are worth stating. First, do not turn displaced staff into rubber stamps. A reviewer who approves everything adds delay without adding control. Give reviewers time, context and real permission to reject, and check whether they ever do. Second, protect the career paths of the people who used to do the tasks. They often make the best context stewards and challengers, because they know where the exceptions hide.

How does Be Human's managed approach handle this?

Be Human designs, implements and operates AI agents whose identity, memory, channels, tools and permissions stay under human control. A first engagement defines one useful role, the memory and tools it needs, the approval boundaries and an observable success criterion, so the human roles described here are written down before the agent starts work. Managed operations can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. Your named owners keep the decisions: what the agent may do, which changes are approved and who answers for the result. The service is remote, in English and Portuguese.

Frequently asked questions

Will agents eventually take over management roles too? Agents will likely take on more coordination, reporting and routine decision work that managers do today. What stays tied to people is accountability, setting direction, and handling conflict and trust. Treat any claim of full replacement as a forecast, and design a structure that still holds if the forecast is wrong.

Do small companies need all these roles as separate people? No. In a small company one person can hold several hats, such as outcome owner and agent operator. What matters is that each role is written down, that the person who controls permissions is not only the person who wants speed where that can be avoided, and that a deputy exists for each role.

How do we keep humans in the loop without slowing everything down? Reserve approvals for decisions that are costly, hard to reverse or relationship-defining, and let the agent handle the rest with logging and sampling. Then watch the approvals themselves. If reviewers approve everything, or reject nearly everything, the boundary is in the wrong place and the rule needs to change.

Who should own the design of these roles? A senior leader with authority across the functions involved, supported by operations and whoever handles risk or compliance. Sponsors set direction, but day-to-day ownership belongs to functional leaders. An outside implementation partner can help draft the structure, but the names in the roles must be people inside your company.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.