Enterprise Architecture for AI Agents: Where Each Piece Sits

Enterprise architecture for AI agents is the practice of placing each agent component (the model, the harness that runs it, its memory, its tools, its permissions and its observability) inside the systems a company already operates. The agent connects to existing identity, data and business systems through defined interfaces instead of becoming a parallel stack that copies them. For most companies the work is mapping and setting boundaries, not rebuilding.

Key takeaways

• Treat the model as a replaceable component behind a stable interface, not as the architecture itself.
• Place the harness, memory and tools at the edges of systems you already run, and give each one a named owner.
• Take permissions from your identity provider and systems of record, never from a copy held inside the agent.
• Make observability record what the agent saw, did and was allowed to do, in logs your company controls.
• Start by placing one role on the map, and extend scope only when that role's evidence holds.

Start with the map you already have

Most companies already run an architecture: an identity provider, a document store, a customer or ticketing system, a finance system, a data warehouse, an integration layer, and people who approve changes. Before you name any agent component, list these systems and their owners. Then, for each fact an agent would need, write down the system that is the source of truth. This list is more useful than a product comparison, because every later decision (what the agent can read, what it can write, where its logs go) depends on it.

A one-page version of this map helps leaders see the whole picture at once. The method is described in Draw Your AI Stack: A One-Page Architecture Map for Leaders, and it works well as the first working session with the people who own those systems.

Where the model, the harness and the memory sit

The model is the inference service that produces language and decisions. The harness is the software around it: it loads instructions, calls the model, runs the loop of steps, enforces limits such as maximum steps and timeouts, and handles failures. Anthropic's engineering guidance on long-running agents describes this scaffolding as the part that keeps work coherent across sessions. In a company, the harness should run in your own runtime, next to the integration layer, not only inside a vendor's chat window. That placement lets you change the model later without rewriting the business logic.

Memory needs more care than most teams expect. Separate three things. The first is the context supplied for one task, which should be as small as the task requires. The second is retrieval, where the agent searches company documents at the moment of a question instead of holding them permanently. Retrieval-augmented generation is the common pattern here, and it should query your existing document store under its existing access rules. The third is persistent memory, such as notes the agent keeps between sessions. Persistent memory is a data store, so it needs an owner, a retention rule and a deletion procedure. If you do not write those down, memory quietly becomes an unmanaged database.

Tools and permissions: thin tools with inherited grants

Tools are the functions the harness exposes to the model: read an order status, draft a reply, create a purchase request for approval. Each tool should be narrow, with a clear name, a small set of inputs and a single purpose. Wrap an existing API rather than exposing a whole database. The Model Context Protocol offers one standard way to describe tools and resources to a model, which reduces custom glue code, but it does not decide who is allowed to use them. That decision stays with your permission model.

The agent should appear in your identity provider as its own non-human identity, with scoped roles and, where possible, grants that expire. OWASP's authorization guidance points toward checking access on every request and denying by default. Each tool should map to one permission in the source system, and every write action should name the human who approves it. Content the agent reads from emails, documents or web pages is data, not instructions, and the OWASP guidance on prompt injection prevention explains why that separation has to be enforced in the architecture, not only requested in the prompt.

• Give the agent its own identity, never a shared employee login.
• Map each tool to one existing permission in the source system.
• Keep read tools and write tools separate, and require approval for every write tool.
• Deny by default, and record an owner and an expiry date for each grant.
• Treat text from external sources as data, and keep it out of the instruction path.

Illustrative scenario: a distributor with a few dozen staff wants an agent to answer order status questions. The agent gets one read tool against the existing order system that returns status fields only, and one drafting tool that prepares a reply for a staff member to send. No tool writes to the order system in the first phase. When the team later wants the agent to update delivery dates, that change is a new tool with its own approval rule, reviewed before it goes live.

Observability: record what the agent saw and did

Observability answers the question every operations leader eventually asks: why did the agent do that? The general concepts in OpenTelemetry (traces across steps, metrics over time, and logs with context) apply directly to agents. Each run should have an identifier. The log should show the model version, the tools called with summarized arguments, the approvals requested and granted, the errors and retries, and the final outcome. Store these logs in your company's observability stack, so an audit does not depend on a vendor dashboard you cannot export.

Logs alone do not prove quality. Pair them with a fixed set of test cases that represent real requests, and run them after each change. The OpenAI evals guide and Anthropic's notes on evaluating agents both describe this pattern: define expected behavior, score the results, and repeat the same checks when the model, prompt or tools change. Minimize sensitive content in logs. You need enough detail to explain a decision, not a copy of every customer record the agent touched.

A placement plan you can run this quarter

The architecture becomes concrete when it is written as a sequence. The steps below are a practical order for a first placement.

• Inventory the existing systems, their owners, and the source of truth for each fact the agent needs.
• Name the one role the agent will fill, and list the decisions it may make without asking anyone.
• Place each component: the model behind an interface, the harness in your runtime, retrieval against existing stores, tools as thin wrappers, and identity from the provider.
• Write the approval boundary for every write action, and name the person who grants it.
• Define the observable success criterion, and the logs and test cases that will show whether it is met.
• Run the agent with a small group of users, review logs on a fixed schedule, and extend scope only through a recorded change.

Each step produces a document that a future auditor, engineer or new manager can read. That is the real output of enterprise architecture work at this stage.

How Be Human handles this in practice

Be Human designs, implements and operates personalized AI infrastructure for companies, with agents that keep their identity, memory, channels, tools, permissions and operating context under human control. A first engagement defines one useful role, the memory and tools it needs, its approval boundaries and an observable success criterion, then places those components inside the systems the company already runs. Managed operations can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements, and changes to scope are handled through a documented request so the placement stays visible to the people who own it. The service is remote and available in English and Portuguese.

Choosing the first task to place

The best first placement is usually a repetitive, well-documented task with a clear owner, a limited set of systems and a result that someone can check in a week. Look for work where the source of truth is clear and mistakes are easy to catch before they reach a customer. If you want help identifying that first task, the free Be Human AI diagnostic is a practical place to start.

Frequently asked questions

Does an AI agent need a separate architecture from the rest of the company? Usually it does not. A well-placed agent uses the identity provider, document stores, business systems and logging that already exist, and adds a small number of new components such as the harness, retrieval and tool wrappers. Building a parallel stack tends to duplicate permissions and create data copies that are hard to govern.

Should the model run inside our own cloud or with the vendor? The answer depends on data sensitivity, contract terms and your internal capacity, and it should be decided per workload rather than once for the whole company. Keep the model behind an interface so the choice can change without rewriting business logic. Document where each workload runs and which data it receives.

Who owns an agent's permissions after launch? A named business owner should own the role and its approval boundaries, while the technical team maintains the tool wrappers and identity configuration. Permissions should be reviewed on a fixed schedule, and every grant should have an owner and an expiry date. Without that review, access tends to accumulate beyond what the role needs.

What should we log to prove an agent behaved correctly? Record a run identifier, the model version, the tools called with summarized arguments, the approvals requested and granted, errors, and the final outcome. Pair those logs with a fixed set of test cases that reflect real requests, and rerun them after every change. Keep sensitive content out of logs unless it is needed to explain a decision.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.