AI Infrastructure for Mid-Sized Companies: A Starting Plan

AI infrastructure for mid-sized companies is the set of shared foundations (identity, data access, tools, permissions, monitoring and clear ownership) that lets several teams run AI agents without each building its own stack. The practical starting plan is to define a few narrow roles, build only the shared services those roles need, put light governance in place with named owners, and postpone everything that does not serve a live role. The order of these decisions matters more than the size of the investment.

Key takeaways

• Start with roles, not tools. Two or three narrow, owned roles show you which infrastructure is actually required.
• Share the services that are costly to get wrong later: identity, permissions, logging and approved data access. Leave workflow detail with the teams that own it.
• Governance at this size means named owners, written approval boundaries and a regular review. It does not need a committee.
• Postpone fine-tuning, company-wide builders, customer-facing roles and broad integrations until a live role demonstrates the need.
• Judge each stage by an observable result agreed in advance, and expand only when that result holds.

Why plan before buying a platform?

Most mid-sized companies reach AI the same way. Sales trials an assistant, finance tries a different one, and someone in operations connects a chat model to a spreadsheet using a personal account. Each choice is reasonable on its own. Together they leave open questions that matter: who can see which data, where company information has gone, which tool made a given change, and who is responsible when something goes wrong.

A company of this size sits in an awkward position. It is too large for informal experiments to stay safe, and usually too small to staff a dedicated platform team. A starting plan fills that gap. It gives each team room to work while the company keeps one consistent answer to the questions above.

A plan is also cheaper to change than a platform. If you commit to a broad platform before you know which roles you need, you tend to shape the roles around the platform. If you define roles first, the infrastructure follows the work. For a fuller map of the pieces involved, see the seven layers of company AI infrastructure.

Which decisions come first, and in what order?

The sequence below is deliberately short. Each step produces something you can read, review and change.

• Name a sponsor and an operational owner. The sponsor is an executive who accepts accountability for outcomes. The owner handles day-to-day questions, reviews and requests from teams.
• Choose two or three candidate roles. Favor work that repeats, has clear inputs, tolerates a reviewed draft and draws on data you can legitimately connect.
• Write a one-page description for each role: its purpose, inputs, tools, what it may do alone, when it must stop and ask, and how success will be observed.
• Decide the identity and access model before connecting anything. Each agent should have its own account with the smallest set of permissions its role needs.
• Select the shared services those roles have in common (covered in the next section).
• Set approval boundaries per role, in writing, before the role goes live.
• Run the first role until it meets its agreed criterion, review the record together, and only then start the second.

The last step is the one most often skipped. Starting every role at once feels efficient, but it spreads the owner's attention thin and hides which design choices worked. If you need a way to rank the candidates, a four-factor method for prioritizing AI use cases can help.

There is also a structural question: one agent for the whole company, or one per team. Neither answer is universal, and one company agent or one per team walks through the trade-offs. For a firm with several teams, a common pattern is separate, narrow roles that sit on the same shared foundations, so the teams stay independent while the controls stay consistent.

What should be shared across teams, and what should stay local?

A useful rule: share what is hard to retrofit and risky to duplicate, and keep local what depends on how a particular team works.

Share these:

• Identity and accounts. Each agent acts through its own account, not a colleague's login, so its actions can be told apart and its access can be removed cleanly. The reasoning is laid out in why a company AI agent needs its own identity and accounts.
• Permissions. Authorization checks should follow least-privilege principles, the same discipline described in the OWASP authorization guidance, and they should be reviewed on a schedule.
• Approved data access. Provide one controlled way to connect document stores, the CRM and shared drives, with a named owner for each source.
• Logging and observability. Keep a common record of what each agent did, which tools it called and what happened next. The concepts in the OpenTelemetry observability primer (traces, logs and metrics) translate well to agent work.
• Model access. Reach models through a layer you control, so changing a model does not mean rewriting every role.
• Change and incident handling. One way to update a role, roll it back and respond when something misbehaves.

Keep these local to each team:

• Role instructions, tone and vocabulary.
• The team's definition of a good output.
• Workflow details and handoffs.
• Who on the team reviews and approves the agent's work.

The boundary is not rigid. If three teams keep writing the same kind of instruction, that is a signal to promote it into a shared service. If a shared rule keeps getting worked around, it is probably too tight for the teams that depend on it.

What does governance need to look like at this size?

Governance for a mid-sized company should be small enough that people follow it. Its job is to make responsibility visible, not to slow down useful work. A workable minimum fits on a few pages:

• An inventory of every active role, with its owner, the systems it can reach and the date of its last review.
• Written approval boundaries: what the agent may do alone, what needs a person's approval, and what it must never do.
• Data rules that say which categories of information each role may use, and where that data is stored and processed.
• An acceptable use policy for employees, so people know what they may share with agents and what they may not.
• A review rhythm, with a fixed moment to decide what to keep, fix or retire.

Approval boundaries deserve the most care. A sensible default is that reading and drafting are allowed, sending anything outside the company needs a person's approval, and payments, deletions and changes to permissions are never done by an agent alone. Loosen these only when the record shows a role has earned it.

If you want a shared vocabulary, the NIST AI Risk Management Framework organizes the work into governing, mapping, measuring and managing risk. You do not need to adopt it in full. Borrowing its structure can help a leadership team talk about the same things in the same terms. For a lightweight start, AI governance for mid-sized companies lays out what to put in place first.

What should you postpone?

Postponing is a decision, and it deserves the same care as the things you build. These items are usually safe to defer until a live role gives you a reason:

• Fine-tuning custom models. Clear instructions and well-organized company knowledge usually come first. Revisit tuning only when the record shows a gap that those cannot close.
• A company-wide agent builder open to everyone. It multiplies roles faster than anyone can review them.
• Customer-facing roles. An internal role is a safer place to learn how your approval boundaries hold up.
• Broad integrations connected just in case. Connect a system when a role needs it, and grant the narrowest access that works.
• Multi-model routing. Begin with one model that fits the role, and add others when you can show the first one is limiting.
• Detailed cost optimization. Track basic usage from the start so you understand what each role consumes, but wait for real usage patterns before tuning.
• A large policy library. Write the pages you need now, and add more when a real situation calls for them.

Some things should not be postponed, even though they are less visible: identity, logging, the record of who owns what, and a plain understanding of how you would export your agent's instructions and memory if you changed providers. These are far harder to add after several teams depend on the system.

What does this look like in practice?

Illustrative scenario: a company has an operations team, a finance team and a sales team. Leadership picks the operations team's weekly project status work as the first role. The agent reads the project tracker, drafts the weekly summary and prepares follow-up messages. A coordinator approves everything before it is sent.

Before the agent starts, the company gives it its own account with read access to the tracker and permission to draft only. It sets up a common log and names the coordinator as role owner. The success criterion is observable: the coordinator reviews the drafts and records which ones needed substantial correction.

Only after that review does the finance team begin its role: preparing draft replies to routine invoice questions for a person to approve. It reuses the same account pattern, the same log and the same approval habit. The finance team writes its own instructions and decides who reviews the drafts. The sales team waits, and nobody has to rebuild anything when its turn comes.

The point of the scenario is the reuse. The second role is cheaper to start than the first because the foundations already exist and the owner already knows how to review. If you want to see how a role moves from trial to routine use, proof of concept, pilot or production describes what each stage should prove.

Be Human follows this same sequence. An engagement begins by defining one useful role, the memory and tools it needs, the approval boundaries and an observable success criterion. Be Human then designs and implements the shared pieces for that role (identity, channels, permissions and memory) under the company's control. Managed operations can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. Later roles build on the same foundations. The service is delivered remotely, in English and Portuguese.

Frequently asked questions

Do we need a dedicated AI team before we start? Not for the first role. You need a sponsor, an operational owner and someone on the relevant team who can review the agent's work. A dedicated team becomes worth considering when several roles are live and the review and maintenance work outgrows part-time attention.

How many roles should we start with? Two or three candidates is a good number to evaluate, but run the first one to its success criterion before starting the next. This keeps the owner's attention focused and makes it clear which design choices worked. Adding a role should feel like reuse, not a fresh project.

Should each department choose its own AI tools? Departments should own their role instructions and workflows, but identity, permissions, data access and logging should be shared. If every department chooses its own tools, you end up with several access models and no single record of what the agents did. A shared foundation with local ownership of the work is the usual compromise.

When is it time to add more infrastructure? Add it when a live role shows a specific need: a repeated workaround, a missing integration, or a review that takes too long without better records. Let evidence from running roles drive the next investment, not a general sense that the setup should look more complete.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.