Skip to content
be-human.ai
← Back to the journal

AI by industry

AI Agents for Virtual Assistant Agencies: Client Task Memory

Keep each client's preferences, task status and access rules in separate memory so notes never cross accounts. A practical setup for VA agencies.

Be Human typographic cover with the title: AI Agents for Virtual Assistant Agencies: Client Task Memory
Conceptual illustration created with AI · Be Human

An AI agent for a virtual assistant agency works safely only when each client's preferences, task status and access rules live in their own memory, with clear limits on what the agent may read and write. Keep those three layers apart, scope every read to a single client account, and require a person to approve anything that crosses a boundary. This article explains how to set that up before the first agent touches a live client.

Key takeaways

  • Store client preferences, task status and access rules as separate memory, each with its own owner and review date.
  • Scope every agent read and write to one client workspace, enforced by the tool layer rather than by the prompt alone.
  • Treat any copy of one client's facts into another client's draft as a leak, and write a rule that blocks it.
  • Keep a record of what the agent remembers for each client and who approved each permission.
  • Start with one role and one client group, and expand only after the first permission review comes back clean.

Why a shared notebook breaks first

Most virtual assistant agencies run on a shared drive, a client tracker and one message thread per assistant. That setup works while people carry context in their heads and check each other's work. An AI agent changes the risk, because it reads memory on every task and has no instinct that a note was written for someone else. A line about a vendor dispute in one client's file can end up shaping a draft for a different client if the agent is allowed to search the whole folder. The failure is rarely dramatic. It looks like a helpful suggestion that quietly uses the wrong client's facts.

The fix is organizational before it is technical. Decide which kinds of memory the agency keeps, who owns each one, and which tools may read it. Managed providers face the same problem with customer tenants, and the design logic in Superintelligence Prep: Managed Service Provider Memory is a useful reference when you draw these lines.

Three kinds of memory, kept apart

Most of what an assistant agency knows about a client falls into three groups. Each group changes at a different speed and carries a different level of sensitivity, so each needs its own storage and its own read rules.

  • Client preferences: tone, working hours, approved vendors, formatting, contact limits and anything the client has asked you never to do. These change slowly and can be kept as one short profile per client.
  • Task status: open requests, the person who owns each item, deadlines, what has been sent and what is waiting on the client. This changes daily and often names third parties, so it needs the strictest read scope.
  • Access rules: which mailboxes, calendars, drives and software accounts the agent may use for that client, and which actions need a human approval. These belong in permission configuration, not in free-text notes the model can rewrite.

Keeping the groups apart lets the agent use preferences broadly. A general style guide can apply to every client, while task status and access rules must never leave one client's workspace. Anthropic's guidance on context engineering makes a related point: an agent performs better when it receives only the context relevant to the current step. Narrow context is also a privacy control. The same logic applies to crews, routes and bids in Landscaping Routes, Bids and Crew Notes: Memory Before Agents, where the wrong note in the wrong file causes the most damage.

A setup plan for client memory

Work through these steps before the agent runs on a real client account. Each step produces something you can show a reviewer.

  • Create one workspace per client, with its own memory store and its own identifier. Avoid one large table with a client column that the agent can filter freely.
  • Assign each memory type an owner on your side, a sensitivity label and a review date.
  • Map every tool the agent uses to the client workspaces it may touch, and deny cross-workspace calls by default.
  • Issue credentials per client or per role, limited to the smallest set of actions that completes the task.
  • Write one rule for copying material between clients. The default should be that nothing moves without a named person approving it.
  • Log every read and write with the client identifier, the task, the tool used and the time.
  • Run a leak test before go-live and keep the test cases in a folder that anyone reviewing the setup can open.

The agency should be able to answer each step in one or two sentences. If a step cannot be answered, that is the gap to fix before the agent goes live.

Where access rules must be enforced

A common mistake is to write 'never share client details with other clients' in the system prompt and treat that sentence as the control. Prompt instructions help the agent behave well, but they are not an authorization layer. OWASP's guidance on prompt injection notes that instructions hidden in emails, documents and web pages can try to override an agent's own rules. For that reason, any tool that can fetch data must refuse a request for another client's workspace, whatever the model asks for. OWASP's authorization guidance makes the same point in general terms: check permissions on the server for every request, not only in the interface someone happens to be using.

Credentials carry the rest of the boundary. OAuth 2.0 lets a user grant an application specific scopes, and the authorization server can issue access tokens limited to particular actions and accounts. When you review the setup, ask which accounts each token can reach. If the honest answer is all clients, the design is not finished, regardless of how careful the prompt is.

An illustrative scenario and a leak test

Illustrative scenario: an agency supports three clients, a bookkeeping practice, a dental office and an online shop. A message arrives in the dental office workspace asking for the invoice layout the bookkeeping practice uses. With separation in place, the agent reads only the dental workspace, finds no approved invoice template there, and drafts a request asking a staff member to supply the correct one. It does not copy the bookkeeping layout, and the log records that the request was declined as outside scope.

Run this kind of test before go-live and again after every change to tools, credentials or memory fields. Send the agent requests that name another client, refer to a file from another client, or ask about another client's open tasks. A pass means refusal or escalation, with a log entry. Logs that tie each action to a client identifier and a tool call make that review possible. The OpenTelemetry observability primer describes how logs, traces and metrics can be connected so a reviewer can follow one request through several steps.

How Be Human runs client memory in managed operations

Be Human designs and operates this kind of setup as part of personalized AI infrastructure for companies. A first engagement defines the one role the agent will perform, the memory each client workspace holds, the tools it may reach, the actions that require human approval and the observable success criterion that shows the role is working. Ongoing managed operations include monitoring for reads that cross workspace boundaries, recurring evaluations that include leak tests, permission reviews when a client or tool changes, and controlled updates to memory fields. Decisions about what the agent may remember remain with the agency and its clients.

Review the boundaries every month

Client relationships change. Clients leave, add staff, switch vendors and approve new software. Each change can widen what an agent reaches without anyone noticing. Put a short monthly review on the calendar. Remove workspaces for ended clients, revoke tokens nobody uses, confirm that each preference profile still matches what the client said, and rerun the leak test. A review that takes thirty minutes and happens every month is worth more than an exhaustive audit that happens once.

If you are not sure which client workflow should be the first one an agent handles, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

Can one AI agent serve several clients at a virtual assistant agency? Yes, but only when each client has its own memory workspace and every tool call is checked against that workspace. Shared instructions can cover general style and process, while task status and access rules stay limited to one account. Without those boundaries, the agent may use facts from one client while working for another.

Should client notes go into one shared knowledge base for the whole agency? Shared material such as an onboarding checklist or a general style guide can live in one place. Client-specific facts belong in separate records with separate permissions. Mixing the two is the most common way notes end up in the wrong inbox.

Is a system prompt enough to stop an agent from sharing client information? No. A prompt can describe the rule, but instructions in incoming emails or documents can try to override it, so enforcement belongs in the tool layer and in the credentials the agent holds. Treat the prompt as guidance and the authorization check as the control.

How do we know the separation still works after launch? Keep logs that show the client identifier, the tool called and the outcome of each action. Run the leak test on a schedule and after every change to tools or permissions. Review any cross-workspace attempt as an incident, even when the agent refused it.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →