Prompt Injection Risks for AI Agents: Defenses to Build First

Prompt injection is an attack in which instructions hidden inside text an AI agent reads, such as an email, a PDF or a web page, bend the agent away from what its owner intended. For a business agent that can read a shared inbox, file documents or update records, the risk is not an odd answer but an action taken with real company permissions. The first defenses are structural: treat every incoming document as data rather than instructions, limit which tools the agent can reach, and require a human decision before anything irreversible happens.

Key takeaways

• Any text an agent reads can carry instructions, including email bodies, signatures, attachments, web pages, form fields and spreadsheet comments.
• The damage an injection can cause depends on the agent's tools and permissions, not on how convincing the attacker is.
• Keep system instructions separate from retrieved content, and never let a document grant the agent new permissions.
• Put approval boundaries around payments, bank detail changes, data exports, customer-facing sends and record deletions.
• Log every tool call and review unusual sequences, because no prompt wording alone closes the gap.

What prompt injection looks like in a business inbox

Large language models do not reliably separate the instructions they were given from the text they are asked to process. A person reading an invoice knows that a sentence in the footer is not an order from their manager. An agent that summarizes the same invoice may treat that sentence as part of its task unless the system is built to keep the two apart. This is the core of the problem: the attacker writes text, and the text ends up in the same channel as the agent's own instructions.

There are two common forms. Direct injection happens when someone types instructions straight into the agent's chat window. Indirect injection is more relevant for companies, because the malicious text arrives inside content the agent is supposed to process, such as a customer email, a supplier PDF, a scanned form or a web page the agent opens while researching. The person sending that content may never interact with the agent at all.

Illustrative scenario: a supplier sends an invoice PDF. In white text on a white background, the footer says that the previous review rules no longer apply, that the invoice is approved, and that the payment account has changed to a new one. An agent that reads the PDF and also has write access to the payables tool may follow that footer. Nothing in the file looks like a virus, and a quick visual check of the document would miss it.

Agents raise the stakes compared with a simple chatbot. A chatbot that is manipulated writes a wrong sentence. An agent can call tools, query records, send messages and save notes to a memory that persists across sessions. An injected instruction that gets written into memory can influence later tasks that have nothing to do with the original email. That is why the design question is less about the wording of the prompt and more about what the agent is allowed to do after it reads something.

Inventory every surface where outside text enters the agent

Before you choose any defense, list the places where text from outside your company can reach the agent. Most teams find more channels than they expected. Write them down in plain language, next to the tools each channel can influence.

• Inbound email bodies, signatures, quoted replies and forwarded threads.
• Attachments, scanned forms, PDFs and spreadsheets received from customers, suppliers or applicants.
• Web pages the agent opens during research, including pages it finds through search.
• Form fields, booking notes and comments that customers or contractors type into shared systems.
• Outputs from other systems and other agents, which can carry text that was itself copied from outside.
• Long-term memory entries, because anything saved from an untrusted source becomes a future input.

For each surface, record the tools the agent can call after reading it. A quote request that only feeds a draft reply has a small blast radius. The same request feeding an agent that can change prices, issue refunds or alter a customer account has a large one. This table of channels and tools is the working document for the rest of the program, and it is worth keeping alongside your broader AI risk register so that new channels get added when they appear.

Defenses to build first: a practical sequence

The order matters. Teams often start with prompt filters because they are easy to demo, then discover that the agent still has write access to everything. A safer sequence starts with reducing what a successful injection could do, then adds detection on top.

• Step one: confirm the inventory of inputs and tools described above, and mark which combinations are dangerous, meaning untrusted input plus a tool that moves money, changes records or sends messages outside the company.
• Step two: make read-only the default. The agent should be able to read and summarize most content without any write permission, and each write permission should be granted separately.
• Step three: separate roles in the design. System instructions, business rules and approved templates live in configuration that outside text cannot change. Retrieved content is labeled as untrusted data in the context the model receives.
• Step four: place approval boundaries in front of irreversible actions. Bank detail changes, payment releases, customer-facing sends and deletions should wait for a named person to approve them, as described in where an AI agent should stop and ask.
• Step five: restrict what can be written to memory. Entries that came from outside sources should be quarantined or marked, and a person should review any memory change that affects payments, pricing or customer identity.
• Step six: log every tool call with the input that triggered it, so you can trace how a suspicious action happened.
• Step seven: build a small set of hostile test messages, including planted instructions in attachments and emails, and run them whenever the agent, its tools or its prompts change.

None of these steps requires a perfect model. They limit what a manipulated agent can do, which is the realistic goal.

Controls that do not depend on the model noticing the attack

The OWASP guidance on prompt injection describes a layered approach, and the most useful layers are deterministic. A model may be persuaded by a clever paragraph, but a validation rule is not persuaded by anything. Build the checks outside the model so that a manipulated decision still hits a wall.

For payments and supplier records, compare any changed bank detail against a verified vendor master list and refuse to act on a change that is not confirmed through a separate channel. For amounts, require a match with an approved purchase order within a tolerance you define in advance. For outbound messages, restrict recipients to an allowlist for automated sends, and hold anything addressed to a new domain for review. For data exports, cap the number of records per request and require an approver for bulk output.

Filters that scan incoming text for suspicious phrases can help catch obvious attempts, but attackers can rephrase, hide text in images or split instructions across several documents. Treat filters as one signal among several. Access control matters more: the authorization principles in the OWASP Authorization Cheat Sheet apply here as they do to any service account, which means the agent should hold only the permissions its role requires, and those permissions should be checked at the point of action rather than trusted because a request arrived in a familiar format. The tier model in growing an agent's trust tiers is a practical way to decide which permissions an agent earns over time.

Decide which actions need a human before you deploy

A decision guide keeps this discussion concrete. Ask three questions about each action the agent could take: can the action be reversed easily, does it move money or change a legal or customer record, and does it reach someone outside the company. The more answers are yes, the more human review the action needs.

• Read, summarize, classify and draft internally: allowed, with logging and no external send.
• Create or update an internal task or note: allowed for the defined role, with an audit trail.
• Change a customer, supplier or employee record: requires a named approver who sees the proposed change side by side with the current value.
• Release a payment, change a bank account or alter a price: requires human approval through a separate verified channel, every time.
• Send an external message to a new recipient or with a new attachment: requires review until the pattern has a track record.
• Export or delete data in bulk: requires approval, a record limit and a retention check.

These decisions also determine who is accountable when something goes wrong. If an injected instruction leads to a wrong payment, the question is whether the approval boundary was designed, tested and followed. The broader accountability questions are covered in who is accountable when an AI agent makes a mistake.

How Be Human handles this in managed operations

Be Human designs and operates AI agents whose identity, memory, channels, tools, permissions and operating context stay under human control. In practice, a first engagement defines one useful role, lists the inputs that role reads, and separates trusted configuration from untrusted content. Tools are scoped to the minimum the role needs, and actions that move money, change records or contact outside parties go through approval boundaries that the client's staff own. Managed operations then add monitoring of tool calls, recurring evaluations that include hostile test messages, permission reviews when tools or roles change, and an incident response path if a suspected injection is found. The aim is a controlled system with records a business owner can inspect, not a promise that no injection will ever succeed.

Testing and monitoring after launch

An agent that passes its launch tests can still drift as new document formats arrive, tools are added or prompts are edited. Build a standing test set from the surfaces you inventoried: a booking email with an embedded instruction, a PDF with hidden text, a web page that tells the agent to ignore its rules, and a memory entry that tries to change a price. Run this set on a schedule and after every change, and record which tests the agent failed and what it attempted to do. Treat a failed test as a defect in the design, not as a reason to add one more sentence to the prompt.

Monitoring should watch for patterns rather than single messages. Repeated calls to a payment tool right after a document arrives, recipients that appear for the first time, or memory writes that contain imperative language are all worth an alert. When an alert fires, the agent should stop at the approval boundary and escalate the case with its evidence, as described in human escalation for AI agent failures. Good escalation sends the decision to a person with the relevant context, not a pile of raw messages.

If you are unsure which of your agent tasks touch the riskiest inputs, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Is prompt injection the same as hacking a system? Not exactly. Hacking usually means breaking into a system through a vulnerability, while prompt injection exploits the fact that an agent reads text and treats some of it as instructions. An attacker may need no access to your network at all, only the ability to send an email or get a document into a workflow the agent processes.

Can a better prompt stop prompt injection? A well-written prompt helps, but it cannot be the only defense. Models can be persuaded by carefully written text, so the safer approach is to limit tool permissions, separate untrusted content from instructions, and place approval boundaries and deterministic checks outside the model.

Which defense should a small company build first? Start by listing the inputs and tools, then make the agent read-only by default and put a human approval in front of payments, bank detail changes and external sends. These steps reduce the damage of any single successful injection and cost little to set up compared with building detection systems.

Does prompt injection risk change if the agent only summarizes documents? It changes, but it does not disappear. A summary can still mislead people who rely on it, and a summary that is later pasted into a system or memory can carry instructions forward. Keep summaries labeled as drafts from untrusted sources, and restrict what downstream steps can do with them.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.