Do You Need a Vector Database for Company AI Agents?

Not by default. Most company AI agents can start with plain search over organized documents and exact lookups in the systems that already hold the facts. A vector database is worth adding when people ask questions in different words than the documents use, and the way to find out is to test real questions before choosing infrastructure.

Key takeaways

• Many agent questions are lookups (an order status, a contract term, an account owner), and a structured record answers them exactly.
• Plain keyword search with filters handles a lot of document questions when documents are titled, dated and organized.
• A vector index helps when meaning matters more than wording, such as long, messy or multilingual text, but it adds something new to maintain.
• Every retrieval layer needs permissions, freshness, deletion and tests, so adding one is an operating commitment, not only a technical choice.
• Decide from a sample of real questions and a retrieval test, not from an architecture diagram.

What does a vector database actually do?

A vector database stores numeric representations of text, usually called embeddings. When someone asks a question, the question is converted the same way, and the system returns the stored passages that sit closest in meaning. The practical result is that a search for "customer wants their money back" can surface a passage titled "refund exceptions", even though the two share no words.

That is a real capability, and it is the retrieval step in the pattern often called retrieval-augmented generation, described in the [Google Cloud architecture guide](https://docs.cloud.google.com/architecture/rag-capable-gen-ai-app-using-vertex-ai) and the [AWS overview](https://aws.amazon.com/what-is/retrieval-augmented-generation/). But notice what it does not do. It does not know which document is current, who is allowed to see it, or whether the text is correct. It finds similar passages. Everything else still has to come from how you organize and govern the material.

It also helps to separate two questions that usually get merged. One is whether you need semantic retrieval, meaning matching by meaning. The other is whether you need a new, separate system to provide it. Many general-purpose databases and search tools can store embeddings, so the answer to the second question is often no even when the answer to the first is yes.

Which questions are lookups, not searches?

Before thinking about indexes, sort what your agent is actually asked. A surprising number of business questions are not searches at all. They are lookups:

• What is the status of this order?
• Who owns this account?
• Which version of the price list applies to this client?
• When is the next delivery for this site?

The answer to each lives in a field in a system of record: a CRM, an order system, a ticketing tool, a spreadsheet with a clear owner. The right design is a narrow, read-only tool that takes an exact identifier and returns the record. The result is deterministic, it respects the permissions of the source system, and it is always as current as the source.

Putting that same information into a vector index turns an exact question into a similarity guess. The agent may retrieve last month's version of a record, or a neighbor that looks alike, and nothing in the retrieval will flag the mistake. If you have not yet decided where each fact is authoritative, start there, as described in how to give an AI agent a reliable source of truth. A well-defined source of truth removes a large part of the retrieval problem before it begins.

When is plain search enough?

Keyword search, combined with filters, is enough more often than the current fashion suggests. It tends to work well when most of these conditions hold:

• The document set is modest and has a clear owner.
• Documents have descriptive titles, dates and a status such as current or archived.
• People use the same vocabulary as the documents: product names, part numbers, clause names, error codes.
• Exact terms matter, and a near miss is worse than no result.

That last point deserves attention. Identifiers and codes are where keyword search is strongest and where embeddings can blur things. A search for one error code should return documents containing that code, not documents about similar problems.

Here is an illustrative scenario. An operations team at a small distribution company gives an agent a few dozen procedure documents covering receiving, returns and carrier claims. Staff ask it things like "carrier claim form" or "returns procedure for the north warehouse". Keyword search with a filter for current documents answers these reliably. Then someone asks "what do we do when a pallet arrives damaged?" and the right document is titled "Freight exception handling". The search misses it.

The tempting reaction is to add a vector database. The cheaper first move is to fix the vocabulary gap: retitle the document, add a short plain-language summary at the top, and keep a small list of aliases (damaged, broken, exception, claim). If misses like this keep appearing across many documents after that, you have evidence for semantic retrieval. If they stop, you saved yourself a system.

When does a vector index start to pay off?

The signal is a persistent mismatch between how people ask and how the material is written, and it usually shows up in a few situations:

• Large volumes of long-form text written by many authors, such as support history, project notes, reports and contracts.
• Customers or staff who describe the same issue in many different ways.
• Questions in one language answered from documents in another, for example English questions over Portuguese material.
• Requests to find similar past cases, not a specific document.

Even then, the strongest setups are rarely vector-only. A hybrid approach runs keyword and semantic matching together, then filters by date, status, client or team before anything reaches the model. Filters matter because similarity alone will happily return a superseded policy that reads very well.

If you do add semantic retrieval, start with the lightest option that fits: an embedding feature in a database or search tool you already run, before a separate product. Move to a dedicated system only when scale, speed or features you have measured demand it. And whichever you choose, prove it with a test, not an impression. How to test retrieval quality in an AI knowledge base walks through building a question set with known correct sources, which is the evidence you need.

A decision guide you can use this week

Work through these in order, and stop at the first step that solves the problem.

• Collect a sample of real questions the agent should answer. Pull them from email, chat and support requests, not from imagination.
• Mark each as a lookup (answer is a field in a system), a document question (answer is in a text), or a judgment question (answer needs a person).
• For lookups, build a narrow read tool against the system of record. Do not index the data.
• For document questions, organize the documents first: clear titles, dates, a current or archived status, and one named owner each. See how to prepare company documents for an AI agent if the material is messy.
• Try keyword search with filters and an alias list. Run the sample questions and record which ones miss.
• Look at the misses. If they come from vocabulary, fix titles and aliases. If they come from meaning (the right passage exists but shares no words), consider semantic retrieval.
• Add semantic retrieval in the smallest form that works, combined with the same filters, and rerun the same sample.
• Keep the layer only if it fixes misses the simpler setup could not, and write down who maintains it.

The point of the order is that each step is cheaper to undo than the next one. A retitled document costs minutes. A new retrieval system costs ongoing attention.

What does each retrieval layer cost to operate?

A vector index is a second copy of your documents, transformed into a form people cannot read. Copies drift, and that creates obligations that are easy to overlook when the demo works.

Permissions come first. If staff can see only some documents in the source, the index must enforce the same boundaries at retrieval time, not rely on the model to withhold what it was shown. For companies serving several clients, this is where leaks happen, and the client data isolation checklist covers it in practical terms.

Freshness comes next. When a document changes, its embeddings must be rebuilt, and when the embedding model changes, much of the index may need to be rebuilt as well. An index that lags behind the source will confidently serve old answers.

Deletion is the third. If a document is withdrawn or a person asks for their data to be removed, it must disappear from the index and from any cached passages too. How to delete AI agent memory explains the difference between correcting, expiring and removing, and the index needs to follow all three.

Finally, there is evaluation. Retrieval quality changes as documents and questions change, so it needs recurring checks, not a one-time launch test. Anthropic's guidance on [context engineering for agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) makes a related point: the goal is the smallest set of high-signal information in front of the model, not the largest. A retrieval layer that returns too much, or the wrong things, can hurt more than a simpler one that returns less.

How does a managed setup handle this decision?

When Be Human scopes an agent role, the retrieval choice follows the questions the role must answer. We list those questions with the team, mark which are lookups against a system of record and which are document questions, and start with the lightest retrieval that answers them, adding semantic search only where tests show a gap. Permissions, document owners and approval boundaries are defined before any index is built. In managed operations, recurring retrieval evaluations and knowledge reviews show whether the choice still holds as documents and questions change, so the setup can be simplified or extended deliberately.

Frequently asked questions

Is a vector database the same thing as retrieval-augmented generation? No. Retrieval-augmented generation is a pattern in which the agent looks up relevant material and uses it to answer. A vector database is one way to do the lookup, and keyword search or direct record lookups can serve the same pattern.

How large does my document set need to be before I need a vector database? Size matters less than the gap between how people ask and how documents are written. A small set with inconsistent vocabulary may benefit sooner than a large, well-titled one. Test a sample of real questions on simple search first, and let the misses tell you.

Will a vector database reduce wrong answers from my agent? Not by itself. It retrieves passages that look relevant, but it does not check whether they are current, correct or permitted. Wrong answers usually fall when sources are clear, owners are named and retrieval is tested, which is why those steps come first.

Can a vector database serve as my agent's memory? It can store material the agent retrieves later, but memory involves decisions about what to keep, who may see it and when to forget it. Those rules matter more than the storage technology. Treat the index as one possible mechanism, not as the memory policy itself.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.