Skip to content
be-human.ai
← Back to the journal

Context & memory

Reindexing Agent Memory After an Embedding Model Change

Test, rebuild and compare retrieval after an embedding model change so old agent memories still resolve correctly. Includes test questions and rollback.

Be Human typographic cover with the title: Reindexing Agent Memory After an Embedding Model Change
Conceptual illustration created with AI · Be Human

When an embedding model changes, every stored vector was produced by the old model and no longer shares a coordinate space with vectors from the new one, so a query can return confident but wrong memories. To reindex safely, freeze a set of real questions with known correct sources, re-embed the full corpus into a separate collection, compare old and new results against those sources, and switch traffic only after the comparison passes.

Key takeaways

  • Vectors from different embedding models cannot be compared, so the old index is not a safe fallback for the new query model.
  • Build a golden question set from real requests before you touch the index, with the correct source and version written beside each question.
  • Rebuild into a parallel collection, keep the old one live, and record the model name and vector dimension on every stored record.
  • Compare retrieval by rank and by correct source, not by raw similarity scores, and review stale or mixed-up results by hand.
  • Cut over only on written criteria, and keep the previous index available long enough to roll back.

Why an embedding model change breaks old memories

An embedding model turns text into a list of numbers. Texts that mean similar things land close together in the space that model defines. A different model defines a different space, often with different dimensions and a different notion of closeness. If you embed new questions with the new model and search a collection built by the old one, the distances no longer mean what the search assumes. The system still returns results, which is why the failure is quiet. The agent answers fluently, using a passage that may not be the right one.

The failure is uneven. Short, exact queries often survive because they share many words with their sources. Paraphrases, questions about exceptions, and requests that name a product by an older title are the ones that drop. A demo with three friendly questions will look fine, which is why a reindex needs a test set before it needs a schedule.

The memories themselves are usually not wrong. The facts may still be correct, and what changed is the lookup. Keep the source records, chunking and metadata as they are, and treat the embedding index as a derived layer that can be rebuilt from them.

A reindexing plan you can run in order

  • Step 1: Name the owner of the memory index and the owner of the test set, and write both names in the change record.
  • Step 2: Export the inventory of source documents with their current versions, so the rebuild starts from the records and not from the old vectors.
  • Step 3: Write the golden question set and mark the expected source for each question.
  • Step 4: Run the old index against the golden set and save the results as the baseline.
  • Step 5: Re-embed the corpus with the new model into a parallel collection, with chunk boundaries taken from the source documents.
  • Step 6: Run the same questions against the new collection and save those results beside the baseline.
  • Step 7: Compare the two runs with the checklist below and log every regression with a suspected cause.
  • Step 8: Fix each cause (chunking, metadata filter, query formatting), then rerun the affected questions.
  • Step 9: Apply the cutover criteria, switch traffic in a controlled window, and keep the old collection available for rollback.

Build a golden question set before touching the index

A golden set is a short list of real questions, each with the answer source written down. Pull them from tickets, chat logs and the questions staff ask most often. Include the awkward cases: a question phrased in internal jargon, a question about an exception to a rule, a question that uses an old product name, and at least one question that should return nothing because no source covers it. The last type matters because a new model can produce a confident match where there is no answer.

Keep the set small enough that one reviewer can read it in a single sitting, and broad enough to cover each document family the agent uses. Store it outside the vector store, in a file the memory owner controls, so the test does not depend on the index it is testing. The method is the same one used to test retrieval quality in an AI knowledge base, applied before and after the model change.

Rebuild into a parallel collection, never in place

Write the new index into a new collection with its own name. Record the embedding model identifier, the vector dimension, the chunking version and the source document version on every record. Those fields let you answer later which model produced a given vector, and they make partial failures visible. Rebuild jobs should be idempotent, so rerunning the job for one document replaces its records rather than duplicating them.

Chunk boundaries should come from the source documents, not from the old index. If a policy was split badly the first time, a reindex is the moment to fix it. The guidance on chunk size for policies and procedures explains why keeping sections whole usually matters more than matching an old setting.

The old collection is the rollback path, so it must not be modified while the new one is built.

Compare old and new results side by side

Compare by outcome, not by raw score. Similarity values from two different models are not on the same scale, so a higher number in the new run does not mean a better match. Compare where the expected source appears in the ranking, whether a superseded version appears, and whether the top result belongs to the right customer, product or department.

  • Does the expected source appear in the top results for each question, in both runs?
  • Has the expected source dropped in the ranking, and does it still reach the answer the agent uses?
  • Does any superseded document version now outrank the current one?
  • Does a result from another customer, product line or department appear where it should not?
  • For questions with no correct source, does the new run return a confident match anyway?
  • Are any questions answered before now returning empty results?

Illustrative scenario: a support agent answers questions about a warranty period. In the old run, the current warranty policy ranks first for every warranty question. After the reindex, a previous version of the policy ranks first for questions that use an older product name, because the new model places that wording closer to the old text. The comparison catches it because the expected version is written in the test set. The fix is not a different model. It is a version filter on the retrieval step, plus a metadata flag that marks superseded documents as such. Read each regression by hand, since a single failed question often points to a bad chunk or a missing alias rather than to the model.

Decide the cutover and keep a rollback path

The cutover decision should be written before the comparison starts, so the result cannot be argued into a convenient shape.

  • Switch when the golden set meets the agreed criteria, no regression touches a restricted scope, and superseded versions never outrank current ones.
  • Hold when any question involving money, legal terms or personal data returns a wrong source, even once.
  • Hold when a regression has no known cause, because an unexplained change tends to return with the next model update.
  • Extend the parallel period when staff report answers that the test set did not anticipate.

Keep the old collection and its configuration until the new one has run in production long enough for the owner to review real questions. Then archive it under the same retention rules as other memory backups. Restores should be tested rather than assumed, and the guide to what to copy from agent memory and how often to test restores covers what belongs in that backup.

How a managed approach keeps the index reliable

At Be Human, managed operations treat the embedding index as one component of the agent's memory, not as a one-time import. The golden question set stays with the memory owner, recurring evaluations rerun it after model updates and content changes, and permission reviews confirm that each role still reaches only the collections it should. When a regression appears, incident response records the cause and the fix before anyone changes the production index. The work stays inside the approval boundaries the business defined, so a model change becomes a reviewed change rather than a silent one.

Keep the index honest after cutover

After the switch, the risk moves from the model to the content. New documents arrive, old ones are superseded, and staff correct answers that the agent got wrong. Review the golden set whenever the corpus changes meaningfully, and add every question that staff or customers surfaced in production. Watch for empty results and for answers that cite a source no longer in use. Treat each new failure as a new test case so the next change is checked against it. If you are unsure which agent task deserves this kind of careful testing first, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

Can I keep the old vectors and only embed new documents? No, not safely. New questions would be embedded with the new model while the old records came from the previous one, so the distances between them do not mean what the search assumes. A full re-embedding into a parallel collection is the cleaner path, and the golden set is what shows whether the result is trustworthy.

How many test questions do I need? There is no fixed number. The set should cover every document family the agent uses and every role that asks questions, and it should include exceptions and questions with no correct answer. A set one reviewer can read in one sitting is more useful than a large set nobody maintains.

Should I compare the new model's similarity scores with the old ones? No. Similarity values depend on the model that produced them, so a score from one model cannot be read against a score from another. Compare ranks, whether the expected source appears, and whether superseded or out-of-scope documents show up.

When can I delete the old index? Only after the new one meets the agreed criteria, has run in production for a period the owner accepts, and a rollback has been tested. Keep the source records and the rebuild process as well, since they are what allow the next model change to be handled the same way.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →