Skip to content
be-human.ai
← Back to the journal

Context & memory

Superintelligence Memory Restore Drill: Prove Your Backup Works

A restore drill shows whether your AI agent's memory can be rebuilt from backup. Learn the scope, steps and checks before an incident.

Be Human typographic cover with the title: Superintelligence Memory Restore Drill: Prove Your Backup Works
Conceptual illustration created with AI · Be Human

A superintelligence memory restore drill is a scheduled test in which your team rebuilds an AI agent's memory from its backup in a separate environment, then checks that the agent answers correctly from the restored material. A backup that has never been restored is an assumption, not a control. Running the drill before an outage, a corrupted import or the departure of the person who maintains the memory tells you whether recovery is possible and how long it takes.

Key takeaways

  • A backup only counts after a restore from it has succeeded in a clean, isolated environment.
  • Test the whole memory set: entity records, document versions, retrieval index, instructions, permissions and the change log.
  • Write the pass criteria before the drill starts, including the questions the agent must answer and the questions it must refuse.
  • Run the drill on a fixed schedule and again after any change to the model, the document system, the tools or the permissions.
  • Keep the results as evidence, and log every failure as a work item with an owner instead of quietly fixing it.

What a restore drill proves, and what it does not

A restore drill proves one narrow thing: that the copies you kept are complete enough to bring the agent back to a usable state in an environment you control. It does not prove that the memory was accurate before the backup was taken, that outdated facts were removed when they should have been, or that the agent's permissions were correct. Those are separate checks. A drill that ignores them can create a false sense of readiness, so the scope should be written down in advance and shared with everyone who will read the result.

Many teams discover during an incident that the backup job has been reporting success for months while producing files nobody can load. A green status from the scheduler only confirms that a job ran. It says nothing about whether its output can rebuild anything. For that reason the drill judges the result, meaning the agent's behavior after restore, rather than the job log.

Decide what the backup must contain

Agent memory has several layers, and each one can fail independently. Restoring the documents without the entity records leaves the agent unable to recognize the customer, job or asset it is working on. Restoring the index without the matching document versions produces answers with citations that no longer point to the right text. A usable backup set normally includes:

  • Entity records for customers, jobs and assets, with their owners and last-reviewed dates
  • Source documents in the exact versions the agent is allowed to cite
  • Retrieval index settings and the name of the embedding model used to build the index
  • Agent instructions, tool definitions and the permission map that says which role sees which memory
  • The write log that shows who added or changed each fact and when

Credentials and secrets should not be copied into the backup as plain values. They should be reissued during restore through your normal credential process, so the drill also confirms that this step works. For a broader view of what to copy and how often to test, see AI agent memory backup: what to copy and how often to test restores.

A step-by-step restore drill

The drill is easiest to repeat when it is written as a runbook that any trained operator can follow. A workable sequence looks like this:

  • Choose the backup set to test and record its identifier, creation time and scope.
  • Provision an isolated environment with no live channel connections, so the restored agent cannot message customers or staff.
  • Restore the instructions and permission map first, then the source documents, then the entity records, and finally the retrieval index.
  • Confirm that the restored index was built from the same document version identifiers recorded in the backup, and that the embedding model matches the one noted in the backup.
  • Run the agreed question set against the restored agent and save every answer together with the sources it cited.
  • Compare the answers with the pass criteria, log each failure as a work item with a named owner, and decide whether the backup is usable.
  • Remove the temporary environment, keep the evidence, and note how long the restore took from start to first correct answer.

The last step matters more than it looks. If the restore takes a whole working day, the business needs to know that before an incident, not during one.

Pass criteria: how to judge the restored agent

Pass criteria should come from real work, not from what is easy to test. Ask the operations lead and the people who use the agent daily which questions they ask most, which answers would cause a problem if wrong, and which requests the agent should refuse. Then translate those into checks. A useful set might require the agent to cite the current version of a policy rather than a superseded one, recognize the correct account owner for a named customer, say that it cannot answer when no approved source exists and route the case to a person, and decline to show records outside the requester's role.

Refusal cases deserve as much attention as correct answers. A restored agent that answers everything has not necessarily recovered well. It may have lost the boundaries that were part of its configuration. The same questions can be reused as regression checks later, and the approach is described in memory test cases for superintelligent agents.

Record the pass criteria with a date and an approver before the drill, and do not change them after seeing results. Changing the bar after a failure makes the drill useless as evidence.

Illustrative scenario: a restore that looked fine

Illustrative scenario: a support agent for a small equipment distributor restores cleanly on the first attempt. The question set passes, and the team marks the drill as successful. Some weeks later a staff member notices that the agent quotes a delivery window that was removed from the policy. The restored index had been built from an older export of the policy folder, and none of the drill questions covered delivery rules. The backup job had worked correctly. The gap was in the checks. The team added a delivery question to the pass criteria and a step that compares document version identifiers before any drill can be marked complete.

The lesson is that a restore can succeed technically and still restore the wrong knowledge. Reviewing what the agent holds before the next drill makes this kind of drift easier to catch, and the approach is covered in AI agent memory audit: how to review what it remembers.

Who owns the drill and how often it runs

Every drill needs one accountable owner, usually the operations leader responsible for the agent, and one technical operator who performs the restore. The business owner approves the pass criteria and receives the result. Frequency should follow risk. An agent that answers customer questions from policy documents needs a drill more often than one that only drafts internal summaries, and any change to the embedding model, the document system, a tool or a permission should trigger a new drill before the change is trusted.

Access during a drill should follow the same authorization rules as production. A restored environment that lets more people see more records than production allows is a new exposure, even though it is a test. The OWASP authorization guidance is a useful reference for checking this, and the NIST AI Risk Management Framework offers a structure for assigning these responsibilities and recording them.

Be Human runs this kind of drill as part of managed operations. For an agent we operate, the team keeps the memory inventory, schedules restores into an isolated environment, agrees the pass criteria with the business owner, logs each failed answer as a controlled change with an approver, and keeps the evidence for review. Permission rules during a drill match production, so a restore never widens who can see what.

If you are not yet sure which agent task would justify this discipline, the free Be Human AI diagnostic can help you find the first task worth handing to an agent.

Frequently asked questions

How often should we run a memory restore drill? There is no universal interval, so the schedule should follow the risk of the agent's work. An agent that answers customers from policy documents should be tested more often than one that only drafts internal notes. Any change to the model, the document system, a tool or a permission should trigger an extra drill before the change is trusted.

Does a restore drill need real production data? Not necessarily, and copying live customer records into a test environment can create its own exposure. Many teams use a representative set of records with personal details removed, provided the structure, the document versions and the permission map match production. The essential requirement is that the drill proves the same kind of recovery the business would need in an incident.

What should we do if the restore fails? Treat the failure as the result the drill was designed to produce. Record the exact step where it broke, the answers that did not pass, and the time spent, then assign each item to an owner with a deadline. Do not mark the backup as usable until a repeat drill passes the same criteria.

Is a restore drill enough to call our agent memory backed up? A drill proves that a specific backup set can rebuild the agent's working state, but it does not prove that the memory was accurate, current or appropriately scoped. Treat the drill as one control alongside regular memory reviews, permission checks and a clear owner for each memory layer. Together these show that the backup is trustworthy for the decisions your team makes.

Your next step

See where an AI agent fits in your company first.

Take the free Be Human AI diagnostic. A few questions about your work show which task to hand to an agent first, what it needs to know and where a person approves.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.

Less busywork. More room for you.

Bring us the work you wish you had time for.

Book a call to try a personalized AI agent free for 7 days ↗Or take the free AI diagnostic →