When an AI Model Is Retired: How to Prepare Your Agents

AI model deprecation is a business event, not only a technical one. The provider announces that a model version will stop working on a set date, and every agent built on it needs a tested replacement before then. Prepare by listing where each model is used, building a test set from real work, choosing a primary and a standby replacement, and telling the people who rely on the agent what will change. Done early, a retirement becomes a scheduled change instead of an outage.

Key takeaways

• A retirement notice is a deadline, not a starting gun. Begin the inventory and the test set the day the notice arrives.
• Know every place a model is used, including tools and vendor products that embed one without telling you loudly.
• Judge a replacement on your own tasks, using a test set built from real cases, not on the provider's description of the successor.
• Choose a primary replacement and a standby, and write down why, so the decision survives staff changes.
• Tell users what will change, what will not, and how to report anything odd. Quiet changes erode trust faster than announced ones.

What does a retirement notice change, and where is the model used?

A retirement notice says a specific model version will stop answering requests after a stated date. Sometimes the provider names a successor, sometimes only a model family. Either way, your agent's behavior is tied to that version: its tone, how closely it follows instructions, how it formats output, when it decides to call a tool, and when it declines to act.

A newer model is usually capable, but capable is not the same as identical. Instructions tuned for one model can produce longer answers, different refusals or different tool choices on another. That is why the work is a controlled change and not a configuration edit.

The business risks are concrete:

• Quality shifts quietly, and nobody notices until a customer or a finance colleague does.
• The agent stops working on the retirement date because nobody owned the notice.
• Speed and cost per completed task change, which affects service levels and budgets.
• Data handling terms differ between models or providers, which matters if you move.

Start with an inventory. For every agent, scheduled job, internal script and vendor tool, record the following:

• The model identifier it calls, and whether it is pinned to a version or points to an alias such as a latest label.
• Where that setting lives, and who can change it.
• The task it performs, and whether the output is customer-facing or triggers actions in other systems.
• The person who owns the workflow and the person who would notice a quality change first.
• How a failure would show up: an error, a wrong answer, a slow reply or a silent skip.

Look for hidden usage. A no-code automation, a plugin or a vendor product may call a model on your behalf. Ask those vendors directly which model they use and what they plan to do about the notice. Pinned versions stop on the retirement date. Aliases may switch on their own, which is more convenient and also riskier, because the change arrives without your review.

How do you build a test set that shows whether a replacement is safe?

A test set is a collection of real situations the agent must handle, each with a clear definition of a good outcome. It is the single most useful thing you can prepare, because it turns the question of whether the new model is fine into something you can check. The method is covered in more depth in Build a golden test set for your AI workflow, and the following is the short version for a retirement.

Draw cases from the agent's actual history. With permission and with personal data redacted, pick examples that cover:

• Typical requests that make up most of the daily work.
• Awkward but legitimate requests, such as missing information, mixed languages or contradictory instructions.
• Cases where the right behavior is to stop and escalate to a person.
• Cases where the agent must refuse or must not reveal something.
• Tool-using cases, where the correct action and its arguments matter, not only the wording.
• Past mistakes. Every error the agent has made and you fixed deserves a permanent place in the set.

For each case, write the input, the context the agent would have, and the expected outcome expressed as checks, not as exact wording. Two good answers rarely read the same, so test for what matters: the right facts were used, the right tool was called with the right values, the escalation happened when it should, the format is valid, the tone fits your guidelines, and nothing was disclosed that should not be.

Run the set on the current model first and keep the results as a baseline. Without a baseline you cannot tell whether a failure is new or was always there. Have a person review the first scoring pass, since a rubric that looks clear on paper often needs adjustment once real outputs appear. After that you can automate the checks that are reliable and leave judgment calls to a reviewer. The evaluation guides from OpenAI and Anthropic listed in the sources are good references for structuring this. For the wider question of measuring agents on actual work, see How to evaluate AI agents on real business tasks in this journal.

One practical habit makes all of this easier: log the model version with every agent run. If your records show which version produced which output, comparing before and after is a query, not an archaeology project. The OpenTelemetry conventions for generative AI describe a common way to name such fields.

How do you choose a fallback model?

Choose on evidence from your test set and on operating constraints, not on reputation. A model that tops a public ranking can still be worse for your invoice-matching task than a smaller one. Use this decision guide in order:

• Start with the provider's named successor. It is the path with the least change, so test it first and adjust instructions if needed.
• If the successor passes but is slower or more expensive than the task requires, test a smaller or faster tier of the same family. Judge on cost per completed task, not on cost per message.
• If the successor fails on cases that matter and instruction changes do not fix it, test a model from another provider. Accept that this adds migration work: tool formats, rate limits, data terms and logging all need a look. The steps are in How to Migrate AI Agents to a New Provider, Step by Step.
• If the task is narrow and stable, ask whether it needs a model at all. Some steps are better as fixed rules, and removing a model dependency removes the retirement risk for that step.
• Keep a standby. Name a second model that has also passed the test set, so an outage or a further notice does not force a rushed choice. The thinking behind this is in Business Continuity Plan for AI Agents: Prepare for Outages.

When comparing candidates, check more than answer quality:

• Reliability of tool calls and structured output.
• The amount of context the task needs and how the model handles long inputs.
• Response time against what the workflow can tolerate.
• Regional availability and the provider's terms on retaining or training on your data.
• Behavior on the escalation and refusal cases, which is where small differences hurt most.

Write the decision down in one page: candidates tested, results, the choice, the standby and the reasons. When someone asks months later why the agent runs on this model, the page answers.

What is a practical preparation plan from notice to cutover?

This routine works for most business agents. Adjust the depth to the risk of the workflow: an internal summarizer needs less ceremony than an agent that drafts customer commitments.

• Record the notice: the model, the retirement date, the named successor if any, and one accountable owner.
• Complete the inventory described above and confirm nothing was missed by asking each vendor and each team that builds automations.
• Freeze a baseline by running the test set on the current model and saving the results.
• Run each candidate on the test set with the existing instructions unchanged. This shows the raw difference.
• Adjust instructions, tool descriptions or context, changing one thing at a time, and rerun. Keep notes on what helped.
• Sort the remaining failures with the business owner into three groups: acceptable, fixable, and blocking. Do not proceed while blocking failures remain.
• Run the candidate side by side on live work without giving it authority. It drafts, a person compares. This is the same idea as shadow mode used before any agent is trusted.
• Decide using the framework in AI Model Update Regression Testing: Adopt, Adjust or Roll Back, then schedule the cutover well before the retirement date.
• Switch the lowest-risk workflows first, keep a way back to the old model while it is still available, and keep approval rules and permissions exactly as they were.
• Watch the first period closely: corrections by reviewers, escalations, tool errors, response time. Compare against the baseline.
• Update the runbook with the new model, the test results, the standby and the date of the next review.

Two cautions. First, do not combine the model change with other changes, such as new tools or new permissions. If something goes wrong you need to know the cause. Second, leave margin. Replacement models can need more than one round of adjustment, and discovering a blocking problem close to the retirement date leaves only bad options.

What should you tell users and staff, and when?

People who work with the agent should hear about the change from you, before they notice it. Silence is the usual cause of lost trust: a reviewer sees slightly different drafts, assumes something is broken and either stops using the agent or stops checking it.

Tell these groups:

• Daily users and reviewers, who need to know what might look different and how to flag it.
• Managers who depend on the workflow, who need the schedule and the rollback plan.
• Customers, only if the agent is customer-facing and the change affects what they experience. Most of the time a short note is unnecessary if quality holds, but a support team should be ready for questions.
• Whoever handles legal, security or data protection, if the replacement involves a different provider or different data terms.

A short message works best, with these parts:

• What is changing, and the date.
• What is not changing: the agent's role, its permissions and the approval rules.
• What you may notice, such as different phrasing or length, during the first period.
• How to report something odd, and to whom.
• Who is accountable for the change.

Illustrative scenario: a distributor uses an agent that drafts replies about order status for its customer service team. The retirement notice arrives, and the team builds a test set from past tickets, including cases where the agent should hand the conversation to a person. The named successor passes the factual checks but writes much longer replies and sometimes offers delivery estimates that the order system does not support. The team tightens the instructions, adds a check on the estimate field and reruns the set. Before cutover, the service team is told that drafts may read a little differently and that any reply with an unsupported promise should be tagged for review. Because the escalation cases were part of the set, the team goes live with confidence that the handoff rules still hold. For setups like this, Human escalation for AI agents: send the decision, not the mess is worth reading alongside.

How does Be Human handle model retirements?

In a managed engagement, Be Human keeps a record of which model each agent uses as part of its operating notes, and monitors provider notices so a retirement is caught on arrival. When one appears, the team runs the agent's existing test set against candidate replacements, adjusts instructions and tools where that helps, and presents the results with a recommendation for the client's approval. The change is made in a controlled way with a path back, the client's approval boundaries and permissions stay as defined, and the operating notes are updated afterward. The client's side of the work is deciding on the recommendation and telling the people who use the agent.

Frequently asked questions

How early should we start preparing for a model retirement? Start the day the notice arrives, and treat the retirement date as the last possible day, not the planning date. The inventory and test set take the longest, and any adjustment rounds need time. If you already keep a test set and a model inventory, the preparation is much shorter.

Do we need to retest if the provider says the successor is a drop-in replacement? Yes. A provider can fairly describe a successor as compatible at the interface level, but your agent depends on behavior, such as tone, instruction following and tool choices. A short run of your own test set is the only way to know whether your workflow is affected.

What if we use an alias that always points to the latest model? An alias removes the deadline pressure but also removes your chance to review each change, so quality can shift without warning. For anything that affects customers or systems, pin a specific version, test new versions deliberately and move the pin yourself. For low-risk internal uses, an alias can be acceptable if you monitor outputs.

Can we keep using the old model after the retirement date? Plan as if you cannot. Providers set their own terms, and requests to retired models normally fail. If a provider offers an extension, treat it as extra margin for testing, not as a reason to postpone the work.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.