No-Regret AI Investments to Make Before Models Get More Capable

To prepare a company for far more capable AI, invest in the assets that stay valuable whichever model you run: well-owned data, durable memory, narrow and reviewable permissions, and a way to test any agent against your own work. These make a stronger model more useful and a weaker model safer. None of them requires you to predict when, or whether, superintelligent systems arrive.

Key takeaways

• Spend on what you own and can carry between models: data quality, memory, permissions and evaluation sets. Spend little on anything tied to one model's quirks.
• Capability raises the stakes of access. Narrow, logged, revocable permissions matter more as agents get better, not less.
• A saved set of real tasks with known good outcomes is the cheapest way to judge any new model fairly.
• Memory should be a governed business record that a person can read, correct and move, not a hidden feature of one vendor.
• Start with one useful role and one observable success criterion. Breadth can follow once the foundations hold.

Why invest in assets instead of predicting capability?

Forecasts about model capability are uncertain, and leaders who plan around a single forecast tend to be wrong in one of two directions. Some wait for a future system and do nothing. Others buy a heavy platform for a capability that does not arrive on schedule. A no-regret investment avoids both errors because it pays off in every scenario.

Test any proposed spend with three questions. If models improve slowly, does this still help next quarter? If models improve quickly, does this make the new capability safer and easier to use? If you switch providers, does the investment come with you? Data hygiene, memory design, permission discipline and evaluation sets pass all three. A custom prompt tuned to one model's habits usually fails the third.

This is also why vague readiness talk is less useful than a short list of concrete assets. The company superintelligence layer idea makes the same point: the durable part of an AI setup is the layer around the model, not the model itself.

Which data investments keep their value?

More capable models do not repair disorganized information. They read it faster and act on it with more confidence, which can make bad source material more expensive, not less. The data work that pays off is unglamorous.

• Name an owner for each core source: pricing, policies, product specifications, customer history, procedures. An owner decides what is current.
• Mark what is authoritative. When two documents disagree, the agent and the person both need to know which one wins.
• Retire stale material. An outdated policy that still sits in a shared folder is a future wrong answer.
• Classify by sensitivity. Know which records contain personal, financial or contractual information before any agent can reach them.
• Keep sources in formats you control, with export paths, so a change of tool does not strand them.

Retrieval design matters here too. Most business agents work best when they look up current company information at the moment of the task instead of having that knowledge baked into the model. The [AWS comparison of retrieval and fine-tuning](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html) explains the tradeoff, and the practical conclusion for most companies is to keep knowledge in sources you can edit, not in a model you cannot inspect.

What should an agent remember, and where should memory live?

Memory is what turns a capable model into a colleague who knows your business. It is also the asset most likely to be trapped inside a vendor by accident. Decide early what the agent should remember, who can read it and how it moves.

Separate three kinds of memory, because they have different owners and risks:

• Standing context: who the company is, how it speaks, what it never does, the roles and escalation paths. This changes rarely and should be written down by people.
• Working knowledge: facts learned during tasks, such as a customer preference or a supplier exception. This needs review before it becomes permanent.
• Task history: what was asked, what was done, what was approved. This is your audit trail and your raw material for evaluation.

The guidance in [Anthropic's work on context engineering for agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) is useful for the underlying principle: an agent performs best when it is given the right information at the right moment, not everything at once. A stronger model benefits from well-organized memory even more, because it can use nuance that a weaker model would ignore.

Two habits keep memory trustworthy. First, make it readable by a non-technical person, so a manager can see what the agent believes and correct it. Second, back it up and test the restore, so that a failed migration or a model change does not erase months of accumulated context. If memory lives only in a proprietary format, you have built a dependency, not an asset. The article on keeping the gains in-house as models change covers the contract and architecture side of this.

How should permissions be set so a stronger model is not a bigger risk?

A more capable agent can do more with the same access. That is the benefit, and it is the exposure. The no-regret move is to make access narrow now, so that growth in capability is matched by deliberate growth in permission, never by default.

Apply the least-privilege principle from the [OWASP authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) to every agent identity:

• Give each agent its own identity, never a shared personal login.
• Grant access per tool and per action: read, draft, send, change, delete are different permissions.
• Require human approval for actions that spend money, contact customers in new ways, change records of value or are hard to undo.
• Log every action with enough detail to reconstruct what happened and why.
• Make revocation fast and rehearsed. If you cannot switch an agent off in minutes, the permission is too wide.

Treat content the agent reads as untrusted input. A stronger model is not immune to instructions hidden in an email or a web page, and the [OWASP prompt injection guidance](https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html) is a good checklist for separating what the agent reads from what it is allowed to do.

Permissions drift as roles and tools change, so review them on a schedule. The quarterly permission review lays out the steps and the evidence to keep.

How do you evaluate a model you have not met yet?

The most valuable thing you can own before a new model arrives is a way to judge it. Vendor demonstrations show the model at its best. Your own evaluation set shows it on your work.

Build the set from real tasks:

• Collect a modest number of representative requests from the role the agent fills, including the awkward ones: ambiguous requests, missing information, conflicting sources.
• Record what a good outcome looks like in plain language, agreed by the person who owns the work.
• Include cases where the correct behavior is to stop and ask a person.
• Add each real failure to the set once it is fixed, so the same mistake is caught next time.

Grade on outcomes that matter to the business, not on how impressive the answer sounds. The [Anthropic article on evaluating agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) is a practical reference for structuring this. For the business side of defining what counts as success, see how to define success criteria for an AI agent pilot.

Illustrative scenario: a distribution company has an agent that drafts replies to supplier delay notices. When a newer model becomes available, the operations lead does not read announcements. She runs the saved set of past notices through both models and compares the drafts with the approved answers. The new model handles the ambiguous notices better, but it changes the tone for one important supplier type and occasionally promises a delivery date it cannot know. She adopts the new model with a rule added to standing context and a new test case for the date problem. The decision took a short review, not a debate.

What is a sensible order of work?

The foundations depend on each other, so sequence matters. This plan fits a mid-sized company and assumes one agent role to start.

• Step one: choose one role. Define the work, who owns it and one success criterion someone can observe without special tools.
• Step two: inventory the sources that role depends on. Name an owner for each, mark the authoritative version and remove what is stale.
• Step three: classify sensitivity. Decide what the agent may never see and what it may see only for specific tasks.
• Step four: write standing context. Put the company's voice, boundaries, escalation paths and approval rules in a document a manager can read.
• Step five: set permissions per tool and per action. Start with read and draft, add actions only when evidence supports them.
• Step six: build the evaluation set from real past work and run it before launch, then after every change to model, prompt, tool or source.
• Step seven: turn on logging and agree on a review rhythm for incidents, permissions and memory corrections.
• Step eight: test a restore and a model swap while the stakes are low, so the first time you do either is not an emergency.

If the work feels large, remember that most of it is writing things down and assigning owners. The stage gates in proof of concept, pilot or production help decide when to widen scope. The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) offers a neutral vocabulary if you need to explain the governance side to a board or an insurer.

What does Be Human's managed approach do with this?

Be Human designs, implements and operates AI infrastructure around these same assets. A first engagement defines one useful role, the memory and tools it needs, the approval boundaries and an observable success criterion. After that, managed operations can include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. When a new model appears, it is tested against the client's own evaluation set before it replaces anything, and the company keeps control of its memory, data and permissions throughout. Service is remote, in English and Portuguese.

Frequently asked questions

Do we need to wait for superintelligent AI before starting? No. The investments in this article are useful with today's models and become more valuable as capability grows. Waiting mostly delays the slow work of cleaning sources, assigning owners and building evaluation sets.

Which single investment should come first? Start with a written definition of one role and how you will know it is working. Without that, data work and permissions have no target. With it, the other investments become straightforward to prioritize.

Should we choose a model provider now or stay flexible? Choose what works for the role today, but keep memory, data and evaluation sets in formats you control. That way a change of provider is a test and a migration, not a rebuild. Using more than one model can also be reasonable when the tasks differ.

How often should we revisit these foundations? Review permissions and memory on a regular schedule, and rerun your evaluation set whenever a model, tool or major source changes. Revisit the role definition when the work itself changes. A short, steady rhythm is more effective than a large occasional project.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.