How to Improve an Existing AI Setup: A Step-by-Step Audit

To improve an existing AI setup, stop rewriting prompts and audit in a fixed order. Define the job and what done looks like, fix the context and sources the AI reads, tighten tools and permissions, then test against real examples and change one thing at a time. Inconsistent results usually come from unclear tasks, stale information and loose access, not from the model itself.

Key takeaways

• Inconsistent output is a symptom. Find which layer causes it (task, context, tools, permissions or testing) before changing anything.
• Write down the job, the inputs and the evidence of done for each AI use. Without that, nobody can say whether a result is right.
• Most quality problems trace back to outdated, duplicated or conflicting sources, so name one source of truth per fact.
• Reduce access to what the task needs, and put a human approval step before any action that is hard to undo.
• Keep a small set of real examples and re-run it after every change, so you know whether the change helped.

Why does an AI setup that worked at first become inconsistent?

Most companies start with a few tools and a few enthusiastic people. One person writes a good prompt, another pastes in a document, a third connects the assistant to the shared drive. It works well enough, and then it spreads. Months later the same question gets different answers depending on who asks, which tool they use and which files happened to be in reach.

The pattern is common because small setups grow without a design. Instructions live in personal chat histories. Documents get edited but old copies remain. Tools are connected with whatever access was easiest to grant. Nobody owns the question of whether the whole thing still does what the business needs.

That is why the fix is an audit rather than a better prompt. A prompt tweak can hide a problem for a week and then fail somewhere else. A short, ordered review shows you which of five layers is responsible: the task definition, the context, the tools, the permissions or the testing. For a fuller picture of how those layers relate, see the seven layers of company AI infrastructure.

Before you start, list every place AI is used in the company, who uses it, and what it is supposed to produce. Keep the list short and honest. Include the unofficial uses, because those are often where the inconsistency lives.

Step one: define the job and the evidence of done

Pick one AI use from your list, preferably one people already rely on, and write its task definition in plain language. A useful definition answers five questions:

• What triggers the work, and who asks for it?
• What inputs does the AI receive, and where do they come from?
• What should the output look like, and who receives it?
• What must the AI never do on its own?
• What evidence shows the work is done correctly?

The last question matters most. If the answer is that it looks good, you have found your first problem. A result that cannot be checked cannot be improved. Evidence might be a cited source, a record updated with a visible change, or a draft that a named person approved. The guide on writing an AI task contract your team can test shows how to turn these answers into something specific.

Expect this step to expose disagreement. Two people may describe the same assistant as a research helper and a drafting tool. Settle it now. An AI use with two purposes will produce results that satisfy neither.

When the definition is written, compare it with the instructions the AI currently has. If the instructions are long, contradictory or full of patches added after each failure, simplify them to match the definition. Short, ordered instructions are easier to maintain and easier to test.

Step two: audit the context, memory and sources

Once the job is clear, look at what the AI actually reads when it does the work. Anthropic's engineering guidance on [context engineering for agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) makes a point worth keeping in mind: the model can only work with what is placed in front of it, so the quality of that material sets the ceiling on the answer.

Run this audit as a checklist:

• List every document, folder, database or note the AI can draw from for this task.
• Mark which ones are current, which are outdated and which duplicate each other.
• For each important fact (a policy, a price list rule, a client status), name the single place where it is authoritative.
• Remove or archive what is stale, and label drafts so they are not mistaken for final versions.
• Check what the AI remembers between sessions, and whether those memories are still true.

The naming exercise is the one most teams skip, and it explains a great deal of inconsistency. If a policy exists in three versions, the AI will quote whichever it finds first. Giving each fact one home is the core idea behind a reliable source of truth for AI agents. Old information that lingers is a separate failure, covered in how to stop stale context from misleading your agent.

Also check the other direction: what is missing. If people keep correcting the AI on the same point, the answer is probably not in anything it can read. Add the missing material in a clean, short form rather than pasting whole documents.

Finally, treat anything the AI reads from outside, such as inbound emails or web pages, as untrusted. The OWASP guidance on [prompt injection prevention](https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html) explains why instructions hidden in content can steer a model, which is one more reason to keep sources controlled.

Steps three and four: what can the AI touch, and who approves it?

The third step is to review the tools and connections. List each one the AI can use: email, calendar, CRM, file storage, messaging. For each, write down whether the AI can only read, can prepare a draft, or can send or change something. These are three different permissions, and many setups quietly grant the strongest one because it was the default.

Then apply a simple rule: give the smallest access that lets the task work. If the AI summarizes client emails, it does not need to send them. If it proposes CRM updates, it does not need to write them directly. The principle is described in the OWASP [authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html), and for agents specifically in least privilege for AI agent permissions.

The fourth step is to place approval boundaries. Use this short decision guide for each action the AI can take:

• Easy to undo and low impact: let the AI act, and log what it did.
• Easy to undo but visible to clients or colleagues: the AI drafts, a person sends.
• Hard to undo, financial or legal: the AI prepares the case, a named person decides.
• Unclear who is affected: remove the permission until someone can answer.

Also review how connections fail. Expired logins, changed fields and renamed folders cause silent errors, where the AI carries on with partial information and sounds confident. Decide in advance what it should do when a tool is unavailable: stop, say so and ask. That is far better than guessing.

Step five: test with real examples, then change one thing at a time

Now you can measure. Collect a small set of real cases for the job you defined, including a few awkward ones: missing information, contradictory sources, a request that should be refused. For each, write what a correct result looks like. Keep the set small enough that someone will actually run it. The article on building a golden test set covers how to choose the cases.

Run the set before you change anything, so you have a baseline. Then make one change, for example cleaning one source or narrowing one permission, and run it again. If you change five things at once, you will not know which one helped, and you may not notice which one hurt. Anthropic's article on [evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) describes this discipline in more depth.

Pair the tests with basic visibility in daily use. You should be able to see what the AI was asked, what it read, which tools it called and what it produced, so that a bad result can be traced to a cause. Without a record, every failure becomes an argument about what probably happened.

Here is an illustrative scenario. A services firm uses an assistant to draft replies to client questions about contract terms. Answers vary between staff. The audit finds three versions of the terms document in the shared drive, instructions that mention two different escalation rules, and an email connection that lets the assistant send directly. The firm names one current terms document and archives the others, rewrites the instructions to a single rule, and switches the assistant to drafting only, with a named person approving. A set of real client questions is run before and after each change. The firm can now see which change improved the answers and which only changed the tone.

After the first job is stable, repeat the sequence for the next one. Do not try to audit everything at once. If you plan to extend the setup, a phased rollout plan helps you decide what comes next.

How does Be Human's managed approach handle this?

Be Human designs, implements and operates AI infrastructure for companies, and an audit like this is a typical starting point. A first engagement defines one useful role, the memory and tools it needs, the approval boundaries and a success criterion that can be observed. Managed operations can then include monitoring, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements. The agents keep their identity, memory, channels, tools and permissions under human control, so the audit is repeated as a routine rather than a one-time rescue. Service is remote, in English and Portuguese.

Frequently asked questions

How long should an audit of an existing AI setup take? It depends on how many AI uses you have, but you do not need to review everything at once. Start with one job that matters, run the five steps on it, and expect the first pass to take a focused stretch of work rather than a long project. Then repeat for the next job.

Should we switch to a different model or tool first? Usually not. A different model will read the same outdated documents and follow the same unclear instructions, so the same problems return in a new form. Audit the task, context, access and tests first, and consider a change only if the evidence points to a real capability gap.

Who should own the AI setup inside the company? Give each AI use a named owner who can say what it is for, approve changes and decide when it is allowed to act. That person does not need to be technical, but they do need the authority to retire old sources and narrow permissions. Shared ownership usually means nobody owns it.

How do we know the improvements are real? Compare results from the same set of real examples before and after each change, and have a person who knows the work judge them against the evidence of done you defined. If you cannot show the difference on those cases, treat the improvement as unproven. Keep the record so the next change starts from a known baseline.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.