Set a latency budget for your AI agent workflow

A latency budget sets how long a workflow can spend on each stage while still meeting the user's need. Measure the time from request to usable result, not just the model response. Retrieval, tool queues, approval and file delivery can dominate the experience.

Match timing to the task

Imagine two requests: find a customer's latest note and prepare a detailed monthly review. The first may need an interactive response; the second may fit an asynchronous handoff. Treating both as the same chat turn can create either unnecessary waiting or rushed, incomplete work.

Agree on what the user should receive immediately and what can arrive later. A truthful acknowledgment can help, but it must not imply work is complete.

Break down the path

• Intake: identify the request and validate required inputs.
• Retrieval: fetch the relevant authorized sources.
• Reasoning: produce the proposed answer or action.
• Execution: call tools and wait for destination processing.
• Delivery: verify the artifact and make it accessible.

Measure typical and slow cases separately. An average can hide occasional delays that make the workflow unusable during a busy day.

Remove avoidable waiting

Cache stable data only when freshness rules allow it. Avoid repeated lookups for the same record within a task. Run independent reads in parallel when permissions and provider limits permit. Keep dependent writes sequential so speed does not introduce conflicting effects.

Before changing models, inspect where time actually goes. A faster generation step cannot fix a slow export job in another system.

Make timeout behavior useful

At the deadline, preserve completed work and show the remaining step. For an unknown write outcome, reconcile rather than repeat. For a long-running report, provide a durable task reference instead of leaving the user in an endless typing state. Good timing design gives people predictable progress and a clear way to continue.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.