API Rate Limits and AI Agents: Avoid Throttling Surprises

To keep AI agents within API rate limits, put a durable queue between the agent and every provider, pace requests below the published limit, and retry throttled calls with backoff that follows the provider's wait signal. Then add priority rules so urgent work goes first and nothing is silently dropped. The goal is not to avoid throttling entirely. It is to make throttling a planned, visible event instead of a surprise.

Key takeaways

• Agents multiply calls: one business task can trigger many model and tool requests, so limits are reached sooner than a pilot suggests.
• A durable queue with a rate limiter in front of each provider turns a hard failure into a short wait.
• Backoff should obey the provider's own wait signal first, add randomness second, and always have a ceiling.
• Priority lanes, reserved capacity and deadlines decide who waits when capacity is short, and who gets a human instead of silence.
• Measure queue depth, wait time and throttle events, so you see pressure before customers or staff do.

Why do rate limits surprise agents more than ordinary software?

A conventional integration makes a predictable number of calls per action. An agent does not. It reads a request, decides which tool to use, calls it, reads the result, and decides again. That loop (explained in more detail in the agent loop article in this journal) means a single email or ticket can become a dozen model calls and several tool calls, and the count changes with the difficulty of the case.

Three things make this worse in practice:

• Pilots run at low volume, so limits are never touched until real traffic arrives.
• Live work and background work share the same allowance. A nightly batch can starve the customer reply that arrives at the same moment.
• Naive retries add load. When a provider says slow down, an agent that immediately tries again makes the situation worse and can extend the throttling period.

The result is a familiar pattern: the agent works well for weeks, then a busy morning produces partial tasks, duplicated messages or a pile of silent failures. Nothing is wrong with the model. The plumbing around it had no plan for scarcity.

A throttled request is a scheduling problem, not a failure.

Which limits apply to one agent workflow?

Before designing anything, list every limit the workflow can hit. They are rarely a single number, and they rarely share the same scope.

• Request rate: how many calls are allowed per interval, often per key, project or organization.
• Token rate: how much text the provider will process per interval. Long documents and large context consume this faster than request counts suggest.
• Concurrency: how many calls may be in flight at once. Slow, long responses hold a slot for a long time.
• Periodic quotas and spend caps: daily or monthly ceilings that stop everything when reached, often with a different error from short-term throttling.
• Tool limits: the CRM, email service, calendar, accounting system or ERP the agent connects to usually has its own limits, and they are often stricter than the model provider's.
• Shared scope: another team's script, a test environment or a second agent may draw from the same allowance.

Read the provider documentation, but also read what the responses tell you. Many providers return headers or error bodies that state remaining capacity and when it resets. Treat the documented limit as a starting point and the observed behavior as the truth. Also check how your provider counts repeated context: features such as [prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) can reduce the cost of a stable prefix, but how that interacts with your limits depends on the provider's rules.

How does a queue keep work from being dropped?

The core design change is simple: the agent stops calling the provider directly. It submits a job to a queue, and a worker takes jobs from that queue at a pace the limit allows. If capacity is short, the job waits. It is not lost.

A useful queue for agent work has these properties:

• Durable: jobs are stored, so a restart, deployment or crash does not erase pending work.
• One lane per limit scope: a separate limiter for each provider, key and tool, so one slow system does not block the others.
• Rate-aware: a token bucket or similar limiter sets the release pace slightly under the real limit, leaving headroom for bursts and for other users of the same allowance.
• Described: each job carries an identifier, a priority, a deadline, an attempt count and a current state.
• Idempotent: each job has a key so that a retry cannot send the same email or create the same record twice. [Stripe's documentation on idempotent requests](https://docs.stripe.com/api/idempotent_requests) is a clear example of how a provider can define this behavior, and it is worth copying the idea even where a provider does not offer it.

The last point matters most for agents that act in the world. A throttled call that actually succeeded on the provider side, but whose response was lost, will be retried. Without an idempotency key, the customer receives two messages.

The queue also needs backpressure. When the queue grows past a defined depth, the agent should stop accepting new low-priority work and say so, rather than quietly stacking hours of delay. Progress should also be saved between steps, so a long task that is throttled halfway resumes where it stopped instead of starting over. [Anthropic's write-up on harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) describes why this kind of saved state matters.

What backoff rules respect the provider?

Backoff is the waiting rule applied when a call is refused. Good backoff is boring and strict.

• Honor the wait signal first. If the response includes a retry-after value or a reset time, wait at least that long. Do not guess when the provider has told you.
• Without a signal, wait with increasing delays and add random jitter, so many workers do not return at the same instant.
• Set a ceiling on a single delay and a limit on total attempts. After that, the job moves to a defined next state, not an endless loop.
• Classify errors. Throttling (commonly a 429 response) and temporary server errors deserve a retry. Invalid requests, permission errors and content problems do not, and retrying them only repeats the mistake.
• Pause the lane, not just the call. When one request is throttled, other workers using the same limit should slow down too. A simple circuit breaker that pauses the lane for the wait period prevents a wave of doomed requests.

The broader rules for when to retry and when to stop are covered in an AI agent retry policy that avoids repeated mistakes. Rate limits are one case of that policy, and they should share the same vocabulary of attempts, ceilings and final states.

Backoff also spends time. If a workflow has a promised response time, waiting is not free, which is why it helps to set a latency budget for the workflow. The budget tells the queue when a job has waited long enough that a person should take over, rather than letting it age quietly.

Priority rules: who waits when capacity is short?

Without priorities, a queue is first in, first out, and the first in is often a bulk job. Decide the order in advance, in writing, with the people who own the work.

A practical decision guide, from first served to last:

• Live conversations where a person is waiting for an answer.
• Work with a hard deadline today, such as a confirmation, a booking change or a payment follow-up.
• Responses to human decisions, such as an approved draft that is ready to send.
• Routine background tasks, such as classification, summaries and record updates.
• Bulk work, such as backfills, re-indexing and evaluation runs, scheduled for quiet periods.

Three rules keep this fair and safe:

• Reserve capacity. Keep a share of the allowance for the top lane, so bulk work can never use all of it, even when the top lane is quiet.
• Age low-priority jobs. A job that has waited long enough rises one level, so it is never starved indefinitely.
• Expire with a handoff. When a deadline passes, the job does not vanish. It goes to a person with the context attached. The principle in human escalation for AI agent failures applies: send the decision, not the mess.

Illustrative scenario: a distribution company runs an agent that drafts replies to supplier and customer emails during the day and reconciles delivery notes overnight. A large batch of delivery notes arrives late and is submitted as a single burst at the start of the morning, just as staff begin to send questions. Without lanes, the burst consumes the token allowance and email drafts stall. With lanes, the reconciliation is capped to a small share of capacity, the drafts keep their reserved share, and the reconciliation finishes later in the day. Nobody sees an error, and the only visible change is that the reconciliation report arrives after lunch.

A step-by-step plan to add rate-limit control

You can apply this to an existing agent without rebuilding it. Work in this order.

• Step one: inventory every provider and tool the agent calls, and record each limit, its unit and its scope (key, project or organization).
• Step two: trace one real task end to end and count the calls it makes. Do this for a simple case and a hard case, because the difference is your burst risk.
• Step three: put a queue and a limiter in front of each provider, with the release pace set under the real limit.
• Step four: add an idempotency key to every job that changes something outside the agent, such as sending, creating or updating.
• Step five: implement backoff that follows the provider's wait signal, with jitter, a ceiling, an attempt limit and a lane-wide pause.
• Step six: define priority lanes, reserved capacity, aging and the deadline handoff to a person.
• Step seven: add alerts on queue depth, oldest job age and throttle events, and review them weekly.
• Step eight: test with a deliberate burst in a safe environment, and confirm that high-priority work still moves, nothing is duplicated and nothing disappears.

Step eight is the one teams skip. A planned burst test is far cheaper than discovering the behavior on a busy day. Running the new controls first in a mode where a person checks outcomes is also sensible, in the same spirit as running an agent in shadow mode before giving it authority.

What should you watch, and what happens when the provider is down?

Rate limit control is only trustworthy if you can see it working. Log enough to answer four questions: how deep is the queue, how long has the oldest job waited, how often were calls throttled, and which lane was affected. The general principles in what to log for AI agent observability apply, and the [OpenTelemetry observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) explains the underlying ideas of metrics, logs and traces if your team is new to them.

A few signals deserve a standing review:

• A rising oldest-job age with a flat queue depth, which means a lane is stuck.
• Throttle events concentrated at the same time every day, which points to a scheduling collision.
• Jobs reaching the attempt limit, which means the retry rules are too tight or the limit is too low for the workload.

Throttling is also a gentle cousin of an outage. If a provider is degraded for a long period, the same queue should have a documented fallback, whether that is a second model, a reduced service level or a manual path. The business continuity plan for AI agents covers how to prepare for that case.

How Be Human handles rate limits in managed operations

Be Human treats limits as part of the agent's operating context, alongside its tools, permissions and approval boundaries. In a first engagement, the limits of each connected provider and system are recorded when the role and its tools are defined, and the success criterion includes work that must not be dropped. In managed operations, queue and throttle behavior is monitored, recurring evaluations include burst cases, integrations are maintained when providers change their limits, and any controlled improvement to the pacing or priority rules is made with the client's approval.

Frequently asked questions

Should an agent retry immediately after a throttling error? No. Immediate retries add load and often extend the throttling period. Wait at least as long as the provider's wait signal says, add jitter if you have several workers, and pause the whole lane so other requests do not hit the same wall.

Is it better to ask for a higher limit or to build a queue? Do both, but build the queue first. A higher limit raises the ceiling, while a queue gives you control when you reach it, and every limit is eventually reached. Requests for more capacity are also easier to justify when you can show real queue and throttle data.

Can a second provider handle overflow when the first is throttled? It can, but treat it as a deliberate design choice and not a quick fix. A different model can behave differently, so overflow should be limited to task types you have tested on both, and the switch should be visible in the logs. For work where consistency matters, waiting in the queue is usually safer than switching.

How do I test rate limit handling before launch? Send a controlled burst in a non-production environment that mimics your real limits, with a mix of urgent and bulk jobs. Check that urgent work keeps moving, that retries do not create duplicates, that expired jobs reach a person, and that your alerts fire before the queue becomes a problem.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.