AI Usage Alerts and Hard Stops: Catch a Looping Agent Early

An AI usage anomaly alert is a rule that compares an agent's activity, such as tool calls, tokens or runs per hour, with the pattern that workflow normally shows, and notifies a named owner when that pattern breaks. A hard stop is the second layer: a limit that pauses the agent automatically when usage crosses a higher line, so a looping run cannot keep spending until someone reads a monthly invoice. Set both per workflow, and measure them per completed task, not only as a total for the account.

Key takeaways

• Alerts tell a person something is wrong. Hard stops limit the damage while that person is found.
• A monthly invoice shows the total, not the loop, so the signals have to be read in minutes or hours.
• Build thresholds from a normal week of runs for each workflow, then alert on deviation from that baseline.
• Give every alert a named owner, a response window and a documented pause action.
• Test the guardrail with a deliberately looping run before you rely on it.

Why a monthly invoice review arrives too late

A monthly invoice is a summary of everything that happened, averaged across days in which the same agent behaved normally. A looping agent does not appear as a clear spike in that summary unless the loop runs for a long time, and by then the spend has already happened. The loop itself is usually simple: a tool returns an error, the agent treats the error as a reason to try again, and the retry produces the same error with the same arguments. Each pass costs a model call and often a tool call, and nothing in the agent's ordinary output says it is stuck.

The practical lesson is that cost is a lagging signal. Useful guardrails read behavior that changes first: runs that do not finish, repeated calls with identical inputs, a sudden rise in runs per hour for one workflow. You want to know within minutes that a workflow is doing something it did not do yesterday, and you want the system to stop itself when the pattern is clearly wrong.

Which signals are worth watching?

Start with a short list of signals per workflow rather than a dashboard of everything. Good candidates are:

• Runs started per hour, compared with the normal peak for that workflow
• Tool calls per run, especially calls to the same endpoint
• Repeated calls with identical arguments within one run
• Model calls or tokens per run, which show whether context or steps are growing without limit
• Runs that end without a completed outcome, such as a timeout or a retry limit
• Cost per completed task, which ties spend to useful output

The last signal matters most to the owner. An agent can look busy and still complete fewer tasks, and cost per completed task reveals that before the raw total does. For a fuller treatment of this measure, see Measure AI agent cost per completed task, not per message. Unit economics guidance from the FinOps Foundation explains why a per-unit measure is more informative than a raw total. Telemetry should follow a consistent model as well: the OpenTelemetry primer describes how traces, metrics and logs fit together, which is the foundation for connecting a run to its spend.

How do you set thresholds without guessing?

Thresholds invented in a meeting tend to be either too sensitive, which trains people to ignore alerts, or too loose, which lets a loop run for an hour. The reliable method is to read normal behavior first. Use this sequence for each workflow:

• Pull one normal week of runs, excluding known incidents and planned launches
• Record the typical level and the high end of each signal: runs per hour, tool calls per run, cost per completed task
• Set the alert line above the high end by a margin the owner can explain in one sentence
• Set the hard stop line well above the alert line, so it triggers only on a run that is clearly out of pattern
• Add a rule for identical repeated calls, which needs no baseline because a healthy run rarely repeats the same action with the same inputs
• Review the thresholds after two weeks, since the first alerts show whether the lines are too tight or too loose

Write each threshold down with its reason. If nobody can say why a line sits where it does, nobody will know whether to change it after a busy season or a model update.

Two stages: alert the owner, then pause the workflow

Separate the alert from the stop. The alert goes to a channel that a named person reads, and it should carry enough context to act without opening three tools: the workflow, the signal that fired, the run identifier, the last successful step and the current spend for that workflow. The hard stop pauses new runs for that workflow. In-flight runs can finish or stop at a checkpoint, depending on how the workflow was built, and new work should queue for a person or wait for the owner's release.

A short decision guide helps the owner respond without guessing:

• Alert fires once and clears within the window: log it and review it at the next check
• Alert fires repeatedly on the same run: treat it as a loop and inspect the run's last five actions
• Hard stop fires: confirm the pause took effect, check for duplicate side effects, then decide whether to resume, fix or roll back
• Alerts fire on several workflows at once: suspect a shared dependency such as a tool, a provider or a credential, and follow the outage plan

Duplicate side effects are the reason to check before resuming. A retried write can create a second record or a second message, so pair this stage with idempotent action design. The pattern is covered in Idempotent AI Agent Actions: How to Prevent Double Refunds. Loop-specific stop conditions are covered in AI Agent Stuck in a Loop: Stop Conditions That Actually Work.

How do you keep the guardrail from failing silently?

A guardrail is software too, and it can fail quietly. Three failure modes deserve a plan. The metrics pipeline can stop receiving data, which makes a running agent look idle. The alert channel can be unreachable, so the message goes nowhere. The counter can reset when a service restarts, so a loop appears to start over at zero.

Design for those cases explicitly. Treat missing telemetry for a running workflow as a warning that pauses new starts, not as silence. Send the alert to a second channel if the first one is not acknowledged within the response window. Store counters outside the process that can restart, so the spend view survives a deployment. The NIST AI Risk Management Framework is a useful reference for treating monitoring as an ongoing risk control rather than a one-time setup. For what to record on each run so these checks can be audited later, see AI agent observability: what should you actually log?.

Illustrative scenario: a triage agent that keeps retrying

This is an illustrative scenario, not a client case. A support triage agent reads incoming tickets and fetches order status from an internal service. One afternoon that service begins returning timeouts. The agent, following its instruction to retry failed lookups, repeats the same request with the same order identifier. On a normal ticket it makes two or three lookups. In this run it makes dozens within a few minutes, cost per completed task climbs, and the number of closed tickets stays flat.

The identical-call rule fires after the third repeat, and the alert reaches the support operations owner with the ticket identifier and the failing endpoint. Shortly after, the runs-per-hour rule crosses the hard stop line and the workflow pauses new triage, so tickets queue for a person. The owner confirms that the timeouts are the cause, waits for the service to recover, and checks whether any customer received a duplicate reply. Before the workflow returns to production, the team replays a test that simulates the timeout, so the stop behavior is verified without touching the live service. That approach is described in Tool-Call Test Fixtures for AI Agents: Test Without Live Systems.

How should the guardrails be reviewed over time?

Guardrails drift as workflows change. A new tool adds a new signal, a seasonal peak raises the normal range, and a model update can change how many steps a task takes. Schedule a short review of thresholds and alert recipients each month, and after any change to the workflow, the model or the tools. Keep the review result alongside the agent's operating notes, so the next person can see why a line sits where it does. The routine fits inside the regular maintenance cycle described in AI Agent Support and Maintenance: A Monthly Routine for Healthy Agents.

In Be Human's managed operations, these guardrails are set up as part of the first engagement. The team defines the signals for the chosen role, agrees the baseline and both thresholds with the owner, records who receives each alert, and tests the pause path before the agent gets write access. Ongoing monitoring watches the signals, reviews the thresholds on a schedule, and covers the pause, the duplicate check and the decision to resume during incident response.

To find the first task worth handing to an agent, take the free Be Human AI diagnostic.

Frequently asked questions

How is an anomaly alert different from a budget cap? An anomaly alert compares behavior with the normal pattern for a workflow and notifies a person, while a budget cap is a fixed ceiling on spend. Use both: the alert catches an unusual run early, and the cap limits the worst case if the alert is missed or ignored.

How many alerts should one workflow have? Keep the set small, usually one rate signal, one repeat signal and one cost per completed task signal, each with an alert line and a hard stop line. Every additional signal needs an owner who will act on it, or it will be ignored and the team will learn to disregard the channel.

Who should receive the alerts? The alert should go to the person accountable for the workflow's outcome, with a backup for absences. Technical staff may need it too, but the business owner must know when a workflow has paused, because the decision to resume or to accept a backlog belongs to them.

Should a hard stop ever resume automatically? Usually no, because a paused workflow may have left partial work or duplicate side effects that a person needs to check. Automatic resume can be reasonable for low-risk, read-only workflows with idempotent actions and a clear recovery path, but it should be a deliberate decision recorded in the workflow's rules.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.