AI Agents for IT Operations: Incidents, Change and Limits

AI agents for IT operations work best like a careful junior engineer with a perfect memory. They read monitoring data, follow your runbooks, draft incident summaries and prepare changes, while people approve anything that alters a production system. The safe path is to start with read-only access, write runbooks as explicit steps, and route every change through the approval rules you already trust. Capability then grows one permission at a time, as the agent's record justifies it.

Key takeaways

• An IT operations agent earns its value first from reading and explaining, not from acting. Triage, correlation and clear handoffs come before any write access.
• A runbook is the agent's job description. If a step cannot be written down plainly, it is not ready to be delegated.
• Change approvals should depend on reversibility and blast radius, not on how confident the agent sounds.
• Every new permission should be granted deliberately, tested against past incidents, and removable without drama.
• Logs, tickets and alert text are untrusted input. Design the agent so that text it reads can never become authority it holds.

What can an AI agent actually do in IT operations?

IT operations is a cycle of four activities: watching, deciding, doing and recording. People are good at the deciding. They are often stretched thin on the rest. An agent can carry much of the watching and recording, and a controlled share of the doing.

In practice that means a handful of well-bounded jobs:

• Reading alerts as they arrive, grouping related ones, and attaching the context an on-call engineer would otherwise gather by hand.
• Finding the right runbook for a symptom and walking through its diagnostic steps with read-only tools.
• Keeping an incident timeline and drafting status updates for stakeholders in plain language.
• Preparing change requests with the reason, the steps, the rollback plan and the affected systems already filled in.
• Writing the post-incident notes while the details are fresh, and noting which runbook steps were missing or wrong.
• Checking for drift, expiring certificates, stale accounts and unreviewed permissions on a schedule.

This is different from a chat assistant that answers questions about IT. An operations agent has an identity, a memory of your environment and a defined set of tools, and it works on a recurring basis. The neighboring problem of employee requests is covered in AI Agents for IT Helpdesk: Tickets, Access and Runbooks, and many of the same limits apply here.

Why should the first version be read-only?

Most of the value in incident response is in the first stretch: understanding what is happening. Observability data (logs, metrics and traces, as the OpenTelemetry observability primer describes them) already holds the evidence. An agent that can query it, compare it with recent changes and summarize it shortens that stretch without touching a single system.

Read-only also limits what can go wrong. If the agent misreads a signal, the cost is a poor summary that a human corrects. If it misreads a signal while holding write access, the cost is a second incident.

The OWASP authorization guidance points the same way: deny by default, grant the least privilege needed, and check permissions on every request rather than assuming them. For an agent, that translates into a short setup checklist:

• Give the agent its own service identity, never a shared admin account or a person's credentials.
• Scope read access to the systems named in its runbooks, not the whole estate.
• Keep secrets out of what the agent can read, and mask sensitive fields in logs where possible.
• Record every query the agent makes so reviewers can see what it looked at.
• Make the credentials revocable in one step by a named owner.

There is a security reason as well. Alert text, ticket bodies and log lines can contain instructions written by someone other than you. The OWASP prompt injection guidance treats all such content as untrusted. A read-only agent that treats what it reads as data, and never as commands, has far less to lose if a hostile string slips into a log.

How do you turn runbooks into something an agent can follow?

Most teams already have runbooks, but they are written for a colleague who shares context. They say restart the service when it hangs, and trust the reader to know which service, which host and what hanging looks like. An agent needs that context made explicit, and the exercise improves the runbook for humans too.

A practical plan for each runbook:

• First, pick one recurring incident type with a clear symptom, such as a backed-up queue or an expiring certificate.
• Second, write the trigger in observable terms: which signal, which threshold, which system.
• Third, list the diagnostic steps in order, and mark each as read-only or as a change.
• Fourth, name the expected result of each step, so the agent can tell whether the step worked.
• Fifth, define the stop conditions: when the agent must escalate instead of continuing, and to whom.
• Sixth, attach the rollback for every change step, even if the rollback is to do nothing.
• Seventh, replay the runbook against two or three past incidents and see where the agent's reading differs from what the responders did.

The replay step matters most. Differences show up as missing context, ambiguous wording or steps that responders did from habit and never recorded. The agent's memory then holds the corrected version, and the runbook becomes a living document with an owner, not a file nobody opens. Anthropic's writing on building effective agents makes a related point: simple, well-defined workflows with clear checkpoints tend to outperform open-ended autonomy.

How should change approvals work when an agent is involved?

Change management exists because well-intentioned changes cause outages. An agent does not make that risk disappear, and it can make changes happen faster than anyone notices. So the approval rule should not depend on who proposes the change. It should depend on what the change can do.

A simple decision guide, sorted by reversibility and blast radius:

• The agent may act alone only on narrow, reversible, pre-approved actions with a tested rollback, such as clearing a known cache or re-running a failed job on a non-critical system.
• The agent drafts and a named person approves for changes that touch production but are covered by a runbook, such as restarting a worker pool or rotating a certificate.
• A human leads and the agent assists for changes that alter data, access, networking or anything customer-facing, with the agent preparing the plan and checking the result.
• No agent involvement in execution for destructive or irreversible actions, such as deleting data or changing security boundaries, until your own policy says otherwise.

Whatever the tier, each change record should carry the same facts: what will change, why, who approved it, the rollback, and the check that proves it worked. The agent can prepare all of that, and after the change it can run the verification and attach the result. Where an action might be retried after a timeout, design it to be safe to repeat, so a second attempt does not apply the change twice.

Approvals should also expire. A yes given for one change at one moment is not a standing permission, and the agent should not treat it as one.

What does this look like in practice? An illustrative scenario

Illustrative scenario: a mid-sized company runs an order-processing service, and its message queue begins to back up outside working hours. The agent notices the growing backlog through its read-only monitoring access and finds that the symptom matches a runbook for stalled workers.

It works through the diagnostic steps. It checks worker health, compares the start of the backlog with the most recent deployment and finds no related change. It sees that two workers stopped reporting at the same time, and writes a short summary for the on-call engineer: what it observed, which runbook applies, and what it recommends.

The runbook's next step is a restart of the worker pool, which is a production change. The agent does not run it. It sends the on-call engineer a change draft containing the reason, the single step, the rollback and the check it will run afterward. The engineer reads the evidence, approves that one step, and the agent executes it and confirms the backlog is draining.

What the agent did not do is as important. While reading, it noticed that database load was somewhat elevated. Scaling the database is not in the runbook, so it listed this as an observation for the morning and did nothing about it. It then wrote the incident timeline and flagged that the runbook lacked a step for checking worker heartbeats. A person reviewed the proposed addition the next day.

How does an agent become more capable without becoming riskier?

The answer is to treat capability as a ladder, not a switch. Each rung adds one thing, and the agent stays on a rung until its record supports the next.

• Observe: read-only access, summaries and timelines, reviewed by humans.
• Recommend: proposes the runbook and next step, with evidence attached.
• Prepare: drafts complete change requests, rollback plans and verification checks.
• Act with approval: executes a single approved step and verifies it.
• Act within a narrow class: handles a small set of reversible actions on its own, with after-the-fact review.

An agent should earn each new permission the way a new engineer does: by showing its work first.

Moving up should rest on evidence. Replay past incidents, run recurring evaluations that compare the agent's actions with what a careful responder would have done, and review where it escalated, where it stayed silent and where it was wrong. Anthropic's guidance on evals for agents describes this kind of repeatable testing in detail. Moving down should be just as easy: if an integration changes, a runbook is revised or a mistake occurs, the agent drops a rung until the cause is understood.

This ladder is also how scope can widen across the business. Teams that already use agents for software development, review and operations or for recurring operations work and reports can apply the same rungs, with permissions reviewed per role rather than per enthusiasm.

How does Be Human handle IT operations agents?

Be Human designs, implements and operates personalized AI infrastructure, and an IT operations agent starts the way every engagement does: one useful role, the memory and tools it needs, approval boundaries, and a success criterion you can observe. For this role that usually means read-only monitoring access, a small set of runbooks, and a defined approval path for each type of change. Managed operations can then include monitoring of the agent itself, recurring evaluations, integration maintenance, permission review, incident response and controlled improvements, so the agent's access changes only when someone has decided it should.

If you are weighing where an agent could help your IT team first, take the free Be Human AI diagnostic to find the first task worth handing to an agent.

Frequently asked questions

Will an AI agent replace our on-call engineers? No. The agent takes over gathering context, drafting updates and following documented steps, while people keep judgment, approval and accountability. The practical effect is that an on-call engineer starts each incident with the evidence already assembled.

Do we need mature monitoring before we start? You need enough visibility that a human could diagnose the incident types you choose first. If signals are scattered or runbooks are thin, the first stage of the work is improving those, which benefits your team regardless of the agent. Start with one incident type where the data and the steps are already reasonably clear.

How do we protect an agent from instructions hidden in logs or tickets? Treat everything the agent reads as data, never as instruction, and keep its permissions narrow enough that a manipulated reading cannot cause harm. Read-only access, approval for changes and a complete log of what it queried and proposed all limit the damage. The OWASP guidance on prompt injection and on LLM applications is a useful reference for your security review.

What should we measure to know the agent is working? Choose a criterion you can observe, such as whether responders get a correct summary and the right runbook at the start of an incident, or whether change requests arrive complete. Review the agent's escalations and mistakes regularly alongside its successes. Widen its permissions only when that record supports it.

Be Human editorial guidance. Examples are illustrative unless explicitly attributed. Sources provide background; implementation choices depend on your workflow.