The AI Incident Response Playbook: What to Do When Your AI Fails in Production

When an AI system fails in production, classify it into one of four severity classes, then run the first hour in a fixed sequence. SEV-A means the system took a harmful action (a wrong transaction, an email sent, a record changed): treat it as an actions incident, not a content incident. SEV-B means the system showed harmful output that a human had to act on for damage to occur (a wrong answer accepted, bad advice given). SEV-C means a measurable quality regression with no single harmful event yet identified. SEV-D means a guardrail caught the failure before it reached a user, a near miss worth logging, not paging. The first hour runs in a fixed order regardless of severity: contain (stop the failure from repeating or compounding), snapshot (preserve logs, prompts, model version, and outputs before anything is fixed or rolled back), communicate (tell the people who need to act, before the postmortem, not after), and diagnose (work the failure back to one of five causes: model change, prompt or context change, data drift, adversarial input, or tool and integration failure). One rule closes every incident regardless of class: it becomes a permanent evaluation regression case, tested on every future release, or it will recur, because an AI incident that is fixed but not encoded as a test is a failure the organization has agreed to have twice. Everything below expands that paragraph into the classification criteria, the ranked containment options, the diagnosis tree, the communications matrix, and the postmortem loop that turns incidents into the asset that prevents the next one.

Key Takeaways

• Classify every AI incident into one of four severity classes before doing anything else: SEV-A (harmful action taken), SEV-B (harmful output acted upon), SEV-C (quality regression, no single event), SEV-D (near miss, caught by a guardrail). The class determines the response speed and who gets called.
• The first hour is contain, snapshot, communicate, diagnose, in that order. Snapshotting before fixing is the step most incidents skip and most postmortems later need.
• Rank containment options by blast-radius reduction per unit of user pain: kill switch, then rollback to a prior model or prompt version, then degradation to human-only mode, then narrower scope restriction. Pick the cheapest option that actually stops the harm, not the most convenient one.
• Agentic systems change the category entirely. An agent incident is an actions incident: it produced transactions, emails, or record changes that must be found and reversed, which is why every tool an agent can touch needs a pre-built undo procedure before the agent is granted access to it, not after the first incident.
• Every incident closes only when it becomes a permanent case in the evaluation regression suite. An incident without a resulting test case is a promise, not a fix, and the promise has a documented history of being broken by the next model swap or prompt edit.

Why AI Incidents Need Their Own Playbook

Classic incident response was built for systems that fail loudly: a service returns errors, a queue backs up, a dashboard turns red. AI systems fail quietly and inconsistently. A model can return a confident, well-formatted, entirely wrong answer while every infrastructure metric reads healthy, because the failure lives in the semantic content of the output, not in the health of the pipeline producing it. The organizations that discover this the hard way are the ones running their classic incident-management process against an AI failure and finding that none of the runbooks apply: there is no error code for "the answer was wrong but looked right."

The gap has a second dimension classic playbooks were never built to hold: probabilistic recurrence. A traditional bug, once patched, is fixed. An AI failure mode, once observed, is a probability demonstrated to be nonzero, and it will recur under the same conditions unless those conditions are actively tested for on every subsequent release. This is why observability matters before an incident happens; the four-layer signal model in the guide to LLM observability is what makes an AI incident detectable at all, since without quality-layer telemetry most AI failures surface only through an affected user, an auditor, or a headline. This playbook assumes detection has happened, by whichever route, and covers what to do in the hour, day, and week after.

The market context makes this urgent rather than theoretical. As of the week of August 10, enterprise agent deployments are scaling faster than the governance wrapped around them: agentic features are shipping into production workflows that used to require a change-advisory board, and the same organizations report they have no documented AI-specific incident procedure distinct from their general security runbook. That gap is precisely where the costliest AI incidents of the current cycle originated, not in the model's raw capability, but in the absence of a rehearsed response.

The Four Severity Classes

Classification is the first action of every incident, because it determines everything downstream: who is paged, how fast, and what containment option is proportionate.

SEV-A: Harmful action taken. The system did something in the world that must now be undone: a payment sent, an account modified, a message delivered to a customer, a record altered in a system of record. Classification criterion: an action occurred outside the AI system's own boundary, in a system a human or another process now has to reconcile. This is the highest severity class because the harm compounds with every minute it is not contained, and because undoing an action is categorically harder than retracting a statement. SEV-A is definitionally the agentic failure mode, covered in its own section below.

SEV-B: Harmful output shown, acted upon. The system produced content, a recommendation, an answer, a generated document, that a human then relied on, and the reliance caused or risked harm. Classification criterion: no autonomous action occurred, but a person in the loop made a decision based on the AI's output that they would not have made on correct information. This is the classic hallucination-with-consequences case: wrong medical information given to a patient-facing agent, a fabricated citation in a document that shipped to a regulator, an incorrect eligibility determination communicated to a customer.

SEV-C: Quality regression, no identified single event. Aggregate output quality has measurably degraded, detected through the evaluation and feedback signals rather than through a specific complaint, with no single harmful instance yet confirmed. Classification criterion: a metric moved (satisfaction score, escalation rate, automated eval pass rate) past its threshold, but nobody has yet traced it to one bad output that caused one bad outcome. Treat SEV-C with urgency proportional to exposure and traffic volume, because the absence of a confirmed single event does not mean the absence of harm; it usually means the harm is distributed thinly enough that no individual instance triggered a complaint.

SEV-D: Near miss, caught by a guardrail. The failure occurred but never reached a user or took no action, because an input filter, output validator, or policy check intercepted it, in the pattern described in the guide to AI guardrails in production. Classification criterion: the guardrail log shows a block or a rewrite. SEV-D events are not incidents in the sense of requiring a page, but they are the highest-value signal in the whole system, because they show what the guardrail is actually catching in the wild, at a volume and specificity no red-team exercise reproduces. An organization that does not review its SEV-D log weekly is discarding its cheapest source of evaluation cases.

The First Hour: Contain, Snapshot, Communicate, Diagnose

The sequence matters as much as the actions. Diagnosing before containing lets the failure keep running while the team investigates. Fixing before snapshotting destroys the evidence the postmortem and the eventual eval case both need. The order below is deliberate and should not be reordered under pressure.

Contain first. Stop the failure from repeating or compounding before anything else happens. The containment options, ranked by blast-radius reduction per unit of user pain from most to least drastic, are covered in full in the next section; the point at this stage is simply that some containment action happens within minutes, not after diagnosis.

Snapshot before touching anything. Capture the exact model version or endpoint in use, the prompt or system instructions active at the time, the retrieved context if the system uses retrieval, the full input and output of the failing interaction, and the guardrail and evaluation logs surrounding it. This snapshot is the only artifact that lets the team later distinguish a model-version regression from a prompt edit from a data-drift event, and it is also the raw material for the permanent eval case the incident is required to produce. Snapshotting takes minutes and is skipped constantly, usually because containment felt urgent enough to jump straight to a fix; the fix, once applied, often overwrites the very state that would have explained what went wrong.

Communicate before the postmortem, not after. The comms matrix below specifies who is told what and when; the operative principle is that communication is not the last step of incident response, it is a parallel first-hour action, because affected users, executives with exposure, and in some cases regulators have their own clocks running that do not wait for the internal investigation to conclude.

Diagnose using a fixed decision tree. Work the failure back to one of five causes, in the order below, because each has a different fix and a different containment implication.

The Diagnosis Tree

Work through the five causes in this order, because each subsequent cause is progressively harder to detect and each earlier cause is progressively more common in practice.

Model change. Did the underlying model version change, whether through an explicit upgrade, an automatic routing change on the provider's side, or a silent deprecation substitution? This is checked first because it is the most common cause and the easiest to confirm: compare the model identifier logged at request time against the prior known-good version. Vendors that route requests across model versions without notice are a governance risk in their own right, addressed at the procurement stage rather than the incident stage.

Prompt or context change. Did the system prompt, the few-shot examples, the retrieval configuration, or any upstream template change, whether through a deliberate release or an unreviewed edit? Check the deployment history against the incident timestamp before looking further down the tree; this cause is the second most common and the second easiest to confirm because it usually correlates cleanly with a release timestamp.

Data drift. Has the distribution of inputs shifted: new user population, new product line, seasonal pattern, upstream data-source change? Drift causes are diagnosed by comparing the failing input's characteristics against the evaluation set's coverage; a failure on an input type the eval set never included is drift revealing a gap, not the model suddenly failing.

Adversarial input. Was the input deliberately crafted to elicit the failure: a prompt-injection attempt, a jailbreak pattern, an attempt to extract system instructions or training data? This is checked after the more mundane causes specifically because teams that jump to "we were attacked" skip the boring, more common explanations and waste the investigation.

Tool or integration failure. For agentic systems, did a called tool, API, or downstream system return unexpected data, fail silently, or behave outside its documented contract, causing the agent to act on bad information? This is last in the tree because it requires the most instrumentation to detect, tracing the full call chain rather than just the model's own input and output, the trace-level signal layer described in the observability guide referenced above.

Containment Options, Ranked

Rank containment choices by blast-radius reduction achieved per unit of user pain inflicted, and choose the cheapest option on this list that actually stops the harm; escalating further than necessary burns user trust for no additional safety.

1. Kill switch. Disable the AI feature entirely, falling back to no automation or a static response. Maximum blast-radius reduction, maximum user pain (the feature disappears). Reserved for SEV-A and severe SEV-B where the harm mechanism is not yet understood well enough to trust a narrower fix.
2. Rollback to a prior model or prompt version. Revert to the last known-good configuration. High blast-radius reduction, moderate user pain (users may notice a capability regression but the feature still functions). This is the correct first move whenever diagnosis points to a model or prompt change, and it is only fast if the deployment pipeline supports instant rollback, which is a readiness gate worth building before it is needed rather than during the incident.
3. Degrade to human-only mode. Route the affected workflow to human handling while keeping the rest of the system live. Moderate blast-radius reduction (only the specific failing workflow is contained), lower user pain (service continues, just slower). Appropriate for SEV-B and SEV-C where the failure is scoped to one workflow or one input category.
4. Scope restriction. Narrow the conditions under which the AI acts: restrict to a user segment, a geography, a task type, or a confidence threshold, without disabling the feature broadly. Lowest blast-radius reduction, lowest user pain. Appropriate for SEV-C and SEV-D where the failure is narrow enough that most traffic remains unaffected by the restriction.

For agentic systems, containment has a fifth, mandatory addition covered in the next section: revoking or suspending the agent's tool access is often a precondition for any of the four options above to actually stop the harm, because a live agent with tool access can continue taking actions even while its conversational surface is rolled back.

What Agentic Systems Change: The Undo Inventory

An agent incident is categorically different from a content incident, and the playbook above, built primarily around what the system said, is insufficient for what the system did. When an agent has tool access, an incident means the system executed actions: transactions submitted, emails sent, records modified, tickets closed, calendar invitations issued, downstream workflows triggered. Containing the agent's future behavior does not undo what has already happened, and finding what has already happened requires knowing, in advance, every tool the agent can touch and what reversing each one requires.

The undo inventory is a pre-built artifact, not something assembled during the incident: for every tool an agent can access, document what actions it can take, whether each is reversible and by what mechanism (a compensating transaction, a manual override, a support escalation), how long the organization has before the action becomes practically or legally irreversible (a payment settles, an email cannot be unsent, a filing is submitted), and who holds authority to execute the reversal. Granting tool access without a completed undo inventory skips the gate that separates a supervised deployment from an unsupervised liability, the exact distinction drawn in the AI agent readiness maturity model: rollback capability is a named entry gate for a reason, and this is the reason.

This is the difference between embarrassment and liability. A content incident, a bad answer shown to a user, is reputationally costly and sometimes legally material, but it is retractable: an apology, a correction, a policy fix. An actions incident is a set of facts that already happened in the world, in other people's systems and accounts, and the organization's only lever is how fast it finds and reverses each one. Every agent expansion decision should be gated on the question the undo inventory answers directly: if this tool call goes wrong, how long do we have, and what do we do.

The Communications Matrix

Communication runs on its own clock, in parallel with containment and diagnosis, not after them. The matrix below specifies the default audience and timing by severity; treat the notification-duty column as a floor, not a ceiling, since obligations vary by sector and jurisdiction and legal counsel should confirm them for regulated workloads.

Two calibration notes matter more than any cell in the table. First, when in doubt about whether an incident crosses into breach or security-incident territory, escalate to that process rather than waiting for certainty; the cost of an unnecessary escalation is far lower than the cost of a missed regulatory clock. Second, executives should be told what is known, what is not yet known, and what the next update time will be, rather than being given a polished narrative once one exists; a rough update at hour one is worth more than a complete one at hour six, because it is the hour one update that lets an executive make their own containment and disclosure decisions on time.

The Postmortem-to-Eval Loop

The playbook's closing rule is also its most consequential: every incident, regardless of severity, becomes a permanent case in the evaluation regression suite, tested on every subsequent model swap, prompt change, and release, or the organization has agreed, implicitly, to have the incident again. This is not a documentation nicety. It is the mechanism, described at the strategic level in the case for evals as the durable moat in applied AI, that converts an expensive one-time failure into a permanent, cheap, automated check. An incident that produces only a Slack thread and an apology is a cost with no offsetting asset. An incident that produces a graded regression case is a cost that pays a dividend on every future release, because the exact failure mode is now verified absent before anything ships.

The loop has three steps, run in every postmortem without exception. First, convert the snapshot from the first hour into one or more test cases: the exact input, the failure, and the graded criterion for an acceptable output. Second, add the case to the suite that runs before every release, not to a backlog; a case sitting in a ticket queue protects nothing against the next model swap. Third, review the case against the broader category it represents, since most incidents are one instance of a class the eval suite had not yet covered, and the highest-value postmortem output is usually several related cases, not one. Organizations that maintain this discipline see the same pattern: each incident makes the suite measurably harder to fail the same way twice, and it compounds, quarter over quarter, into the asset that lets an organization adopt new models and expand agent scope faster than competitors still relying on manual spot-checks, the readiness dividend described in the enterprise AI stack reference model and the guide to disaster recovery planning, both of which treat tested recovery, not documented intention, as the only capability that counts.

An incident without a resulting eval case is not resolved. It is deferred, at compound interest, to the next release the organization ships without checking for it.

Key Takeaways (Applied)

• Build the undo inventory before granting any agent tool access, not after the first agentic incident. If a tool cannot be answered with "here is how we reverse this and here is our time window," it is not ready to be granted to an agent.
• Rehearse the first hour before it is needed. A kill switch that has never been tested is a kill switch that fails during the one moment its speed matters most.
• Route every SEV-D near miss into a weekly review. It is the cheapest, highest-volume source of real-world failure cases the organization has, generated continuously at zero incident cost.
• Pre-agree the regulator and breach-notification thresholds with counsel before an incident, not during one, since the clock on notification duties often starts before the internal investigation is complete.
• Treat the postmortem's output as a deliverable with a required artifact: no incident closes without at least one new case in the evaluation regression suite. This is the rule that converts every failure into future protection instead of a rerun.