AI incident response: when the model is the problem

Most incident response assumes an adversary. A significant class of AI incident has none - the system behaved exactly as built, and the output was wrong, biased, or harmful anyway. That breaks several assumptions in a standard plan.

The incident types worth planning for

• Harmful or incorrect output at scale - the model gives wrong advice, produces discriminatory outcomes, or fabricates confidently. No compromise occurred.
• Data leakage through output - the system reveals information from its context, training data or another tenant.
• Prompt injection consequences - instructions in processed content cause an action you did not authorise.
• Model or data poisoning - the training or retrieval corpus was manipulated.
• Degradation and drift - accuracy falls quietly as the world changes.
• Supplier-side change - a provider updates a model and your behaviour changes without any deployment on your side.

That last one deserves attention. Your system can change materially without you changing anything, which no traditional change control catches. Version-pin where the provider allows it, and monitor output quality continuously rather than only at release.

Detection is different

There is often no alert. Detection typically comes from user complaints, support tickets, output sampling, or an external party noticing before you do.

What helps: sample outputs continuously against a quality baseline, monitor refusal and error rates as leading indicators, track user feedback signals as a security and quality feed rather than a product one, and log inputs and outputs so you can reconstruct what happened.

Containment

1. A kill switch. Disable the feature immediately without a full deployment - build it before launch, not during the incident.
2. Fall back to a known-good state - a previous model version, a simpler deterministic path, or human handling.
3. Reduce autonomy rather than switching off entirely where possible: drop from acting to suggesting.
4. Preserve evidence - inputs, outputs, model version, retrieval context and tool calls, before anything is redeployed.

Investigation questions that differ

Beyond the usual: which model version and configuration was live, what was in the retrieval context, was the input adversarial or merely unusual, is the behaviour reproducible, and how many outputs are affected. That last question is often the hardest and the most important - the blast radius of a bad model is every response it produced, not a single event.

Remediation and disclosure

Where the model produced harmful outputs at scale, consider whether affected users need telling and whether decisions made on those outputs need revisiting. That is a genuinely different obligation from a data breach and it is easy to overlook because no data left.

Regulatory considerations: personal data exposure triggers privacy notification as normal; automated decisions affecting individuals may carry rights to explanation or review; sector rules and emerging AI regulation may impose their own duties. Establish which apply before an incident.

Update the plan

Add AI-specific roles - who can disable a model, who judges whether output is acceptable - and run a tabletop on an incident with no attacker. Teams accustomed to intrusion scenarios find that exercise genuinely disorienting, which is exactly why it is worth doing.

Key takeaways

• Many AI incidents have no attacker - the plan must handle that.
• Build a kill switch and output logging before launch.
• Your behaviour can change when a provider updates a model.
• Blast radius is every affected output, not a single event.