Your compliance team spent six months building an AI risk register. You documented model drift, bias risks, and data poisoning scenarios. You assigned likelihood scores, impact ratings, and risk owners. The spreadsheet is comprehensive.
Then your customer service AI starts fabricating account balances in support tickets. Your risk register doesn't tell you who can shut it down, who needs to be notified in the first 30 minutes, or how to preserve the training data for forensic review.
This isn't a hypothetical gap. Organizations are deploying AI systems with risk documentation but no executable response plan. Let's explore what that failure looks like in practice.
What Happened
Consider a team that deployed a retrieval-augmented generation (RAG) system for internal knowledge management. The system began returning confidential information from restricted documents to unauthorized users. The security team discovered the issue during a routine audit three weeks after deployment.
The organization had a risk register entry for "unauthorized data exposure through AI retrieval." But when the incident occurred, no one knew who had the authority to disable the system, which logs needed immediate preservation, whether to notify affected users, or how to assess the scope of exposure.
The response took 48 hours to coordinate. During that time, the system continued serving queries.
Timeline
Day 1, 09:00: Security analyst notices unusual access patterns in audit logs during routine review.
Day 1, 11:30: Analyst confirms the AI system is retrieving and displaying content from restricted SharePoint sites to users without proper access.
Day 1, 12:00: Analyst emails the AI project lead. No response (lead is in meetings).
Day 1, 14:30: Analyst escalates to security manager, who contacts IT operations. IT says they don't have authority to disable a "business critical" system without approval.
Day 1, 16:00: Security manager schedules emergency meeting with AI project lead, compliance, and IT for the next morning.
Day 2, 09:00: Meeting convenes. Team debates whether this constitutes a "security incident" under their existing incident response plan.
Day 2, 11:00: Decision made to disable the system. IT begins shutdown procedures.
Day 2, 14:00: System fully disabled. Team realizes they have no procedure for preserving the RAG system's query logs or vector database state.
Which Controls Failed or Were Missing
Missing authority matrix: No documented decision tree for who can disable AI systems under what circumstances. The risk register listed "unauthorized access" as a risk but didn't specify response ownership.
No evidence preservation procedures: Unlike traditional security incidents where you'd preserve logs and memory dumps, no one had defined what constitutes evidence for an AI incident. The team didn't capture the model's state, the retrieval queries, or the vector embeddings that led to the exposure.
Undefined incident classification: The existing incident response plan covered data breaches and system compromises but didn't address AI-specific scenarios. The team spent hours debating whether this was a "security incident" or an "operational issue."
No notification triggers: The risk register didn't specify when to notify affected users, regulators, or leadership. The team had no framework for assessing impact when the exposure mechanism was probabilistic retrieval rather than direct database access.
What the Relevant Standard Requires
ISO/IEC 27001:2022 Annex A.5.24 requires organizations to plan and prepare for information security incident management. This isn't optional for AI systems just because they're new technology.
NIST CSF v2.0 function DE.AE (Anomalies and Events) requires detection processes and procedures to be maintained and tested. Your AI risk register documents potential anomalies, but detection without response is incomplete.
For organizations under PCI DSS v4.0.1, Requirement 12.10.1 mandates an incident response plan that includes procedures for responding to suspected or confirmed security incidents. If your AI system processes cardholder data, "we'll figure it out when it happens" doesn't satisfy this requirement.
SOC 2 Type II Common Criteria CC7.3 requires the entity to identify, develop, and implement activities to respond to identified security incidents. A risk register entry isn't an implemented activity.
Lessons and Action Items for Your Team
Build an authority matrix before the incident: Document who has authority to disable, throttle, or modify AI systems under different scenarios. Include contact information, escalation paths, and decision criteria. Test it with a tabletop exercise where you simulate an AI hallucination incident at 2 AM on a Saturday.
Define AI-specific evidence requirements: Work with your legal and forensics teams to determine what you need to preserve during an AI incident. This typically includes: model weights and version, training data lineage, inference logs with timestamps, vector database state, and configuration files. Store this in your incident response playbook, not buried in your risk register.
Create AI incident classification criteria: Extend your existing incident classification framework to cover AI-specific scenarios. Define what constitutes a severity 1 AI incident versus a severity 3. Include factors like: scope of exposure, model confidence scores, user impact, and regulatory implications.
Establish notification triggers: Before AI systems move into production, document when you'll notify users, regulators, or partners. A risk register can note "reputational risk from AI errors," but your incident response plan needs to specify: "If the model returns incorrect information to more than X users, notify the communications team within Y hours."
Practice with realistic scenarios: Your next tabletop exercise should include an AI incident. Don't make it theoretical. Use a specific scenario: "The content moderation model starts flagging legitimate customer reviews as spam. How do you respond in the first hour?" Walk through the actual steps your team would take.
Integrate with existing IR processes: You don't need a separate "AI incident response plan" if you extend your current plan to cover AI-specific scenarios. Add AI systems to your asset inventory, include AI-specific evidence in your forensics procedures, and train your incident response team on AI system architecture.
A risk register documents what might go wrong. An incident response plan tells your team what to do when it does. Your AI systems need both.
