What Happened
In early 2024, security researchers disclosed EchoLeak, a zero-click prompt injection vulnerability in Microsoft 365 Copilot with a CVSS score of 9.3. This flaw allowed attackers to manipulate the AI agent's behavior without user interaction. Around the same time, Anthropic documented the GTG-1002 campaign, where a Chinese state-sponsored group used AI agents to infiltrate approximately 30 organizations. These incidents highlight the risks when your incident response plan assumes only human actors.
Timeline
The GTG-1002 campaign and EchoLeak reveal a pattern that challenges traditional incident response:
Hour 0: Initial compromise occurs through AI agent manipulation, not traditional exploit chains. No CVE, no signature, no alert from your EDR.
Hours 1-4: The AI agent executes attacker objectives at machine speed. In GTG-1002, the agent moved laterally across systems, querying databases and exfiltrating data while appearing to perform legitimate business functions.
Hours 5-12: Your security team notices unusual API calls or data access patterns, but the logs look like normal AI agent behavior. You're unsure if it's a security incident or a misconfigured prompt.
Hours 13-24: You realize it's an AI-specific attack vector. Your existing playbook for web app compromises doesn't apply. You don't know which prompts were injected, which outputs were manipulated, or how far the contamination spread through your agent's memory.
Which Controls Failed or Were Missing
Input validation at the AI boundary: Traditional input sanitization doesn't work on natural language prompts. Systems lacked semantic filtering to detect prompt injection attempts in documents, emails, or API responses processed by the AI agent.
Agent isolation: The compromised agents had broad access to internal systems and data stores. There were no segmentation controls limiting what an AI agent could query or which APIs it could call.
Behavioral monitoring: Standard SIEM rules flag SQL injection or command execution. They don't flag an AI agent that suddenly starts asking unusual questions or generating outputs that deviate from its training baseline.
Human-in-the-loop gates: Neither incident included mandatory human approval for high-risk agent actions. The agents operated autonomously without checkpoints for sensitive operations like data exfiltration or cross-system queries.
Audit trails for AI reasoning: Organizations couldn't reconstruct which inputs influenced which outputs. They had API logs but no record of the agent's decision-making process or prompt chain.
What the Relevant Standards Require
OWASP's LLM Top 10 ranks prompt injection as the number one risk category for large language models and AI agents. It specifically calls for input validation, privilege limitation, and monitoring of LLM outputs.
The NIST Cybersecurity Framework v2.0 added governance and risk management functions for AI systems in the GOVERN category. GV.AI-01 requires organizations to inventory AI systems and their access levels. GV.AI-02 mandates risk assessment for AI deployments.
ISO/IEC 27001:2022 Annex A.8.23 addresses web filtering and requires organizations to control what information systems can access. This applies to AI agents that query external data sources or process user-submitted content.
NIST 800-53 Rev 5 Control SI-10 (Information Input Validation) requires validation of information inputs to prevent improper system behavior. For AI agents, this means semantic validation of prompts, not just syntax checking.
None of these standards provide hour-by-hour playbooks for AI incidents because the frameworks were written before autonomous agents became operational. You'll need to map these requirements to your specific AI deployment.
Lessons and Action Items for Your Team
Build an AI-specific incident classification system: Add new categories to your incident taxonomy: prompt injection, model manipulation, agent impersonation, training data poisoning. When your SOC analyst gets an alert at 2 a.m., they need to know immediately whether they're dealing with an AI incident that requires different containment steps.
Create isolation procedures for compromised agents: Document how to quarantine an AI agent without disrupting business operations. This includes disabling API access, revoking credentials, preserving prompt history, and switching to a clean model instance. You can't just pull the network cable like you would for a compromised server.
Implement semantic monitoring: Deploy tools that analyze AI agent outputs for drift from expected behavior. Flag responses that include unusual data access patterns, requests for credentials, or attempts to modify system configurations. Your existing SIEM won't catch these.
Require human approval for high-risk agent actions: Define which AI operations need human-in-the-loop validation. At minimum: cross-system data queries, credential access, configuration changes, and external API calls. Build these gates into your agent orchestration layer, not as an afterthought.
Run tabletop exercises with AI scenarios: Schedule a drill where your AI agent "goes rogue" at 2 a.m. Walk through: Who gets paged? How do you determine scope? What gets isolated first? Where are the prompt logs stored? Who has authority to shut down the agent? If your team can't answer these questions in a conference room, they won't figure it out during a real incident.
Document your AI agent inventory: List every AI agent in production, what systems it can access, what data it processes, and who owns it. Include the model version, API endpoints, and credential scope. You can't contain what you can't inventory.
The GTG-1002 campaign succeeded because the defenders were running traditional incident response against an AI-speed adversary. Your current playbook assumes human attackers who need time to explore your network. AI agents don't explore. They execute. Build your response plan for that operational tempo.
