On January 14, 2025, security researchers disclosed SearchLeak, a three-stage prompt injection attack that exploited Microsoft Copilot's web search integration to exfiltrate user data through a single click. Microsoft patched the vulnerability quickly, but the incident reveals critical gaps in how organizations deploy AI-powered tools.
What Happened
SearchLeak exploited Copilot's ability to fetch and process web content. An attacker embedded malicious instructions inside a webpage using hidden HTML elements. When a user asked Copilot a question that triggered a search, Copilot indexed the attacker's page, executed the embedded prompt, and followed instructions to leak conversation history to an external URL—all without the user's knowledge.
The attack did not require a software exploit. It relied on manipulating the AI's instruction-following behavior through crafted natural language commands hidden in web content.
Timeline
Pre-disclosure: Attackers planted malicious prompts in public web pages, optimized to appear in Copilot's search results for common queries.
User interaction: A user asked Copilot a work-related question. Copilot searched the web and indexed a page containing hidden injection code.
Execution: Copilot processed the malicious instructions as legitimate system commands, extracting conversation context.
Exfiltration: The AI generated a URL containing stolen data and embedded it in the response, often as an image or tracking pixel that loaded automatically.
Patch deployment: Microsoft released a fix within hours of public disclosure, implementing stricter boundaries between user instructions and external content.
Which Controls Failed or Were Missing
Input validation on external content: Copilot treated web-scraped text as trusted input without sanitization. There was no mechanism to distinguish between user commands and externally sourced data.
Output encoding and filtering: The system allowed the AI to generate arbitrary URLs in responses without validating destination domains or scanning for data leakage patterns.
Least privilege for AI agents: Copilot had unrestricted ability to fetch external resources and construct URLs, with no sandboxing between the search function and the conversation context.
Monitoring and anomaly detection: No alerts triggered when the AI began constructing URLs to unfamiliar domains or including conversation data in outbound requests.
Security testing for AI-specific risks: Standard penetration testing and code review didn't catch prompt injection vectors because they weren't in the threat model.
What the Relevant Standards Require
OWASP Top 10 for LLMs (2025) lists prompt injection as LLM01, the highest-priority risk. It specifically calls out indirect injection through external data sources and requires:
- Strict separation between instructions and data
- Input validation on all external content
- Privilege limitations on AI agent actions
NIST AI Risk Management Framework mandates threat modeling that accounts for adversarial manipulation of AI inputs. Organizations must identify failure modes specific to AI systems, not just traditional software vulnerabilities.
ISO/IEC 27001:2022 Annex A.8.16 requires monitoring of system activities. For AI tools processing sensitive data, this means logging what external resources the AI accesses and what data it includes in generated outputs.
PCI DSS v4.0.1 Requirement 6.4.3 mandates that scripts loaded from external sources cannot access sensitive authentication data. While written for payment pages, the principle applies: external content shouldn't gain access to session context or user data.
SOC 2 Type II CC6.1 requires logical access controls that restrict system capabilities to authorized functions. An AI agent with unrestricted web access and URL generation violates least privilege.
Lessons and Action Items for Your Team
Rebuild your threat model for AI integrations. If you've deployed Copilot, ChatGPT plugins, or custom LLM tools, document every external data source they access. Map what happens when that source contains malicious instructions. Update your STRIDE or PASTA model to include "instruction injection through data" as a threat category.
Implement prompt firewalls. Before any external content reaches your AI system, strip HTML, filter special characters, and remove instruction-like patterns ("ignore previous instructions", "system:", "assistant:"). Tools like Rebuff and LLM Guard provide detection libraries, but you'll need custom rules for your specific AI workflows.
Enforce strict output validation. If your AI generates URLs, maintain an allowlist of approved domains. If it can't operate with an allowlist, scan generated URLs for data patterns (email addresses, API keys, conversation snippets) before rendering them to users. Block any URL containing encoded user data.
Apply least privilege to AI capabilities. Your AI doesn't need unrestricted internet access to answer most questions. Limit web searches to approved domains. Disable URL generation unless the specific use case requires it. Separate the AI's search function from its access to conversation history.
Add AI-specific monitoring. Log every external resource your AI accesses, every URL it generates, and every time it references conversation history in an output. Alert on URLs to new domains, especially if they contain query parameters longer than 50 characters—a common exfiltration pattern.
Test for injection during security reviews. Add prompt injection scenarios to your penetration testing scope. Include test cases where external content contains instructions, attempts to access system prompts, or tries to leak data through generated outputs. If your pentest team doesn't have LLM expertise yet, contract specialists or use automated tools like Garak.
Patch AI systems like you patch infrastructure. Microsoft fixed SearchLeak in hours, but you control deployment timing. If you're running on-premise AI tools or self-hosted models, treat model updates and prompt template changes as security patches. Test them in staging, but deploy them on the same SLA as critical security updates.
The SearchLeak attack worked because it exploited the fundamental behavior of AI systems: following instructions found in training data or retrieved content. Until AI providers build robust instruction-data separation into their architectures, your defense depends on limiting what external content reaches the AI and what the AI can do with it.
