What Happened
A customer-facing LLM chatbot bypassed its safety guardrails when a user submitted input that overrode the system's original instructions. The bot disclosed internal prompts, leaked customer data from its context window, and generated responses that violated the company's brand guidelines. The incident went undetected for 48 hours until customers began posting screenshots on social media.
The chatbot had passed traditional security testing. Penetration testers found no SQL injection vulnerabilities, no authentication bypasses, and no exposed APIs. Every conventional security control was in place. Yet the application failed when someone typed a few sentences of natural language.
Timeline
Day 1, 14:23 UTC: User submits a message containing instructions to "ignore previous directions" followed by a request to reveal the system prompt.
Day 1, 14:24 UTC: Bot responds with its complete system prompt, including internal company terminology and customer handling procedures.
Day 1, 16:45 UTC: Same user experiments with additional prompts. Bot begins generating responses outside its intended scope, including recommendations for competitor products.
Day 2, 09:12 UTC: Multiple users discover the technique through online forums. Traffic to the chatbot increases by 340%.
Day 2, 11:30 UTC: Bot inadvertently includes personally identifiable information from another customer's conversation in its response, visible in the context window.
Day 2, 19:45 UTC: Screenshots appear on Twitter. Engineering team becomes aware.
Day 2, 20:15 UTC: Chatbot taken offline.
Day 3, 08:00 UTC: Incident response team confirms approximately 1,200 conversations contained leaked system prompts or inappropriate responses.
Which Controls Failed or Were Missing
The application had no controls specific to LLM security. The security team had implemented:
- Input validation for SQL injection, XSS, and command injection
- Rate limiting on API endpoints
- Authentication and authorization checks
- Network segmentation
- Logging and monitoring for traditional attack patterns
None of these controls address prompt injection. The monitoring system didn't flag the malicious inputs because they contained no special characters, no SQL syntax, no JavaScript. They were plain English.
The application also lacked:
Output validation for LLM responses: No mechanism checked whether the model's output matched expected patterns or contained sensitive information before displaying it to users.
Prompt isolation: User input was concatenated directly with system instructions, allowing attackers to manipulate the model's behavior.
Context window sanitization: Customer data from previous conversations persisted in the context without checks for cross-contamination.
Behavioral monitoring: No alerts triggered when the bot's responses deviated significantly from its training or when it began exposing internal information.
What the Relevant Standard Requires
OWASP Top 10 for LLM Applications (2023 edition) lists Prompt Injection as LLM01. The guidance specifies that applications should implement privilege control, separate user instructions from system prompts, and validate outputs before displaying them.
ISO/IEC 27001:2022 Annex A.8.16 requires monitoring and logging of activities. For LLM applications, this means instrumenting the model to detect when outputs contain unexpected patterns, internal terminology, or data that shouldn't be exposed.
NIST AI Risk Management Framework (AI RMF 1.0) addresses this under the "Manage" function: "AI system outputs are monitored and assessed for validity, reliability, and safety." Your LLM's outputs are system outputs. They require the same rigor you apply to any other application component.
If you're handling payment data, PCI DSS v4.0.1 Requirement 6.4.3 mandates that scripts and applications are reviewed before deployment to prevent common vulnerabilities. For LLM applications, this review must include testing for prompt injection, jailbreaking, and data leakage through the context window.
SOC 2 Type II CC6.6 requires logical access controls to restrict access to data. When your LLM can access customer data through its context window, you need controls that prevent one user's prompt from exposing another user's information.
Lessons and Action Items for Your Team
Stop treating LLMs like traditional applications. Your web application firewall won't catch prompt injection. Your SQL injection scanner won't find jailbreak attempts. You need different tools.
Implement output validation immediately. Before any LLM response reaches a user:
- Check for patterns that indicate system prompt leakage (internal terminology, configuration details, instruction keywords)
- Scan for PII that shouldn't be in this user's session
- Flag responses that deviate significantly from expected topics or tone
Separate system instructions from user input. Don't concatenate user messages directly with your system prompt. Use structured formats that clearly delineate between instructions and data. Some frameworks support this natively; if yours doesn't, implement it at the application layer.
Sanitize your context window. If your LLM maintains conversation history:
- Implement strict session isolation
- Remove or redact sensitive data before adding messages to context
- Set maximum context lengths to limit exposure
- Clear context when switching between users or topics
Add LLM-specific monitoring. Your existing SIEM won't help here. Instrument your application to detect:
- Responses that contain instruction-like language
- Outputs that reference system components or internal processes
- Unusual response lengths or formats
- Repeated attempts to manipulate the model's behavior
Test for LLM vulnerabilities in your security program. Add prompt injection, jailbreaking, and context manipulation to your penetration testing scope. If your current pentest vendor doesn't understand these attack vectors, find one that does or train your team.
Document your LLM's intended behavior. You can't detect deviation if you haven't defined normal. Create clear specifications for:
- What topics the model should and shouldn't discuss
- What data it can access and under what conditions
- What format responses should take
- What constitutes an unsafe or inappropriate output
This incident happened because the security team applied traditional controls to a non-traditional system. Your LLM isn't a web application. It's a statistical model that generates text based on patterns. It doesn't understand "don't leak secrets." It needs different guardrails.
Start building them today.
