Incident Overview
An AI-powered customer service system fell victim to a malicious prompt that extracted internal knowledge base content and exposed it to an unauthorized user. The attack succeeded because the application relied on post-inference moderation instead of request-time enforcement. By the time the moderation layer flagged the response as problematic, the large language model (LLM) had already processed the hostile instruction and generated the sensitive output.
The application's architecture placed its security decision point after the expensive inference call. When a user submitted a prompt designed to override system instructions, the request passed through authentication and rate limiting without inspecting the prompt content itself. The LLM processed the instruction, the moderation API reviewed the output, and only then did the system attempt to block the response. The damage was already done.
Incident Timeline
T+0 minutes: User submits prompt containing instruction override attempt through authenticated API endpoint.
T+0.3 seconds: Request passes authentication, rate limiting, and input validation checks.
T+0.4 seconds: Application forwards prompt to production LLM without content inspection.
T+2.1 seconds: LLM processes hostile instruction and generates response containing internal documentation.
T+2.8 seconds: Moderation API flags response as policy violation.
T+2.9 seconds: Application blocks response from reaching user.
T+3.2 seconds: Application logs show successful inference call with token consumption.
Post-incident: Review reveals 47 similar attempts over the previous two weeks, all consuming inference tokens before moderation.
Failed or Missing Controls
The application had several security controls, but none addressed prompt content at the enforcement boundary:
Missing pre-inference inspection: No mechanism existed to analyze prompt content before the LLM processed it. The application treated all authenticated requests as safe to forward.
Misplaced moderation layer: The moderation API operated on LLM outputs, not inputs. This architecture ensured that every hostile prompt would consume inference resources and potentially expose sensitive information before any security decision occurred.
No application context integration: The security model didn't use session state, user identity, or request patterns to inform security decisions. Each prompt was evaluated in isolation after inference.
Absent decision point in request lifecycle: The application lacked a gate to reject requests before expensive operations. Authentication verified identity but didn't assess request safety.
Relevant Standards and Requirements
OWASP Top 10 for LLM Applications (2023) lists prompt injection as LLM01, the highest-priority risk. The guidance explicitly calls for input validation and sanitization before prompts reach the model. Waiting until after inference violates this principle.
NIST AI Risk Management Framework emphasizes controls at decision boundaries. Section 2.3 requires organizations to "implement technical controls at points where AI systems interact with external inputs." Post-inference moderation doesn't meet this requirement.
ISO/IEC 27001:2022 Annex A.8.16 requires organizations to monitor and log security events. While this application logged moderation failures, it didn't log or act on suspicious prompt patterns before inference. The control existed but operated too late in the request path.
PCI DSS v4.0.1 mandates that applications validate input before processing, particularly for systems handling sensitive data. If your AI system processes payment card data or related customer information, you can't defer input validation until after the LLM generates output.
SOC 2 Type II CC6.1 requires logical access controls that prevent unauthorized access to information. When your control mechanism only triggers after the AI has already processed and generated a response, you're not preventing access. You're cleaning up after it.
Lessons and Action Items for Your Team
Move Enforcement to the Application Boundary
Inspect prompt content before it reaches your LLM. This isn't about adding another service call. It's about placing your security decision at the first point where you can act on it. If you're using a gateway or API layer, that's where prompt inspection belongs.
Your authentication middleware already sits in the request path. Add prompt content analysis at the same layer. You need a binary decision: forward this to the model or reject it now.
Stop Relying on Moderation as Your Primary Control
Moderation APIs serve a purpose, but that purpose isn't preventing attacks. They're designed to filter outputs for policy compliance, not to block hostile inputs. If moderation is your only AI security control, you're running every attack through to completion before you notice it.
Review your current architecture. If your first security decision happens after model.generate(), you've already lost. Shift that decision point to before the inference call.
Integrate Prompt Inspection with Application Context
Your application already knows who's making the request, what their session state is, and what they're authorized to access. Use that context when evaluating prompts. A request that looks benign in isolation might be clearly hostile when you consider the user's role or recent activity pattern.
Don't treat prompt security as separate from your existing security model. If you've built rate limiting, authentication, and authorization controls, extend them to cover prompt content.
Measure What Matters
Track these metrics weekly:
- Prompts blocked before inference vs. responses blocked after moderation
- Inference token consumption on requests that ultimately got blocked
- Time between prompt submission and security decision
- Percentage of requests that reach your LLM without content inspection
If your "blocked requests" metric only counts post-moderation blocks, you're measuring cleanup, not prevention.
Test Your Enforcement Boundary
Send a test prompt designed to extract system instructions or override context. Where in your request path does it get stopped? If the answer is "after the LLM processes it," you don't have enforcement. You have detection.
Run this test monthly. Your AI security posture isn't static, especially as you add new endpoints or modify prompt templates.
Document Your Decision Points
Map every path a user prompt can take through your application. Mark where security decisions happen. If you can't draw a clear line showing "prompts get inspected here, before this function call," you need to add that line.
This isn't theoretical architecture work. It's operational documentation that your team will reference during the next incident.
