Scope - What This Guide Covers
This guide provides the security controls and response protocols your team needs when deploying large language models (LLMs) in production environments. The rapid jailbreaking of models like Anthropic's Fable 5 highlights vulnerabilities that can affect any AI system integrated into your infrastructure.
You'll find specific guidance for:
- Pre-deployment security assessments for LLM integrations
- Runtime monitoring and constraint enforcement
- Incident response protocols when model guardrails fail
- Compliance mapping to existing security frameworks
This guide focuses on what you can control as the security engineer responsible for AI systems in production, not on model training security or adversarial machine learning research.
Key Concepts and Definitions
Jailbreaking: Techniques that bypass an AI model's safety constraints to produce outputs the model was designed to refuse. Jailbreaks often use natural language manipulation rather than code injection.
Guardrails: Output filters, input validation rules, and behavioral constraints implemented by model providers. These operate at the API layer, not within the model weights.
Prompt Injection: A class of attacks where user input manipulates the model's instruction context, similar to SQL injection but targeting language model context windows.
Constitutional AI: Training methodology that attempts to build safety constraints into model behavior. The Fable 5 incident shows that trained-in safety alone is insufficient.
Requirements Breakdown
Map your AI security controls to your existing compliance obligations:
NIST CSF v2.0 - Govern (GV) Function
GV.RR-03 requires you to determine and communicate legal, regulatory, and contractual requirements regarding cybersecurity. When deploying an LLM, document which requirements apply to AI-generated outputs. If your model processes payment data, PCI DSS v4.0.1 Requirement 6.4.3 applies to any scripts or code the model generates.
ISO/IEC 27001:2022 - Control 8.16 (Monitoring Activities)
Monitor AI system outputs as you would any automated system with security implications. Set up logging for prompt patterns, output filtering events, and guardrail bypass attempts.
SOC 2 Type II - CC6.1 (Logical and Physical Access Controls)
Treat your AI model API keys as credentials. Rotate them, scope them to minimum necessary permissions, and log their usage.
Implementation Guidance
Pre-Deployment Assessment
Before integrating any LLM into your production environment:
1. Threat model the integration point
Identify where user input enters the prompt and what downstream systems consume the model's output. A model generating SQL queries presents different risks than one drafting customer emails.
2. Test guardrail effectiveness
Don't rely solely on the provider's safety claims. Run adversarial prompts against the specific use case. Document what gets through and what doesn't.
3. Build your own output validation
Never trust model output directly. If using an LLM to generate firewall rules, parse and validate the output before applying it. If generating customer communications, scan for PII leakage and policy violations.
Runtime Controls
Input sanitization layer
Strip or escape characters commonly used in jailbreak attempts. Filter obvious patterns: instructions to "ignore previous instructions," requests to "roleplay" as an unrestricted system, or attempts to inject system-level commands.
Output filtering
Implement keyword scanning, pattern matching, and anomaly detection on model outputs. Flag outputs that are significantly longer than baseline or match sensitive data patterns.
Rate limiting and quotas
Jailbreak attempts often involve iterative testing. Implement per-user rate limits on API calls. If someone makes 50 requests in 60 seconds with varying prompts, that's a signal.
Monitoring and Detection
Set up alerts for:
- Repeated output filter triggers from the same user or API key
- Prompt patterns matching known jailbreak techniques
- Unusual output length, format, or content type
- API errors indicating guardrail enforcement
Log these events with enough context to investigate: user ID, timestamp, sanitized prompt, output classification, and which filters triggered.
Common Pitfalls
Assuming provider guardrails are sufficient
The Fable 5 jailbreak shows that even well-resourced AI labs cannot prevent all safety bypasses. Layer your own controls on top of provider safeguards.
Treating AI outputs as trusted
Model outputs are user-generated content. Apply the same validation you'd use for any untrusted input before acting on them.
Security through obscurity
Hiding your system prompts or hoping attackers won't find your API endpoints is not a control. Assume your prompts will leak and your endpoints will be probed.
Overlooking indirect prompt injection
If your model processes external content (emails, documents, web pages), attackers can embed instructions in that content. A malicious email could contain hidden instructions that manipulate your email-processing AI.
Compliance checkbox approach
Mapping AI systems to ISO/IEC 27001:2022 or SOC 2 Type II isn't about checking boxes. It's about applying the same risk-based thinking you use for traditional systems.
Quick Reference Table
| Control Type | Implementation | Relevant Standard |
|---|---|---|
| Input validation | Strip jailbreak patterns, limit prompt length, sanitize special characters | OWASP ASVS v4.0.3 (V5.1.1) |
| Output filtering | Keyword scanning, PII detection, content classification | ISO/IEC 27001:2022 (8.16) |
| Access control | API key rotation, least-privilege scoping, MFA for admin access | SOC 2 Type II (CC6.1) |
| Monitoring | Prompt/response logging, guardrail bypass detection, rate limit violations | NIST CSF v2.0 (DE.CM-01) |
| Incident response | Escalation path for jailbreak attempts, model version rollback procedure | NIST 800-53 Rev 5 (IR-4) |
| Documentation | Approved use cases, risk assessment, control mapping | ISO/IEC 27001:2022 (5.1) |
When Jailbreaks Happen
Despite your controls, assume jailbreaks will occur. Your incident response plan should include:
Immediate containment: Can you disable the affected model integration without breaking critical business functions? Document your rollback procedure now.
Evidence collection: Preserve the prompts, outputs, and any intermediate states. You'll need this for root cause analysis and potential provider escalation.
Impact assessment: What data did the jailbroken model access? What outputs reached downstream systems or users? Map this to your data classification scheme.
Provider notification: If you're using a third-party model, your contract may require breach notification. Know your SLA terms.
The speed of the Fable 5 jailbreak—days, not months—tells you that your detection and response timelines must compress accordingly. Monthly security reviews aren't sufficient when vulnerabilities emerge this quickly.
AI security best practices
