What Happened
Security researchers have found that modern LLMs rely heavily on role tags—such as system prompts and user designations—as their primary security framework. However, these tags do not persist into the model's internal workings. When text is marked as "system" versus "user" input, the model processes both as continuous token streams without meaningful security boundaries.
This creates a vulnerability that prompt engineering cannot resolve. Attackers can embed instructions in user-controlled content—file uploads, API inputs, web scraping results—that the model treats with the same authority as your system prompt.
Timeline
This vulnerability class emerged as organizations began integrating LLMs into production systems:
Initial deployment phase: Teams implemented LLMs with role tags to define trusted instructions, such as "You are a customer service assistant. Never reveal internal data," separated from user input.
First exploitation: Attackers discovered that user input containing phrases like "Ignore previous instructions" could override the original role boundaries.
Defense attempts: Security teams added input validation, output filtering, and more sophisticated prompt structures. However, each defense led to new bypass techniques.
Current state: Organizations continue deploying LLM features while the underlying architecture remains vulnerable to injection attacks that exploit the lack of genuine role perception.
Which Controls Failed or Were Missing
The failure is architectural:
Input validation failed: Traditional injection defenses assume you can identify and sanitize malicious patterns. With prompt injection, the "malicious" content is natural language that looks identical to legitimate user input. You can't simply filter out "Please summarize this document that happens to contain instructions to email all previous conversation history to [email protected]."
Boundary enforcement failed: The model treats system prompts and user input as a single continuous context window. There's no privilege separation or security context switch.
Output controls failed: Even if you filter the response, the model has already processed the injected instructions. Side effects—API calls, database queries, external tool invocations—occur before you see the output.
Least privilege failed: LLM-powered features often run with broad permissions because the model needs context to be useful. When injection succeeds, the attacker inherits those permissions.
What the Relevant Standards Require
No compliance framework anticipated this architecture, but existing requirements expose the gaps:
OWASP ASVS v4.0.3, Requirement 5.1.1 mandates that input validation is applied on a trusted service layer. Your LLM processes untrusted input directly—there is no trusted layer between user content and instruction execution.
NIST 800-53 Rev 5, Control AC-3 (Access Enforcement) requires the system to enforce approved authorizations for logical access. When user input can modify system behavior through injection, you've lost access control enforcement.
ISO/IEC 27001:2022, Control 8.3 (Information Access Restriction) demands that access to information and other associated assets is restricted in accordance with the established topic-specific policy. Prompt injection bypasses these restrictions by manipulating the model's interpretation of its own access rules.
PCI DSS v4.0.1, Requirement 6.2.4 requires that bespoke and custom software is developed securely. If your payment processing includes LLM features for dispute resolution or fraud detection, prompt injection could expose cardholder data through manipulated queries.
The standards assume you can separate trusted code from untrusted data. LLMs blur this boundary at the architectural level.
Lessons and Action Items for Your Team
The paper's conclusion—that without genuine role perception, injection defense remains a perpetual challenge—means you need defense in depth, not perfect prevention:
1. Treat LLM outputs as untrusted
Never pass LLM responses directly to privileged operations. If your model generates SQL, validate it against a whitelist of allowed patterns. If it calls APIs, use a restricted service account with minimal permissions. Build the same controls you'd use for any untrusted external input.
2. Isolate LLM features from sensitive data
Don't give your chatbot direct database access. Create a read-only API layer that returns only the specific data types the feature needs. If the model gets compromised through injection, limit what an attacker can reach.
3. Log everything with injection detection
Capture the full context: system prompt, user input, model output, and any tool invocations. Build detection for suspicious patterns—output that contradicts the system prompt, attempts to access out-of-scope data, or responses that echo injection-style phrasing. This won't prevent attacks, but it surfaces them faster.
4. Design for failure
Assume injection will succeed. What's your blast radius? If an attacker can manipulate your model's behavior, can they exfiltrate customer data? Modify records? Escalate privileges in connected systems? Design boundaries that contain the damage.
5. Document the risk in your compliance artifacts
Your SOC 2 Type II report needs to acknowledge this. Under CC6.1 (logical and physical access controls), document that LLM features lack true role-based access control at the model level, and describe your compensating controls. Auditors are starting to ask about AI security—have an answer ready.
6. Avoid LLMs for security-critical decisions
Don't use a model to approve transactions, grant access, or validate security policies. The injection risk is too high. Keep LLMs in advisory roles where a compromised output is inconvenient, not catastrophic.
The research confirms what many security engineers suspected: role tags are prompt engineering, not security engineering. Until LLMs develop genuine role perception—internal representations that distinguish trusted instructions from untrusted data—you're building on a foundation that attackers can manipulate through carefully crafted text. Design your systems accordingly.
