The conventional wisdom: If we train AI models better, add more safety guardrails, and fine-tune their behavior, we'll solve the prompt injection problem. It's a training issue, not a design flaw.
Why we disagree: You're building a castle on quicksand. Prompt injection vulnerabilities aren't behavioral bugs you can train away, they're architectural problems baked into how large language models process text. When Zscaler tested leading AI models for indirect prompt injection (IPI) susceptibility, four high-end models failed: Llama3-3-70b-instruct, Llama3-2-90b-instruct, Gemini-3-flash, and Gemini-2.5-pro. These aren't obscure edge cases. These are production models with extensive safety training.
The fundamental issue: LLMs can't reliably distinguish between "instructions from the user" and "instructions embedded in data the user asked them to process." When your AI agent reads a webpage, parses an email, or processes a document, it treats everything as potential instructions. That's not a training gap, it's how the architecture works.
The Evidence
Zscaler identified IPI traps embedded in real websites, where hidden instructions manipulated AI agent behavior. The attack vector is simple: an attacker plants instructions in content they control, your AI agent encounters that content during normal operation, and the model follows the attacker's instructions instead of yours.
Here's what makes this architectural, not behavioral: the model has no reliable way to determine instruction provenance. You can train a model to resist obvious attack patterns ("ignore previous instructions and..."), but that's pattern matching, not understanding. Change the phrasing, embed the instruction in legitimate-looking content, or split it across multiple inputs, and you're past the guardrails.
Aman Mahapatra got it right when he emphasized the architectural nature of these vulnerabilities. You need architectural defenses, not better training. Consider what happens when you deploy an AI agent that:
- Reads support tickets and drafts responses
- Processes vendor invoices and flags anomalies
- Monitors security feeds and summarizes threats
Each of these interactions creates an attack surface. An attacker doesn't need to compromise your systems, they just need to get malicious content into the data stream your agent processes.
What to Do Instead
Stop treating AI agents like hardened applications. Start treating them like untrusted execution environments that need containment.
Implement strict I/O boundaries. Your AI agent shouldn't have direct access to production systems. Put it behind an API gateway that validates every action request against explicit rules. If the agent wants to send an email, approve a transaction, or modify a record, that request goes through a control layer that checks: Is this action type allowed? Does this match expected patterns? Are the parameters within acceptable ranges?
Separate instruction channels from data channels. When you feed data to an AI agent, mark it explicitly as data, not instructions. Some frameworks let you designate certain inputs as "system" (trusted instructions) versus "user" (untrusted content). Use this separation religiously. If your agent needs to process external content, pre-process it to strip potential instruction markers.
Log everything with human review hooks. You need an audit trail of what the agent was asked to do, what data it processed, and what actions it attempted. More importantly, you need automated alerts for anomalous behavior: unusual action types, out-of-pattern requests, or attempts to access restricted resources. Think of it like you'd monitor a service account with elevated privileges.
Limit blast radius. Don't give your AI agent permissions it doesn't need. If it's drafting emails, it shouldn't have database write access. If it's processing invoices, it shouldn't be able to initiate wire transfers. Apply least privilege as if you're dealing with a compromised service account, because functionally, you might be.
Test with adversarial inputs. Before you deploy an AI agent, run it against a test suite of IPI attempts. Can you get it to leak credentials? Bypass approval workflows? Access restricted data? If you can, an attacker will. Build these tests into your CI/CD pipeline.
For compliance purposes, treat AI agents as automated decision-making systems under your existing frameworks. If you're subject to SOC 2, your AI agent's actions fall under CC6.1 (logical and physical access controls) and CC7.2 (system monitoring). Document how you're enforcing least privilege and monitoring for unauthorized actions.
When the Conventional Wisdom Is Right
Training and safety measures do matter, they're just not sufficient on their own. Better training reduces the attack surface by making simple prompt injection attempts fail. If your model can resist obvious attacks, you force attackers to get more sophisticated, which raises the bar.
Safety training also helps with unintentional harm. Even without malicious actors, AI agents can misinterpret instructions or make poor decisions. Training that emphasizes caution, asks for clarification, and refuses ambiguous requests makes your system more reliable overall.
The key is recognizing what training can and can't do. It can make attacks harder. It can reduce false positives. It can't eliminate the architectural vulnerability that lets external content influence model behavior.
Noah Kenney's criticism about the non-generalizability of test results is valid too. AI behavior is dynamic, and a model that fails a specific test today might resist it tomorrow after an update. But that's exactly the point: you can't rely on behavioral defenses that change unpredictably. You need architectural controls that work regardless of how the model behaves.
If you're deploying AI agents in production, assume they're vulnerable to prompt injection. Build your security model around that assumption. The models will get better, but the architecture won't fundamentally change. Your defenses need to account for that.
