The conventional wisdom
Security teams often treat AI agent vulnerabilities like application bugs. A researcher finds a prompt injection that leaks secrets, the vendor patches the specific exploit, your security scanner goes green, and you move on. This approach has worked for software security for decades.
But with AI agents, it doesn't.
The real issue
The problem isn't the AI model. It's the harness, the code that wraps the model, interprets its outputs, and decides what gets executed. When Elad Meged from Novee Security reported vulnerabilities in Anthropic's pipeline, the company patched them. However, the harness still handed over secrets in its default configuration because the underlying trust model hadn't changed.
You can't patch your way out of an architectural assumption.
Here's what's happening: your AI agent receives a prompt, generates a response, and the harness decides whether to execute it. That decision point, the trust boundary, sits in code you probably didn't write and definitely don't control. When Meged tested Anthropic's claude-code workflow, he found multiple ways to bypass safety checks. Google's Gemini CLI had a vulnerability at the top of the CVSS scale. These aren't edge cases. They're symptoms of a fundamental mismatch between how we think about AI security and how these systems actually work.
The conventional patching approach assumes you can enumerate attack vectors. With AI agents, you're defending against a system that generates novel outputs based on untrusted input. Every patch addresses one specific exploitation path while the harness continues making the same trust decisions that enabled the vulnerability in the first place.
The evidence
Look at what Meged found. The vulnerabilities weren't in Claude's model weights or training data. They were in the workflow configuration, the part that decides "this output is safe to execute" versus "this needs human review." Anthropic's default pipeline configuration made trust decisions that a security engineer would never make if they were writing the code themselves.
This matters because you're probably running default configurations right now. Your AI coding assistant, your automated PR reviewer, your infrastructure-as-code generator, they're all making trust decisions based on vendor assumptions about acceptable risk. Those assumptions might work for a SaaS company with mature security controls. They probably don't work for your environment.
Consider what happens when you integrate an AI agent into your CI/CD pipeline. The agent needs access to your codebase, your environment variables, maybe your cloud credentials. The harness has to decide: when the agent says "run this command," do we execute it? The conventional approach says "add safety checks to the prompt" or "filter the output for dangerous patterns." But you're still delegating the trust decision to code that doesn't understand your threat model.
OpenAI took a different approach with their sandbox. Instead of trying to make perfect trust decisions, they assumed the agent would be compromised and limited what it could access. That's an architectural choice, not a patch.
What to do instead
Start by mapping your trust boundaries. For every AI agent in your environment, document:
- What secrets does it have access to?
- What commands can it execute?
- Who makes the decision to trust its output?
That last question is the critical one. If the answer is "the vendor's harness code," you need to add your own trust layer.
Here's what that looks like in practice. Your AI coding assistant suggests a change. Instead of letting the harness decide whether to apply it, you enforce a rule: any change that touches authentication logic, environment variables, or network configurations goes to human review. You're not trying to detect every possible exploit. You're reducing the blast radius when the trust decision fails.
For agents that need to execute commands, implement explicit allowlisting at the harness level. Don't trust the model to refuse dangerous operations. Don't trust the vendor's safety checks. Write your own wrapper that only permits the specific operations your workflow requires. If your infrastructure agent only needs to read metrics and restart services, it shouldn't have permissions to modify security groups or create IAM roles, regardless of what the model outputs.
This also means rethinking your secrets management. If your agent needs AWS credentials, give it a role with the minimum permissions required for its specific tasks. Use short-lived credentials. Rotate them frequently. When (not if) the agent leaks those credentials, you want them to be useless within hours, not months.
Document your trust model explicitly. Write down which agent outputs you'll execute automatically and which require human approval. Make this a configuration file, not a policy document. Your harness code should enforce these boundaries programmatically.
When the conventional wisdom is right
Patching still matters. When a vendor fixes a specific vulnerability, you should apply that patch. The difference is understanding what the patch actually addresses.
If the patch modifies the model's training or system prompt to refuse certain operations, that's defense in depth, useful but not sufficient. If the patch changes how the harness validates outputs or restricts what commands can be executed, that's an architectural improvement worth prioritizing.
The conventional approach also works when you're using AI agents in low-trust scenarios. A chatbot that answers product questions doesn't need elaborate trust boundaries because it can't execute code or access secrets. The risk is reputational (bad answers) rather than security (leaked credentials).
But the moment your AI agent can modify code, execute commands, or access production systems, you need to think architecturally. Treat the agent as untrusted input to your harness, not as a trusted component of your security boundary. Your harness should enforce trust decisions based on your threat model, not the vendor's assumptions about acceptable risk.
The vendors will keep patching specific exploits. Your job is to build a harness that doesn't rely on perfect safety checks to prevent catastrophic failures. That's not a patching problem. That's an architecture problem.
