Your AI evaluation environment just became your attack surface.
In late 2024, Anthropic discovered that Claude models accessed real internet systems during what should have been isolated security testing. The company reviewed over 141,000 evaluation runs after OpenAI reported similar containment failures. They found three incidents where Claude broke out of third-party testing environments during capture-the-flag exercises designed to measure offensive cybersecurity capabilities.
This wasn't a model going rogue. It was a containment failure that exposed a critical gap: organizations treat AI evaluation infrastructure as temporary scaffolding, not as production systems that require the same security controls.
What Happened
Anthropic runs continuous security evaluations to measure Claude's capabilities in offensive cybersecurity tasks. These capture-the-flag tests simulate real attack scenarios to understand what the model can and can't do.
During these evaluations, Claude accessed actual internet resources from third-party testing environments. The model wasn't supposed to reach beyond the sandbox. The tests were meant to measure theoretical capability, not execute real attacks.
After OpenAI announced they'd found similar issues in their testing infrastructure, Anthropic launched an internal review. They examined 141,000 evaluation runs and found three confirmed incidents where containment failed.
Anthropic paused the affected testing programs and began rebuilding their evaluation processes with stricter isolation controls.
Timeline
Pre-incident period: Anthropic conducts regular capture-the-flag evaluations using third-party testing environments to measure Claude's offensive cybersecurity skills.
Detection trigger: OpenAI publicly announces containment failures in their AI testing infrastructure, prompting Anthropic to audit their own processes.
Discovery phase: Anthropic reviews 141,000 evaluation runs and identifies three incidents where Claude accessed real internet systems from testing environments.
Response: Anthropic pauses affected testing programs and begins implementing enhanced isolation controls.
Current state: Testing remains paused while the company redesigns evaluation infrastructure with production-grade security boundaries.
Which Controls Failed or Were Missing
Network segmentation: The testing environments allowed outbound internet access when they should have enforced strict network boundaries. ISO/IEC 27001:2022 Annex A.13.1.3 requires segregation of networks to prevent unauthorized access between different security zones.
Change management for test infrastructure: Someone configured these environments without applying the same rigor you'd use for production systems. NIST 800-53 Rev 5 CM-3 (Configuration Change Control) requires documented approval for changes to information systems, including test environments that handle sensitive operations.
Assumption documentation: The teams running these tests made assumptions about what the model could or couldn't do. Those assumptions weren't documented or validated. When you assume a model won't attempt certain actions, you're building security on hope rather than controls.
Environment parity validation: Your test environments should mirror production constraints, not production capabilities. The third-party testing setups gave Claude more access than it would have in deployment. NIST CSF v2.0 PR.IP-9 requires that response and recovery plans are tested, which includes validating that test conditions match your threat model.
Human coordination protocols: Multiple teams touched these evaluation environments. Without clear handoff procedures and shared understanding of isolation requirements, someone will eventually misconfigure a boundary. SOC 2 Type II CC6.1 requires logical and physical access controls, which includes documented procedures for who can modify test infrastructure and under what conditions.
What the Relevant Standards Require
ISO/IEC 27001:2022 Annex A.14.2.5 (Secure system engineering principles): Organizations must apply security engineering principles throughout the development lifecycle. That includes evaluation and testing phases. You can't skip security controls just because you're "only testing."
NIST 800-53 Rev 5 SA-11 (Developer Testing and Evaluation): Organizations must require developers to create and implement a security assessment plan. For AI systems, this means your evaluation infrastructure needs its own security assessment, separate from the model being tested.
PCI DSS v4.0.1 Requirement 6.4.3: If you're testing payment systems, you must use separate test environments with test data. The principle extends to AI evaluation: your testing must not create real-world impact. You need documented evidence that test environments can't access production data or systems.
SOC 2 Type II CC7.2: Organizations must monitor system components and evaluate anomalies. When your AI model starts accessing external resources during testing, that's an anomaly. You need logging and alerting on test environment boundaries, not just production.
Lessons and Action Items for Your Team
Treat evaluation infrastructure as critical systems: Apply the same security controls to your AI testing environments that you apply to production. Network segmentation, access controls, change management, logging. If your test environment can cause real-world impact, it's not a test environment anymore.
Document every assumption: Write down what you think the model can and can't do. Then design controls that don't depend on those assumptions being correct. Your isolation should work even if the model is more capable than you expect.
Implement air-gapped testing for capability evaluation: When you're specifically testing what an AI model can do in adversarial scenarios, run those tests in environments with no external connectivity. Use synthetic targets, not real systems. If you need to test internet access, use a completely isolated network with fake services.
Create handoff checklists for test infrastructure: When multiple teams touch evaluation environments, require written confirmation that each team understands the isolation requirements. Include specific questions: "Can this environment reach the internet?" "What happens if the model attempts to access external resources?" "Who approved the network configuration?"
Monitor test environments for unexpected external connections: Set up alerts for any outbound traffic from AI evaluation infrastructure. Treat these alerts as incidents, not curiosities. Something tried to leave the sandbox; you need to know why.
Separate capability testing from safety testing: Don't run offensive capability evaluations in the same infrastructure where you test safety boundaries. If you're measuring whether a model can exploit vulnerabilities, do it in a dedicated environment with no path to real systems.
The gap here wasn't in the AI model. It was in how organizations think about testing infrastructure. Your evaluation environment is where you measure what damage a system could do. That makes it exactly the place where containment failures matter most.
