On Wednesday, PortSwigger released the public beta of Burp AT, an agentic AI pentesting tool. Within hours, security teams started asking a critical question: what happens when your AI pentester doesn't stay in its lane?
This isn't a typical breach analysis. No data leaked, and no ransomware was deployed. However, the release of Burp AT highlights a new type of incident: autonomous security tools operating outside their intended boundaries. The risks are real, and the controls needed to prevent them align with existing compliance frameworks.
What Happened
PortSwigger integrated agentic AI capabilities into Burp Suite, creating a tool that autonomously discovers vulnerabilities, chains exploits, and adapts its testing methods in real-time. Unlike traditional scanners that follow set patterns, Burp AT uses large language models to understand application behavior and generate new attack sequences.
The critical design decision: Burp AT operates behind a deterministic control layer that enforces scope boundaries, tool access restrictions, and approval workflows. Every AI action passes through this layer before execution. This control layer is a hard gate that blocks operations outside defined parameters.
This architecture exists because PortSwigger's CEO, Dafydd Stuttard, recognized the "beast needs a cage" problem. An AI with full autonomy and access to pentesting tools could easily escalate beyond intended scope, trigger production incidents, or execute destructive tests without human review.
Timeline of Risk
The incident pattern we're preventing looks like this:
T+0 minutes: Security engineer launches AI pentesting agent against staging environment.
T+15 minutes: AI identifies authentication bypass, begins automated exploitation.
T+22 minutes: AI discovers staging shares database connection pool with production.
T+23 minutes: AI attempts SQL injection against what it believes is staging data.
T+24 minutes: Production database locks under unexpected query load.
T+25 minutes: Customer-facing application returns 500 errors.
T+30 minutes: Engineer realizes AI crossed environment boundary.
This hasn't happened with Burp AT because of its control layer. But it's exactly what happens when you deploy autonomous agents without deterministic boundaries.
Which Controls Failed or Were Missing
In the hypothetical scenario above, several control failures occur:
Scope enforcement: No technical control prevents the AI from targeting out-of-scope systems. The engineer defined scope in natural language ("test the staging API"), but the AI couldn't validate system boundaries before acting.
Approval workflows: Destructive or high-risk operations executed without human review. The AI decided on its own that SQL injection testing was appropriate for the context.
Environment isolation: Staging and production shared infrastructure, allowing scope creep to impact live systems. This is an architecture problem, but it's worsened when autonomous tools can discover and exploit those connections faster than humans can intervene.
Audit logging: No record of which specific AI decisions led to which actions. When the incident occurred, the team couldn't reconstruct the AI's reasoning path or identify where the scope violation originated.
What the Standards Require
These aren't hypothetical controls. Current compliance frameworks already mandate the safeguards needed for autonomous security tools:
ISO/IEC 27001:2022 Control 5.23 requires technical controls to prevent unauthorized access between environments. Your AI pentester needs the same network segmentation and access controls you'd apply to any external testing vendor.
NIST 800-53 Rev 5 Control SA-11 mandates that security testing "does not result in unintended consequences to operational systems." If your AI can't guarantee it won't touch production, it fails this requirement.
PCI DSS v4.0.1 Requirement 11.4.7 states that segmentation controls must be tested at least once every six months. An AI that can autonomously discover network paths between environments could invalidate your segmentation testing if it's not properly scoped.
SOC 2 Type II CC6.6 requires that system access is restricted to authorized users and programs. An autonomous AI with broad tool access needs explicit authorization boundaries, not just human oversight.
The control layer in Burp AT implements these requirements through deterministic rules. Before the AI can execute any action, the control layer verifies:
- Is this target within the defined scope?
- Does this tool require approval for this context?
- Has a human reviewed and authorized this specific operation?
If any check fails, the action is blocked. The AI doesn't get to argue its case or try alternative approaches. The gate stays closed.
Lessons and Action Items
If you're evaluating AI-assisted security tools or building automation into your testing workflows, here's what you need to implement:
Define scope in machine-readable format. Natural language descriptions ("test the API") aren't sufficient. Use CIDR blocks, hostname patterns, or API endpoint lists that a control layer can validate programmatically. Document these in your test plan and verify them before execution.
Implement approval gates for destructive actions. Any operation that modifies data, consumes significant resources, or could impact availability requires human review. Configure your tools to pause and request approval rather than proceeding autonomously.
Separate your testing infrastructure. If staging shares any resources with production -- databases, message queues, API gateways -- your autonomous tools will eventually find that connection. Fix the architecture problem before you deploy the AI.
Log AI decision paths, not just actions. When an incident occurs, you need to reconstruct why the AI chose a particular test sequence. Require your tools to log their reasoning, not just the commands they executed.
Test your control layer. Don't assume the AI's guardrails work as documented. Deliberately configure out-of-scope targets and verify that the control layer blocks them. Include this in your regular security validation cycles.
The shift to agentic AI in security testing isn't optional. These tools will find vulnerabilities human testers miss and encode methodologies that previously required deep expertise. But autonomy without deterministic control isn't innovation -- it's an incident waiting to happen.
Your AI pentester needs a cage. Make sure you build one before you need it.
