What Happened
Google's Mandiant used an AI-driven tool called the Agentic Vulnerability Discovery Harness (AVDH) to scan production codebases. In just two days, it found over 100 verified, high-severity vulnerabilities. This tool has been operating inside Mandiant for ten months, scanning tens of millions of lines of code.
This isn't a typical breach analysis. It's a controlled experiment revealing how much critical code remains unpatched because your current tools can't efficiently filter out false positives.
Timeline
Here's a brief overview of the events:
- Months 1-10: AVDH operates internally at Mandiant, scanning extensive codebases.
- Test Period: A two-day focused assessment on target codebases.
- Outcome: Over 100 verified high-severity flaws identified.
- Post-Discovery: Mandiant researchers publish findings, emphasizing the need for manual validation.
Which Controls Failed or Were Missing
This wasn't a single team's failure. It's a systemic issue in vulnerability management:
Static Analysis Noise Overwhelmed Human Capacity. Your SAST tools have been identifying vulnerabilities for years, but they also produce thousands of false positives. When the backlog grows too large, engineers lose trust in the scanner and may ignore it.
Vulnerability Review Cadence Couldn't Keep Pace with Code Velocity. PCI DSS v4.0.1 Requirement 6.3.2 requires reviewing custom code before release to identify security vulnerabilities. If your review process takes weeks while you're deploying daily, you're not compliant.
No Domain-Specific Rule Sets. Generic SAST rules miss context. A SQL query might be dangerous in one context but safe in another. Your scanner can't differentiate because it doesn't understand your architecture.
Human Validation Was a Bottleneck, Not a Checkpoint. Mandiant researchers note that manual validation is essential. However, if it's your only quality gate, humans can't review thousands of findings effectively.
What the Relevant Standard Requires
Let's align this with the requirements you should already be meeting:
PCI DSS v4.0.1 Requirement 6.3.2: Custom code must be reviewed before production release to identify security vulnerabilities. The focus is on identifying actual vulnerabilities, not just generating alerts.
OWASP ASVS v4.0.3, V14.2.1: All code changes must be reviewed for security issues. Your review process should match your deployment speed to avoid accumulating security debt.
ISO/IEC 27001:2022, Control 8.25: Secure development lifecycle requirements include vulnerability assessments during development. The process must find vulnerabilities before they reach production.
NIST 800-53 Rev 5, SA-11: Developer security testing must include static code analysis. SA-11(1) specifies using tools that identify common flaws and document results.
Lessons and Action Items for Your Team
Here's how you can improve, starting next sprint:
1. Audit Your False Positive Rate. Review your SAST findings from the last quarter. How many were false positives? How many were ignored? Knowing these numbers is crucial to understanding if your tools are effective.
2. Build Domain-Specific Rule Sets. Different systems require different rules. Start with one critical system, define what "vulnerable" means, and configure your scanner accordingly. Expand this approach over time.
3. Implement Human-in-the-Loop Validation at the Right Stage. Don't have humans validate thousands of findings. Use automation to narrow down to high-confidence candidates, then validate those.
4. Measure Time-to-Validation, Not Just Time-to-Detection. Finding a flaw is just the start. Track how long it takes to validate and fix it. If you're stuck in a backlog, your process needs change.
5. Test Your Review Process Against Deployment Velocity. If you're deploying daily but reviews take weeks, either slow down deployments or speed up reviews. AI-assisted tools can help here.
6. Document What "Verified" Means in Your Context. When Mandiant says they found "verified, high-severity flaws," they mean manual confirmation of real, exploitable vulnerabilities. Define "verified" in your pipeline and make it repeatable.
The AVDH results highlight how much vulnerable code is in production because your current process can't keep up. You don't need to wait for a commercial release of Google's tool. Fix the process that's letting critical vulnerabilities slip through. Start by addressing the false positive rate. Everything else follows from there.
