Overview of the Issue
This analysis explores the pitfalls when security teams deploy AI-powered vulnerability scanners without understanding their limitations. OpenHack, an MIT-licensed tool by Hadrian, is an open-source security scanner using AI to identify vulnerabilities. It operates within coding environments, using file-based workflows and includes 12 expert families aligned with OWASP and MITRE standards.
The problem: teams expect comprehensive coverage from these AI scanners, but they can't assess runtime access controls, authentication bypass vulnerabilities, or business logic flaws that appear only in production. The scanner reports "clean," and vulnerable code is released.
Timeline of Events
Week 1: The security team integrates OpenHack into the CI/CD pipeline. It performs static analysis on application code, identifying several SQL injection and XSS vulnerabilities. The team addresses these issues.
Week 2-3: Subsequent scans show minimal findings. Confidence in coverage grows, and the code review backlog decreases as reliance on automated scanning increases.
Week 4: The application is deployed to production with authentication middleware that checks user roles but fails to validate role hierarchy. Administrative functions become accessible to any authenticated user who modifies their session token.
Week 6: A security researcher reports the privilege escalation vulnerability through responsible disclosure. The flaw wasn't in the codebase analyzed by the scanner but emerged from interactions between three microservices at runtime.
Missing or Failed Controls
Runtime Security Testing: File-based static analysis can't observe authentication flows during service communication. The team lacked dynamic application security testing (DAST) or interactive application security testing (IAST) to catch runtime issues.
Threat Modeling: There was no mapping of data flows between services to identify where privilege checks should occur. The AI scanner analyzed individual repositories but couldn't assess cross-service authorization.
Code Review for Business Logic: Engineers assumed the scanner's "clean" status meant security was assured. No human reviewed whether the role-checking middleware effectively prevented unauthorized access.
Security Testing in Staging: Scanners were run in CI but security controls weren't validated in an environment mirroring production's service topology.
Relevant Standards
OWASP ASVS v4.0.3, Section 4.1.1: "Verify that the application enforces access control rules on a trusted service layer, especially if client-side access control is present and could be bypassed."
Your AI scanner analyzed code files but didn't verify if your API gateway, service mesh, or middleware enforced those rules when requests arrived.
NIST 800-53 Rev 5, AC-3 (Access Enforcement): "Enforce approved authorizations for logical access to information and system resources in accordance with applicable access control policies."
The control requires enforcement, not just implementation. Authorization code might be perfect, but if deployed incorrectly or bypassed by direct service-to-service calls, AC-3 fails.
PCI DSS v4.0.1, Requirement 6.4.2: "For public-facing web applications, ensure that either automated technical solutions are deployed that continually detect and prevent web-based attacks, or the application is subject to manual or automated vulnerability security testing."
The requirement acknowledges that automated solutions have limits. Sole reliance on static AI-powered scanning doesn't meet the intent when runtime vulnerabilities are missed.
Lessons and Action Items
Action 1: Layer Your Testing Approaches. Use OpenHack and similar AI-powered static analyzers in your CI pipeline, but also include:
- DAST tools for running applications (e.g., OWASP ZAP, Burp Suite)
- Quarterly manual security testing for business logic
- Chaos engineering exercises to test authorization under failure conditions
Action 2: Define Scanner Limitations. Create a matrix: rows are vulnerability classes (OWASP Top 10), columns are testing methods (static, dynamic, manual). Identify which tools cover which classes. The gaps are your risk.
For OpenHack: it excels at finding code-level vulnerabilities like injection flaws. It cannot validate runtime behavior, service-to-service trust boundaries, or configuration errors.
Action 3: Require Threat Models for Multi-Service Features. Before deploying features spanning multiple services, document:
- Which service validates user identity
- Which service enforces authorization
- What happens if service B trusts service A's assertion without re-validation
Action 4: Test in Production-like Environments. Ensure your staging environment mirrors production's architecture. Deploy security testing tools in environments that match your production topology.
Action 5: Calibrate Your Confidence. A scanner reporting zero findings confirms it found nothing within its scope. Train your team to ask: "What could this tool not have seen?"
AI-powered security tools like OpenHack democratize advanced vulnerability research, making it accessible for mid-sized teams. However, accessibility doesn't equal comprehensiveness. Use these tools for code-level vulnerabilities and build complementary controls for broader security coverage.
Your compliance framework likely requires defense in depth. One AI scanner, no matter how sophisticated, isn't depth. It's a single layer. Treat it accordingly.
OWASP ZAP
Burp Suite
