Inconsistent Results from LLM Security Reviews
Snyk conducted 250 identical security reviews using Claude on the same JavaScript codebase. The outcome was startling: 80 of 161 unique findings appeared in only one of five repetitions. Despite using the same prompt, code, and model, different vulnerabilities were flagged each time.
This isn't a malfunction; it's how LLMs operate. If your security pipeline relies solely on AI-driven code reviews without a deterministic backup, you're on unstable ground.
The benchmark, VulnBench JS 1.0, compared Claude's performance against Snyk Code (a SAST tool). The highest-recall LLM configuration identified only 81% of the reference vulnerabilities that Snyk Code detected. More concerning: running the same LLM review five times yielded five different result sets.
The Testing Process
Benchmark Setup
Snyk developed VulnBench JS 1.0 with a reference set of known JavaScript vulnerabilities. They configured Claude with identical prompts and ran it against the same codebase 250 times across five repetitions per configuration.
Detection Phase
The LLM identified vulnerabilities across all runs, but consistency was lacking. Of 161 unique findings that didn't match the SAST baseline, half appeared in only one of five identical attempts.
Analysis
When Snyk compared LLM findings to their SAST reference set, even the best-tuned configuration missed 19% of known vulnerabilities. This variability was not an anomaly but a central finding.
Missing Controls in LLM Security Reviews
Deterministic Verification
The main issue: treating LLM output as a standalone security control. LLMs use probabilistic models, sampling from a distribution of possible responses. Running the same analysis twice can yield different results due to this inherent variability.
Your security pipeline assumed repeatability that the tool cannot provide.
Baseline Coverage Validation
The 81% detection rate means one in five known vulnerabilities went undetected even in the best case. If you're using LLM reviews as your primary detection mechanism, you have no way to know which 19% you're missing — and that percentage will shift with each run.
False Negative Tracking
The 80 single-appearance findings indicate a measurement problem. When your tool produces different results on identical inputs, you can't distinguish between:
- Real vulnerabilities it found once and missed four times
- False positives that appeared randomly
- Edge cases where the model's attention mechanism focused differently
Without deterministic behavior, you lose the ability to track false negative rates over time.
Compliance Standards and Their Requirements
PCI DSS v4.0.1 Requirement 6.3.2
"Security vulnerabilities are identified and addressed as follows: New or custom software are reviewed prior to release to production."
The requirement doesn't specify the review method, but it does require that you identify vulnerabilities. When your review tool misses 19% of known issues and produces inconsistent results across runs, you cannot demonstrate that you've met the identification requirement.
OWASP ASVS v4.0.3 Section 1.14
"Verify that all components are up to date with proper security configuration(s) and version(s)."
ASVS expects your security verification to be reproducible. If your compliance auditor asks "how do you verify secure configuration?" and your answer is "we run an LLM review," they'll ask for evidence of consistency. You won't have it.
NIST 800-53 Rev 5 Control RA-5
"The organization scans for vulnerabilities in the information system and hosted applications... and when new vulnerabilities potentially affecting the system/applications are identified and reported."
RA-5 requires scanning and remediation tracking. Deterministic tools let you track: "We scanned on Monday, found 12 issues, fixed 12 issues, rescanned and confirmed zero findings." With non-deterministic LLM results, your scan-fix-verify loop breaks down. You can't confirm remediation if your verification tool gives different answers each time.
Actionable Steps for Your Security Team
1. Use SAST as Your Baseline
Deploy a deterministic SAST tool (Snyk Code, Semgrep, or similar) as your primary detection layer. This provides:
- Repeatable results you can track over time
- A consistent baseline for compliance reporting
- The ability to measure false negative rates
Configure it to run on every pull request and block merges on high-severity findings.
2. Add LLM Review for High-Signal Patterns
LLMs excel at identifying complex exploit chains that SAST tools might miss, such as business logic flaws and authentication bypasses. Use Claude or similar models as a supplementary review for:
- Critical authentication and authorization code
- Payment processing logic
- Any code path that handles PII or cardholder data
Run the LLM review three times on the same code and flag findings that appear in at least two of three runs. This filters out the single-appearance noise while catching patterns the model consistently identifies.
3. Document Your Hybrid Workflow
Your auditor will ask how you ensure secure code. Document:
- "Primary detection: [SAST tool] runs on every PR, blocks merge on high/critical findings"
- "Secondary review: LLM analysis on authentication, payment, and PII-handling code, with manual validation of repeated findings"
- "Verification: SAST rescan after remediation confirms zero findings"
This shows you understand the limitations of each tool and have controls to compensate.
4. Track What Each Tool Catches
For the next quarter, log every vulnerability finding with:
- Source (SAST, LLM, manual review)
- Whether it was a true positive
- Whether other tools detected it
This data will highlight your gaps. If your LLM consistently flags business logic issues your SAST misses, that's valuable. If it's just duplicating SAST findings with less consistency, you can deprioritize it.
5. Never Use LLM Output for Compliance Evidence
When you need to demonstrate "we reviewed this code for vulnerabilities," use your SAST scan results. They're deterministic, timestamped, and repeatable. LLM findings can supplement your internal review process, but they cannot serve as compliance evidence because you cannot reproduce them on demand.
The Snyk benchmark didn't reveal a flaw in LLMs — it highlighted a mismatch between how they work and how security teams need to operate. LLMs are probabilistic; security controls must be deterministic. Use each tool for what it does well, and build your compliance story on the one that gives consistent answers.
