Scope
This guide addresses the security implications of AI-assisted development tools like GitHub Copilot, Amazon CodeWhisperer, and Cursor on your application security program. It covers changes in threat modeling, testing requirements, and control adjustments necessary when developers generate 30-50% of their code through large language model (LLM) prompts.
If your team has adopted AI coding tools in the last year, you'll need different controls than before.
Key Concepts
AI-Generated Code: Source code produced by LLMs in response to natural language prompts or partial code context. Unlike traditional code generation, the output is non-deterministic and trained on public repositories.
Prompt Injection in Development: Occurs when an LLM incorporates malicious patterns from its training data or context window into generated code. This happens at write-time, not runtime.
Dependency Hallucination: AI tools may suggest libraries, packages, or APIs that don't exist, creating typosquatting opportunities.
Context Leakage: Proprietary code or secrets fed into an AI tool's context window could be stored or reproduced in other users' outputs.
Requirements Breakdown
Your existing controls don't adequately cover AI-generated code. Here's what changes:
OWASP ASVS v4.0.3 Verification 14.2 (Dependency)
Standard: "Verify that all components are free from known vulnerabilities."
AI addition: Verify suggested dependencies exist in official registries before installation. AI tools suggest non-existent packages at rates you wouldn't see with human developers.
PCI DSS v4.0.1 Requirement 6.3.2
Standard: "An inventory of bespoke and custom software, and third-party software components is maintained."
AI addition: Tag AI-generated code blocks in your inventory. When a training data lawsuit or license issue emerges, you need to know which modules were human-written vs. generated.
SOC 2 Type II CC6.6 (Logical Access)
Standard: "The entity implements logical access security software."
AI addition: Your access controls must now cover AI tool configurations. Which developers can use tools that send code to external APIs? What's allowed in the context window?
NIST CSF v2.0 ID.RA-5 (Threat Intelligence)
Standard: "Threats, both internal and external, are identified and documented."
AI addition: Document threat scenarios specific to AI code generation, such as training data poisoning, context manipulation, and suggested vulnerable patterns.
Implementation Guidance
Phase 1: Visibility (Weeks 1-2)
Audit which AI coding tools your developers are using. Don't rely on self-reporting; check IDE telemetry, browser extensions, and API usage logs. You'll find tools you didn't approve.
Create a code tagging convention:
// AI-GENERATED: [tool-name] [date] [prompt-hash]
// HUMAN-REVIEWED: [reviewer] [date]
Your static analysis pipeline should flag untagged AI blocks.
Phase 2: Control Mapping (Weeks 3-4)
Map your current SAST rules against common AI code patterns. Tools like Semgrep and CodeQL need new rules for:
- Deprecated function calls (AI training data is old)
- Incomplete error handling (LLMs optimize for the happy path)
- Hard-coded credentials in example code
- SQL queries built with string concatenation
If you're using SonarQube, enable the "AI-Generated Code" quality profile when it ships. Until then, create custom rules.
Phase 3: Review Process (Weeks 5-8)
AI-generated code needs different review criteria than human code. Your PR template should include:
- Dependencies verified in official registry
- No credentials in generated examples
- Error paths implemented (not just happy path)
- License compatibility confirmed
- Generated test cases reviewed for coverage gaps
Train reviewers to spot AI patterns: overly generic variable names, tutorial-style comments, suspiciously complete boilerplate.
Phase 4: Tool Configuration (Ongoing)
If you're allowing AI tools, configure them defensively:
Context window limits: Restrict what code the AI can see. Don't let it ingest your entire codebase.
Output filtering: Use tools that strip common vulnerability patterns before showing suggestions to developers.
Logging: Capture what prompts generated which code. When you find a vulnerability, you need to trace it back.
Common Pitfalls
Treating AI code like vendor code: Vendor code has a support contract and liability model. AI-generated code doesn't. You own every line.
Assuming SAST catches everything: Your SAST tools were tuned on human-written code. They miss AI-specific issues like hallucinated dependencies and training data artifacts.
Ignoring license compliance: AI tools trained on GPL code might generate GPL-like snippets. Your legal team needs to review generated code for license conflicts, especially if you're shipping proprietary software.
Skipping threat modeling updates: Your threat model probably doesn't include "attacker poisons LLM training data to inject vulnerabilities into our codebase." It should now.
Over-trusting "secure by default" claims: AI tools don't guarantee secure output. They optimize for code that compiles and runs, not code that resists attack.
Quick Reference Table
| Control Area | Traditional Approach | AI-Augmented Approach |
|---|---|---|
| Code Review | Focus on logic, style, security | Add: dependency verification, license check, AI pattern recognition |
| SAST Rules | Human code patterns | Add: hallucinated APIs, incomplete error handling, training data artifacts |
| Dependency Management | Verify known packages | Add: verify package existence, check for typosquatting |
| Threat Model | Developer mistakes, supply chain | Add: training data poisoning, context leakage, prompt manipulation |
| Access Control | Repository permissions | Add: AI tool API access, context window scope |
| Compliance Evidence | Code inventory, review logs | Add: AI-generated code tags, tool configuration records |
| Incident Response | CVE tracking, patch management | Add: AI tool version tracking, prompt audit logs |
Your application security program needs an AI appendix. The controls you built for human developers don't transfer directly. Start with visibility, add tagging, and update your review process before AI-generated code becomes 50% of your codebase.
