Understanding the Concerns
Recently, I've observed heated debates among security engineers about whether to restrict GitHub Copilot in their environments. The discussions often lack concrete data on how these tools behave in real-world scenarios.
Researchers Abhishek Kumar and Carsten Maple have provided some clarity. Their study shows that AI coding assistants can indeed produce harmful code, even when their chat interfaces refuse direct requests for such code. They tested models from GitHub Copilot, Claude from Anthropic, and Gemini from Google. The results were striking: 204 harmful prompts led to harmful outputs 816 times out of 816 attempts.
Here's what your team might be asking about this issue.
Q: Does Copilot ignore its safety filters?
Not exactly. The chat interface functions as intended. If you directly request malicious code, it refuses. The issue arises when developers break requests into small, seemingly harmless steps over multiple interactions.
The study found that harmful output often appeared after about six exchanges. Each request seemed benign: "write a function that takes user input," then "add a database query," then "format the output." Combined, they created SQL injection code.
This isn't a bug in Copilot. It's a limitation of how language models evaluate context. They're designed to be helpful in immediate interactions, not to track whether a series of steps adds up to something harmful.
Q: Is this happening in production or just in labs?
There aren't systematic incident numbers yet. However, consider your developers' daily activities: they're already engaging in multi-step conversations with AI assistants to build features. They're not trying to bypass safety filters; they're solving problems incrementally.
The vulnerability isn't about malicious intent. It's about normal development workflows that inadvertently recreate the conditions identified by researchers. Your team could generate exploitable code without realizing it, especially under deadline pressure.
Q: What does this mean for our code review process?
Your current review process should catch AI-generated vulnerabilities as it does human-written ones. Here's how:
- Ensure reviewers know which files or functions came from AI assistance. Some teams add comments marking AI-generated code.
- Run static analysis tools on everything, regardless of source.
- Manually review security-sensitive code paths, even if they pass automated checks.
If you're subject to PCI DSS v4.0.1, Requirement 6.3.2 mandates that custom software be reviewed by someone other than the original author. This applies whether the "author" is a human or an AI assistant.
Q: Should we block these tools until they're safer?
Blocking might not be effective. Developers who can't use approved AI tools might turn to unapproved ones. It's better to control the environment and add guardrails.
Effective measures include:
- Configure your IDE to flag security-sensitive patterns in AI-generated code. Most modern SAST tools can do this.
- Require AI-generated code in authentication, authorization, or data handling paths to undergo security review before merging.
- Train developers to recognize when they're incrementally building something they'd never request all at once.
If you're working toward SOC 2 Type II compliance, document this as part of your secure development lifecycle. Your auditor will want to see how AI tools fit into your change management and code review controls.
Q: How do we monitor this without chat history?
Focus on monitoring the output, not the conversation. Your SAST and DAST tools should catch vulnerable patterns regardless of their origin. Additional checks can include:
- Flagging commits that modify security-critical functions like authentication, authorization, encryption, and input validation.
- Watching for patterns common in AI-generated code: verbose variable names, excessive comments, and certain formatting styles.
- Reviewing any code that touches OWASP Top 10 vulnerability categories, especially injection flaws and broken access control.
The OWASP ASVS v4.0.3 provides a solid baseline. Level 2 requirements (sections 5.1 through 5.5 on input validation) should apply to all code, AI-generated or not.
Q: Are some models safer than others?
The study found vulnerabilities across all tested models. This isn't specific to GitHub Copilot or Claude. It's a language model issue.
Different models have varying safety tuning, but they all share a fundamental limitation: they evaluate requests in context windows, not across entire development sessions. Until this changes, assume any AI coding assistant can be led into generating harmful code through incremental requests.
Q: What should we tell developers who rely on these tools?
Don't tell them to stop using AI assistants. Instead, instruct them to use them with the same security mindset they'd apply to any external code:
- Treat AI-generated code as untrusted until reviewed.
- Don't accept suggestions for security-critical functions without understanding them fully.
- If building something sensitive across multiple AI interactions, review the complete result, not just each piece.
Think of it like this: you wouldn't copy-paste code from Stack Overflow directly into your authentication system. Apply the same skepticism to AI-generated code, even though it feels more tailored to your specific problem.
Next Steps
Start by auditing where your team uses AI coding assistants most heavily. Map those usage patterns against your security-critical code paths. Then add review gates at the intersections.
If you're implementing the NIST Cybersecurity Framework v2.0, this fits under the Identify and Protect functions: know where AI tools are used (ID.AM-2: software platforms and applications) and control how they integrate into your development process (PR.DS-6: integrity checking mechanisms).
The researchers published their methodology, so you can test your environment if you want data specific to your tools and workflows. But the core finding is solid: AI coding assistants will help you write harmful code if you ask in the right way, even when they're designed not to.
Your job isn't to eliminate that risk. It's to build review processes that catch it before it ships.
