You've got an AI agent writing production code. It passes the linter and looks clean. Then it ships a dependency that phones home to an expired domain, or it refactors an authentication check into a decorator that never actually runs.
Linting catches syntax errors and formatting issues. It won't catch an agent that generates well-formatted code with broken logic, introduces a vulnerable library version, or rewrites a critical security control in a way that silently fails. If you're letting AI agents write code, you need verification layers that go deeper than style guides.
The Problem: Speed Without Understanding
AI agents produce code that's syntactically correct and idiomatically sound. The linter gives it a green checkmark. But agents don't understand your threat model. They don't know that your authentication middleware must run before every controller method, or that your database connection pool has specific timeout requirements tied to PCI DSS v4.0.1 Requirement 6.4.3.
When an agent refactors code, it might move a security check from a function's entry point to a conditional branch that rarely executes. The linter sees valid Python. Your integration tests might even pass if they don't cover that edge case. You won't know there's a problem until an auditor asks why unauthenticated requests are hitting your cardholder data environment.
What You Need Before Starting
Before you build verification layers for agent-generated code, you need:
A baseline understanding of your security controls. Document which functions enforce authentication, authorization, input validation, and audit logging. If you can't list these, you can't verify an agent didn't break them.
Static analysis tools that go beyond linting. You need control-flow and data-flow analysis. Tools like Semgrep, CodeQL, or Snyk Code can trace how data moves through your application and flag when sensitive operations lack required checks.
A dependency policy. Define allowed package registries, minimum security scores, and license restrictions. Tools like pip-audit, safety, or Dependabot can flag vulnerable dependencies, but you need a policy that says what's acceptable.
Test coverage for security-critical paths. If your agent changes authentication logic, you need tests that verify unauthenticated requests get blocked. If it touches database queries, you need tests for SQL injection patterns.
Configuration as code. If agents can modify environment configs, infrastructure definitions, or service meshes, those changes need to be tracked and validated just like application code.
Step-by-Step Implementation
1. Add Control-Flow Analysis to Your CI Pipeline
Install a semantic code analysis tool. For Python projects:
pip install semgrep
semgrep --config=auto --error
For JavaScript/TypeScript:
npm install -g @github/codeql-cli
codeql database create mydb --language=javascript
codeql database analyze mydb --format=sarif-latest --output=results.sarif
Write rules that verify security controls remain in place. Here's a Semgrep rule that flags authentication decorators removed from API endpoints:
rules:
- id: missing-auth-decorator
pattern: |
@app.route(...)
def $FUNC(...):
...
pattern-not: |
@require_auth
@app.route(...)
def $FUNC(...):
...
message: "API endpoint missing @require_auth decorator"
severity: ERROR
2. Implement Dependency Verification
Create a pre-commit hook that scans for new or modified dependencies:
#!/bin/bash
# .git/hooks/pre-commit
# For Python
pip-audit --requirement requirements.txt --format json > audit.json
if [ $? -ne 0 ]; then
echo "Dependency audit failed. Review audit.json"
exit 1
fi
# For Node.js
npm audit --json > audit.json
if grep -q '"severity": "high"' audit.json; then
echo "High-severity vulnerabilities found"
exit 1
fi
Set a policy: no dependencies with known CVEs rated high or critical. No packages from unmaintained projects (no commits in 18+ months). No licenses incompatible with your product's license.
3. Add Behavioral Tests for Security Controls
Write tests that verify the behavior an agent might break:
def test_unauthenticated_request_blocked():
response = client.get('/api/users', headers={})
assert response.status_code == 401
def test_sql_injection_prevented():
malicious_input = "1' OR '1'='1"
response = client.get(f'/api/user?id={malicious_input}')
assert response.status_code in [400, 404]
# Should not return all users
Run these tests on every commit. If an agent refactors your authentication middleware and breaks it, these tests fail before the code merges.
4. Track Configuration Changes
If agents modify Kubernetes manifests, Terraform files, or environment configs, treat those as code:
# In your CI pipeline
terraform plan -out=tfplan
terraform show -json tfplan > plan.json
# Check for security-sensitive changes
jq '.resource_changes[] | select(.change.actions[] == "delete") | select(.type == "aws_security_group")' plan.json
# Flag if security groups are being deleted
For Kubernetes, use admission controllers like OPA Gatekeeper to enforce policies:
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: require-security-labels
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
parameters:
labels: ["security-tier", "data-classification"]
Validation: How to Verify It Works
After implementing these layers, validate they catch real issues:
Introduce a known vulnerability. Remove an authentication check from a test endpoint. Your control-flow analysis should flag it. Your behavioral tests should fail.
Add a vulnerable dependency. Temporarily add a package with a known CVE to your requirements file. Your dependency scanner should block the commit.
Modify a security-critical config. Change a firewall rule or remove a required label from a Kubernetes pod spec. Your policy checks should reject the change.
If any of these slip through, you've found a gap in your verification layers. Fix it before an agent finds it for you in production.
Maintenance and Ongoing Tasks
Weekly: Review flagged changes. Check your CI logs for security warnings. If agents are consistently triggering the same false positives, refine your rules.
Monthly: Update analysis rules. As your application evolves, your security controls change. Add new Semgrep or CodeQL rules when you introduce new authentication patterns or data handling requirements.
Quarterly: Audit your verification coverage. Run a coverage report on your security tests. If you've added new API endpoints or database queries, make sure you've added corresponding tests.
When onboarding new agents: Establish guardrails. If you're adding a new AI coding assistant or expanding an agent's permissions, document which parts of the codebase it can modify and which verification layers apply.
Linting is still useful. It catches typos and formatting issues before they waste your time in code review. But when an agent can rewrite your authentication layer in thirty seconds, you need verification that understands what the code does, not just how it looks.
