Stop Using LLMs to Generate Secrets

Your CI/CD pipeline might be generating passwords right now. If you're using AI coding assistants, there's a good chance those passwords are predictable and undetectable by your secret scanners.

Research by Irregular found that Claude Opus 4.6 generated only 30 distinct passwords across 50 sessions. That's not randomness; that's a pattern. A 16-character password from a cryptographically secure pseudorandom number generator (CSPRNG) has about 98 bits of entropy. Claude Opus 4.6 achieves roughly 27 bits.

Here's how to find these passwords in your codebase and prevent new ones from appearing.

Why This Matters

LLMs generate passwords using next-token prediction, not true randomness. They don't meet NIST SP 800-90A Rev. 1 requirements for secure entropy generation. Standard entropy meters like zxcvbn can't detect this weakness because they measure character complexity, not generation method.

Your secret scanners won't catch these either. They're looking for patterns (AWS keys, GitHub tokens) or high-entropy strings. LLM-generated passwords look random enough to pass these checks while being predictable enough to enumerate.

If you're using GitHub Copilot, Cursor, or similar tools to scaffold configuration files, database connection strings, or initialization scripts, you've likely got these passwords in your repository history.

What You Need Before Starting

Access and Permissions:

• Read access to all repositories in your organization
• Admin access to your CI/CD platform
• Ability to rotate credentials across development, staging, and production

Tools:

• Git with full history access
• trufflehog or gitleaks (you'll modify their behavior)
• A CSPRNG-based password generator (openssl, pwgen -s, or secrets.token_urlsafe() in Python)
• Your secret management platform (Vault, AWS Secrets Manager, etc.)

Documentation:

• Inventory of all services using generated credentials
• List of AI coding tools approved in your environment
• Current password rotation schedule

Step-by-Step Implementation

1. Identify LLM-Generated Passwords in Existing Code

Standard secret scanners won't work here. You need to look for structural patterns.

Create a custom ruleset for your scanner. LLM-generated passwords often:

• Use dictionary words with number/symbol substitution (Tr0ub4dor&3)
• Follow predictable patterns (capital-lowercase-number-symbol)
• Appear in comments alongside AI-generated boilerplate
• Exist in files created or modified by AI tools

# Search for files modified by AI tools
git log --all --pretty=format:"%H %an %s" | grep -i "copilot\|cursor\|ai"

# Extract those commit hashes and examine changed files
git show <commit-hash> | grep -E "(password|secret|key|token)\s*=\s*['\"]"

Look for passwords in:

• Docker Compose files
• Kubernetes secrets (base64-encoded)
• Database initialization scripts
• Configuration templates
• Test fixtures

2. Build a Reference Set of Known LLM Outputs

Prompt your organization's approved AI tools to generate passwords. Run 50+ sessions per tool. Document the outputs.

# Example: Test your LLM's password generation
import anthropic
client = anthropic.Anthropic(api_key="your-key")

passwords = []
for i in range(50):
    response = client.messages.create(
        model="claude-opus-4",
        messages=[{"role": "user", "content": "Generate a secure 16-character password"}]
    )
    passwords.append(response.content[0].text)

# Check for duplicates
print(f"Unique passwords: {len(set(passwords))} out of {len(passwords)}")

If you see fewer than 45 unique passwords in 50 attempts, you've confirmed the predictability issue.

3. Replace Identified Passwords

Don't rotate in place. Generate new credentials with a CSPRNG and update references atomically.

# Generate a proper password (Linux/macOS)
openssl rand -base64 32

# Or using Python
python3 -c "import secrets; print(secrets.token_urlsafe(32))"

Update in this order:

1. Generate new credential with CSPRNG
2. Store in secret manager
3. Update application configuration to read from secret manager
4. Deploy updated configuration
5. Verify application connectivity
6. Revoke old credential
7. Remove hardcoded value from code

4. Modify Your Development Workflow

Add pre-commit hooks that reject LLM-generated patterns:

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: check-password-generation
        name: Check for LLM-generated passwords
        entry: scripts/check-llm-passwords.sh
        language: script
        files: \.(yaml|yml|env|conf|json)$

In scripts/check-llm-passwords.sh:

#!/bin/bash
# Check for common LLM password patterns
if grep -rE "(password|secret|key)\s*[:=]\s*['\"][A-Z][a-z]+[0-9!@#$&*]" "$@"; then
    echo "ERROR: Possible LLM-generated password detected"
    echo "Use: openssl rand -base64 32"
    exit 1
fi

5. Update AI Tool Policies

Document in your secure coding standards:

PROHIBITED: Using LLMs to generate passwords, API keys, or cryptographic secrets
REQUIRED: Use openssl rand, secrets module, or your secret manager's generator
EXCEPTION: None. This applies to development, testing, and production.

Add this to your AI coding assistant prompts if your tool supports system instructions:

Never generate passwords, API keys, tokens, or cryptographic material. 
When asked for credentials, respond: "Use openssl rand -base64 32"

Validation - How to Verify It Works

Test Your Detection:

Plant a known LLM-generated password in a test repository. Run your modified scanner. It should flag it.

Verify Your CSPRNGs:

# Generate 1000 passwords, check for duplicates
for i in {1..1000}; do openssl rand -base64 16; done | sort | uniq -d

You should see zero duplicates.

Audit Your Secret Manager:

Review creation timestamps and sources. Anything generated by an AI tool needs rotation.

# AWS Secrets Manager example
aws secretsmanager list-secrets --query "SecretList[?contains(Description, 'copilot')]"

Check Compliance Mapping:

If you're subject to PCI DSS v4.0.1 Requirement 8.3.6 (passwords must be strong), LLM-generated passwords with 27 bits of entropy don't qualify. Document your CSPRNG usage for auditors.

Maintenance and Ongoing Tasks

Monthly:

• Review git commits for new AI-generated credentials
• Audit secret manager for non-CSPRNG sources
• Test pre-commit hooks on a sample of recent commits

Quarterly:

• Re-test your approved AI tools for password generation behavior (models change)
• Update your reference set of LLM outputs
• Review and rotate any credentials created before this policy

When Onboarding New AI Tools:

• Test password generation behavior before approval
• Document entropy characteristics
• Update pre-commit hooks if new patterns emerge

Incident Response:

If you find an LLM-generated password in production:

1. Treat it as a compromise (assume enumeration is possible)
2. Rotate immediately using your CSPRNG
3. Review access logs for the affected service
4. Check for lateral movement
5. Document in your incident log

The fix is straightforward: stop asking LLMs to generate secrets. They're not designed for it, they don't meet cryptographic standards, and your existing tools can't reliably detect the problem. Use a CSPRNG for every credential, every time.