What Happened
Your security operations team integrated an LLM-based assistant to speed up alert triage and vulnerability analysis. This tool accessed your organization's vulnerability management system to provide context-aware recommendations. Within weeks, an attacker used prompt injection techniques to extract details about unpatched vulnerabilities, including specific version numbers and internal asset identifiers. The breach went undetected until the attacker exploited one of the disclosed vulnerabilities to gain initial access.
This scenario, while composite, reflects well-documented failure modes. OWASP maintains a catalog of common LLM application failures. The UK Department for Science, Innovation and Technology has mapped vulnerabilities across the AI lifecycle. The 2024 finding that AI-generated phishing emails performed on par with human experts shows how quickly these tools can be weaponized.
Timeline
Week 1: Your team deploys an LLM assistant with read access to the vulnerability database and SIEM logs to improve triage speed.
Week 2-3: Team members use the assistant for routine tasks. No anomalous behavior is detected in application logs.
Week 4: An attacker gains access to the LLM interface through compromised credentials from a separate phishing campaign.
Week 4-5: The attacker conducts reconnaissance using crafted prompts to extract vulnerability data, including:
- Specific CVE identifiers for unpatched systems
- Internal hostname patterns
- Software versions and patch status
Week 6: The attacker exploits a disclosed vulnerability to establish persistence.
Week 7: Your team discovers unauthorized access during routine log review, tracing it back to LLM prompt history.
Which Controls Failed or Were Missing
Access Control Boundaries: The LLM had unrestricted read access to the vulnerability database. No one questioned whether an AI assistant needed the same privilege level as senior security analysts.
Input Validation: The application accepted arbitrary prompts without content filtering, sanitization, or detection of potential extraction attempts. Practices that would catch SQL injection or command injection were absent.
Output Filtering: The system returned raw database results through the LLM without redaction or classification checks. Sensitive internal identifiers and version numbers flowed directly to the user interface.
Audit Logging: Prompt history existed but wasn't monitored. No alerts triggered on unusual query patterns, repeated vulnerability lookups, or systematic enumeration attempts.
Privilege Separation: The LLM operated with a single service account that had broad read permissions across security systems. No attempt was made to implement least-privilege access based on the requesting user's role.
Session Management: Compromised credentials provided the same level of access as legitimate users, with no additional verification for sensitive operations.
What the Relevant Standard Requires
NIST 800-53 Rev 5, AC-6 (Least Privilege): Requires that users and processes operate with the minimum privileges necessary. Your LLM service account needs role-based access control, not blanket read permissions. If an analyst can only see vulnerabilities for systems they manage, the LLM should inherit those same boundaries.
NIST 800-53 Rev 5, AU-6 (Audit Review, Analysis, and Reporting): Mandates review and analysis of audit records for inappropriate or unusual activity. This applies to your LLM's prompt logs just as much as your database query logs. Set up detection rules for systematic enumeration patterns.
ISO/IEC 27001:2022, Control 8.2 (Privileged Access Rights): Requires allocation and use of privileged access rights to be restricted and controlled. An AI assistant with database access is a privileged account. Treat it as such.
NIST Artificial Intelligence Risk Management Framework: Provides structure for managing AI-specific risks including data privacy, security vulnerabilities, and trustworthiness. Map your LLM deployment against the GOVERN, MAP, MEASURE, and MANAGE functions. NIST AI RMF
OWASP ASVS v4.0.3, Section 4.1 (Access Control): Specifies that access control mechanisms must deny by default and enforce authorization on every request. Your LLM's data access layer needs the same controls as your API endpoints.
Lessons and Action Items for Your Team
Implement Semantic Access Control Now. Your LLM needs context-aware permissions, not database-level credentials. Build an intermediary layer that:
- Maps user identity to data access boundaries
- Filters LLM queries before they hit your systems
- Redacts sensitive fields from responses
- Logs every data access with user context
Write Detection Rules for LLM-Specific Attacks. Monitor for:
- Prompts containing SQL-like syntax or system commands
- Repeated queries for vulnerability or configuration data
- Attempts to bypass filters ("ignore previous instructions")
- Systematic enumeration patterns
- Requests for bulk data extraction
Classify Your Data Before You Connect It. Tag fields in your vulnerability database:
- Public: CVE identifiers, general descriptions
- Internal: Hostnames, IP addresses, software versions
- Restricted: Exploit details, compensating controls, business context
Configure your LLM integration to respect these classifications. If you can't redact restricted fields, don't connect that data source.
Treat LLM Deployments as Privileged Access Reviews. Before you give an AI assistant access to a system, ask:
- Would you give a contractor this access on day one?
- Can you audit what data it retrieved?
- Do you have controls to prevent bulk extraction?
- Can you revoke access if behavior becomes suspicious?
Test Your Defenses with Red Team Exercises. Have your security team attempt:
- Prompt injection to bypass filters
- Social engineering the LLM to reveal system details
- Extracting training data or fine-tuning artifacts
- Using the LLM to generate reconnaissance data
Document what works. Fix what doesn't.
Separate Your LLM Environments by Sensitivity. Run different instances for:
- Public-facing applications (no access to internal systems)
- Internal productivity tools (read-only, filtered access)
- Security operations (full access, heavy monitoring)
Never reuse the same LLM deployment across trust boundaries. The convenience isn't worth the risk.
The core lesson: LLMs are not passive tools. They're active components with data access, processing logic, and attack surface. Apply the same rigor you use for service accounts, API keys, and database connections. Anything less turns your AI assistant into an information disclosure vulnerability with a chat interface.
