The Security Gap in AI Systems
Cobalt's State of Pentesting Report highlights a significant gap between AI system security and traditional application security. When pentesters examined AI and LLM implementations, they found that 32% of vulnerabilities were high-risk, compared to 13% in legacy systems. Alarmingly, only 38% of these high-risk LLM issues are remediated.
Organizations are deploying AI systems with inadequate security measures, treating them as if they were still in the proof-of-concept stage, yet connecting them to production data and customer-facing workflows.
Emerging Patterns in AI Security
The report identifies a troubling pattern across multiple pentests:
Initial Deployment: Teams integrate LLMs into applications without a dedicated security review, treating them as libraries rather than potential attack surfaces.
Discovery: Pentesters find vulnerabilities such as prompt injection, insecure model access, and insufficient input validation—issues not typically found in traditional code.
Remediation: The process often fails here. With 62% of high-risk LLM findings left unpatched, existing vulnerability management workflows are inadequate.
Ongoing Exposure: Organizations continue to run vulnerable AI systems in production, often unaware that OWASP now ranks prompt injection as the top risk for LLM applications.
Missing or Failed Security Controls
Input Validation: Traditional input validation methods fail with LLMs, which interpret context and intent rather than just syntax. Regex-based filters designed for SQL injection are ineffective.
Least Privilege: AI systems often have broad database access without query-level controls, violating the principle of least privilege.
Segregation of Duties: A single team often manages the AI feature from concept to production, bypassing security reviews due to perceived low risk.
Change Management: Model updates occur outside standard deployment pipelines, leading to untested security controls.
Monitoring and Detection: Traditional SIEM rules do not detect prompt injection attacks, which appear as normal user input. AI systems lack logging for prompt patterns, model outputs, and data access.
Compliance Standards and Requirements
OWASP ASVS v4.0.3, Section 5.1 mandates that all untrusted data be validated, sanitized, or escaped. For LLMs, semantic validation is necessary to evaluate the intent of prompts.
ISO/IEC 27001:2022, Annex A.8.2 requires information security throughout the system lifecycle. AI implementations need a threat model before coding begins.
NIST CSF v2.0, Function: Protect (PR.AC-4) emphasizes managing access permissions with least privilege and separation of duties. LLMs should authenticate with credentials limited to their function.
PCI DSS v4.0.1, Requirement 6.4.3 requires review of custom code before release, including prompt engineering and system prompts.
SOC 2 Type II, Common Criteria 6.6 demands logical access controls to prevent unauthorized access, ensuring queries are verified against user permissions.
Actionable Steps for Your Team
Reevaluate AI Features: Treat AI systems as critical components once they interact with production data or customer input. Conduct security reviews before deployment.
Implement AI-Specific Security Controls:
- Use prompt filtering that evaluates semantic intent.
- Log all prompts and model outputs for forensic analysis.
- Rate-limit AI interactions to mitigate automated attacks.
- Establish a separate data access layer enforcing row-level security.
Clarify Ownership: Assign a directly responsible individual (DRI) for AI security to coordinate between data science, application security, and infrastructure teams.
Adapt Your Threat Model: Include questions about user manipulation, data exposure, and potential harmful queries in design reviews.
Test for AI-Specific Vulnerabilities: Update your pentest methodology to include tests for prompt injection and other AI-specific vulnerabilities.
Instrument Before Deployment: Ensure logging of user prompts, model outputs, data source access, and failed access attempts before launching AI features.
The 32% high-risk finding rate reflects a lack of preparedness in securing AI systems. Your team can either retrofit security controls post-deployment or integrate them from the start. Choose the latter to avoid becoming part of the statistic of unprepared organizations. AI Security Guidelines
