Security engineers have spent decades developing reliable methods to measure software security. However, with the advent of AI systems, these measurements have become ineffective.
The issue isn't that AI security can't be measured—it's that the wrong tools are being used. Traditional benchmarks that work for software don't capture the vulnerabilities specific to AI systems. Your team needs a different approach, and the evidence suggests that process-driven standards are the solution.
The Shift in Security Measurement
Over the last 30 years, security engineering has evolved from black box penetration testing to process-driven standards like the Building Security In Maturity Model (BSIMM). This shift occurred because teams realized that security outcomes depend on repeatable processes, not just point-in-time assessments.
AI systems disrupt this model. Benchmarks that measure AI capabilities, including security, don't work effectively. AI systems change behavior based on training data, prompts, and context in ways that traditional software doesn't. A vulnerability assessment valid today may become irrelevant after your next model update.
Key Findings
1. Static Benchmarks Miss Dynamic AI Behavior
Testing a web application's authentication provides results that remain valid until the code changes. In contrast, testing an AI system's resistance to prompt injection only applies to the specific inputs tested. Change the phrasing, add context, or modify the system prompt, and you're measuring something different. There's no "security meter" that provides a stable reading.
2. AI Attack Surfaces Don't Fit Traditional Categories
Existing security frameworks organize controls around networks, applications, and data. AI systems introduce attack vectors that span all three: training data poisoning affects the model before deployment, prompt injection exploits the application layer, and model inversion attacks extract training data. The NIST Cybersecurity Framework v2.0 added AI considerations, but many teams are still figuring out how to implement them.
3. Process Maturity Predicts AI Security Outcomes Better Than Technical Tests
Teams that excel in BSIMM activities—security training, threat modeling, code review—tend to ship more secure AI systems. This mirrors traditional software security: the presence of security activities is more important than the results of any single test. If your team lacks a process for reviewing AI system prompts before production, you have a gap that no benchmark can measure.
4. Compliance Frameworks Lag Behind AI Risk
ISO 27001 includes information security controls but doesn't address AI-specific risks like model theft or adversarial examples. PCI DSS v4.0.1 requires secure coding practices, but those requirements assume deterministic software behavior. SOC 2 Type II audits verify that you follow your stated security processes—but most organizations haven't defined AI security processes yet.
What This Means for Your Team
You need to build AI security processes before you can measure them. Start with three assumptions:
Assume AI Systems Will Behave Unexpectedly. Traditional software does what you program it to do. AI systems do what they learned to do, which isn't always the same. Your security controls need to account for this uncertainty.
Assume Existing Security Tools Won't Detect AI-Specific Attacks. Your WAF won't catch a prompt injection. Your SIEM won't flag training data poisoning. Your vulnerability scanner won't identify a model inversion risk. You need new detection capabilities built around AI system behavior, not just network traffic or application logs.
Assume Compliance Doesn't Equal Security. Meeting ISO 27001 or SOC 2 requirements doesn't mean your AI systems are secure. Those frameworks weren't designed for AI risks. You need to layer AI-specific controls on top of your baseline compliance program.
Action Items by Priority
Priority 1: Document Your AI Security Processes
Create written procedures for:
- How your team reviews AI system designs for security risks
- Who approves production deployment of AI systems
- How you test AI systems for adversarial robustness
- What monitoring you run on AI system behavior in production
These processes form your audit trail. When an auditor asks how you secure AI systems, you need documented answers.
Priority 2: Adapt BSIMM Activities for AI Systems
Apply three BSIMM practices to AI:
- Security Training: Add AI security topics to your developer training—prompt injection, data poisoning, model theft
- Threat Modeling: Run threat models specifically for AI systems, covering training pipeline, inference endpoints, and model storage
- Code Review: Review not just the code that calls your AI models, but the prompts, guardrails, and output validation logic
Track completion rates for these activities. Process metrics give you something to measure when technical benchmarks fail.
Priority 3: Build AI-Specific Security Gates
Add these checks to your deployment pipeline:
- Prompt review: Someone other than the developer must review production prompts for injection risks
- Output validation: Every AI system must validate outputs against expected formats and content policies
- Behavioral testing: Test how your AI system responds to adversarial inputs before each production release
Document who performed each check and when. This creates the audit trail that compliance frameworks require.
Priority 4: Establish AI System Monitoring
Deploy monitoring that tracks:
- Unusual patterns in AI system inputs (potential injection attempts)
- Changes in AI system output distributions (potential model drift or poisoning)
- Access patterns to AI models and training data (potential theft attempts)
Set baseline metrics during your first 30 days of production operation. Alert on deviations.
Your AI security program won't look like your application security program. But the underlying principle remains: you secure what you can measure, and you measure what you can repeat. Focus on building repeatable processes first. The metrics will follow.
