Mapping Security Accountability Across the AI Stack

Scope - What This Guide Covers

This guide defines security responsibilities for teams deploying AI systems from third-party vendors. It addresses:

• Which security controls belong to your team vs. the vendor
• How to translate cloud security frameworks to AI deployments
• Specific accountability boundaries when incidents occur
• Documentation requirements for compliance audits

Out of scope: Custom-built ML models, research environments, and AI systems developed entirely in-house.

Key Concepts and Definitions

Shared Responsibility Model: A framework that divides security obligations between service providers and customers. In cloud computing, AWS secures the infrastructure; you secure what you run on it. AI requires a similar split.

AI Stack Layers:

• Foundation layer: Pre-trained models, training data provenance, model weights
• Platform layer: APIs, inference endpoints, rate limiting, authentication
• Application layer: Prompt design, input validation, output filtering, access controls
• Data layer: Training data you provide, fine-tuning datasets, user queries

Security Incident Categories:

• Prompt injection attacks
• Model extraction or inversion
• Data leakage through model outputs
• API authentication bypass
• Unauthorized access to training data

Requirements Breakdown

Vendor Responsibilities

Your AI vendor must secure:

1. Model integrity: Prevent unauthorized modification of model weights
2. Infrastructure security: Patch management, network isolation, compute environment hardening
3. API authentication: Implement OAuth 2.0 or equivalent, provide key rotation mechanisms
4. Rate limiting: Prevent resource exhaustion attacks at the platform level
5. Audit logging: Capture all API calls with timestamps, caller identity, and request/response metadata

Compliance mapping: These align with NIST CSF v2.0 functions PR.AC (Identity Management) and PR.DS (Data Security).

Your Team's Responsibilities

You own security for:

1. Input validation: Filter malicious prompts before they reach the model (OWASP ASVS v4.0.3, Section 5.1)
2. Output sanitization: Prevent injection attacks when displaying AI-generated content
3. Access controls: Define who can query models, view responses, or access fine-tuning capabilities
4. Sensitive data handling: Ensure PII doesn't leak into prompts or training data (PCI DSS v4.0.1 Requirement 3.3 if processing cardholder data)
5. Monitoring and alerting: Detect anomalous query patterns or unexpected model behavior

Documentation requirement: Maintain a responsibility matrix for SOC 2 Type II audits. Map each control to either vendor or internal ownership.

Implementation Guidance

Step 1: Request Vendor Security Documentation

Before deployment, obtain:

• Security whitepaper: How they secure the model training pipeline
• Incident response SLA: Maximum time to patch critical vulnerabilities
• Compliance certifications: SOC 2 Type II, ISO 27001 reports
• Data handling policy: Where your prompts are stored, retention periods, whether they're used for model improvement

If the vendor deflects with "our system is secure" without specifics, that's a red flag.

Step 2: Define Boundary Controls

Map each attack vector to an owner:

Attack Vector	Vendor Responsibility	Your Responsibility
API key compromise	Provide key rotation API, alert on unusual geographic access	Rotate keys quarterly, store in secrets manager
Prompt injection	Implement content filtering at platform level	Validate inputs before sending, sanitize outputs
Model extraction via repeated queries	Rate limiting, query pattern detection	Monitor your application's query volume
Training data poisoning	Validate data provenance, scan for malicious samples	Review fine-tuning datasets for sensitive content
Unauthorized model access	Authentication, authorization at API layer	Implement application-level RBAC

Step 3: Establish Incident Response Protocols

When vulnerabilities surface, you need clear escalation paths:

Your runbook should include:

• Vendor security contact (not general support)
• Threshold for disabling AI features (e.g., if authentication is bypassed)
• Evidence collection process (capture malicious prompts, API logs)
• Notification requirements (customers, regulators if PII exposed)

Vendor commitments to negotiate:

• Vulnerability disclosure timeline (30, 60, 90 days?)
• Patch deployment process (automatic or manual?)
• Breach notification SLA

Step 4: Implement Detection Controls

Deploy monitoring for:

1. Abnormal query patterns: Sudden spike in API calls, especially outside business hours
2. Sensitive data in prompts: Regex patterns for SSNs, credit cards, API keys (automate redaction)
3. Model output anomalies: Responses containing internal system paths, configuration details
4. Failed authentication attempts: Brute force attacks against your application's AI endpoints

Tool integration: Connect AI API logs to your SIEM. Treat them like any other third-party service.

Common Pitfalls

Pitfall 1: Accepting "Social Engineering" as the Default Explanation

AI vendors often categorize security incidents as user misuse or social engineering attacks. When a prompt injection bypasses your input validation and the vendor's content filter, that's a shared failure. Document both parties' contributions to the incident.

Pitfall 2: Assuming Vendor Compliance Covers Your Use Case

A vendor's SOC 2 Type II report covers their infrastructure, not your implementation. If you're processing payment card data, you must ensure prompts don't contain cardholder data (PCI DSS v4.0.1 Requirement 4.2).

Pitfall 3: No Contractual Security Requirements

MSAs often lack specific security obligations. Negotiate:

• Maximum time to patch critical vulnerabilities
• Your right to audit their security controls
• Data deletion guarantees when you terminate service
• Liability caps that reflect actual risk

Pitfall 4: Treating AI APIs Like Static Services

Models get retrained. Behavior changes. Version your prompts and test after vendor updates. What worked last month might produce different outputs—or new vulnerabilities—today.

Pitfall 5: Ignoring the Data Layer

If you fine-tune a model with proprietary data, you've created a new security boundary. That training data becomes part of the model. Ensure your vendor's data isolation guarantees prevent other customers from extracting it.

Quick Reference Table

Scenario	Vendor Action	Your Action	Compliance Reference
API key leaked in public GitHub repo	Provide key revocation API	Rotate immediately, scan all repos	NIST 800-53, IA-5
Prompt injection returns internal data	Investigate model behavior, patch if systemic	Implement output filtering, report to vendor	OWASP ASVS, 5.2.1
Unauthorized access to AI endpoint	N/A (application-layer attack)	Review authentication logic, check for credential stuffing	PCI DSS, Req 8.3
Model outputs PII from training data	Investigate data leakage, retrain if necessary	Stop sending PII in prompts, audit existing queries	ISO 27001, A.8.11
DDoS against AI API	Absorb attack, maintain availability SLA	Implement application-level rate limiting	NIST CSF, PR.PT-4

---

Start here: Review your current AI vendor contracts. If security responsibilities aren't explicitly defined, you're operating in a gray area. Schedule a call with your vendor's security team this week—not their sales team—and document who owns what. The next vulnerability disclosure might involve your deployment.