Skip to main content
Application Security Standards
Category: Application Security

Artificial Intelligence Security

Also known as: AI Security, AI Cybersecurity, GenAI Security
Simply put

Artificial Intelligence Security encompasses two related concerns: protecting AI systems themselves from threats, and using AI tools to strengthen an organization's security posture. On one side, it addresses risks to the integrity, confidentiality, and reliability of AI models and the data they depend on. On the other side, it involves applying AI-driven capabilities such as automated threat detection and prevention to improve defensive operations.

Formal definition

Artificial Intelligence Security is a dual-faceted discipline. The first facet covers the protection of AI systems, including models, training data, inference pipelines, and supporting infrastructure, against threats that may compromise their integrity, confidentiality, or operational reliability. Attack categories relevant to this facet include adversarial inputs, model inversion, data poisoning, and supply chain threats targeting the AI stack. The second facet covers the use of AI and machine learning techniques as security controls, typically to automate threat detection, behavioral analysis, and prevention workflows within an organization's security infrastructure. AI-based detection tools in this second facet may produce false positives, flagging benign activity as malicious, as well as false negatives, missing threats that fall outside their training distribution or that are novel in nature. Both facets apply across traditional application environments and generative AI deployments, and effective AI security programs typically address model governance, data protection, and runtime monitoring as complementary controls.

Why it matters

AI systems are increasingly embedded in critical application workflows, from automated decision-making to generative content pipelines, making their integrity and reliability a direct concern for application security practitioners. Threats such as adversarial inputs, data poisoning, and model inversion can compromise AI outputs in ways that may not be immediately visible through conventional monitoring, potentially affecting downstream business processes or exposing sensitive training data. Because AI components often interact with other software and cloud infrastructure, vulnerabilities in the AI stack can propagate risk across an organization's broader attack surface.

On the defensive side, AI-driven security tools offer meaningful capability improvements for threat detection and behavioral analysis, but these tools carry their own limitations that practitioners must account for. AI-based detection systems may produce false positives, flagging legitimate activity as malicious and adding noise to security operations workflows. Equally important, they are susceptible to false negatives, failing to identify threats that fall outside their training distribution or that represent novel attack patterns not previously encountered. Treating AI-based detection as infallible introduces operational risk, and effective programs typically supplement these tools with human review and complementary controls.

Generative AI deployments introduce an additional layer of concern, as models built on large-scale training data and exposed through APIs or user-facing applications present attack surfaces that differ from traditional software. Governance over model behavior, data protection practices for training pipelines, and runtime monitoring of inference activity are all areas where organizations are building out dedicated programs. The field has matured enough that frameworks addressing AI-specific risk, such as AI Security Posture Management, have emerged as recognized practice areas.

Who it's relevant to

Application Security Engineers
Application security engineers working on systems that incorporate AI components need to account for attack surfaces that differ from traditional software, including adversarial inputs at inference time and supply chain risks introduced by third-party models or datasets. They are also increasingly responsible for evaluating and integrating AI-based security tooling, which requires understanding its false positive and false negative characteristics.
Security Operations Teams
Security operations practitioners who use AI-driven detection and behavioral analysis tools must understand the scope boundaries of those tools, particularly their tendency to miss novel threats or flag benign activity as malicious. Calibrating these systems, reviewing their outputs critically, and maintaining escalation paths for edge cases are core operational responsibilities.
ML and Data Engineers
Machine learning engineers and data engineers are responsible for the integrity of training data and the reliability of model pipelines, making them important stakeholders in AI security programs. Data poisoning attacks and supply chain threats targeting pre-trained models or libraries are categories of risk that fall within their operational scope.
Cloud and Infrastructure Security Teams
AI workloads typically run on cloud infrastructure and depend on APIs, storage systems, and orchestration layers that must be secured independently of the models themselves. Cloud security teams need visibility into how AI components interact with broader infrastructure and where sensitive data, including training datasets and model weights, is stored and accessed.
Security Architects and Risk Leaders
Architects and risk leaders are responsible for establishing governance frameworks that address both the use of AI as a security capability and the protection of AI systems the organization relies on. This includes defining policies for model governance, data protection in AI pipelines, runtime monitoring requirements, and the acceptable use of AI-based detection tools given their known limitations.

Inside Artificial Intelligence Security

AI Model Threat Modeling
The process of identifying and analyzing threats specific to AI systems, including adversarial inputs, model inversion, membership inference, and data poisoning attacks that do not have direct equivalents in traditional software threat modeling.
Training Data Security
Controls and practices governing the integrity, provenance, and confidentiality of datasets used to train AI models, addressing risks such as data poisoning, label manipulation, and unauthorized inclusion of sensitive or proprietary data.
Adversarial Robustness
The degree to which an AI model maintains correct behavior when presented with inputs that have been deliberately crafted to cause misclassification, evasion, or unexpected outputs. Robustness is typically measured empirically and may vary across input domains.
Model Supply Chain Integrity
Verification and provenance controls applied to pre-trained models, fine-tuned weights, and third-party model components to detect tampering, backdoors, or substitution before deployment.
Inference-Time Attack Surface
The set of vulnerabilities exposed when an AI model is serving predictions, including prompt injection for large language models, model extraction via repeated queries, and denial-of-service through computationally expensive inputs.
AI-Specific Access Controls
Restrictions on who or what system may submit inputs to a model, retrieve outputs, access embeddings, or query model metadata, recognizing that AI APIs introduce unique exfiltration and abuse vectors beyond those of conventional software APIs.
Output Validation and Guardrails
Runtime controls that inspect or constrain model outputs before they are acted upon or surfaced to users, intended to catch harmful, hallucinated, or policy-violating content that static analysis of the model cannot reliably predict.
AI Detection Tool Limitations
Characterization of the false positive and false negative behavior of tools that use AI to detect security issues. Such tools may flag benign code patterns as threats (false positives) and, critically, may miss novel attack patterns, obfuscated malicious code, or issues requiring runtime context (false negatives). Both failure modes must be accounted for in any deployment.

Common questions

Answers to the questions practitioners most commonly ask about Artificial Intelligence Security.

Does adding AI capabilities to an application automatically make it less secure than a traditional application?
Not automatically. AI components introduce specific risk categories, such as prompt injection, model inversion, and training data poisoning, that traditional applications do not face. However, these risks are manageable through established controls including input validation, output filtering, access restrictions on model endpoints, and supply chain verification of pretrained models. An application with AI components that are properly scoped, tested, and monitored is not inherently less secure than a comparable traditional application. The risk differential comes from whether AI-specific threats are accounted for in the threat model, not from the presence of AI itself.
Can existing application security tools fully cover AI-specific threats without any changes to tooling or process?
No. Existing static analysis, dependency scanning, and DAST tools address the conventional application security surface of an AI-enabled application, such as injection flaws in surrounding code, vulnerable libraries, and insecure API configurations. However, they typically cannot detect AI-specific threats such as adversarial inputs crafted to manipulate model outputs, data poisoning in training pipelines, prompt injection through user-controlled inputs to language models, or model extraction attempts at runtime. Covering AI-specific threats generally requires additional controls, including runtime monitoring of model inputs and outputs, evaluation frameworks for model robustness, and threat modeling that explicitly addresses the machine learning pipeline.
When evaluating AI-based security detection tools, what false positive and false negative risks should practitioners account for?
Practitioners should account for both directions of error with equal weight. False positives occur when the tool flags benign inputs or behaviors as threats, which can cause alert fatigue and erode confidence in the tooling. False negatives, which are cases where genuine threats are not flagged, are equally significant and often underemphasized. AI-based detection tools may fail to flag novel attack patterns that differ from training data distributions, adversarial inputs specifically crafted to evade the model, or low-and-slow attacks that individually fall below detection thresholds. Scope boundaries also matter: most AI-based detection tools operate on observable signals at a specific layer, such as network traffic or application logs, and cannot detect threats that do not produce observable signals at that layer.
How should organizations integrate AI security considerations into an existing secure development lifecycle?
Organizations should extend existing SDLC phases rather than create a parallel process. In the design phase, threat modeling should explicitly include the machine learning pipeline, covering training data sources, model provenance, and inference endpoints as attack surfaces. In development, dependency and supply chain controls should extend to pretrained models and datasets, not only code libraries. In testing, in addition to conventional security testing, evaluation should include adversarial robustness testing and prompt injection testing where applicable. In deployment and operations, runtime monitoring should cover model input and output anomalies in addition to standard application telemetry. Existing change management and incident response procedures should be updated to account for model updates and dataset changes as events that may affect the security posture.
What controls are most effective at reducing prompt injection risk in applications that use large language models?
Effective controls typically include strict separation of trusted system instructions from untrusted user inputs at the architectural level, output filtering and validation before any model-generated content is acted upon or rendered, privilege minimization so that the model's access to downstream systems or actions is restricted to what is necessary, and monitoring of inputs and outputs for patterns consistent with injection attempts. No single control eliminates prompt injection risk entirely. Defense in depth is the recommended approach because prompt injection can occur through indirect channels, such as content retrieved from external sources during retrieval-augmented generation, not only through direct user input. Controls must account for both direct and indirect injection paths.
How should training data and pretrained models be treated from a supply chain security perspective?
Training data and pretrained models should be subject to supply chain controls comparable to those applied to third-party code dependencies. This typically includes verifying the provenance and integrity of datasets and model weights before use, maintaining an inventory of models and their sources analogous to a software bill of materials, assessing the trustworthiness of sources from which pretrained models are obtained, and scanning datasets for poisoned or adversarially manipulated samples where feasible. Pretrained models obtained from external repositories may contain embedded behaviors that are not detectable through static inspection of weights alone, so additional validation through behavioral testing is generally warranted before deployment. Updates to pretrained models or fine-tuning datasets should be treated as changes that require security review, not only functional review.

Common misconceptions

AI-based security detection tools provide comprehensive coverage with manageable error rates in both directions.
AI-based detection tools exhibit both false positives and false negatives. False positives create alert fatigue and may block legitimate code, while false negatives are often more dangerous because they silently miss novel, obfuscated, or context-dependent threats. Neither error type is inherently rarer, and practitioners must tune and validate tools against both failure modes rather than focusing on only one.
Securing the model weights and API endpoints is sufficient to protect an AI system.
AI system security spans the full lifecycle, including training data integrity, supply chain provenance of pre-trained components, inference-time input and output controls, and monitoring for model extraction or inversion. Perimeter controls on weights and APIs address only a subset of the attack surface.
Traditional application security controls are directly transferable to AI systems without modification.
While foundational controls such as access management and logging apply, AI systems introduce attack classes (adversarial examples, prompt injection, membership inference, data poisoning) that require AI-specific threat modeling, testing methodologies, and runtime guardrails that have no direct analog in conventional software security.

Best practices

Perform AI-specific threat modeling at design time to enumerate threats such as adversarial inputs, data poisoning, model inversion, and prompt injection, treating these as distinct from conventional software vulnerabilities rather than assuming existing threat models cover them.
Establish and verify provenance for all training data, pre-trained model weights, and third-party model components, applying integrity checks and reviewing licensing and sourcing before ingestion or deployment.
Deploy runtime output validation and guardrails as a control layer separate from the model itself, because static analysis of model weights typically cannot predict harmful, hallucinated, or policy-violating outputs without execution context.
When using AI-based security detection tools, explicitly measure and account for both false positive rates and false negative rates. Tune detection thresholds with awareness that suppressing false positives may increase false negatives, and validate tools against representative samples of both benign and malicious patterns.
Apply least-privilege access controls to model inference endpoints, embedding APIs, and model metadata, treating AI APIs as a distinct attack surface with unique exfiltration risks such as model extraction through repeated querying.
Continuously monitor deployed AI systems for behavioral drift, anomalous query patterns, and signs of adversarial probing, recognizing that AI system security posture can degrade over time as attackers discover exploitable input regions that were not present in initial testing.