Category: AI Security

Hallucination Exploitation

Also known as: Slopsquatting, Hallucination Squatting, Package Hallucination Attack

Simply put

Hallucination exploitation is an attack technique where adversaries take advantage of the tendency of AI models to fabricate information, such as inventing nonexistent software package names, by registering those fabricated names as real malicious packages. When developers follow AI-generated code suggestions that reference these hallucinated packages, they may unknowingly install attacker-controlled software. This represents a novel software supply chain threat that leverages the predictable nature of AI-generated errors.

Formal definition

Hallucination exploitation refers to a class of supply chain attacks in which threat actors identify software package names that large language models (LLMs) consistently hallucinate (fabricate with apparent confidence) and then register those names on public package registries with malicious payloads. Because LLMs can generate plausible but nonexistent package references across multiple programming languages, attackers can anticipate these hallucinated names through systematic analysis of model outputs and pre-register corresponding packages. The attack surface is shaped by the reproducibility and frequency of specific hallucinated package names across different models, prompt configurations, and language ecosystems. This technique, sometimes called slopsquatting, differs from traditional typosquatting in that it exploits AI-generated fabrications rather than human typographic errors, and its effectiveness depends on developer trust in AI-assisted code generation without independent verification of package provenance.

Why it matters

Hallucination exploitation represents a fundamentally new category of software supply chain attack that emerges directly from the widespread adoption of AI-assisted code generation. As developers increasingly rely on large language models to suggest code snippets, dependency lists, and package references, the fabricated but plausible-sounding package names these models produce become a predictable and exploitable attack surface. Unlike traditional typosquatting, which depends on developers making manual typing errors, hallucination exploitation leverages systematic patterns in AI model outputs, meaning the same fictitious package names may be suggested to thousands of developers independently.

The threat is amplified by the reproducibility of hallucinated package names. Research, including work published through USENIX, has demonstrated that LLMs can consistently hallucinate specific package names across repeated queries, different prompt configurations, and multiple programming language ecosystems. This reproducibility allows attackers to systematically identify high-frequency hallucinated names and pre-register them on public package registries (such as npm, PyPI, or RubyGems) with malicious payloads. When developers follow AI-generated recommendations without independently verifying package provenance, they risk installing attacker-controlled code into their projects.

This matters because it erodes a trust assumption that many development workflows now depend on: that AI-generated code suggestions reference real, legitimate software components. Organizations that lack verification controls for AI-suggested dependencies face exposure to malware injection, credential theft, and other supply chain compromises through a vector that did not exist before the era of LLM-assisted development.

Who it's relevant to

Software Developers Using AI Coding Assistants

Developers who use LLM-based tools (such as code completion assistants or chatbot-style code generators) to suggest dependencies or generate code are the most direct targets of this attack. Without independent verification of every suggested package name against official registries, developers risk installing malicious packages that were registered specifically to exploit AI hallucinations.

Application Security Teams

Security teams responsible for securing the software development lifecycle need to account for hallucination exploitation as a distinct supply chain threat vector. This includes establishing controls such as dependency allowlists, automated provenance checks, and developer training that specifically addresses the risks of unverified AI-generated code suggestions.

DevSecOps and Supply Chain Security Engineers

Engineers who manage CI/CD pipelines and software composition analysis (SCA) tooling should evaluate whether their existing controls can detect newly registered packages with no meaningful history, which is a common characteristic of slopsquatting packages. Integrating checks for package age, maintainer reputation, and download volume can help surface suspicious dependencies introduced through AI suggestions.

Package Registry Maintainers

Operators of public package registries such as PyPI, npm, and RubyGems face the challenge of identifying and responding to bulk registrations of packages whose names correspond to known LLM hallucination patterns. Proactive detection of suspicious registration activity and coordination with AI security researchers may help mitigate this threat at the registry level.

AI/ML Model Developers and Providers

Teams building and deploying large language models for code generation should consider hallucination exploitation when evaluating model safety. Reducing the frequency and reproducibility of fabricated package names in model outputs, or adding verification layers that cross-reference suggestions against real package registries, can help reduce the exploitable attack surface.

Inside Hallucination Exploitation

LLM Hallucination

The tendency of large language models to generate outputs that appear plausible but are factually incorrect, fabricated, or unsupported by training data, creating a foundational vulnerability that attackers can leverage.

Package Hallucination Attack

A specific exploitation technique where attackers identify fictitious package names that LLMs repeatedly hallucinate in code suggestions, then register those package names in public repositories with malicious payloads, targeting developers who trust AI-generated recommendations.

Prompt Manipulation for Hallucination

Adversarial prompt crafting designed to increase the likelihood or control the content of hallucinated outputs, steering the model toward generating specific false information that serves the attacker's objectives.

Trust Exploitation Vector

The social engineering dimension of hallucination exploitation, where attackers rely on users or automated systems treating LLM-generated outputs as authoritative without independent verification.

Software Supply Chain Poisoning via Hallucination

The downstream impact when hallucinated dependencies, libraries, or code patterns are incorporated into software builds, potentially introducing malicious code into the supply chain through packages that were created to match hallucinated names.

Common questions

Answers to the questions practitioners most commonly ask about Hallucination Exploitation.

Is hallucination exploitation just a reliability problem rather than a real security concern?

While hallucinations are often discussed as an accuracy or reliability issue, hallucination exploitation is specifically a security concern. Attackers can deliberately trigger or leverage hallucinated outputs to cause harm, such as when a model fabricates nonexistent software packages that an attacker then registers with malicious code, or when hallucinated API endpoints or URLs are used to redirect users to attacker-controlled infrastructure. The security impact goes beyond mere inaccuracy when adversaries actively exploit the predictable patterns of model confabulation.

Can hallucination exploitation be fully prevented by improving model training or using better prompts?

Improved training and prompt engineering can reduce the frequency of hallucinations but cannot eliminate them entirely. Hallucination is an inherent characteristic of how large language models generate outputs, not simply a defect that can be patched away. Attackers may craft inputs specifically designed to increase hallucination likelihood, bypassing general mitigations. Effective defense requires layered controls including output validation, dependency verification, and human review rather than relying solely on model-level improvements.

How can development teams detect when AI-generated code suggestions reference hallucinated packages or libraries?

Teams should implement automated verification checks that validate any suggested package names against known, legitimate package registries before installation. This typically involves cross-referencing recommended dependencies with curated allowlists or checking package metadata such as publication date, download counts, and maintainer history. Software composition analysis (SCA) tools can help flag unknown or suspicious packages, though they may not catch very recently registered malicious packages that exploit hallucinated names. Integrating these checks into CI/CD pipelines provides a practical enforcement point.

What specific patterns should security teams monitor for to identify hallucination exploitation attempts in their environments?

Security teams should monitor for installation attempts of packages that do not exist in vetted registries, requests to API endpoints or URLs that the organization does not own or recognize, and references to nonexistent documentation or configuration patterns that could lead to insecure setups. Logging and alerting on failed dependency resolution attempts can surface cases where hallucinated package names are being acted upon. Additionally, monitoring for prompt patterns known to increase hallucination rates in any AI systems integrated into development workflows can provide early warning.

How should organizations adjust their software supply chain controls to account for hallucination exploitation risks?

Organizations should treat AI-generated dependency recommendations with the same scrutiny applied to any untrusted input. This includes enforcing the use of private or curated package registries, requiring manual review or approval for any new dependencies not already in an approved list, and implementing namespace reservation for internal package names to prevent dependency confusion scenarios amplified by hallucinated suggestions. Package pinning and integrity verification through checksums or signatures provide additional layers of defense against acting on fabricated recommendations.

What role does output validation play in mitigating hallucination exploitation, and where are its limitations?

Output validation serves as a critical control by checking AI-generated content against known-good references before it is acted upon. This may include verifying that referenced URLs resolve to legitimate domains, that suggested packages exist in trusted registries, and that code patterns conform to established security standards. However, output validation has scope limitations: it typically cannot detect semantically plausible but subtly incorrect security guidance, and it may produce false negatives when hallucinated outputs closely resemble legitimate resources. Validation is most effective as part of a layered defense strategy rather than as a standalone mitigation.

Common misconceptions

Hallucination exploitation only affects end users who naively trust AI chatbot responses.

Hallucination exploitation poses significant risks to software development workflows and supply chains. Developers relying on AI-generated code suggestions may unknowingly reference hallucinated packages or libraries, which attackers can register and populate with malicious code, affecting downstream applications and their users.

Improving model accuracy will eliminate hallucination exploitation as a threat.

While reducing hallucination rates lowers the attack surface, hallucinations are an inherent characteristic of current generative AI architectures. Even models with low hallucination rates may still produce fabricated outputs in specific contexts, meaning exploitation remains viable. Defense must also occur at the consumption layer through verification and validation controls.

Hallucination exploitation is purely a theoretical or low-impact risk.

Researchers have demonstrated practical attacks, particularly package hallucination attacks, where fictitious package names consistently hallucinated by popular LLMs were registered in public repositories. These attacks are reproducible and can scale, making them a concrete threat to organizations that integrate AI-generated code without adequate review.

Best practices

Verify all AI-generated package names, library references, and dependency recommendations against trusted, authoritative package registries before incorporating them into projects.

Implement automated dependency validation in CI/CD pipelines to flag packages that are newly registered, have minimal download history, or lack established maintainership, as these may correspond to hallucinated names claimed by attackers.

Apply namespace reservation or internal package registry controls (such as scoped packages or private registry proxies) to prevent dependency confusion when AI tools suggest packages that may not exist in trusted sources.

Establish organizational policies requiring human review of AI-generated code, particularly for security-sensitive components, rather than allowing direct integration of model outputs into production codebases.

Monitor LLM outputs for patterns of repeated hallucination, especially in code generation contexts, and maintain an internal blocklist of known hallucinated package names to prevent their accidental adoption.

Educate development teams on the specific risks of hallucination exploitation, including package hallucination attacks and fabricated API references, so that practitioners apply appropriate skepticism to AI-assisted development outputs.